My name is Navdeep Martin. Um I am today going to talk about uh specifically towards product managers and direct this talk on how to build out A I products.Um So while this is for a product manager, I think this could be applicable to basically any of you guys building out products. Um These are tips of the trade that I've kind of learned along the way. Um A little bit of my background. Um I have a seven and eight year old. Um I uh practice this duck on front of their stuffed animals. Um So hopefully that should have led to some, some good content. You should have seen them all, they were all watching me. Um But on a more serious note, um I have a computer science background. I did an engineering path right out of school. Um I worked at the CIA for 10 years. Um and then my machine learning product management expertise actually really started at the Washington Post. Um I was there at the right place at the right time. Um Bezos had wanted to uh basically start a personalization program and I got the opportunity to lead that. Um and then moved over to Comcast to try out machine learning product management in a new vertical um which was TV, and movie recommendations. And after that moved over to two start ups in the space.

Um At Primer A I was one of them, I led the machine learning strategy with our flagship product um and then joined Blackbird A I um just recently Q four of last year as their VP of product. Um We are using A I to detect online narratives and risks for fortune 500 companies and national security agencies to give you a sense of where I'm headed. Um I'm gonna start out with some definitions of terms that I mentioned in my deck. So just wanna make sure we're all speaking the same language. Um After that, I'll go through my five tips for A I product management. Um My ninth grade teacher used to say to show and not tell. So I promise to give as many practical examples as I can. And finally, I'll leave you with my own reflections for the future of women, leading breakthroughs in A I. Um So let's get started on, on my definitions here. Um First off, just kind of get us on the same playing field. Machine learning. Um Machine learning is a branch of A I, you're giving a computer examples of data and it's driving inferences to make predictions in the future. Um This is an example of what machine learning could look like in a product. Um This is an example from my own experience and recommendations. Um What you see here is a recommendation module and everything you see here is a recommendation um from the products chosen. Um In this case, the products here are news article.

Um You also also the images are recommendations as well as the headings. So the titles uh you know the titles of the articles themselves. Um A machine learning algorithm is determining the optimal recommendation for these three different situations to yield a click from your from users which would be you. And I um backing up though. How do we get to this? Um First you need training data. Um This is kind of a crude example here um where I have a list of news articles, um their titles, um you see a first paragraph and a URL to go skim the article if I desire. Um So what I would do here is have a subject matter expert to go in and tag whether each article is about fashion parenting or national security. Um The three columns to the right. Um The finished version of the spreadsheet is essentially my training data. A data scientist will take this and train a supervised machine learning model to classify if these articles are about fashion parenting or national security. And the finished result is a classifier for these categories. Machine learning is prevalent in your everyday life. Two, here's some examples kind of from my own, um, my own feeds. Um So the first one to the left is, um, a grocery store app. Um, if you think about it, grocery stores only have, um, the, a certain amount of real estate in their apps.

Um And so they really need to make the right recommendation at the right time. Um And then the middle um, section is, is actually my Pinterest board. Um, what I found really interesting about this is um about in January or so my husband and I decided to go vegetarian and I didn't like mark some setting in Pinterest uh to let Pinterest know I'd gone vegetarian. But somehow um over time, uh the the recommendations shown to me were act actually began to be a vegetarian. Um And I mean, what's happening here behind the scenes is is there's a recommendation module and that's taking the the input my data input of not clicking on meat recipes as uh data when deciding to decide on the optimal recommendation for me. Um On the right, you'll see an example from the company. I work for Blackbird A I. This is an example of image analysis. Um So what can happen here is that customer searches for brands and risks. Um And through um our machine learning image analysis, we can show images that match with that search. All right. Um This is those definitions were not comprehensive obviously, but it'll give you kind of a starting point for my five tips which I'll go into next.

Um First off design matters a lot. Um How big you make your text? What did you name your heading? Um Most importantly, how will customers use this? Um Some industry examples from my own past um companies uh like Comcast found that titles like more like this and people also watched were much better headings for the recommended program sections over something like you might like. Um The Washington Post um found that having an image preview next to each article actually increased their click rates. Another company found that if they showed their work and allowed users to understand how they came up with the categorization, even if they got that categorization wrong users were forgiving of that error because they understood how the machine came up with it. The point is your designer needs to play center stage in building out your product, which leads me to my second tip. You want to design and build your ugly prototypes in parallel. Um What I like to do is actually have my designer tinkering away while my machine learning solution is being built. Um My process that I follow is actually to work with my machine learning team to deploy an ugly prototype.

Um I call this an ugly prototype because no thought should be put into the design at all. The point is to showcase your solution um in parallel, I'll provide my designer with some loose requirements and ask them to develop wire frames of what the workflow could be. So let's, let's give an example here. I think this makes this make more sense. Um Let's say I'm building a news article, virality predictor. My marketing team wants to amplify possible viral articles and they want an alert every two hours of the day to get a ranked list of anything they should act upon and they want some level of explainability here too. They want to understand why we came up with uh the recommendations on this list. So what I would do here is work with my machine learning team to explain the need by writing a one pager. Um My one pager might describe my users uh the use case, what else is on the market that might do this? Um I might also include how my users do this work today and how they could do that work in the future. At the same time, I'll begin working with my designer to begin creating design prototypes to share. When I finally have something ready to show, I'll begin my roadshow.

Um I'll essentially uh schedule user interviews back to back as much as I can um and conduct user interviews where I'm showing both the ugly prototype and the design prototype. Um I'll change the design prototype incorporating user feedback in between interviews. And at the end of this exercise, I have validation that the machine learning model is doing what my user needs and I know I've built the right user experience. Let's move on to tip. Number three, you, you kind of want a team that likes one another. Uh You might be thinking, how do I control for this or this isn't my job. Um But as A PM, I would argue your job is the success of the product. Um If your team doesn't have basic respect for one another, you'll never be great. Um Few examples from my past, I've, I've seen situations where A PM and designer cooked something up for a couple of months without ever involving an engineer when the engineers were finally brought in, they tore it apart, creating a permanent finger pointing scenario or I've seen where the team self selected into the cool kids versus everyone else, the environment deteriorated before our eyes.

Um In this case, the PM began to assume the designer role. And if you remember my tip number one, designers go to school and have reasons for everything that they do. Design matters. A lot, a good team will appreciate the value of good design. It doesn't matter how good your machine learning models are. If no one wants to use your product, um or the worst one I've experienced was when the engineering team spent four months building out new functionality on a deadline, there wasn't enough time. Um I'll put that in quotes, enough time for the PM to do product discovery. And at the last minute, the PM and designer were brought in out of realization that no one could even explain the functionality to external users. There were countless design emergency meetings with the team to try to determine the U I based on a very limited back end. Um Because the team, the engineering team had spent four months building up the functionality, it was really too late to reverse product decisions that had already been made. Um My point is here that you'll never build something great without a positive relationship of all four roles you see here. But let me kind of dig into that because I've been to organizations where it's actually not clear who does what.

Um So as A PM, as I mentioned, you'll need to use your soft skills, figure out how to break down barriers. Um Whether that's virtual team building that you tag an organization development person to help you with if you have someone like that at your organization, um, or you can conduct them yourself. Um, in the height of the pandemic, I've introduced doing Latin dance workouts off screen, of course, um Scavenger hunts, um, casual phone walk and talks where just, uh have a one on one with someone while we're walking outside. Um I've also done outdoor in person gatherings wherever I could. My hope is that, that I can get everyone's guard down and just have natural conversations flow, um, where people are, are just talking to one another without my needing to be there. Um On A P A team, the PM also needs to set realistic and inspiring goals to build great products for customers and work with each team member to get that product to fruition. At the end of the day, the PM is the decision maker. Um But uh we're, we're essentially trying to bring everyone um you know, along the same ride. Um Let's so let's hit each of the other roles here.

The data scientists for me has often been the originator of like the great ideas that I've uh ended up putting into products. Um You want them with you on your customer calls to hear directly from customers to improve their, their own machine learning models. Um Meanwhile, your designer, um they're making large amounts of data accessible to a human being. Um They need to be involved right from the start. Um If you're B to B that means they should be involved with all user meetings. Um If you're B to C, you'll want them intimately familiar with uh regards to what people are clicking on so that the user metrics, your front end engineer is bringing your designer sketches to life and they'll offer your designer additional ideas. My favorite kind of relationship that I've seen between front ender and designer is when they're actually kind of pair programming. Um you know, elbowing one another. Hey, try this, try that. Um in order to get to the best possible solution. Let's go to tip number four. This is really important. Does this problem even mean machine learning? I have to say oftentimes when you've hired a machine learning team, the mindset goes to how can I build the coolest thing to solve this problem? Versus the more basic, what's the best path to solve this?

The best times to use machine learning is when a person can complete the same actions, the machine would I found uh that regular expressions for instance, um which is a rule based search pattern um can be used in place of machine learning with very positive results. I found that there are often open source data sets, um open source algorithms or papers written solving the same problems a new team might be facing. I found that customer or interns can be your subject matter experts tagging data with positive results. I found that you can use rule based logic to remove things that are problematic. Um For example, in a previous company, we were analyzing news articles, we found that stock ticker trades were showing up um in our results. Um So these were articles that had a very peculiar pattern and there were thousands of them and they were messing up our data results. Um We found that writing a simple rule based if then statement allowed us to remove these articles from our analysis and we could spend our machine learning engineers to solve the more really hard problems. My point here is the takeaway is take a look at the problems you're trying to solve and determine the fast and most effective solution. You might not even need machine learning. Moving on tip number five, machine learning models are costly. Um storing and transferring data is costly.

Um You really need to look at whether you have the customer demand to support this. I'll give you a couple of examples in my past life. Um I've seen a situation where the company's leadership initiated a very costly relat effort. Um This effort involves the entire engineering team and it took a year to do the team did not conduct a cost analysis until the effort was nearly complete. What ended up happening is once deployed. They ended up with a very costly infrastructure to maintain and they couldn't find customers to cover the costs. I've seen another situation where GP US are used in development to improve features, but they never get into the product because of the expense of running a GP U.

My point. My conclusion here for this is do some back of the envelope cost estimates before going too far to ensure your customer demand will justify your costs. I have a bonus tip here. Um Bone tip number six, if you will as A PM, you really need to roll up your sleeves and tag data. How can you explain to taggers how you want data to be tagged without doing it yourself. Um In my experience looking at classifier results, there are always edge cases that aren't clear from the start. Um I'll give you a couple of examples here. Um I've seen a situation where a data scientist on the team trained to finance classifier. Um They only use documents with the financial context, unfortunately, and when a completely different customer came on a national security customer, uh and they were trying to review those same documents. Um We found that there were a lot of false positives. Um And I had to send that model back to the drawing board for retraining. Um Another situation I found that we had decided to, to create an elections classifier. Um But the peculiarly, we found that the, the results were often showing um up for articles that were also about polling and voting and that had nothing to do with elections.

Um In this case, as the product manager I realized from our user's perspective, they'd be OK with having polling and voting results show up within this classifier. So I I renamed it. It was a simple renaming of elections uh to elections and voting. Um My point here is there are product management decisions to be made for classifiers and you'll want to be very in tune with the data, at least from the start so that you can be educated on what's happening and make decisions. I wanted to, to leave you all with uh a few words on women trying to advance in their career in A I. And I'll have to say, as I sat down to think about what I wanted to say here, I really struggled with not wanting to appear to generalize. And I still also wanted to, to be authentic with, with my advice. And so what I'm sharing here is my own growth story. Um And uh this may or may not apply to you but wanted to share with you how I've kind of um gone through my own challenges. Um So I will take you back. Um If you can, uh, we're go, we're going on a trip uh to college, uh, junior year for me willing to marry. I decided I was gonna major in computer science. So I'm sitting there junior year.

Uh, this is a, a college in, in Williamsburg, Virginia if you're not aware. Um, and I'm sitting in, in a class and it's, it's, I don't know, maybe the first day of school actually. And our professor asks a question. He says, what is the meaning of X? I actually don't remember what his question was. Um But forgive me for, for not remembering the exact question. Um But he was asking what's the, the, the meaning of X? And he was challenging the class and I, I looked around and I didn't know the answer but no one knew the answer. No one was answering this question and the professor was like, no, come on guys, you know, this, what's the meaning of X? And finally a guy in the back row. I still remember turning this way to look at him and I remember exactly what he looks like. Um, he says, is it blah and sorry, I, that terrible story. I don't remember what he, he said either, but he said something like, is it blah? And the professor turns and says, well, that's a stupid answer. The whole rest of the year, I didn't say a word in that class. And in fact, that was my personality. Anyway, I was afraid to ask questions. I didn't want to ask something that I should already know. So I proceeded and you know, did well, got, got it out of school in my first job.

And somehow I over time, I learned to pull a close colleague aside when I had questions and I'd ask them um in a private manner, my question. But over more time, I, I learned that I do in fact know what I'm talking about. And I felt more confident to ask questions in larger settings. I don't want to generalize and state that women are afraid to ask questions. But as I sat thinking back, what advice I wanted to give this group, I found that from my own experience, in order for me to thrive in this field, my advice for you is to begin asking your questions. Tech is constantly evolving and that you gain more from a situation by asking your question and pushing the conversation. And I guarantee you there will be someone in the room who has that same question and they're happy that you asked it.