Um Good afternoon, good evening, everyone. Uh My name is Evelyn. May really glad to have you join my session at the Women Tech Global Conference 2022. My topic for today is deploying A I in scale at global enterprises. Who am I?Uh I am a senior product strategy manager at Oracle. I've been with the company for the last five years and uh really worked on the 0 to 1 process for two A I based products at Oracle, uh the Oracle Logistics Machine learning solution as well as the Oracle Logistics, Digital Assistant solution. Both are A I applications within the field of supply chain management, especially supply chain logistics management. Um Both products took us several

years for understanding the use case,

developing it, testing it with a prototype iterating through several cycles and then launching it and deploying it uh with real customers and iterating with the customers. So my talk today is about sharing uh my whole experience in that process and also

provide you with an overview of how the process typically

looks like. So start with some customer success. Um This

is one of the

customers that we have worked with um this customer had seen value in the A I product. And um at the end of the product deployment, they actually worked with us and had a joint uh customer success presentation with us where they shared some of the results they that, that they saw. So this is a public video that's available on youtube. If you want to know the customer perspective, how the customer experienced it from their angle, I would also highly recommend that you check it out. And similarly, um the products success has also been quoted as part of the oracle press release where uh a customer had spoken out of their success with the product. Um Here on the machine learning capability for this particular customer had delivered enough value that the customer um made this public announcement that it had helped improve. So in this case, is a supply chain estimated time of arrival accuracy from 64% to 93% uh which is an invaluable result uh especially in the time of COVID disruptions because we know um right now there's a lot of supply chain disruptions, variabilities in supply chain lead times.

So the capability of using machine learning to predict the supply chain lead times really helped the customer cope with the uh with the chaos in this pandemic. OK. So just lead with that. Here is my agenda for today. I want to cover what is the typical process for developing an A I based product, deploying it at customers and how to make sure it's future ready. What are some of the considerations for this A I based product? First of all, I would like to um highlight the typical development life cycle, what I call a four step process. Number one is A I on paper. Uh Meaning um we are still in the process of defining the use case and working out um the overall skeleton of A I product. And then the number two step in A I in the lab, we're testing and prototyping A A I model with a sample data set and experimenting and try to find out the best parameters, best model structures. And then once we get a good enough um accuracy, we then deploy it into the real production code and make that A I application available to customers.

But we want to make sure that we do the, we, we expose the product in a controlled environment so so called A I in the zoo. So we deploy the product to a limited set of key customers and iterate with the customer work very closely with the customer side by side. So our development team working with the customer, it team to deploy and, and train models and, and test the model back, test models with different settings and then find out if there's any enhancement ideas and uh or if there's any instability in the code. And then finally, when when, when the product is more polished, unleash the power of A I in the wild to a broader range of customers. So here is a summary of the development life cycle A I on paper A I in the lab A I in the zoo and A I in the wild. So we'll go into each step in more detail here. The first step A I on paper, it actually took us almost a year to fully define the product in terms of the business use case.

Um We have worked with customers very closely and many of the customers have have been urged by the internal management teams to uptake so called innovative technologies. They are looking to deploy A I and machine learning solutions and they see um many problems in their day to day life that can be addressed by A I and machine learning, but they don't know how to um how to build or uh an, an A I product. And uh if they want to procure an A I product, which company to go to. So it took us a year to first of all, talk to um different kinds of enterprises, large scale global companies um like for example, Coca Cola and Unilever and so on, but also smaller scale companies um to, to find out um what business problems they want to get addressed away with A I product.

And then secondly, we usually get a collection of many different problems and some of the problems actually do not require A I machine learning prediction. So we need to prune out many still valuable but not relevant um product ideas and just focus on those uh really high value problems that can be addressed by A I and then delve into the second problem. So what kind of data is required for building an ML solution? Uh Typically, customer either do not have access to the data um that's necessary for prediction or they're not yet collecting the data. Um Many of the data collection is actually done um in a manual uh in a uh in a manual process. So the data quality is not good or some of the data um sits in another department, for example, for supply chain prediction. Um The the demand, supply chain demand, the customer demand data sits in another department and another software or different legacy Softwares where there's not no no good mechanism like API S to expose these data. So there are many layers. First of all, what what problem is valuable.

And secondly, whether it's feasible or not in terms of access to the data and also quality of the data. So yeah, the business problem um definition is very, very important before we dive into solution. And the second step A I in the lab is the first level solution to the problem. First of all, we will need to find a real world data set to build the model on. Uh Typically we work with um key strategic customers who would trust us and willing to cove this innovative solution with us and they will share the data with us. Now, um once you have a data, it is first of all, um very important to understand the business context of that data. It's, it's not a simple data set that's available on, on, on Cagle and so on. Um When it comes to an enterprise A I product, it involves the enterprise business, business process, different columns from the data set may come from different department or might come from different systems and the name of the column that the name of the data label also is not easily interpretable.

Um So we had to go through many rounds of discussion with the business person, for example, the supply chain planner to go through. What does it, what does this particular day field mean? Um Is it the ship day? Is it the promised date or what is the downstream uh um implication? Uh And what is the target field you want to predict? Um So understanding the business problem is the first step to interpreting the data. And then the second step for the data set processing is to eliminate outliers. As I said, many of the data that's collected in enterprise space is manually entered and there are either fat finger errors or convenience errors. For example, we have noticed that um let's say a particular shipment has a planned delivery date and it also has an actual delivery date in an, in an ideal world. We will want the, the actual to meet the plant, but there's always deviation. So

the, so these two fields

should be closed but not exactly the same, right, because the actual should have or, or typically has some deviation from the plant. Um, but what the, um, the, um, the operator may do due to some convenience, um um reasons she would just copy the plan over and filling the actual. So in that case, the actual is not the um the, the um the actual real world value. So it's not trustworthy. Um So in that case, we also need to eliminate those and find out what records, data records are, are actually um usable for machine learning. So it's also a very um thorough analysis that uh we have to conduct. And then what is uh special about um working with real customers on machine learning A I model building is that the legal process and the compliance process is very important. So, um it um different from getting a public data set from um from, from Cagle or from a research um university. When we work with these customers, we had to um establish a legal agreement to protect both of us. And the legal agreement is usually drafted by the oracle legal.

And then this customer um also needs to have their own legal team review it and it specifies the term, um, as well as the, the, the nature of the data as well as, um, um, the, um, the right to cancel as well. Um And, uh, so this is a, I would say at least three months of process back and forth between the legal teams to negotiate what's on the agreement. And I would say 70% of our customers who, uh, who we, um, worked, try, try to work with in the first place, dropped out of the process because of all the legal hurdles. Um Our customer contact, um let's say the supply chain department really want to work with us and have us help them address the problem, build A I solution for them, but they don't typically have the right to sign the legal protection agreement. They have to get their CEO or CIO to sign the agreement. So there are many levels of questioning and one eventually gets to the CEO level. It might be like um several months after or it gets shut down by some uh some middle level management or the legal agreement just never got signed in time.

So 70% of customers just dropped, um and, and um abandoned the project and then the other 30% of customer put additional limitations on, on top of the agreement, uh which made the data sharing term super short um to the point that we have to speed up and use their and, and, and work on their data with a very short time frame and then delete the data after the project is done so that we, we make uh we, we remain complying to the legal agreement.

So that's the uh the legal protection aspect. And the third aspect is about compliance because when it comes to getting access to customer data, everyone is concerned of data privacy and data integrity. Um Sometimes our customer contact, let's say the supply chain department, they are not sensitive to the data. They think that they think the um the data is no big issue. So they would just send us, for example, uh the they would just want to send us the data in a CS V file by email. Are we allowed to take that? No, even if the customer wants to share data with us in an email, we're not allowed to take that because of all the data priva privacy and protection issues. We as a product team, uh we have to go through a, a uh a secure environment, a secure data pipeline and have the uh the customer comply with the, you know, with a secure process as well. Um So overall, yes, the legal and the compliance process is also added overhead to the overall to the project. Um But I think it is totally worthwhile because um um data issue is getting more and more um under attention right now under government uh regulations.

So it is important to get to get things done, right? OK. So A I in the lab everyone knows um hope like many people knows how to build A I models once you have the data. So I was, I was um skip that part about how we actually work with the data scientist uh through different iterations to build a good accuracy model. But then once you have the model, the next step for us is to deploy the model in actual production code and make the production software available to the customer. So in this case, we are shipping a uh an A I solution. Uh It's like a framework and it allows customer to plug in their data into, into this framework and then train a specific model for this particular customer. So um our solution does not deliver a pre trained model, but we, we um support the customer in the model training process and trains a um tailored model to the particular customer. So once we are ready to release this product, um in uh as I have said in the previous step, we've already worked with a number of customers and they have worked with us in the lab process.

And then when the solution was released and delivered, these customers are naturally the early adopters, those who have already worked with us in the lab process are ready to take the production software and use A I in their actual business processes. And in this controlled environment, we worked with the customer very, very closely. Uh We had daily meetings between our development team, product management team, data scientist with the customer side, not just it people, but also business people. Because usually the business people define the business problem, they will tell us what kind of model they want to train and what kind of accuracy is good enough. And then the it people on the customer side um takes care of the configuration of the systems. Um and they are, they are closer to um to uh to the manipulation of the data and they work very closely with us on the technical aspects. So there's a lot of hand holding care and feeding and what's really important here is close partnership with the customers because as you, as you all know, um the first one or two releases of the software is typically not as stable and there's a lot of enhancements to be done.

So to be very frank um with uh with the customer um is uh is an important aspect here. Um The customer is just a development partner and innovation partner and the relationship building uh aspect is, is key. OK? And then once um the software is more stable and we have proved out the, the, the value it's ready to unleash A I into the wild. And um as a product manager, it's really important to um conduct regular health checks uh with, with any customers for product enhancements. And also um a model is a living and a living and breathing um uh model, it, it, it gets retrained over time. So we will need to go and follow up with the customers. See as the world evolves as the supply chain issue gets um resolved. Uh post pandemic, how, how, how is the model uh behaving and if there's any additional enhancements that, that we need to add to the platform. And on top of that, uh also uh a lot of the time I um I, I would uh spend uh with the sales people um with uh with also prospect customers to walk them through the solution and address any questions that they have because very typically many customers and many prospect customers, uh let's say they are a uh supply chain team within a uh uh fast moving consumer goods company.

Um They are very curious about uh what A I or machine learning solutions that we are offering and they want to hear about them and then they want to take some time internally to uh to, to discuss and, and, and um and make a decision if and when they were deployed. So, um I have these ongoing conversations with these customers to just share the latest status uh of the product. So here's a um a summary. Uh Let me just go back to the first uh slide of the section. The life cycle being a I on the paper, use case definition and data set definition. Uh Second step A I in the lab, it's working with data scientists and to clean the data, understand the data and build out useful models. And in this step, the legal and compliance aspect is really important

because we are working with real customer data.

And number three A I in the zoo is deploying the solution with um a limited set of key customers. And then the last step is A I in the wild. OK. With that, I would go into the next section which is about unique challenges in building enterprise AIS in the session abstract. Um I, I had mentioned that many of the A I solutions right now are being deployed in the consumer space only. Um Whereas the enterprise space is ripe for A I deployment, but it has its own unique challenges, many unique challenges. And I would like to go through each one of them. The first challenge being legal consent. When I use any consumer product, let's say uh Facebook or a mobile app when it pops up a user agreement and asks me to agree. Usually I would just agree without even reading it because if I don't agree, I mean, uh I may not be able to actually use the product. So whatever I just agree. So from the tech company's perspective, it's really easy to get the user to agree to share data in the consumer space. Um But in the enterprise software space, you're getting another company to agree to share data.

So um you will need to go through many, many rounds of negotiation with their legal team. And also, and also many people get involved in the process, not just the supply chain department who has a burning need for your solution, but also the other departments that may be more conservative and they, they don't feel uh such a burning need. So they may actually block um the uh the, the data sharing um process. So it's uh uh I, I would say it's exponentially more complex to get the legal consent from an enterprise. And second is the collaboration dilemma. Now, when I use Google map and you, you also use Google map, we our data gets pulled together to train one common model, right? So um the model really gets scaled as more people get to use it. But when it comes to the enterprise space, let's say, um CV S um uses a particular supply chain prediction model, by the way CV S is a uh is a grocery um or, or pharmacy um chain in the US. And then let's say Walmart also uses the same supply chain management uh A I product.

But typically, they would use two different instances of the same product and they will not share data with each other or they will not share data with A um uh with um with, with us the software developer in order for us to train a global model that may benefit both CBS and Walmart because the data sharing um the the data sharing itself exposes some of their business information, right?

Um For example, let's say CV, S and Walmart had shared their, their data and pulled uh into a bigger pool and trained one common model CV S may be able to infer business activities or, or even um the pattern of business activities uh at Walmart just by calling the model which leverages some of the data.

Um uh Let's uh uh uh uh yeah, uh some of the data from Walmart. So there is some sensitivity in data sharing which may not exist in the consumer space. OK. So the pros of collaboration between customer and training, a common model is that you may predict more accurately but some of the companies, some of the members of the collection may be free riders or even competitors. So how, how, how do we get the different companies to share data? It's the second dilemma. Uh And the third, the third challenge here for the enterprise A I space is model flexibility. How do we build one ML system that fits all? Uh When we come to uh when it comes to uh consumer products, let's say youtube recommends the next video for me. The the use case is simple and my desire and your desire also also simple. Uh We just want to see the most relevant video um and no matter um category or genre, but the model structure is the same

right for a

consumer uh for a consumer product um in front of different

users.

But when it comes to enterprise product, uh different enterprises may use a collection of many different enterprise systems. So the software landscape is different between companies. Uh In my previous example, let's say CV S may use, I'm just uh um as an example may use uh uh uh an SAP er P system and then the and then, and uh and sales force uh for uh for, for customer relationship management and then oracle for um for example, uh financial reporting. Um But then Walmart may use their homegrown system for uh for, for financial reporting and oracle for er P. So different enterprises use a different set of systems. And then if the machine learning model needs to pull data from all these different systems, right, we will need to build different connectors for different customers. And then the connectivity piece is very time consuming um because different customers may use a diff um may, may use the same data field differently. Also, let's say uh a very simple example, the idea of the order, the order id, typical customer will just use a business number generator, a serial number and it just um increments from 1 to 10, for example.

So this ID generates by itself with a simple rule but some other customer may might fill in the id field with another um with data that's pulled from a different system. So they were in my copy paste from another system and pull that id field to fill in this field. So if that is the case, what the id field means also differs from different by, by different customer. So if the machine learning system needs to leverage that particular field, we also need to define what role that field plays into the machine learning model. So I I hope that makes sense model flexibility. How do we build one ML system that fits all customers? Because each customer, each company is a distinct animal. And number four challenge is integration compatibility. Uh Corporations usually have pre-existing systems, as I said, they might be already using SAP for this and sales force for that. And then when you release this A I software, it needs to get integrated with all the other systems in this um and the software landscape and then the uh the, the input into the machine learning model may come from a specific set of systems and then the output also needs to be fed into another or a few other systems.

Um You will need uh to work with the customers uh to find out the specific software landscape. And in that case, we typically need to rely on a um solution or a or a it consulting company and to,

to, to come

up with a, um, a tailored solution for, for each company for each customer. Ok, so I, I see I have, uh, 10 more minutes and, um, that's all the presentation material I have prepared, but it is a very deep topic and I look forward to receiving your questions or comments in the chat. Let me stop here so I can, so I can

see the chart. Ok. Yeah, I'm Cindy Dustin. Thank you for responding. Ok, let me know if you have any questions or comments. So um if you want to get in touch. Thank you, Mary. Thank you, Cindy. Yeah, thanks Mary. We are all ready. Thank you, Carly. I will stay here to uh another six minutes. Ok. Got a question. Yes, Cindy, what do you do when companies lack data for training models, et cetera? So for this question, there are two answers to it. Uh First of all is to find a proxy for the missing data. Sometimes the customer contact who had approached us is in one department and he or she may not know um whether the stimulator would exist in the other departments. So make a recommendation to her or him to explore internally or explain the process to her and bring some with uh with her to find out. Uh Maybe there's other fields that other data that can be used to, to sub in uh in this case. Uh So yeah, uh so different proxy for the data. And then secondly, uh we are also find a so called collaborative mode where constant or companies can all hold their, their the data together to train the common model. But you can imagine there may be many considerations here.

But if a customer um agrees to join this collaborate uh this consortium, this uh this company will need to sign the agreement and join the construction and then enjoy the benefit from the pre trainin collaborative model. So in that case, if company A does not have the data, but the company A joins the consortium, um this company will be able to still use machine learning by using a pre trained model uh based on construction. Hope that answers the question. And yeah, when there's no data limit, really nothing more, we can do, we have to help the customer find data. Thank you, Cindy. Yeah. Thanks everyone for a stay. Feel free to send your questions here. What is the most challenging problem when implementing A I that you've come across? Well, that's another great question.

Uh Most challenging problem. When implementing A I, there are many challenging problem and I can um go through each one of them in the initial stage. It was about getting the customers to share data because as I said, many of the customers just dropped out of the process and they kept chasing and chasing the customer and they also um they're also frustrated with their internal legal process. Um So, so I was little um anxious about not being able to get any data to get the start to have to get the project started. So that was the first challenge uh once we had the data and was in the model building process, it, it was about uh finding out the commonality between customer data sets because as I mentioned, different companies do business, do businesses differently and um they may have a different set of data fields, data columns and how, how do we come up with a common approach that fits most of the customers.

Uh That is challenge number two and challenge number three was about deploying uh the model at a customer site. Um Sometimes the customer

has a really high expectation of machine learning.

When we train the model, it says accuracy is, for example, 87% the customer was like, no, we won't deploy a model unless it's 95% accurate. So it is really hard to get a model uh accuracy up to 95% accurate. Um But we also need to be very um honest about what the model accuracy is. Um So it's also about meeting the customer expectation. And number four, a challenge I would say is, is the general public's uh expectation of A I. Many of the prospective customers who approached me wanting to hear about A I, they have a total uh different mindset. They, they were expecting something like sci fi or um or A I being able to just do anything for them. Uh Whereas in this particular stage, we are still building task specific and domain specific prediction models rather than general purpose A is. So the, so when I hear about customers throwing very ambitious uh tasks of, to me, I have to sort of taper down their expectations and, and, and talk to them what's realistic. Yeah, thanks Cindy. I see your next

question. How do you guide a customer as to what problem

to use A I to solve other than

the obvious, such

as repetitive tasks? Yeah, that's also a great one. How do you guide the customer as to what problem to use A I to solve? Yeah. Um so we will first find out the customer goal. Sometimes when they talk to me, they already have a solution and they say please spell this solution, but the proposed solution may not be the most efficient one. There may be a simpler solution to the problem. So the first step is to find out what the real problem, real goal is without actually trying to jump into the solution part and build out the customer requested solution. And then secondly, um it, it's about getting the customer um through the A I process. Yes. Um Actually many of the problems that customers raise can be solved by a business intelligence tool. Uh They just want to see the patterns or, or get an uh an alert when, when the um when something happens or when the pattern changes, when the trend changes, when they get an alert that actually does not require uh machine learning solutions. Um What uh what, what I did usually is to, is to find out what existing solutions they have and what simple add ons or, or, or extensions um can be introduced in that process.

If uh after all these validations, we agreed that a new A I or machine learning model needs to be built, then I will say please share a sample data with us. And that, that, that data set would require all the necessary input

features into the model. So the

influencing factors uh of of the model and also the target variable that needs to be predicted. Um and then go through the process with the business contact um to find out what, what are the outliers and um how, how, how do they like the model to perform? Because I can give you a simple example um in supply chain transit time uh in supply chain lead time prediction problems. Um There can be two models. The first kind of model predicts tries to, to predict the transit time and tries to be as accurate as possible. But there can also be a second kind of model which tries to predict the optimal um promise date such that when you tell the end customer this promise date, um uh you will uh your your likelihood of being on time is above 95%. So these two different business problems use the same, the same input data set as well as um the same data fields. Um But they, but they address slightly different problems. One is to get to uh is is trying to predict the transit time as accurate as possible. The second is trying to get um it is trying to deliver on time, right? Um They are similar problem but different angles so that will also train different kind of um machine learning models in that process. Um Talk with the customer in detail. What do you actually want to do? Because in in the beginning, they may say we want to do X, we want to get as accurate as possible, but then they change their mind and saying we want to get there on time.

Um So there are actually two models we need to train in this case and need to be very patient with the customer too. Yeah. Thanks Cindy. Thanks for the great um comments and and questions very thought provoking. Ok, I see that my time is up. Thank you everyone for joining today's session and um please feel free to get in touch. Thank you. Bye. Have a great day.