#187 Mohamed Elgendy: Systematic Testing for Generative AI Models with Kolena

0:00So we are working with the industry, with our customers and with the regulators to put this standard out. for everybody. There's no sensitivity in that and it's in our best interest that it's copied. Everybody built on it. For now, our customers trust us because they see that we have a framework and they see the results from our work. We are planning to build our own eval LMs, so we think that quality LMs. And this quality LMs has two jobs. One is understanding what are the metrics that are used everywhere. We have millions of test runs running every month. on a wide set of applications, even within LMS, so a wide set of applications of that.

0:41Hi, I'm Craig Smith, and this is Eye on AI. In this episode, I speak with Mohamed Eljendi, CEO and co-founder of Kalena, an AI quality platform that helps teams test, validate, and ensure their AI products perform as expected. Mohamed shares his insights on the challenges of testing machine learning systems, the importance of having a systematic framework for AI quality assurance, and how Kalena is working to define a gold standard for AI evaluation. We discuss the role of human evaluation, the potential of using adversarial models for testing, and the future of AI quality standards. Kalena and the MLOps community are joining forces to host the inaugural AI Quality Conference, or AIQCon, the first conference in the industry to focus on advancing new standards and best practices for AI model testing and quality.

1:44Taking place on June 25th at the Hibernia in San Francisco, AIQCon will bring together leaders from Cruise, NVIDIA, Google AI, Anthropic, U.com, and many more, as well as top VCs and journalists to discuss how the industry can consistently build high-quality AI systems, prevent model failures, eliminate biases, and drive greater adoption. I hope you find the conversation as interesting as I did. Hi. Good tech solves problems that you've thought about. Great tech solves problems that you haven't even thought of. What can the commerce platform trusted by millions of merchants do for you? It's time for Shopify.

2:31The commerce platform revolutionizing millions of businesses worldwide. Whether you're a garage entrepreneur or IPO ready, Shopify is the only tool you need to start, run, and grow your business without the struggle. Shopify puts you in control of every sales channel. So whether you're selling satin sheets from Shopify's in-person point of sale system or offering organic olive oil on Shopify's all-in-one e-commerce platform, you're covered. Shopify powers 10 % of all e-commerce in the United States. And Shopify's truly a global force, powering Allbirds, Rothy's, and Brooklyn Lennon, and millions of other entrepreneurs of every size across over 170 countries.

3:19Plus, Shopify's award-winning help is there to support your success every step of the way. So sign up for a$1 a month trial period at shopify.com slash IonAI. That's Shopify, S-H-O-P-I-F-Y.com slash I on AI, that's E-Y-E-O-N-A-I, all lowercase, all run together. Go to Shopify.com slash I on AI to take your business to the next level today. Give them a try. They support us, so let's support them. AI might be the most important new computer technology ever. It's storming every industry and literally billions of dollars are being invested. So buckle up. The problem is that AI needs a lot of speed and processing power.

4:11So how do you compete without costs spiraling out of control? It's time to upgrade to the next generation of the cloud, Oracle Cloud Infrastructure, or OCI. OCI is a single platform for your infrastructure, database, application development, and AI needs. OCI has four to eight times the bandwidth of other clouds, offers one consistent price instead of variable regional pricing, and of course, nobody does data better than Oracle. So now you can train your AI models at twice the speed and less than half the cost of other clouds. If you want to do more and spend less, like Uber, 8x8, and Databricks Mosaic, Take a free test drive of OCI at oracle.com slash IonAI.

5:05That's E-Y-E-O-N-A-I, all run together. Oracle.com slash IonAI. That's oracle.com slash IonAI. Well, thanks, Greg, for having me. A bit of background, I'm Mohamed, CEO and co-founder of Colena. Before Colena, I'm a technical founder. Before Kalena, I was in machine learning leadership roles for a big part of my career. I worked at the AI teams at companies like Amazon, Trilium, Rakuten, worked on different set of machine learning problems. Before that, I was a software engineer, and my education was in biomedical engineering, so software and hardware engineering and healthcare field. Okay. Yeah.

5:51Tell us about Kalena. How did you come to start it, and what does Kalena do? Yeah, so things started with us when, well, Kutena is an AI quality platform that takes care of the testing and validation and ensures that the team is building a product and AI product that they know exactly how it's going to perform and they have tested it regularly in a systematic way. things started with us before that when we like a few years ago many years ago started we were working on building AI systems we started noticing that every time I worked with a team we have to build some kind of solution for testing the reason here is machine learning is not a deterministic process or not a deterministic product these algorithms they define the logic as opposed to engineers defining the logic in software So that ends up making the testing process very tedious and time-consuming.

6:49And more importantly, it's an ad hoc process that heavily relies on the scientist's intuition and their experience in machine learning and experience in the domain and in the company. Because they need to remember, oh, we failed in these areas last year. We failed in these areas this year. So I need to look into that. The nature of this process being not a systematic process and a very ad hoc process leads to a lot of blind spots in the test, leads to misalignment between teams. And then the end product is something that nobody knows whether the scientist is building, the product team that is actually speccing the product, or the user.

7:28Nobody knows really how it's going to perform in all scenarios, which led to the term that's known as long tail wedge cases. Now, this is compounded now by the generative models that started now, where it's trying to achieve more of a generic purpose that makes it really, really hard to start relying on the scientist situation. it was really hard before and now it's becoming impossible and we're seeing in the industry how things are going uh with the we'll probably talk about it if you bring it up like with all these models and how things are going uh so we with that said we have uh we started with uh looking into how we've been building models whether it's language models vision models uh structured uh data or time series like recommender systems and structured data side we started seeing that okay there's a need happily for us as builders to have a framework first of all not even tooling i think what is that framework for testing this happened in software which is machine learning is software so what happened in software when it was started early beginnings of say three decades ago or more more in the early beginning of software everything was still manual and ad hoc and still were not reliable even at the end until the software community started thinking about quality assurance processes and frameworks and started thinking about, okay, we need to have some unit tests and functional tests and user acceptance tests.

8:50Now we have stages and everybody understands. So when you talk to people in this encoders working in software, it's clear. What stage is your software? And we're in beta. So we lost, but still in beta and we're expecting what happens there. That's not the case for AI. It's still we don't know really. The biggest question we hear from customers or practitioners in space, it's not like what's your tool doing? It's more of the first question is, how do we do it? What is that framework? So we're focusing first on defining that framework. Now, what comes after the framework is the tooling process of the infrastructure that automates that framework.

9:31So there are two stages in our head that we're working on in Kulana. One is defining that quality framework and then defining this whole standard of what quality means. What is astrophilic criteria look like? What are you testing? Here, if you're testing a computer vision model, you need to be testing, let's say it's an autonomous vehicle in a perception context. You are testing whether the vehicle or the sensors or the algorithm is able to identify these obstacles or objects. And now you want to test how it's doing for different scenarios, right? Like the occlusion, not occluded, day, night, rain, snow, and so on.

10:08And now you have defined what's called a test coverage. This is what it's called in software. So when you look at this, Mohamed and Craig are working on building their product, you see Mohamed is engineer, Craig is a product manager, for example, and we work together and define what these functional requirements should look like as best as we could. And now they are defined, everybody in the team start, you start having these alignment and trust with everybody in the team who understand what we're building. So you built the specs for the first time in machine learning. So that's the example in autonomous vehicles.

10:37and then you add another layer. How are you testing this? Not just the test coverage. What are the metrics? Is it the traditional metrics that we know about in machine learning, like the precision, recall, accuracy, F1, and so on? Or for the specific application, which in this example, it's autonomous vehicles, then do we need specific metrics? We call this product level metrics. You're not necessarily want to test. If you rely on recall, which is looking at false negatives, you don't necessarily want to test at this level are the recall metric, which is how many objects you have missed or the model has missed.

11:12But a more effective metric, which is a product level metric, you want to call it collision risk because that's the goal. And then collision risk is not just missing the object. It is a function of depth to how far this object is. This way, if you're looking at the collision risk as opposed to recall, which is looking at missed objects, you start seeing that this model A or system A is really good at identifying objects when they are in the collision zone. And then another one is which has better score for recall, but it's really poor at collision. So that's the example here. And that same thing for question answering for LMS.

11:55If you use the traditional language models, metrics like the blue and rouge and others versus custom metric, If it's a Q &A application, then correct answers to test factualness. So it's a function of BERT and BLUE and others and maybe a threshold that you want to set in there. That's what we call a gold standard, which is defining the test coverage. Part of it is some scenarios. Some part of it is data leak. Part of it is hallucinations and others and jailbreak and so on. So I divide the risk with NIST calls this risk management framework. So you map, this is the first function. You map these risks or the functions that you want.

12:32And then the second step is identifying what metrics you're going to use to test each of these functions. And then the third thing is, what is the path of your criteria? So this is what we call a framework to test. Once this is defined in the industry and inside the organization, now everything becomes a systematic process. And then you start evolving the process to more areas and evolving the framework. The models that you're evaluating or testing, are they proprietary models built from scratch, or are most of your customers fine -tuning open-source models or some instance of a proprietary model?

13:17I mean, what, or are you being used by Google and OpenAI to test the foundation models that they're building? Where in that market are you guys? Yeah, we are, we support, like we built Kulina to support the three layers that you mentioned. The teams that is building their own generative models, whether it's language or vision now. and then teams that are using these language models to fine-tune into the application because there's a layer here you need to find the best of these tools for your own application so you cannot just rely on these benchmarks on the internet and you can test it on your own data and then the third one is these foundation model providers we don't have like we don't work with OpenAI and Google specifically but we work with the other two layers and some of them are building their own foundation models for their own applications or they are tuning these applications.

14:18But the step before tuning is what is the best one to start with? Because there are strengths and weaknesses that are different for that. So you want to test it on your own, figure out which is the best out of the tens that are available now, and then guide the fine-tuning process. Because fine-tuning process here, you will want to send data to the provider and here retrain on this data. So you need to understand where the weaknesses are. And this will dive deep into this when we dive into our framework. But we serve these two layers as of now, and we support all the layers. Yeah. And when you talk about coverage, it's not like unit testing where you have a code base and you need to cover a certain percentage of that code base with unit tests.

15:06What do you mean by coverage in this context? Yeah, that's a good question. We've been going back and forth on what is the term because you mentioned unit test. We call it unit test. We understand the confusion here that might cause unit test in software. You'll talk about code coverage. So you're testing the code. unit test we mean here by and on the machine learning side scenario level testing you are not looking at that aggregate metric you wouldn't say llama versus gpt4 llama 2 versus gpt4 90 92 or 93 that is what's done now which is the aggregate metric unit testing is looking at every scenario how is this model doing for hallucinations and then you break you break have no solution down into these stages.

15:52And how is this model doing for jailbreak? How is this doing for data leak? Because data leak here is defined, even like, for example, differently, or even bias. So bias can be gender bias, can be race bias, can be just bias for the kids or in healthcare application. We have one of our customers building their own gen AI for that. So bias here, you can be biased towards a department in the system versus other. So bias here will be defined based on the application. So this is usually what goes with that. Yeah. And how do you do that? I mean, I was just talking to somebody about whether or not large language models will develop a true understanding of the world.

16:43and what they were saying, which was interesting, this researcher, is, yeah, they probably will, but we'll never know for sure because you can ask it a bunch of questions and it can give answers that demonstrate understanding, but you don't know if you've asked the right questions. If you ask five more questions, it may break down. And so you're never going to have a certainty. So how do you deal with that? I mean, you can, in scenario testing, you can come up with, you know, a dozen scenarios and test them. But there may be a 13th scenario that you didn't test in which the model breaks down. Yeah.

17:37Yeah, so I read your article on Forbes last week about the understanding, and I totally agree with his thoughts. Because we need to define what's understanding first, before we say whether these models would understand. From the perspective of, so what these large models do, or in general deep learning, they start painting or drawing this mind map, if you will, if you try to visualize it. And if you have an object like a car, it knows that it takes, when it's training, it takes, okay, a car is an entity. It doesn't know what a car is, but it's an entity. And it falls under the vehicles, types of vehicles, and falls under like a means of transportation.

18:18And then within vehicles, you break it down into cars, trucks, buses, and others. And then within cars itself, you start adding characteristics of cars, what they look like for wheels and so on. And then you start adding how to drive. And then from that context, it understands, right? Maybe, probably, definitely, it recalls more information. So from the information perspective, more than a human, because within this entity, all types of cars are there, better in specific areas and how it respects the car. So from that perspective, they do understand, and they will understand or recall information or put the map of information about these entities better than a human.

19:00Now, whether the understand is something that we need to define, like exact language or intelligence, what is that? So we need to define what this is. From the perspective of a data scientist building a product or testing a product, it should be the same. The idea of how much it understands that this is a car, how much it understands and predicts the next token and the next word, from us, we want to understand based on the product. and we work with our customers instead of going ahead and saying, okay, here's a model or here's an algorithm, then we test it. That is a very big task. A QA is not something that software just started.

19:38Every product we have has some QA process. It doesn't go out to consumers this way. So with that, you need to define the specs first. This is what quality is built to specs. You cannot say this$1 pen is higher quality than$100 pen because it's built for that purpose. So it is high quality. So the way we approach that is we work with customers and start saying, let's build two specs. You are building a, we have a customer that's doing like really, really exciting work automating the drive through process. You know, you go in and you give your order and then actually several of those because it's a very hot area.

20:19Now, if you start and they are in the applications is based on what you asked before they are doing that. They are testing these LLMs and finding the best and then fine-tuning internally and through the provider. So when you start thinking about this, you can't just... How are you going to test it? Because you need to collect data for this kind of test or prompt it as you see results. So you need to think about first, okay, how much is it able to extract the order that I'm ordering? And how much does it understand confusion here? because if you are going, give me a burger, you don't want to be confused by something that's outside the menu, like a burglar.

21:01That doesn't make sense. It should be able to map that somebody see the burger and map it to the menu. Now, that menu, this process is called Rack, which is the retrieval augmented generation. So you want to build first two specs. You define, okay, I want to be able to extract what the order is. I want to be able to handle corrections when somebody or having their customer changing their mind. So now you start building the specs. Now with these specs, and then you want to handle angry customers. So that's another spec. So with that said, you will build your product, your first version based on these specs.

21:35And then you throw it out to your customers and handle those, obviously, the biggest risks that you're worried about. And then give it out to customers. And then it's an iterative process. It becomes the monitoring piece, which our solution involves three main pieces. The data quality for training and then pre-deployment test and then post-deployment evaluation and monitoring. And then the shortest answer that I'll give you is it's an iterative process that starts with, like any product and software specifically, it starts with defining the risks and functional requirements, build the metrics and test coverage for that, and then start testing.

22:10And then now you are systematically chasing the long tail and you're not relying on data scientists in the company with their own experience and leave and come back and build memory, this whole thing becomes a systematic framework. Right. And how much are humans involved in this? How automated is this? So there are several stages. We automated the biggest parts of most of them. So the first part is the data quality, understanding the quality of your test, test data. some known properties of quality of data, like missing values or interesting differences in values and so on. So you clean up, you make sure that you have hyperdensity tests.

22:56So you want to make sure, okay, my test is this way. And then within that, the functional performance that you have just defined, there is the area of identifying blind spots. So the long tail of edge cases here, you have a customer that's been working on a problem for a couple of years. So we have a long list of these functional requirements for the experience. Now you have another customer that just started today or you have another team machine that just started today. You have a much smaller background and much smaller knowledge of the system. So the way we automate this is we are looking into these applications domain by domain and building a library.

23:33It's a as a recommender system, a library of all the, what we can see as known now across the industry, not known for your own company, all the known edge cases across the industry. And it comes as a recommendation system. So you are building, again, this drive-through application, automation application, and then you will have to find 20 of those and you find when I recommend another 40 articles and mapping them to your data. The example of this, like you are looking at it, Like I said, an angry customer and defining an angry customer as how to handle that. So you want to, another customer that is, think of this.

24:08So we wrap this and we tell them within your data, you have 100 ,000 pumps. Let's say, for example, we saw that there's 15 ,000 pumps that are representative of this angry customer. We grouped them for them and tell them, take that or not. And then we can ignore that this doesn't apply to the reputation. So that's the data quantified. Make sure that you have covered as much as you, the industry collectively has known about these unknowns and loans to speed up this product evolution. And then identifying points whilst increasing these test cases. Now, the second, the train, if they train internally, like OpenAI or others, for the fine tuning, and then which is outside Kulana training and everything is outside Kulana.

24:52And then we test on Kulana. So the first component data quality, we talked about it, and then the testing part here, we start defining what are the test coverage, the test cases in test coverage, we just talked about that. And then what are the metrics here for each one? Now, these metrics here, matrices, this is where we get into a fusion between automated matrices and human eval. So, for example, we all use several tools like the GPT tools, question answering tools, if you will, chat GPT and others. And then we have our preferences. You know what, this tool always answers better. It's a situation, this other tool talks more and doesn't follow prompts, for example.

25:42So that is more subjective. And so some test cases or some testing scenarios will require human eval. Now, that human eval here will look at the framework for this. And then you're able to define the questions that you want to ask Craig and your users or your internal testers. And then they go these lists of results for the models and one or more. And then give some eval. The system learns that you are looking for data leak. And in data lake here, you mean you don't want the person's name or the person's age to be retrieved if you're reading from your HR system folder or anything. So having defined the metrics, and a big part of the metrics can be automated.

26:26Another part of the metrics has to be human evolved, but it's then defined now. And then the third thing is defining pass or fail criteria, which is the last thing here. Pass or fail criteria here, it's impossible for a non-deterministic system like machine learning models in general. to tell it, okay, when you reach 90 % automatically deploy, or when you reach 99%, you cannot say that because there's no 100%. So now with that said, there are different ways that you want to define passive-critical theory to get the full automation of that evaluation. So the several approaches that we have is you define critical scenarios or test cases, and then you say, okay, then for any models that I'm updating now, pass or success when it's not regressing on these critical scenarios.

27:18We can never deploy without any regressions on these scenarios. Now, that kind of automation allows us in the training, which is outside our product, that was outside Polina, allows us to throw out 50, 100 or so experiments, push them all together in training, and then automatically when the model converges, where the training and validation set started to agree on the results, then you go ahead and run a test on Kulena. And instead of looking at these hundreds of models, how they're doing, you're looking at only the five or ten that did not fail in your passive criteria or did not progress in your critical cases.

27:55Now, this automation allows faster iteration in training and faster validation process. Right. On the human evaluation, you say you have a framework and then do you plug into some sort of a crowd sourcing platform like Mechanical Turk? I mean, where do you get your human evaluators? Yeah, so that's the plan. As of now, the last two years we've been working with select customers. We're actually about to announce something next week. I'll be more open. But for now, our customers are AI-first companies and they have the resources and they want to evaluate their models by specialized people in the domain, like we are in the healthcare domain and so on.

28:47So with that, for now, our customers are taking care of this quote-unquote labeling, but it's actually called evaluation. Now, soon enough, we're planning probably sometime in May or around May Q2 timeframe, We will start providing these, whether plugging into MTurk from AWS or others, plugging into this human evaluators for other cases. And then applications that are in production, that you have ChatGPT as an example, now we can utilize this massive user base that we have and then build in the feedback based on the customer's response to the prompt. or they give feedback, but it's much more better feedback than thumbs up and thumbs down.

29:34Right. And on the automatic side, is each evaluation strategy unique? I mean, or do you have templates at work? a customer comes in, they have a particular problem, and you have an evaluation protocol that off the shelf that will apply to that? Or do you have to craft a new evaluation strategy for each customer? How general can these solutions be? Yeah, so it's a hybrid of a automated template. It's not really a template. I will explain it a little bit more now. And configurable by the customer to add their own metrics. So the example of that is, let's say we have a text summarization problem.