#207 Noa Srebrnik: How to Mitigate AI Risk in High-Stakes Industries

0:00I see that most of the organization are understanding that sometimes not one model can deal with all their data and all their customers. And then when we are identifying specific places, they can use different models to different population and have just something like a router that's saying that if you are from this kind of population, you should go to this model. We built our platform in a way that it can be agnostic to the type of the model. You don't need a different technique, a different settings in order to work with different. And we want to allow you to have one inventory when all your AI and machine learning are inside in order to really know what are the risks in all your model, in all your use case.

0:44AI might be the most important new computer technology ever. It's storming every industry and literally billions of dollars are being invested. So buckle up. The problem is that AI needs a lot of speed and processing power. So how do you compete without costs spiraling out of control? It's time to upgrade to the next generation of the cloud, Oracle Cloud Infrastructure, or OCI. OCI is a single platform for your infrastructure, database, application, development, and AI needs. OCI has four to eight times the bandwidth of other clouds, offers one consistent price instead of variable regional pricing, and of course nobody does data better than Oracle.

1:32So now you can train your AI models at twice the speed and less than half the cost of other clouds. If you want to do more and spend less, like Uber, 8x8, and Databricks Mosaic, Take a free test drive of OCI at oracle.com slash IonAI. That's E-Y-E-O-N-A-I, all run together. Oracle.com slash IonAI. That's oracle.com slash IonAI. So first of all, Craig, it's very nice to be here and thank you for this opportunity. I don't know how many of your audience are started in the tech industry, moved to work for the government and went back to the tech industry. So I'm one of those. And I started my career in tech.

2:25I worked for VMware company from for a cyber company. But then I understood that a lot of what's interesting is the politics and economy. so I did a degree in PPE philosophy politics and economy and a master degree in economy and I joined the national bank in order to allow them to do the risk so I joined the risk department in the national park and then I worked for the ministry of finance so all my life I'm in the intersection between regulation risk technology and this is what led me to go back to the tech industry but to go in the understanding of what are the regulators are looking for and how we can really be able to bring innovation but care about our customers, our users.

3:16And I started by working in a synthetic data company, allowing the financial industry to use the private data in a safe manner. And there I met with the problem that we are trying to solve now in Citrus X. So all my career is about risk and regulation and high-risk organization. So this is what interests me. It's all about what I'm doing and looking to solve for us in order to be innovative, but also to be trustworthy and be safe. And so tell me about Citrus X. What is it offering, a service or a product? So we are offering a product. We are a platform company, and we are some kind of an AI enabler.

4:08We are enable the high-risk organization, if it's a financial organization, insurance, digital health, security, but we are mainly focused on the finance and insurance and allowing them to bring innovation and bring new products and services to their clients, but doing it with knowing what are the risks, knowing how they can comply with regulation and they can make sure it's doing properly. So we are kind of allowing these companies to doing risk assessment and validation of their model, but mitigate the risk, but in scale, not doing just one model, two models, but to be able to really implement a lot of services, a lot of AI and machine learning for different use cases and to be trustworthy and responsible with them.

4:58Yeah. And the regulatory environment is evolving. It remains very fragmented. What markets are you focused on? Because each market is a little bit different. Yeah. So we are focusing mainly in North America, but also in some of Europe and the UK. and Europe is like being that as we saw before in the GDPR so the same is happening now with the EUAI Act. So they are the frontier of the regulation and they really created what is high-risk organization and we are targeting those high risk. So for us they did the job to divide who needs us in the most critical way. And we are working for them in order for them to allow better services, but also to be able to do it complying with regulation, but not only complying with regulation, but also responsibly and trustworthy.

6:00Because in order for people to use AI, they have to trust it and really be able to understand it. And this is what we are trying to solve for those organizations and provide them. Yeah. In the U.S. market, they're following the EU in some respects, as they did with GDPR. But the U.S. market's very fragmented. Different states are passing different AI safety laws, and they're not all aligned. And so do you track each state or how do you manage that? So it's a very good question, but we see, like we saw before with the GDPR and CCPA, that in the end, a lot of the companies prefer to go to the more, you know, the extreme of the regulation in order to make sure that they cover everything.

7:05So the same we see it in the U.S. But I have to say that some of the regulations that we are seeing that the companies are aware of are regulations that already exist. Because now companies that are doing lending with machine learning, they still have to be compliant with Fair Lending Act. So although it's like an old regulation, they still need their new and innovative products and models to comply with it. So we are allowing them not only to comply with the new AI regulation, but also with the existing one in order to continue the work and continue bringing the services that they want to bring, especially, you know, now with the interest rates and everything that anyone wants to lend money and give those things.

7:51and and so tell me about the platform what does what does it do is it uh yeah i mean i i could speculate but what does it do so uh citrus six is an end-to-end platform it was really important for us to be in all the ai life cycle in the prep production and in the prep production in the production sorry and we are validating monitoring and explaining ai models in order for companies to to make sure that their model is accurate, robust, and governed. So we are doing all the procedure from the minute the model is developed or bought from a third-party vendor, allowing us to validate the model, explain it, making sure that also the non-technical people really understand how the model is acting.

8:43afterwards we're doing all kind of governance measurements fairness testing all the things that you need documentation reporting for the regulation and afterwards after all the different stakeholders approve the model understand the risk of the model and make sure that they can take it to production we are monitoring the model in order to make sure that it's keep doing what they thought it would do. Also, it's remaining with the same risk level. And you know when you need to retrain it. And all of this is based on our proprietary technology that we put a lot of efforts in order to create. We saw that the open source solutions are not sufficient.

9:24So we understood that we need to have a different paradigm in order to solve this complex problem. So we put a lot of efforts in the beginning of the company to really research and have a different and a unique way using synthetic data in order to really understand what are the risks, how you can validate, and especially how you can mitigate those risks in order to take the best model that you can have to production. Yeah. And the models that you're looking at, what percentage of them have an element of generative AI in them, which is notoriously difficult to explain? So at the moment, most of the clients that we have are having in production already machine learning models for lending models, mortgages, KYC, fraud, and those kind of use cases.

10:18So this is what is already in production. But meantime, we are working with design partners. And if some of your listeners wants to work with us to try it out, mostly on RUG models in LLM, because we see that those higher risk organizations will go in the LLM world to the RUG, to the knowledge graph, in order to keep a little bit safeguard of what they are doing with the Gen AI. Right, right. Right. So on the existing models that are in production, those are supervised learning models or, you know, rule-based models. What kinds of models are those? So most of the models that we are working with are the more traditional machine learning models.

11:08So there are supervised models that can be random forest XGBoost and so on, even neural networks. It depends on the use case and we built our platform in a way that it can be agnostic to the type of the model. You don't need a different technique, a different settings in order to work with different. And we want to allow you to have one inventory when all your AI and machine learning are inside in order to really know what are the risks in all your model, in all your use case. and to be able to really know how to manage it and work with it and not to have like a procedure of six to nine months until you're taking the model to production, but to reduce the time to market in order to have the best and most updated model with the updated data going to production.

11:59Yeah.

12:04So walk me through a use case. I'm a bank and I'm using a recommendation engine to decide whether or not I'm lending to somebody. but I would like to transition to a Gen AI system to, you know, leverage a foundation model to, you know, maybe inform the lending officer of policies and help make decisions. What do they do? Do they upload the model to the platform? Do they? So, yeah. So the first stage that when they deploying our platform, most of our platform and most of our existing clients are doing a putting us in their private cloud or on prem, because of the high risk of the organization they are.

13:05So they are onboarding to the system, the model and the training data in order to our work to start working. We are generating synthetic data in order to map all the data space and in order to be able to show and identify for our clients what are the risks, where are they, and how they can identify and give them action items, how to reduce them and mitigate them. And then from the moment that the data scientist finish his work, his onboarding to the system, the model risk management, who are the ones that checks the model and need to validate them, get an alert that the new model is in the system and he needs to start to do his work.

13:50He can create his own thresholds because of the specific use case that he has. And after he creates the threshold, he sees all the results and sees what is good or not or bad, comparing to what he created and he decided. And because of our unique technology, we are not giving only the performance metrics, but we also providing a robustness metrics, a certainty metrics, vulnerability metrics, all kinds of additional metrics that will give you a full picture of how your model is really acting. After this has been done, we are moving to the explainability step. When the business unit or the risk officer needs to check if the reasoning behind the model is what's relevant to what he knows from the business case.

14:44And after this been done, we are moving to the governance metrics. And we check for all kinds of governance metrics, especially if it's lending about fairness. It can be sensitive features that are inside the model that could be sensitive features that are outside, but are some correlation to the features that are inside of the model. and we're showing them all kinds of the fairness metrics. We're identifying where are the fairness issues and how you're supposed to deal with them. And after everyone approving the model, the model can go to production and start working in production and give better service and better lending propositions.

15:27Right. And for each implementation in each industry, Does the platform have access to like the Fair Lending Act or FTC rules or, you know, FDA rules or whatever it is, whatever industry it is? Is that built into the model for that implementation or is that something that's done on top of this other analysis? So we started by giving and providing our platform, especially for the financial and insurance industry. So we work mainly with those legislation and regulation. We did work with a legal firm about the Fair Lending Act because it's something that is coming up from most of our clients. And we are showing them how the metrics that you need to show in the Fair Lending Act is aligned to the metrics that we are showing in order for them to feel comfortable with complying with this regulation.

16:39So in this case, we are really focusing on those things. but also because sometimes there are a regulation or specific things that companies want to take on themselves if it's easel or those kind of things we are allowing them to change all the thresholds depending on what they want we are suggesting what the threshold should be but it can be customized by the client as he wants to take what is the level of the risk he wants to take and how he wants to work with the model in order for all the people and all the stakeholders in the end to feel comfortable by using the system. And also different stakeholders can create different thresholds and see the measurements in a different way in order to get different alerts that are relevant to what they're doing.

17:31I just want to add that it was important for us to give information for different stakeholders So we have a Python SDK that is relevant to the data scientist, a web UI that is more for the risk officer, the management, but also reports that can be relevant to the board of directors, for the regulators, because in the end, a lot of those regulations, the directors will be responsible for them. Right. And so the risk officer has the dashboard or whatever and can watch everything. But when it comes time to file a report with regulators, is it literally like I need a report for the, I don't know who you report to the, you know, for the Fair Lending Act.

18:23and you click a button and it spits out a report that's formatted that makes sense to that regulator? Yes, exactly. So all that you can in advance, if you want to create your own reports, you can customize it. And then when you are in the system in a specific model, you just push and click a button and then you can get and export the report with the information. It can be report for validation report for explainability report for fairness report for specific regulation it can be or about a use case or about performance metrics it depends on what you want to have in your reports yeah do you how often do you run across a situation where um you the whatever system is in the, that the client has doesn't meet the regulatory standards, that your system does the review and the analysis and it falls short of regulatory standards.

19:29And what do you do in that case? So it's a very good question because nowadays it's the end. Some people knew how to do those kinds of metrics, but they are doing it only for the average of all the models. And what is unique about us that we can identify the places that you cannot see. And this is why most of the time when we are doing like a POC, when we want to show you in your data what the value that we can bring. And with your model, we are showing that although you thought everything is okay, you're still having a specific place with a problem. And those are really eye-opening for most of the clients that afterwards becoming our clients from the POC because they understand how much they didn't know because of using the average scores.

20:21So with us, we can really identify that specific population that they have less customer from those might be the ones that are problematic. And because we're generating this synthetic data, we can overcome the lacking of the data in one place by showing them what's really problematic there. yeah and then what sort of do you do you advise on mitigation or uh is it then up to them to fix that so we are giving action items how you should mitigate the problem and reduce the risk we are not doing that change in the model because we want to be the independent risk validator so we don't want to touch the model, but we are telling you and give you all the tools how you can really solve this.

21:13And this is why it was important for us to have the Python SDK connected and speaking the same language to the UI because then the risk officer can tell the data scientist take a look at this in this area, this segment, and the data scientist can use our system to really target these segments, identify the problem, and afterwards to solve it. Yeah. And does that result in, for example, in a lending organization, in adjusting their lending policies? I mean, ultimately, that's what has to happen. You can't massage the data or tweak in their, I'm not talking about your platform, But in their model, you know, you can't just kind of hide it.

22:03You have to, if they're biased against an underserved population, they have to do something proactive to address that with the underserved population so that the data is more balanced. I mean, do people do, do organizations do that? I see that most of the organization are understanding that sometimes not one model can deal with all their data and all their customers. And then when we are identifying specific places, they can use different models to different population and have just something like a router that's saying that if you are from this kind of population, you should go to this model.

22:46And if you are from this population, you should go to this model. And in each of the model, you will have much higher accuracy, performance, fairness scores, because you didn't put everyone in the same pile. Right, right. Actually, that makes sense. Yeah. Yeah, that's fascinating. And it seems to be coming at a good time. So what are you doing about, what is in the tech in your product? Is it, and what are you doing about this generative AI wave that's coming? And frankly, with the new reasoning agents that Sam Altman's talking about, we're going to go through another big shift, I suspect. and a lot of systems are going to come online.

23:44So how do you keep up with those changes? So it was really important for us to work with our clients and to see how they progressed in the Gen AI and to which use cases and what interests them. Because, you know, the Gen AI is still like a lot of the people don't know what is explainability there what you want to gain and understand from there so we didn't want to create something that won't be relevant so we are work working with our current clients as they working towards the gen ai in order to create the right solution and this is why we saw that we need to focus in the reg rag models because this is the direction that they will go to.

24:31Most of the high-risk organizations won't go to the open-source kind of models. They will go more to the co-pilots one or those kind of models that it's based on their own data. So this is how we are targeting this market and targeting our solution in this world. Yeah. And then, excuse me, in your product, is there a Gen.AI component? what what kinds of models do you have in the background in in the under the hood so i really like your question because it was important for us not to have ai in our platform because we don't want to be another black box that's right so we don't want no one needs to validate our our technology we are using statistic and math in our technology which is a open box white box how you want to call it in order for people to feel comfortable using us so from day one we understood that in order to have a solution for this problem we have to get far from doing another ai on top of it we can use you know afterwards gen ai to just generate documentation or reports but in the base of our technology is proper math statistic and things that we can really have the evidence and showing it and let people to feel very confidence with.

26:02Actually, that's fascinating. Yeah. And what are the toughest models to satisfy the explainability demands of regulators for that was a convoluted question but what are the where do you run into a problem with explainability so i have to say that at the moment the only problem sometimes with explainability it's some sometimes that it's not the same language that people are thinking because sometimes it's the language of the features and you need to move it to the language of the clients that it's understand. So we created some kind of a dictionary that takes some of the features to our words that people can understand and interact with and know what does it mean.

27:02But other than this, it was important for us comparing to the open source solutions to get something different, that it won't be problematic to do the explainability for local explanation, global clusters, for groups, for all of those in order to give real confidence, like 100 % confidence that you really understand how models are working. Yeah, I didn't mean the explainability of your product. When you're dealing with a client and they need to explain how they reach a certain conclusion, uh which what kind of models do you run into over and over again and it's like yeah that's a problem because you really can't explain that or or how do you explain it again i'm talking about your the clients uh models that you're evaluating so we are giving them the explainability layer in order for them to be able to explain it.

28:13So we see that they can explain the model in a way that it will be in the end relevant and understandable to the non-technical people. As we see the model that in production and meet the clients beforehand, it was only like they felt that they need only to have like the default kind of explainability just to give a reasoning about like what the model is made a specific decision but nowadays when the regulation is becoming tougher and there's starting to be lawsuits and all kind of talking about this companies understand that they need to be able to give a specific explainability and to be able not to get to to have feature importance and feature impact.

29:01And to give you the understanding of the counterfactional means like how you can change the decision. What are the changes in your features that you need to do in order to change the decision that you receive? So they need to give a much more wider ways and definitions of explainability, less than just what is the feature that's affecting the model. Yeah. And I'm thinking about even supervised learning, just neural networks. You can explain how it works, but you can't really, when you're dealing with very large systems, you can't really uh track the uh the changes of the weights and parameters that eventually get to an outcome because there or maybe you can uh it seems to me you can't because there's so uh there's so many and then in generative AI, I'm thinking about Llama 3.1.

30:14I mean, a lot of your clients presumably are using open source with a RAG component. But Llama 3.1 meta has never shown it's revealed its underlying training data. so even if you have llama connected to a rag how do you understand how llama works i mean those kinds of are those the kinds of problems that you face or or am i off so so probably we will face those but at the moment in production we are facing with newer networks and machine learning which are all kinds of trees. And we are able with our technology because we are creating this data space also in neural networks to be able to give them the definition of the explainability behind each and every answer.

31:19And we can do it in production or in prep production in order for you to understand how the model is really going to act and what are the features that are affecting those. And as many features that you will have in the model. So you will have, you know, different proportions for different features. But we also created something that we called feature bias, because sometimes you see that there is a prediction that is affecting by one or a few features. And it's not really, it's not good because you want to have some kind of way that each and every feature that you put in the system have some effect on the decision.

32:02because if you have one or two features that affecting too much, a small portion of changing those features can change all your answer. So a lot of the time we also alerting you to know, to look, that you have a few features that affecting too much and you should take care of it in order not to have too many vulnerabilities or changes that you might happen in the real time. Yeah. Have you gone through a full cycle with any of the clients where they're reporting to regulators and with reports generated by your platform? And I mean, presumably, as you start this process of clients reporting to regulators, regulators are this is all new to them as well coming up with questions even if they have a report i mean how how much of a learning curve is there on the reporting side so i have to say that it was important for us to sit with the regulators so we saw we met with some of the regulators we setting their round tables in different countries in order to hear also from the regulators what they are looking for to get i have to say that some of them don't really know yet what they are looking for to get but because we are starting in a way that in some use cases that already have regulation so we started we try to keep as a in the the known language that they familiar with So we are taking, in fairness, the measurements that they know or things that they are familiar with in order to put in our report that they can feel comfortable with.

33:57We are not trying to create a new language that they will get lost with. We are trying to let them have all the information and feel comfortable about it. Yeah. And in that process, do you sit or do you communicate with specific regulatory bodies, show them the reports that you generate and say, would this satisfy you or is there something missing? or yeah so i have to say that some of the regulators are are feeling more comfortable to say their own opinion because sometimes they don't want to be you know for one company or the other but they're giving you like a some kind of thumbs up of you are in the right direction but i have to say as i mentioned before i'm not sure that they really know 100 what they're looking for So a lot of the regulation, for example, in North America are looking for the European to see what they are looking for.

35:00But we see that at the end, all the regulators want to have information allowing them to see that the company did a procedure. They don't really know, you know, check every bit and bite, but they want to see that they know and it's important for the company to know what is the data, how it was collected, what is the procedure that the validation went through, what is the governance measurements, the fairness, to see that there is a process, there is a way that they really making sure that companies feel that they need to be accountable to the process to the to the responsibility of using ai and machine learning correctly yeah uh that's uh the um

35:52how crowded is this space because it's a new space for for your company um and i've spoken to a few people that are attacking it in different ways. And some of the big companies, I know IBM has a whole suite of tools for fairness and explainability. So is it just, is the market as this regulation comes on so vast that you don't worry about competition because there's enough business for everybody or is it a tight market uh because they're big players and you're you're trying to uh you know get attention or get credibility or get whatever to to play so of course it's uh it's not it's not yet a crowded market because it's a new market it's true that what you said about the big companies But in the end, when you were speaking to the risk officer, the business unit, they are not capable to use really the very complex technical things of all the big companies.