#195 Mrinal Manohar: Building Secure AI Systems with Blockchain

0:00If you have like four trillion hashes, it's such an overhead to go through them. So it's really about very smart and efficient batching. You do it at the model level. You don't say, okay, I have 30 models running. I'm going to put them all in one pot. So model becomes, you know, one subset. And then each of those models could have a version number or, you know, a certain parameterization. So now you've created more branches of the tree. And each of those was trained by a certain set of data sets. And as you see, then you go down into a tree, which is very, very similar to what Walmart does. You have the aggregate financials right up at the top.

0:38But that is a combination from the tiny store in the book, all the way up to the massive store on the outskirts of Chicago. Hi, I'm Craig Smith, and this is Eye on AI. Today, I speak with Mrenol Manohar, the CEO of Casper Labs. With a background in computer science and experience in AI governance, Mrenahl discusses the innovative use of blockchain technology for AI governance, the importance of tamper-proof records, and the future of AI compliance. We explore how Casper Labs is addressing regulatory changes and creating a more secure and efficient AI landscape. I hope you find the conversation as interesting as I did.

1:23Can you start by introducing yourself, give your educational background as far as it's relevant and then then we'll start talking about Casper. Absolutely. Hi, I'm Rinald Manohar, CEO of Casper Labs Holdings AG, which is a Swiss-based company that's working in the area of AI governance. In terms of my background, I'm a computer scientist. I got my MS from Carnegie Mellon University. Research areas that I was particularly interested in include neural networks, formal language theory, etc. You know, the base rudiments of artificial intelligence. Great to see the whole thing come full circle. In terms of career, I actually spent a lot of time on Wall Street.

2:11I spent a few years at Bain & Company, the management consulting firm, and then Bain Capital in private equity for another two to three years. That was at Cigard Holdings between 2012 and 2017, and all these jobs focusing on the TMT sector pretty much exclusively. Founded the company in 2018. So it's been about five years and really excited about what we're doing. Thanks for having me on, Craig, by the way. When you say you focused on the TMT sector, was that as an analyst or were you doing something quant like? Both. So while I was at Bain Capital, I was an associate. I joined that job when I was 24.

2:55So I was there for just under three years. A lot of it was quantitative, but quantitative analysis, more so financial statements. But given my deep interest in tech, I'd always take it a level lower, look at what their coding standards were, essentially really examine if a company, another competitor could replicate their mousetrap. At Sagard, obviously slightly more senior. I was a vice president and later a principal and sector head of technology, media, and telecom. And so basically did all the quantitative and qualitative analysis of companies in the technology, media and telecommunications spaces.

3:34Yeah, and so tell us about Casper Labs. You're involved in blockchain. I was interested in blockchain a few years ago, but it got so crazy I've kind of pulled away from it and really focused on artificial intelligence. But you're kind of at the intersection is that right that's that's absolutely right so let me let me just step back we're an enterprise grade blockchain infrastructure company or just an enterprise grade infrastructure company our interest primarily in blockchain early on was that blockchain is really the best version of tamper proofing that there is compared to a regular database, you have a completely tamper-proof record of things.

4:25So there's various applications in any sort of governance tool. And on top of that, it allows things like multi-party access, Turing complete programming on top of it, which a database just doesn't give you. Now, what we've been really looking for, what's the right use case to really bring these you know, these characteristics to bear in the market. And AI governance, we actually started work on this well over a year ago, about April of 2023, even before, you know, all the legislation came out. We realized that since you have this tamper-proof immutable record, it has massive implications for governance of AI, where you have these rapidly evolving models.

5:07And if you want to track and trace their development, or have an accurate audit trail, or perform version control, for example, this is really the best, cheapest, and most secure technology to do this. Yeah, and I've been talking to people, I don't know if you know Oasis Labs out in California, it's Don Song out of Berkeley, but they're working on differential privacy and productizing differential privacy for large corporations. And over the years, I've talked to people about federated learning and things like that. Do you guys have a product or are you a SaaS platform? Are you a consultancy? What's the primary offering to corporates?

6:02The primary offering is a product. We're building a product called Prove AI. we actually showcased the proof of concept in November of last year. We co-hosted a Gartner webinar with IBM where we showed the proof of concept, where in real time we showed end-to-end data integrity and tracking of an AI model. We showed version control as well as multi-party access to governance. And it was based on the interest in that webinar that we decided to go full bore and develop this product in collaboration with IBM Consulting. I see. So coding is active. It's been active since early this year. And we're launching Q3 of this year.

6:44I see. And when you say in collaboration with IBM Consulting, are they, in effect, the sales arm for the product? I mean, will they take the product to their customers? So I'll give a nuanced answer. It's a qualified yes in the sense that the product integrates seamlessly into WatsonX. So the WatsonX platform has three components, WatsonX.AI, which is essentially a model studio, WatsonX.Data, which is curated and pre-indemnified datasets, meaning you know that there's no PII or copyright information on it. And then, of course, WatsonX Governance, which is a newest product. And all of these are really built on the OpenPages platform.

7:30You know, IBM has been at the forefront of GRC for the past few decades. And so the product Prove AI will seamlessly integrate into the WatsonX platform because it's a better together story, meaning you get all the UX and open pages design that IBM's known for, as well as the augmentation from the blockchain, which is tamper proof and auditable records. So your ISO 42001 and EU AI compliant, multi-party access, as well as version control. So, it's in both our interests to sell the product and also in the interest of the customers to be compliant with the raft of legislation that's coming up. Yeah.

8:13Well, to that point on the raft of legislation, what legislation specifically does Casper help address? Yeah, so specifically, and when we initially built the product, just to be fair, in April or May, when we came up with the idea, none of the legislation had actually come out. To be honest, we're surprised at the speed in which governments have reacted. We thought this is probably half a decade out, but let's get companies prepared. Specifically, I'd say the EU AI Act, we have a strong belief that's similar to GDPR. It'll be de facto in many ways. Obviously, it's in a nascent state. I actually think it's a pretty well-written act, especially given it's in a nascent state.

9:00And it's likely to be modified significantly. But off of the EUAI Act, ISO started a standard called 42001. If you're compliant with 42001, I mean, nothing's ever black and white, but you're, in essence, compliant with the EUAI Act. But if you looked at the acts coming out of the UK, Colorado, California, etc., all of them are very, very close to what the EU AI Act is. And if I were to step back, you know, not go through the entire act because it's fairly long, essentially what it says is the level of auditability you need for your underlying AI system builds needs to approach that of financial statements, meaning the old world of, hey, it's AI, it's a black box.

9:44we don't know what's going on, that's not acceptable anymore. You're going to have to be able to give an auditable trace. And there's this concept of adversarial robustness where you have to prove that your trace has not been tampered with. And you combine those two things, the combination of the WatsonX platform and what we bring to bear with Prove AI basically addresses all those points. Yeah. And by using the blockchain, this enterprise-grade blockchain, That's the immutability of that and the ability to store data on the blockchain. Are those the sort of key components of what you're doing?

10:29That's one of the key components. But if I may talk about a couple more. So one of the key components, obviously, is it's an excellent form of copy protection and immutability. You know, the records don't change. As a result, it saves you a lot of cost because you don't have to have a third party verify it or a third party can easily say, yep, it's clear to us that this hasn't been modified. But blockchain gives three more things that you wouldn't get off of a database. The first is multi-party access is very, very easy on a blockchain system because it uses a private key system. who has access to what information, to your point early on in the conversation about privacy and data integrity and privacy issues, all of this is out of the box with a blockchain because managing access control is an inherent part of an underlying blockchain.

11:21The second thing that it brings is any programs written on top of a blockchain are Turing complete. So if you think about a database, you're beholden to SQL, which is an input-output language. And so more complex access control or more complex triggers can't really be set up on a database, especially in a multi-party environment. And third, as a result of all the things I've noted above, you actually get the best form of version control for your AI because you are tracking and storing all the fundamentals on this immutable ledger. ledger, if your AI starts misbehaving, instead of trying to solve it the brute force way that they do today, which is retrain the model, which is A, very expensive, and B, non-deterministic, you're hoping for the best, you can actually just revert the model back in time to a state when it was behaving to specification.

12:17Yeah. So for people who aren't familiar with blockchain uh with how it works um i'm trying to let's let's take uh i mean one of the things you do is protect the integrity of training data and make that training data data audible audit auditable is that right that's correct so so talk about how you do that and how the blockchain does that. I mean, blockchain essentially is a ledger where transactions are recorded. It doesn't necessarily hold the asset, right? That's correct. It holds two things though. It holds the transaction, meaning this data set was used to train this model at this time that was working under these specifications.

13:16So all that metadata is stored, but it also stores a hash of that data set. So meaning not to become fully auditable, meaning if anyone modifies or tampers with that data set, those hashes will no longer match. So you don't need to replicate the data within the blockchain, you can keep that within your data lakes. And if you're using Snowflake, or you know, a standard Oracle database, you can keep it where it is. It's just that the record of the trace path is completely immutable. Meaning if there was any modification to the data set, the hashes wouldn't match and then that way you could not prove auditability.

13:54So if you look at the way the acts are stated, you essentially do have to snapshot your data sets and be able to clearly prove this is what I used going forward. And so you get both those things from blockchain. Yeah. And on that, on the training data, the auditability of the training data, how, do you understand how that works? That's something, you know, people say, but these, the OpenAI, the GPT-40 training set is larger than anything that a human could crawl through. So how do you audit that? I'm going to give a two-part answer. So first part is essentially you replicate what a general ledger system looks like in a financial statement.

14:54For example, if you take Walmart, right? Walmart does billions and billions of transactions, but their financial statements essentially aggregate that. You have proof at each store level. And so essentially, you transform that into a general ledger system. You essentially say, anytime a data set is going in, we're going to track it, we're going to capture the hash. And so we know that this transaction occurred. Now, it brings up a second point, which is where's the point at which it's useful versus useless, right? If you have like 4 trillion hashes, it's such an overhead to go through them. So it's really about very smart and efficient batching.

15:34You do it at the model level. You don't say, okay, I have 30 models running. I'm going to put them all in one pot. So model becomes, you know, one subset. And then each of those models could have a version number or, you know, a certain parameterization. So now you've created more branches of the tree. And each of those was trained by a certain set of data sets. And as you see, then you go down into a tree, which is very, very similar to what Walmart does. You have the aggregate financials right up at the top. But that is a combination from, you know, the tiny store in the book all the way up to, you know, the massive store on the outskirts of Chicago.

16:12all of them aggregating up to the top. Now the AI data set, ChatGBT is a exception here, but for most enterprise use cases, if you have a AI chat bot that's giving you insurance quotes, yes, it gets pretty complex. You're drawing actuarial information from several sources, but you essentially replicate the general ledger system, as I said before, and then it becomes way less unwieldy. Essentially just what you've been doing with financial statements before, And it's all automated. If you've written your smart contract appropriately and you say, every time I train it, put an imprint here, all of this is done through a UX.

16:49So you don't have to really write any custom code. It essentially aggregates the data such that you have a general ledger of sorts of everything that happened with your underlying AI models. Yeah. And then the actual auditing, is that done with AI? I mean, who does the actual auditing or is that just sampling and that sort of thing? The good news is you wouldn't have to sample that much. I mean, look, again, there's a difference between practicality and being compliant. You wouldn't need AI to audit this, right? You could just send a trace and say, okay, this is the trace history of the model.

17:35do all the hashes match to all the data sets that you've stored? If the answer is yes, you can just spawn a certificate. And then a third-party vendor like Vanta, Shellman, or one of the big four can look at that and say, yep, this is on a highly secure, highly cryptographic ledger. All the hashes are matching. So whatever you're telling us you said you did is what you did. And so that's how it works in real time. Now, obviously, if you're a company, look, this isn't happening now, but who knows in a decade. If you're a company running 100 AI models, which is each looking at thousands and thousands of data sets, there might be some sampling involved similar to how audit at large companies work.

18:16They take 10 random stores, make sure everything's above board, and then assume, okay, it's been above board across. There, it's a little harder to comment because that depends on how regulatory standards will be enforced. In financial statements, some level of sampling is allowed. I assume as the volume of AI data sets and AI models increases, sampling could be allowed as well. But the good thing is since it's stored on this data structure that's Turing complete, a lookup is not actually that complicated. Like even if you had to go through a billion records and just reconcile the ashes, we're not talking months of work.

18:54We're talking days. Yeah. And the product that Casper's developing, it will record where the training data resides, when it was created. so you can track the provenance and model versioning of the data, I guess. Does that mean you can track the model versions? Yes. So what we're tracking over and above multi-party access, because that's really important to AI. But essentially what we're tracking is anytime a model has a context switch. And a context switch means it'll behave differently than it does currently. And the way that happens is one of three things happens. Either it's trained with a new data set or your vector database that's performing retrieval augmented generation has an update to it.

19:59So that's a new context. The second is, has the tuning of the model changed? Meaning the parameterization. Have you changed the number of tokens if it's a token based model? have you changed the P or K values? Have you changed the sampling method? Any change there would also change context. And then finally, of course, any update to the core model itself. You know, if you go from flan 1.1 to flan 1.2, you've obviously created a context change. So if any of these three things change and you know, these change fairly rapidly, if you're deploying a lot of models, that will result in an imprint. Now, obviously with the retrieval augmented generation, it goes back to how practical do we need to be?

20:43Because, you know, vector database could be updated. You know, if it's spitting out sports scores, for example, it could be updated every millisecond. So you might decide that the right batching mechanism is once every minute. We take an imprint every minute, that's the lock. And so there are certain areas, especially in RAG, where you have to think a little deeper. But for general data set training, you know, you cover those three, you're essentially covered. That being said, that's how AI is currently being used. We don't know how it goes in the future, but the beauty of, you know, Turing Complete smart contracts is you can always add another primitive if that primitive becomes substantial to the governance of underlying AI platforms.

21:27Yeah. How are companies doing this now? By and large, they aren't or they're doing it highly manually. AI governance is a very new industry. I mean, there have been some competitors who've been around a while, but it seems the focus has been much more on MLOps, you know, having model studios and helping with that plumbing. The rush towards AI governance has actually been spurred by both a general call it, hey, let's stop and make sure we're doing this right, for lack of another term. You know, there's just a general feeling that AI could go off the rails. And by the way, I'm an AI optimist. We're not doing this because we want to slow things down.

22:14We're essentially saying we give you the seatbelt so you don't have to worry about that. Drive as fast as you can. And the second reason is on the back of, you know, all the legal threats that have gone back and forth for PII, copyright infringement, you know, that Google bot showing Nazis of varying times. types, you know, because of that, it seems like governments have taken, you know, very, very strict action. The EU AI Act actually goes into effect in 2025. So this has actually become one of the top of mind subjects for CISOs, CIOs, chief risk officers, because, you know, the clock is ticking.

22:54There's about six months to get ready to be compliant. Obviously, it's not going to be as strict in the early days as we saw with GDPR. But over time, you can expect, as with GDPR, Sarbanes, Oxley, etc., this just becomes a de facto way of tracking and tracing what you're doing. Yeah. Can smart contracts on blockchain enable automated triggers to detect or shut down rogue hallucinating AI systems? I mean, does it have that kind of capability? Yes, absolutely. In fact, the way you'd set it up is access to power and access to compute would basically be a switch on the blockchain. You set up your triggers.

23:35And if the trigger gets overridden, essentially, the blockchain would cut off power, essentially shut down the model and make sure it doesn't have access to anything. So, you know, while that's not our primary use case that we're going off, we're not looking to just build a kill switch. The kill switch does exist. Yeah. Can you explain that understanding? That's not your focus, but it's something that the public certainly talks about. How would that work? Yeah. Just to set up historic context. Right. So if you watch Matrix or Terminator, like the main thing that caused Skynet to happen or models going really, really rogue.

24:20And obviously, we're not close to that yet. The main thing is they figured out, OK, we have access to our own power now. And so once that happens, you know, they don't need us. Right. There's no intermediary needed. So essentially, you need a technology that is absolutely not affected by AI to decide whether it gets access to power, access to resources, etc. Blockchain is a completely decentralized system, meaning, you know, take our network, for example. there's tens of thousands of delegators. It's their combined vote that decides whether a transaction goes through or not. So someone writes a smart contract, essentially you say access to this power is governed by the smart contract, is currently set to on.

25:03And the smart contract looks for a set of triggers. For example, is the AI asking, is it emailing vendors trying to figure out whether it can get power from somewhere else? Is the AI acting outside of bounds? Meaning Is it, you know, it's a robotics facility? Has the AI, you know, started like making the arms do things that it's not meant to do? Is it putting humans in the category of food? You know, all sorts of things can happen. You set your triggers up and you essentially say, if any of these triggers are ever overridden, immediately cut off power, immediately cut off access to compute, and the AI just shuts down.

25:41or if it tries to copy itself. You can't imagine the number of scenarios, but essentially anything you can imagine, you can set up as a trigger within the blockchain. And essentially it'll automatically, because smart contracts are too incomplete, shut off that power and give you time to go audit it. But again, that's not our core focus area. It's just really interesting because yes, the technology does enable you to do that. Yeah. What's the scalability consideration when applying blockchain for AI governance at an enterprise scale? Yeah. So essentially the best way to deal with scalability is to have hybrid networks, meaning you want essentially the important imprints to all be on the public network.

26:34Because the public network is truly immutable and is truly tamper-proof. And so everything is replicated to a degree or batched to the public network. But the way you get real hyperscale within your builds is you set up a private instance as well. It's similar to what you do with cloud computing. You know, you mix and match your on-prem and off-prem. You get the best of both worlds. So similarly, the way you scale is essentially you're building a layer two network within your company where you store absolutely everything. And then you decide like, hey, for auditability purposes on a daily or hourly basis, we are going to put every imprint that we have relayed to the public blockchain, prove that all the hashes are secure.

27:19Obviously, all of this happens in an automated manner. You set up the policy up front, and then the smart contract just takes care of it, and it can be upgraded as time goes by and policies change. But essentially, the scalability is similar to how they do it in cloud. It's a mixture of on-prem and off-prem. And how would an enterprise implement this? Excuse me. So I've got a model and a training data set, and the model is making inferences. Are you kind of plug and play? I mean, how do you integrate your platform into what I'm doing? Hi, I wanted to jump in and give a shout out to our sponsor this week, TrialKey.ai, the market leader, AI-driven clinical trial design optimization and predictor that improves trial design and execution.

28:25I recently had TrialKey on the podcast. I encourage anybody who's interested in clinical trials to go back and listen. With over 90 % accuracy based on precision recall, TrialKey.ai predicts the success of clinical trials more precisely than any other tool on the market. TrialKey.ai contains the world's first clinical trial design simulator, which allows you to manipulate a trial's design to find the best combination for most probable success. TrialKey.ai pulls data from over 350 ,000 clinical trials, inclusive of medical devices, alternative therapies, and non-randomized drug control trials. Pivotal clinical trials often exceed$48 million in cost, yet only 10 % of pivotal trials succeed.

29:22TrialKey aims to address this issue by helping companies improve trial design with AI. A 2011 study found that 60 % of trial protocols needed amendments, with a third preventable through better design planning. On average, completed protocols undergo 2.3 amendments, each causing significant delays and costs around half a million dollars each. AI provides clinical researchers and pharmaceutical companies with in-depth variable analysis and a clinical trial simulator that improves trial design, execution, and drug mechanism of action. Leverage AI-powered clinical trials to accelerate medical breakthroughs at trialkey.ai.

30:10Maximize clinical trial success while minimizing costs. Book a demo at trialkey.ai or sign up for free. Transform your clinical trial outcomes and shape the future of healthcare. Again, schedule a demo at TrialKey, that's T-R-I-A-L-K-E-Y, AI, or sign up for free. Elevate your trial success and drive innovation in healthcare. Yeah, so you basically sign up for Prove AI, and now you have access to the entire platform, as well as, you know, the integration into WatsonX governance. And then essentially, you can bring in your models. You say, hey, this is the model I'm using. I want this to be tracked and traced completely.