Controlling AI Models from the Inside

0:03Welcome to the Practical AI Podcast where we break down the real world applications konsine intelligence and how it's shaping the way we live, work, and create. Our goal is to help make AI technology practical, productive, and accessible to everyone. Whether you're a developer, business leader, or just curious about the tech behind the buzz, you're in the right place. Be sure to connect with us on LinkedIn, X, or Blue Sky to stay up to date with episode drops, behind-the-scenes content, and AI insights. You can learn more at practicalai.fm. Now, on to the show.

0:48Welcome to another episode of the Practical AI Podcast. This is Daniel Whitenack. I am CEO at Prediction Guard, and I'm joined as always by my co-host, Chris Benson, who is a principal AI research engineer at Lockheed Martin. How are you doing, Chris? Hey, doing great today, Daniel. How's it going? It's going well. Well, done a little bit of snow shoveling on the ground. As we speak, we're kind of headed into winter break or the holiday Christmas season here in the US. And I think this episode will be released in the new year. So if you're listening to this, you're listening in the future. To be honest, I'm really excited about talking about the future because our guest today is really thinking very innovatively about how we can secure our AI models and have safety.

1:36as we move into that future. Really excited to welcome to the show today, Ali Khatri, who is founder of RINX. Welcome, Ali. Thanks. Thanks for having me, Daniel. Yeah. Yeah. We met earlier this fall. I'm really fascinated by your kind of line of work and technological innovation at RINX. But I also know that you've been thinking about these topics around AI safety, guardrails, disallowed content, et cetera, for quite some time. Could you give us a little bit of a background of how you got into these topics and what you've done in the past? Yeah. So I have been in the machine learning for AI safety or anti-abuse use cases in general for the past eight or so years of my career.

2:26I spent about three years at Meta where I built infrastructure that serves about half of the world's population. Basically, any safety, anytime you type a message on Facebook, it goes through tens of safety checks, which are powered by thousands of models. I built the infra that these models run on. And then I moved on to Roblox, where I built systems to protect about$3 billion in payments against fraud. So I've been in this space. I've been using AI models to sort of protect against abuse. And during this time, I realized that the models that I'm using themselves are susceptible to abuse. So that's what led me to founding ranks.

3:13And I know that now you're thinking about those actual models. So I often tell people also on the AI security side, there's kind of AI for security and then there's security for AI. It sounds like there's something similar kind of in, I guess, model or safety anti-abuse space. Could you give us a little bit of an understanding of like when we're talking about the safety or security of AI models? Could you kind of define that for us? What do you have in mind as the kind of bad case scenarios or worst case scenarios of what a model could do, why it might not be secure or safe? Yeah. So thanks for making that distinction between AI for security and security for AI.

4:04And they're two very different things, right? Like AI for security basically means using AI to solve existing security challenges in a more effective or a better way, right? So that's a very different and a very linearly separable body of work from security for AI, which focuses on making the AI models themselves and like AI-based use cases secure. As models have entered the tech stack, they also bring in a bunch of security challenges, and that is what security for AI solves. Now, in terms of the safety aspects that we talked about, models today will generate anything. We'll literally tell, like, these generative models, like, if it's a tech model, it'll generate any form of vile content known to man.

4:54And it could tell you, like, openly I got into trouble for, like, where a teen was asked to commit suicide, right? Or was encouraged to commit suicide. So you have self-harm. You have different other categories of harm where you can, you know, generate pornographic content. You can generate other forms of inappropriate content like violence, gore. And sometimes it doesn't even have to be inappropriate because safety is very context-specific, right? As a law firm, safety looks very different for you than what it does for a medical shop versus, say, a customer service setting versus like a co-generation environment.

5:29So each one of these use cases come in with a set of permissible and non-permissible behaviors. And that's kind of what safety really is, that making sure that the technology works in ways that you intended to and minimize the unintended outcomes of it. And I guess people are addressing, I mean, these are known issues in the sense that at least a segment of the people that are working on these models know about these issues. Could you give us a little bit of a sense of like, as we speak, how these, at least in a production sense, like what's capable now? Like how does the landscape look in terms of capabilities to defend against or align with certain policies or allowed content, disallowed content?

6:28What are our choices right now? If I'm looking at the availability currently of both kind of open source projects and what's in maybe closed platforms, what's available to me to deal with this issue? Okay. So before I answer that, I'll start with an analogy, which will help make the rest of the response much more clearer and add some context. So imagine you are in a, like a story, like a built, like a giant apartment building with over, like say a thousand condos, right? Now, let's say your neighbor for some, for some reason or no reason at all, really decides to pull out a golf club and starts violently assaulting you with it.

7:13Now for a situation like this, will the good guys be able to protect you just by checking IDs at the gate? Obviously not. That's kind of where we are. You're past that point. So that's kind of where we are with AI safety today. That's kind of how jailbreaks work, right? So here, the giant thousand-story building is the model. So today, what we are able to do is just today's solutions, analyze what's going into the model, also known as the prompt, and analyze what's coming out of the model, which is the response. But by then, the damage has already been done. So now, if you're using video generative models, right, You can put in a text prompt, get a video output.

7:52Your video output is going to be, it's too expensive to analyze, number one. And the video has already been generated. So we already spent a large amount of compute generating that bad content. Again, if you're talking with audio models, you can trick audio models into generating bad content. Like a seemingly innocuous looking prompt can be tricked today using a multitude of techniques to generate really malicious output. So now, unless you have visibility into what's going on inside of the model, you're not going to be able to catch a lot of these things. That's where jailbreaks come from. That's where adversarial machine learning comes from, if you look at it through the context of predictive models versus generative models.

8:36It's essentially the same core phenomenon where operating these models as black boxes, and we have no idea of what's going on inside of them. So we're trying to change that. Looking at that, it seems like in the way that you just kind of phrased it with the context there, it seems like a very intractable problem. As a follow-up to Dan's question, like how should I as a new user potentially or someone coming into a use case in my company where a model is desirable, but I'm worried about whatever bad or abuse means in the context that I'm operating in, how should I start off thinking about that?

9:14Like, like, what's my starting point? Because I, I gotta say, you know, coming into it, I'm not even sure where to start. So like, can you level set that a little bit in terms of how, you know, what, what's, what's square one? Yeah. So that's a good question. So normally what you do is the way I like to think about it is there's a general category of bad stuff that no one really wants or the law doesn't allow and things like that. So there's like a general category of stuff like that, right? Like here, I'm including porn, hate speech, yada, yada, yada, which they're a general category of undesirables that no one wants on the platform, like child safety, for example.

9:50That's non-negotiable, no matter what context you're in. Now, there's also another aspect of categories, which is about context-specific safety. So now if you're in a banking use case, you've got to think about, okay, money laundering. You might not have to think about money laundering, say, in a code generation setting, for example. So you want to think about these very specific categories of risk or issues that come from your use case. And people tend to usually have a very good idea of that. If you understand your use case well, which most people do, which is why they're exploring models and they're trying to solve some problem within their use case.

10:28So within that use case, you also would understand the problems that you're facing. And that's another category of risk that you want to think about. Then once you've thought broadly about these two, then you've got to figure out about once you've identified both of these categories, then you want to think about mitigations, detections and things like that. And I guess there's like if we're just kind of defining some terms for people that they might have heard of this approach that you talked about related to kind of guarding the gate to the apartment complex, right? Or the inputs or outputs, managing the prompts and the outputs.

11:08Is that what people refer to as like guardrails, safeguarding? What is some of the terminology that's being used? And then I know that you all that you have a different way of thinking about this. I guess just setting the stage jargon wise. How do we define these kind of guarding the gate things? And then as we filter into actual safety within the apartment complex or within the model, what kind of terminology? And I guess like there's a body of research building up to what you're doing. what terms are used to describe that? And as people wanted to research that, what would they look for? Yeah, so guardrails essentially are a catch-all term.

11:52They can refer to prompt and response filters. Now today, one of the ways of doing, there are multiple guardrail type solutions out there. There are guard models out there. Meta has one, IBM has one, Google has one, OpenAI has one. And I'm talking about public releases, right? Internally, most people have their own. But these essentially are prompt and response filters. So they look at the data going in, they look at the data coming out. Now, so that is like one thing that guardrails are used to refer to. Less commonly, guardrails are also used to use, refer to like static checks, where you just look at the output of the model and say something like, okay, the word, like forbidden word, let's say the F word, right?

12:35The F word appeared in output, this is not permissible. So that's a simple regex filter that you can use. So that would also be called a guardrail. In terms of looking at the internal state of the models, there's a whole field of research, area of research that's developing. It's called interpretability. There's a subset of that called mechanistic interpretability, where they try to figure out what subcomponent of the model led to this particular output and try to change it at the source or try to alter or modify behavior while it's happening as opposed to after or before.

13:29Well, friends, here's your hot take of the day. Your team's AI tools, they might be making collaboration messier, not faster. You probably know this. You feel this. Think about it. You've got AI literally everywhere now summarizing, generating, suggesting. But if there's no structure, no shared context, you're just creating more noise, more outputs, It's more stuff to wade through. The gap between having a great idea and actually shipping that idea, that gap isn't a speed problem. It's a clarity problem. Well, this is where Miro comes in. And honestly, it's shifted how I think about team workspaces.

14:06Miro's innovation workspace is not about brain dumping everything into an infinite canvas and just hoping for the best. It's about giving your work context, intentional structure, So your team knows what to focus on and where to find what they need without playing detective across 12 different tools, clicking and moving and tabbing. Just too messy. And the AI piece, well, Miro AI actually gets this right. They've got these things called AI sidekicks that think like specific roles, like product leaders, agile coaches, product marketers, reviewing your materials and recommending where to double down or where to clarify.

14:46You can even build custom sidekicks that are tailored to your team's exact workflow if you desire. And then there's Miro Insights. It sorts through sticky notes, research docs, random ideas in different formats, and synthesizes them into structured summaries and product briefs. And Miro prototypes, they let you generate and iterate on concepts directly from your board. Test 20 variations before you ever touch your hi-fi design tools, saving you time, giving you ideas, and getting it right. This whole thing is built around the idea that teamwork that normally takes weeks can get done in days, not by going faster, but by eliminating the noise and the chaos.

15:29So help your teams get great done with Miro. Check it out at Miro.com to find out how. That's M-I-R-O dot com. Again, Miro dot com.

15:44So Ali, you were just getting into these ideas of interpretability, mechanistic interpretability, I think you called it. We've talked about interpretability on the show before, I think mostly in relation to trying to figure out why a model made a certain decision in relation maybe to certain concerns around bias or other things. So like if I have a risk model for approving insurance or something like that, then I might need to have some interpretability around that. Or maybe in the case of healthcare, there's a burden for interpretability of like how decisions were made. Here, it sounds like kind of the interpretability is being applied to, I guess, why the, is a good way to put it, why the model generated some problematic output?

16:41Or is there a better way to think about that? So there's multiple overlapping aspects here, right? Interpretability is like an umbrella research area. It's an umbrella term. So what you alluded to earlier is more of, it's also described as explainability. So why was this credit card denied. So you're trying to explain it in human concepts. That is a part of interpretability, no doubt. It's a subset of it. There's another subset, which is how was this generated? So for example, if your generative model outputs, say, if you say, how are you? Hi, and the model says, how are you? You want to care about how was this generated internally?

17:25How were these tokens produced? And the reason you care about that is you want to know if, instead of saying, like responding to, like responding with how are you, it could respond with howdy or with something else, right? So you want to know what caused those differences and you want to be able to control that. So that aspect is also interpretability. Now, where this interplays with safety is when you have these prompts, which look bad, which look good to a human, but bad, but result in bad outputs, which is how jailbreaks work. When you analyze how the data flows inside of this black box, you're able to control it and stop it at the source.

18:06So think of this as, continuing our analogy from earlier on, think of this as cameras at every gate or every path. So you know that, okay, well, this is what's happening in this hallway and we got to stop it. We got to put an end to it. So it's a very different class of defenses. As you're saying this, you're actually talking about manipulating the internals of the model and the flows that are there, kind of the cameras on the doors and stuff like that, and making it maybe a gray box rather than a black box to some degree, as opposed to kind of the more traditional guardrail approach where you have programmatic, you know, you use the word guardrails around the models, inputs and outputs to try to handle things that way.

18:49So it's kind of a whole different thing about instead of treating the model as a black box, you're saying you're diving into it and trying to affect an improvement there? Yeah. So intervening is one, like stopping generation or modifying that is one form of intervention, right? Intervention does not have to necessarily be in that form. It can take various other forms. So once you understand today, we have no idea of what's going on inside of the model. We provide an input to the model, get an output. We have no idea what's going on. So this, what we're trying to build, or what Interpretability tries to do is understand what's happening inside.

19:29Now you could control it, or you could use that to make a risk quantification and use that downstream. You don't have to do anything in the moment necessarily. It can be leveraged downstream. So now it's basically like you have a whole new set of information or a whole new set, like a whole new class of data points that you can leverage in creative ways downstream. This is something that's not available today. And this is what interpretability builds really at runtime. Am I correct? So part of my assumption in the past, and I'm fascinated by this whole subject, is like some small changes in the, for example, the weights or individual layers or individual pieces of the model can produce very large changes in the output behavior of the model, which it's kind of...

20:24I'm blanking on this. What is the thing? It's like a butterfly flaps, it swings and Oh, the butterfly effect. Yeah. The butterfly effect or, or, or whatever. So like these, cause you can make a change, whether that's quantization or other things to the model. And that, you know, may produce unclear and sometimes catastrophic changes in the, in the behavior of, of the output. And so if I'm understanding what you're saying, right, Ali, it's one way you could try to use the information about how the model is producing certain outputs is to intervene by actually making a modification or preventing something in the model.

21:05But that could produce other changes that you may not want, I'm assuming. But you could also instrument the model to understand potentially when it is kind of firing those certain neurons or lighting up in a certain way that is indicative of problematic behavior. Am I understanding that right in terms of like various ways of, I guess, either intervening or instrumenting? I don't know if I'm using the right terms. We are instrumented. Like the way we're approaching this is we are trying to understand what happens inside of a model at runtime. Models are a monolith, right? But we're breaking it down into different spaces or subspaces.

21:47And we look at the subspaces that get activated during bad generation. So now when you're, let's say, generating non-permitted content versus permitted content, different sub-regions of the model get triggered. So we're building visibility into that, and we're trying to identify them at runtime. So now there are some sub-regions that you wouldn't care about, right? Like, for example, if you take a general-purpose LLM, it's trained on everything ranging from Python code to 15th century Chinese poetry, right? Now, when you're using it in a customer service setting, you care about neither one of them.

22:26And if those sub-regions of the model are getting activated, then you want to be able to arrest it while it's happening. So this is similar to, if you go back to the analogy that I had made about the apartment building, you want to have visibility at all times into what's going on at each level, right? So you find out that a bad thing is going to happen way before it actually happens. Like, for example, if someone's going to conduct a bank robbery, people just don't get up and conduct a bank robbery, right? There's some searching going on. There's cycles of planning going on. There's purchases of firearms or whatever going on.

23:02So now if you stop them at these bad activities at different levels, the police don't have to deal with a shootout situation in a bank at the very end. So similarly, defense works in depth, and we're building a whole new layer of safety that hasn't been tapped into just yet. Yeah, that's fascinating. And I'm guessing certain folks are probably wondering, like I am out of curiosity, like this might be the first time that they're hearing about such an approach to this kind of problem. And they might be thinking, you know, how is this possible? Like how. So one thing I think the general concept makes sense, right?

23:44Like instrumenting the interior of the apartment complex, understanding what's happening, kind of retrieving that, you know, that intelligence to for you to make decisions or determine if you want to mitigate something. What are the, I guess, have there been multiple attempts to try this sort of thing? And from your perspective, in terms of how you all are approaching it, what is kind of needed, I guess, from the customer side to create this sort of instrumentation? So I could imagine, like in one scenario, like I could train, I could tell the customer, well, we're not going to train on Chinese poetry or code.

24:28we're just going to train a whole new model. And that burden on the customer is very, very heavy, right? And on another scenario, you could say, oh, take this model off the shelf and do this to it. And then, you know, that's a less burdensome. So there's probably a spectrum of here. Could you help us understand like the, I guess the burden and what might be required to get to this instrumentation? Yeah. So the way we're approaching this today is we do not modify the model. We do not even build models nor require the customer to build a model. We take an off-the-shelf model and we build a safety module that sits on top of it.

25:08So essentially for the customer, it's like a very low friction approach where they take a model that they use and love, like Lama or Granite or any Mistral or any of the open-weight models for image generation. You have WAN and you have a host, other host of Chinese models for the audio video setting. So any model that you love, we sort of make it more secure and tailor it for your context. So now, again, remember, if you're like a lower law firm, right? An off the shelf Lama or off the shelf Mistral is not going to have the protection that you have. Like if you're like, say, a shoe company, let's say you're Adidas or Nike, right?

25:49You as a user want to talk about Nike, but not talk about Adidas, right you can't expect the model maker to put that in for you because the model maker is trying to sell to everybody so we help build that customization and we do that without changing your primary model like your primary model will continue to be as it is if you make any modifications to it uh like fine tuning or anything like that that's on you you control that we don't require you to do it but even if you do that we can still support you could could you uh so totally recognizing that there's proprietary stuff that you're not going to dive into and respect that.

26:26Could you talk a little bit about just kind of clarifying as we were kind of talking earlier about kind of the buttressing with guardrails on the external side versus the going into the model. And as you're talking about adding a component, like, so in my conclusion, it seems a little bit like it's on the outside. Can you talk a little bit about what you mean by that without diving into places you can't go? Yeah. So I'll give you a very, I'll try to address that as much as possible without going into the specific. Fair enough. Specific details. So today what you have, right? You have these filters, which analyze the inputs and the outputs.

27:04Now, analyzing the economics here is all messed up. Like if you were to analyze video or audio, right? Those tend to be very expensive computationally. Those models are very expensive. So now if your inference itself costs X, and if you're expecting someone to pay another X to analyze it, number one, it's slow. Number two, it's like paying someone$1 ,000 to guard a$100 bill. You're just not going to do that, right? So what people end up doing is people end up like shipping unsafe models. So today, we have tested models from different audio companies. We've tested models from different video companies, different image generation models, each and every one of them with little to no trickery, which means that average user can just go there and ask them to generate bad stuff and they will do it.

27:51There are little to no guards there. And that's because the economics does not make sense the way thing is today. So what we've built is a scientific, like we've sort of had a research breakthrough where we can build safety about thousand times cheaper. So just to give you some concrete numbers. What we've done with LAMA, we've taken a LAMA model, which is like an 8 billion parameter off-the-shelf LAMA model. And today, if you had to protect it, you would have to use LAMA Guard 3, which is an 8 billion parameter model. And assuming it generates 10 tokens, that is, you're running about 80 billion parameters of inference at runtime.