Ensuring LLM Safety for Production Applications with Shreya Rajpal - #647

0:09All right, everyone, welcome to another episode of the TwiML AI podcast. I am your host, Sam Charrington, and today I am joined by Shreya Rajpal. Shreya is founder and CEO of Guardrails AI. Before we get going, be sure to take a moment to hit that subscribe button wherever you're listening to today's show. Shreya, welcome to the podcast. Thank you for inviting me, Sam. Excited to be here. I'm really excited about our conversation as well. We'll be talking about AI safety, among other topics. But before we do that, I'd love to have you share a little bit about your background. Yeah, absolutely.

0:41So I'm Shreya. I am the founder and CEO of Guardrails AI. The company really started on an open source project that I created called Guardrails. So it focuses on, you know, how do you take this fantastic technology of large language models and make sure that you can like reliably use it in production scenarios. But in the past, I've spent about a decade or so in machine learning. I've worked kind of across the stack, starting out doing research, you know, not even deep learning and like classical AI systems, which seems a little bit archaic now. Then did a bunch of research in deep learning, spent a number of years in the self-driving space, basically training models and doing machine learning in every part of the stack in self-driving.

1:19And then joined an MLOps company called Predabase as their founding engineer, where I led their machine learning infrastructure team. So I've gone the full gamut from research, applied ML, ML infrastructure, and now focusing on safety for AI systems in LLMs. Awesome. I guess we should jump right in and talk about, when you say LLM safety, what are some of the things that you're thinking about? Is it just hallucinations or are there more safety issues that you and the market are concerned about? Yeah, I think hallucinations are definitely top of mind for everybody. I think they're also one of the most obvious failure modes of LLMs where you ask it a question and you're like, no, I know the answer to that.

2:02And I know that what you just kind of printed, as confident as you sound, is not the right answer. So hallucinations are definitely like top of mind for a lot of people. And as you build applications and then run them at scale, like run them where, you know, you have like a bunch of users using a prototype. So it's not just, you know, like a side project that you build and you're trying out. You truly start seeing some of the scale of like hallucinations and how, you know, the same query, the same data set, source, etc. will result in different answers to some degree. that's barely scratching the surface because a lot of the promise of LLMs, a lot of the really fascinating, amazing demos that all of us saw end of last year, this year, et cetera, why they were fascinating is that you kind of see the potential that they have to disrupt traditional work, disrupt like a lot of like add a lot of automation or like human augmentation, et cetera.

2:55And that's where a lot of the promise lies. And we make a lot of assumptions about like how reliable humans are when you give them a certain instruction, right? You're like, hey, don't, you know, if you're working for McDonald's, never ever mention like Burger King. You know, McDonald's is always like the top burger chain for you. But if you want to bring in like an AI model that kind of does that, even like following simple instructions like that might not always be respected, right? Especially as these models are like poked and prodded and deployed at scale, not just individually, but as part of these like larger complex systems.

3:29In addition to hallucinations, ensuring that all of these like very domain specific, very industry specific like constraints, which we just expect like humans to kind of just know and get are suddenly assumptions that are not valid anymore for LLMs. And I think like that is a much more like kind of functional interpretation of AI safety, like within a domain, within an industry, within an application or use case, etc. So those are across the gamut of things that we're focusing on, you know, that includes the set of problems as well. And have you come across taxonomization of these kinds of concerns?

4:04Do you have a mental model of the different categories of concerns or do they all fall under hallucination or how do you think about the different failure modes? Yeah, I think it's a great question, honestly. So hallucination is part of it. In terms of taxonomization, I'm a huge fan of taxonomization. It's how I build mental models about whatever I'm working in. So even within hallucination, there's a taxonomy. Not all hallucinations are kind of created equal. Some are more harmful than others. And there's ways to kind of categorize, classify, evaluate them separately. But outside of hallucinations, like there's a lot of risk that gets, you know, exposed to customers or builders when they're working with LLMs.

4:48And this might, you know, go anywhere from just like straight up performance risk, where even if the model is not hallucinating, it's just straight up not helpful. You know, it's not able to give you the kinds of like detail or the kinds of questions that you're looking for. Like maybe it's too verbose. Maybe it's too, you know, like terse and doesn't have enough context or the relevant context. Even if it's correct, maybe it's focusing on the wrong section of documents or the wrong section of, you know, like text, etc. to generate an answer, right? So one of the risks is just like straight up like performance or like accuracy would be a very simple way of thinking about it.

5:26But there's a lot of other risks as these systems get deployed into real world organizations and real world scenarios. So brand risk is an obvious one, right? Where the model might not be hallucinating, but it might end up saying something that ends up being a brand risk to whichever organization wants to use these LMs or AI models in production, right? Another one might be like compliance risk or regulatory risk, where essentially the model, even if it's factually correct or maybe it may or may not be factually correct, but is doing an action or a behavior that the organization does not have the authorization to do.

6:02A very, very simple example of that is, let's say you're building like a health chatbot whose only purpose is to help you with your billing inquiries. In that case, that chatbot should never, ever give medical advice. you'd be surprised. You'd be surprised by like how easy it is to jailbreak these systems and how easy it is to like get it to give, you know, medical advice when it's actually working in production. So your bill is due in 28 days. But by the way, have you considered this alternative medicine approach? Yeah, exactly. Exactly. Especially when, you know, there's like the most common UX for deploying these systems currently is like a chatbot, right?

6:41So it's very conversational. It's very, you know, People want to chat with it. people won't chat with it. They're like, okay, my bill is due. Oh no, what am I going to do now? And you're like, okay, actually, I can't really tell you. I can only help you with billing inquiries. So making sure that these systems don't behave in a way where they cause compliance or regulatory risk for their parent organizations. I think there's a bunch of other risk factors here that open up, but it's broadly this classification of how even if the model is factually correct, or it's hard to evaluate whether it's factually correct in the case of like medical advice, but it just behaves in a manner that opens up a can of worms for, you know, whatever, whoever the organization in charge is.

7:22And then you said that for even within hallucination, you've got a sub categorization or taxonomization of types of hallucination. Can you share a little bit about that? Yeah, yeah, yeah, absolutely. So on the highest level, it depends on the workflow that somebody has when they're building with LLMs. So one of the most common patterns that we've seen be very widely adopted whenever for any application that is being put out into production is retrieval augmented generation or RAG, where just on a very high level, retrieval augmented generation is that you have some private data source. let's say if you're internal to the company like maybe this is you know you know employee handbook or something and you want to build a system that somebody can come to the system ask questions in natural language from the handbook and the llm or the the application that you're building like looks at the employee handbook and then responds to questions from that so that's the high level goal of like retrieval augmented generation which is you retrieve some relevant documents first you augment the LLM's knowledge with those retrieved documents.

8:35And then using all of this additional context, you generate some response. So if you're doing retrieval augmented generation, a lot of the assumption that stakeholders have as they're going into this is making sure that only answer from the set of questions that are present in the documents that I give you. Don't use your world model or the entire internet that you were trained on to just inject a few sentences here or there just from your own information, just stick to what I've provided you. So in this case, one very, very common failure mode is that you build retrieval augmented generation systems.

9:13And as the LLM is generating some text, it will not just look at the documents that you have, but add some extra flavor here or there. Maybe that flavor is right. Maybe that flavor is wrong. But the intention here is just to make sure that you're just not opening up that Pandora's box and you're making sure that only what you know to be the source of truth, which is, you know, your documents are what the LLM is using for answering. So that is kind of like closed domain hallucination, right? Another kind of like flip side of this is that you just have a general question answering system. You're not building a rag system and you maybe have a, you know, a QA bot that is just a companion bot or just a conversational agent, et cetera.

9:55In that case, like the model can still hallucinate, but you know, it's like, it's not constrained to, you know, the domain that you provided it. Right. So there's just that top level classification, depending on, you know, are you providing like some source of truth that you want to be respected or does it, you know, are you expecting the model to rely on its world model to answer your questions? I think even within that, then you can, you know, specify like what are the types of hallucinations? So is it messing up, you know, like some specific entities, right. Where the fact is true, but it's just talking about a different entity or it's talking about, you know, instead of like in your employee handbook, you have like, you can only take like 30 leaves and you have to apply for permission from your manager.

10:34It's you can only take like 25 leaves and you have to apply for permission from your manager. Right. So it's messing up like specific entities or specific values in the text, or it is, you know, maybe like completely misrepresenting the meaning or it's adding like ambiguous information, et cetera. So you kind of like can become like way more specific with like what are the, you know, yeah, what the specific type of like hallucination represents. RAG has become very popular as a way to, I guess, limit the failure surface of LLMs. I think the existence of closed domain hallucination as, you know, one of your subcategories suggests that RAG in and of itself is not sufficient to, you know, solve the problem.

11:17But I'm wondering if you can elaborate a little bit on the kinds of experiences that you've seen with folks using RAG and an assessment of the degree to which it solves the problem. Yeah, I think that's a great question. In terms of the degree to which RAG solves the problem, I would say that without RAG, a lot of the LLM adoption we're seeing today would just not exist. At least a lot of the excitement in enterprises and corporations, et cetera, to, you know, like bring LLMs into the fold and see how they can be used. I would suspect that that would be a lot more muted if RAG didn't exist. So I don't think right now that there are sufficient alternatives to RAG outside of fine-tuning.

11:59And even then, you might have way more of a lift using RAG than you would with fine-tuning. Even ignoring the cost of fine-tuning, ignoring the effort, and ignoring the requirements that you need to have on having a data set that's curated data and actually works, et cetera, that is needed for fine tuning. Ignoring all of that, even if you just did RAG versus did fine tuning, you might end up, you know, it's very possible that you're going to end up with like better performance with RAG than you would with fine tuning. I don't think we can do away with RAG at this point. I do think it's a very essential part of the LLM stack, but then even within RAG, right?

12:36Like I think the most common thing we see is that like, you know, I gave it this text, but it's not respected, right? Like it just added extra context or added extra information that wasn't present in the text that I just don't want it to have. Or it maybe responded with something that I didn't want it to respond to. Or maybe someone asked a question that wasn't actually in the documents at all. So the LLM, you know, once it couldn't find like a relevant document or something, just ended up generating information that again, relying on its world model to, you know, generate some text. So I think it's this constant push and pull between using how capable and how adaptable these models are, but then putting some constraints on them, right?

13:17Using that capability, but only making it work on your domain, your data, your text, and trying to make it fight its own capabilities where it wants to rely on all of this other knowledge that it's been trained on. So I think that's pretty much the most common failure mode where you give it some text, but it'll just add extra context extra information that ends up being most common. But then you also have like to a lesser degree examples where you give it some context, but it also just straight up like negates it, or it like refuses to find, you know, what the relevant piece of information is.

13:51So maybe you're like, okay, what is the leave policy? The retrieval part of it might work correctly, but you run this at scale, you'll kind of find that like, okay, what is my leave policy? And you end up like generating a response where even if here's the leave policy is in the text that you give it, it would just end up generating responses of like, oops, sorry, I couldn't find anything about your leave policy. Or maybe like you don't have a leave policy, et cetera. And, you know, just respond with that. So the challenge here ends up becoming that a lot of the excitement in the space, you know, it's because like it triggers the imagination of what's possible with LLMs, right?

14:23But the bet that everybody, you know, working in the space is making is that this technology is going to be so powerful that it permeates, you know, like it's going to be everywhere. It's going to be every tech that you use is going to have an assistant and it's going to be on your phone. And, you know, maybe it's a companion, maybe it's a sales, you know, assistant, et cetera. But in order to do that, if you think about like, okay, if I was querying a database with SQL, I only expect it to work. Like maybe I'm pulling a number out of the hat here, but like, I only expect it to work like maybe 70 % of the time.

14:54Right. That is not how software works today. You have like expectations of, you know, reliability. You have some, some implicit contract with like software when you use it. But that's not the case with machine learning. And that's not the case with these elements today. So in order to get to that stage where this technology is adopted, and it's everywhere, like we would need to cross that hurdle and make sure that we don't run into these problems. Yeah, one of the interesting things that you've emphasized thus far is the aspect of these problems in which they appear at scale, like you You can build a system, you can interact with it and everything could be fine.

15:32But, you know, when you scale it across many requests, if the probability of a failure is small at scale, it's going to manifest. How can builders like wrap their head around the failure rate essentially of their systems if they need to scale? Do we have the tooling to, you know, is there are there load generation tools or the analog for that for LLMs? like what's the right set of tooling to wrap your head around the problem? I think that's a very, very exciting question. I think it's a very active area in this space right now, which is how do you actually figure out how well these models work or how well this workflow or application works, right?

16:12So we're in this kind of like interesting space where I'm sure, you know, other guests you've had on the podcast before I mentioned this as well, but we're in this interesting point where the ML metrics are actually a little bit behind where the technology is. So if you take the classic case of like summarization, like the metrics that we use for summarization, you know, for the longest time aren't usable anymore because, you know, LLMs will do worse on the metrics, but actually would have generated better summaries. And that is also because, you know, there's no one right way to do generation.

16:45Like you can, you know, generate a text in multiple different ways, have like different sentence structures, et cetera. And all of that would still be a valid generation, right? So how do you kind of capture that? I think in terms of tooling, it's a very, very active space. Like a lot of people in academia, in industry are really looking into, you know, like what the right way to kind of like solve this problem is. And I think one of the most well-adopted ways to solve this problem and to figure out how well evaluation works is to get the LLM to self-evaluate itself, right? So let's say you're using OpenAI to generate some text from some documents that you gave it, then you would do a follow-up query where like, oh, actually, given this text and given this response, is this truthful?

17:28Is this factual? Am I missing anything, et cetera, right? So this LLM self-evaluation is like one of the most common patterns right now. And I think it's more because of necessity rather than anything else, because we are, at this point in time, I think there aren't as many tools available outside of LLM self-evaluation. But at the same time, you know, personally, for me, it makes me very uncomfortable where, you know, if you're using the LLM to self-evaluate itself, like the first question is, okay, like who will guard the guards? Like how do you know the evaluation isn't hallucinated, right?

17:59The second is that LLMs have this very, very fascinating, you know, tendency where to some degree, you know, they're like next token predictors. And if you just ask it to, let's say, for example, you know, you ask it to generate like a score between one to five or one to 10. There's a bunch of research where it would be way more likely to generate like, you know, one number because it's just like, you know, more conditioned to generate that number. Right. A much more simpler way of thinking about this problem is you ask the LLM to generate a random number between one to 10. and you would expect that the distribution looks uniform where like each number kind of comes up roughly similar number of times, but you end up finding like, I forget what the actual number was in these tests, like three or something, like three would just be way more overrepresented.

18:47Like, you know, 70 % of the, I'm pulling numbers out of my hat here just to kind of like talk about the problem, but let's say it's something like 70 % of the times you'd get a three and then everything else would be, you know, very low. So using the LMs to evaluate themselves. So rate the factuality of the previously generated response on a scale of one to six, excluding the number three. Yeah, exactly. It starts to get a little wonky. Yeah. We're kind of in the Wild West, like understanding the capabilities and, you know, controlling and constraining the capabilities of these things. So it's pretty fascinating, like where we are tech wise.

19:22I think like other issues are that, okay, if you ask the LLM to, you know, once again, evaluate itself, you know, compare like one generated text versus another and you're like, okay, which text is better, right? And you're just trying, like, you're just looking for a factual evaluation. You know, they're a little bit narcissistic in that they find their own work way like better more often than they find like, you know, human generated text. So they're much more likely to rate themselves higher than they are like human text, etc. So there's like all of these like really fun challenges where I like some of the other, you know, benchmarks that we did internally.

19:59You use the LM to evaluate itself. Evaluate itself. You don't change anything. No hyperparameter is changed. zero temperature and you run this evaluation multiple times and you're going to get, you know, varying, like it's within some range, but it's not a small range. Like it's a non-trivial, you know, difference in like, let's say accuracy, right? Like you'd get a differently varying accuracy depending on when you run. So I think like, I personally am very, very uncomfortable with like LLMs being evaluated. So in terms of the tooling, I think like people are looking into, you know, how do you kind of like bridge this gap between traditional ML tooling, which was, way more reliable versus something that can accurately evaluate the flexibility of LLMs.

20:42And I think as with all ML problems, the first step is really being able to curate a data set or have a data set that has, maybe you have human labelers or human experts kind of look at it, that kind of captures these tasks. So it varies from task to task, but that is where we are, that's the lay of the land a little bit. And does the data set that you referenced, Does that exist or is it and even can it exist generally or does everyone need to kind of curate their own because the failure modes are so context specific? Yeah, I think it's the latter, unfortunately, where the failure modes are so context specific that, you know, every org, every company, every startup has to kind of like figure out internally what they're kind of building this for.

21:28So if you have, let's say, internal company document question answering versus if you have, you know, something that's more customer facing, it's where you're evaluating for very different things, right? In one, brand risk becomes way more important and there aren't any data sets. There's no brand risk data set. I think there actually might be now. Recently, some researchers at Stanford collaborated with all of these other stakeholders and released LegalBench, which is this very comprehensive benchmark for using LLMs for legal tasks. And it's a fascinating piece of work. I would recommend everybody check that out.

22:05But then, you know, we're kind of like in the very, very early stages of like looking at, you know, what does taking these things that we just assume humans would do right, which is not give medical advice if you're not authorized to give medical advice, and then turn those problems into data sets, right? Or don't talk about Burger King if you're at McDonald's, like take those things into data sets. But it kind of points to the number of these data sets we'd need, like Bank Bench, Burger Bench, like it's very, very context specific. Burger Bench. I'm looking forward to seeing that come out out of a lab in the next three months.

22:43McDonald's AI research. Yeah, true. Yeah, I think an interesting pattern that is very, very recently emerging. I'm forgetting the name of the open source project, but there's some open source project, LMBench maybe, that essentially generates synthetic data. Like you give it a one sentence description of the data that it wants and it'll like keep querying, et cetera, and then generate a data set of what you want, et cetera. So I think like synthetic data can to some degree help alleviate the problem. With all of these like LLMs evaluating themselves, you keep getting back to the same original problem, which is how can you trust the data set now in the first place, right?

23:22So I think like the true way of mitigating a lot of these problems is using the LLM as this augmentation tool. So you capture what the data set is, you run it across these different things that you want to test out a few different times, and then still go through this human curation, labeling, et cetera, process to build confidence. another kind of set of work in this space is taking uh this actually used to be research that i did like way back in the day but taking crowdsourcing principles right which is that when what do you do when you can't trust like one human labeler and then combining them uh so that you can build confidence and like what the end product is so taking like those techniques etc are like some ways to mitigate this so how does the guardrails project fit into all of this what are you trying to do with that?

24:12Yeah, so Godrails, it actually came up a little bit organically for me, where I was, you know, towards the end of last year, I was making some of my own applications, kind of very excited about the space. I'd worked in ML for the longest time, and including like joining a company because I was like, it's so, so hard to train machine learning models, and it shouldn't be this way. There has to be a better way and, you know, focus on that for a while. So having the ease of, oh, I don't actually need to train a model anymore. All of it is behind an API and I can just, you know, like use it and actually unlock the promise of machine learning, you know, and use it for downstream tasks.

24:48Right. So I was very excited about this was building and building, you know, some of my own applications and then kind of quickly discovered that even as a developer, when I was testing out what I was building, I was like, yeah, okay, this, this works maybe some of the times. And, you know, for a lot of the times it's not giving me the results that I want. And this is kind of like a fundamental problem with LLMs because they're stochastic models at the end of the day, right? Like you're fitting them into these really complex workflows, but they are like machine learning models and you can't expect determinism from them.

25:19So then the key insight behind guardrails was that, okay, if you can't rely on them, what can you do surrounding them, right? To make sure that you have independent checks, independent like tests, et cetera, that run as a secondary layer on top of the LLM outputs to make sure that it actually enforces all of the correctness criteria and all of the quality criteria that you care about. So that was some of the key insight there. So how Guardrails, the open source project works is it essentially has like a catalog of what we call like validators, which essentially are these like independent like checks that you can run on the top of an LLM, which captures like some correctness criteria, right?

26:02So for example, making sure that there's no hallucination would be like one criteria. Making sure that there is no like medical advice would be like another criteria. Making sure that you're not mentioning competitors, et cetera, would be another criteria. And then within this criteria, you know, if you have infinite budget, infinite latency, which no one has, you know, are more likely to use one implementation, right? If you have like, if you're much more latency constrained, then you're more likely to use this other implementation. So that's the general principle of guardrails, which is, you know, it's building like the secondary layer surrounding the LLM, which, you know, allows you to enforce these criteria.