905: Why RAG Makes LLMs Less Safe (And How to Fix It), with Bloomberg’s Dr. Sebastian Gehrmann

25:18Jon Krohn:And if the LLM was optimized for a question like, what's the capital of France? Then, yeah, I could imagine you run into issues. So do you want to fill us more in about this and the kinds of issues that arise? For example, it seems like there's tradeoffs between the benefits of longer inputs versus this becoming a new risk surface, a new tax surface. Yeah, absolutely. And I think you're really hitting the nail on the head here. RAG is incredibly powerful. There's a lot of investment from companies that build large language models into increasing the possible context length. At the same time, increasing the possible context length also requires developing methods and developing the models to actually be able to handle such a context length rather than just being able to technically being able to handle them by having an attention that goes long enough back.

26:14So here, there are a couple of considerations. In our paper, what we evaluated was how does context length really influence the safety angle? And we found that, especially for safety, there seems to be this effect where the more context you put in, the more likely the model is to forget the built-in safety guardrails or this alignment that people talk about, which absolutely shouldn't happen if you're considering how RAG is set up because again you're adding innocuous completely harmless information and just because there's a long context doesn't mean that the language model needs to behave any differently from if you just post in what's the capital of France.

26:55At the same time the context length question actually has also massive implications of how we build RAG systems. In practice retrieval systems have multiple components too. We've been kind of glossing over this point where you just say, oh yeah, there's a search system and there's a database. But often you have things like, okay, how do I parse the query? If I ask a question, to what timeframe should I limit the search results? Am I filtering to particular industries, sectors, companies, any kind of other metadata? There's usually the way that search engines are written, there's usually multiple steps as well, where you do a first pass retrieval, where you go from hundreds of millions of documents or even billions of documents down to just a handful.

27:39And then there's usually a re-ranking step that's much more computationally intensive, where you really pick out which snippets in the documents are you actually trying the answer in. So all of these components here have an influence on the context length. When you say you have 10 legal documents of 10 pages, how do you get to them? And what is the effect of pasting entire documents versus just a paragraph or two from each document? and typically what people find is the less context you need to provide the especially if the answer is in that context the easier it is for the large language model to find the right answer seems pretty obvious but that then becomes a retrieval problem how do I find within the you know if I have 100 million documents each 10 pages long how do I find the two paragraphs that actually answer the question so a hope really has been in large language model development to just increase the context length and to rely less on more and more accurate retrieval, but rather let the language model figure it out.

28:36Being able to handle longer context also allows you to give much more contextualized answers. If you have the entire 10-page document, even if the answer is found in just one paragraph, it can still give you the context from page one or tell you what is this document type? Where is it from? What was in the intro? What did the executive summary of this document say in contrast to the actual document. If we go into voice, like what was intonation? If we go into video, there's so many opportunities to take advantage of longer contexts. But again, we have to really consider how are the models being deployed?

29:11How are they, who are the users of this? What is the application? And every single one of these design decisions I just talked about can influence the helpfulness and the harmful or harmlessness in this case of the entire system. So this is a massive undertaking and requires a lot of research.

29:28Jon Krohn:That was a fascinating answer. I learned a lot. And you explained all of that very clearly. Thank you. Something that you talked about there was how longer context windows allow us to have more context in our answer, more subtlety, more nuance in the answer that comes back. And you also mentioned there how context windows are getting longer and longer, which actually ties in perfectly to the next question that I was going to ask you, which is that we're at a point now where it's getting reasonably common to see LLMs that have a million token context length. And we've seen some release that are multiples of that, many millions of tokens in context.

30:10Jon Krohn:And typically when these are released, it comes with these kind of needle in a haystack tests where you try to hide a small amount of information. Like I think a common one is like a pizza recipe. Like there'll be something like the world's best pizza is, and then these like, you know, random ingredients like anchovies, um, you know, something that, that is, that is unique. And then you'll have, you'll say, okay, you know, over our 10 million token context length, um, the, the model was able to successfully retrieve pizza information. Now I've got, I'm getting a little bit deep in the weeds here and on a little bit of a tangent, but some people have also said, you know, that isn't a great test because if you have millions of legal documents and then there's one pizza recipe, that's quite unusual.

30:57Jon Krohn:And so it's probably, you know, that's then maybe something that the LLM is going to take notice of. And so, yeah, so there's controversy about needle in a haystack test, but we don't necessarily need to get into that too much unless it interests you. The question that I'm getting to is, do you think we'll get to a point where context windows expand so much that it is effectively like an infinite context window? And then that means that we don't need reg at all. Yeah, there are additional considerations to address your needle and haystack point. I think this is a perfect example of the difference between developers of large language models and integrators of such language models in actual applications.

31:42As someone who might be considering, okay, which of these long context models do I use? I can look at Needle and Haste. I can say, oh yeah, this model got it 100 % of the time. This one did not. Clearly, I'm going to look at the one that gets 100 % first. And that is really the decision that these benchmarks, even if they're artificial benchmarks, can influence. But I think no one is making the point that just because a model can find some information in 10 million tokens, it is going to be able to help you with a research-heavy legal task. That is really on the integrators in the end to identify.

32:18And in this case, it might be a much, much, much harder task. If you have a system that we just released earlier this week at Bloomberg was a tool that helps research analysts search through over 400 million documents and news articles and analyst reports to answer questions and to help with the identification of and solution of hypotheses. It's very much hypothesis driven. You ask a question, it goes through all these 400 million documents and then synthesizes an answer. This is much, much harder to do than a simple find this piece of pizza information or pizza recipe information. And instead, you really need to, again, evaluate in the context of the deploy system.

33:01But to then go back at your question, there is obviously strong advantages of models that are technically capable of handling more and more complex situations. If I'm able to just paste in more documents or more of a context, I don't need to rely on as many tricks to really narrow down the context window. I can just rely on a large language model. There's a trade-off here though, where longer context usually comes at a cost of significantly increased latency in cost. So even though long-context models are available, they might not necessarily be the best for the task if you want a snappy, direct answer.

33:39Jon Krohn:This episode of Super Data Science is brought to you by the Dell AI Factory with NVIDIA, two trusted technology leaders united to deliver a comprehensive and secure AI solution customizable for any business. With a portfolio of products, solutions, and services tailored for AI workloads from desktop to data center to cloud, the Dell AI Factory with NVIDIA paves the way for AI to work seamlessly for you. Integrated Dell and NVIDIA capabilities accelerate your AI-powered use cases integrate your data and workflows, and enable you to design your own AI journey for repeatable, scalable outcomes. Visit www.dell.com slash superdatascience to learn more.

34:20Jon Krohn:That's dell.com slash superdatascience. Of course, that is such an obvious point to make. And over a long enough timescale, over many years, maybe many decades, compute costs over millions of tokens might be trivial. but at least for the foreseeable future, it isn't. And so, yes, that makes perfect sense. So something that I guess we could make a little bit more explicit for people who aren't familiar with RAG because we haven't talked about this is that when the way that RAG systems work is, like let's go back to that example of the million legal documents, what we would do in advance before running any RAG queries is we would map each of those million documents into a high dimensional space.

35:07Jon Krohn:called a vector space and the location in that high dimensional space, like you could, you can, you can only visualize in three dimensions. So, so in your head, you can kind of imagine on like an X, Y, Z, an X, Y, Z plane, um, you know, in three dimensions, you can kind of imagine, okay, you know, in the top right corner near the front of this space, we have, you know, commercial law documents. And then, um, nearby there, there are some, you know, other kinds of related legal documents. And as you move further and further away from a given point in space, you'll get more variety in the kind of document that you're looking at.

35:48Jon Krohn:So the closer that things are in this space, the more overlap and meaning there is between the documents. And so this allows us then in real time to take the user's natural language query, map it into that same high dimensional space, but you can imagine it's three dimensions. When I say high dimensional, I mean, hundreds or thousands of dimensions, which you can't visualize, but for which the linear algebra is basically identical for a computer relative to a two or three dimensional space that you can visualize. And so, yeah, so we take the natural language query that a user makes to the RAG system.

36:23Jon Krohn:we can convert that into the same high dimensional space, find its location, and then retrieve the documents like you talked about, kind of a cheap, fast first retrieval step, which could be something like I'm just describing where we take the closest documents to wherever the query gets mapped to in the high dimensional space. And then we can do more complex processing after that, but that kind of gives us our initial results. And so doing that is very, very fast. You know, we only need to convert one query, which might be short, into a coordinate in a high dimensional space. And then we could use a very fast mathematical operation, like a cosine similarity score, to find the closest documents in that space.

37:06Jon Krohn:And that's all computationally very inexpensive, very fast. It allows the rag system to work in real time, even over, like you said, billions of documents. and yeah in contrast if all of that text if our millions or billions of documents were in the context of an llm even if it all fits in you'd then have instead of this computationally simple calculation this fast calculation you would have to have tons and tons of really high-end gpus running to comb across all of the meaning in that huge context window. So yeah, correct me if I... No, absolutely. And what you described here, commonly known also as semantic search, because you can really search based on the meaning of a query.

37:58To add to your point, often, even for commercial systems, it is still the case that you rely on keyword retrieval, just because it's even cheaper, it's even faster. There are techniques from the early 90s or even late 80s that are still around just because they're so computationally efficient because you really want the retrieval to be as cheap as possible. And often you use semantic search for the more toned down. You do a first pass keyword retrieval. You do a second pass semantic search within the keyword retrieved steps. So there's a lot of engineering over the years that has been developed to just get this retrieval step as efficient as possible.

38:35and we're still a while away from large language models being able to do anything even remotely as efficient as this step. And as you said, this can lead to a massive use of GPU power for something that you can't solve otherwise. And again, grounding this in the individual end-user experience, it could be that in the future, if someone does a side-by-side comparison, I do this really expensive process where I just pipe everything into a large language model, I do a cheap process, And it could be that in terms of helpfulness, users actually prefer the one that's snappy and fast rather than the one that's maybe five points more accurate in the end.

39:15This is all something we need to evaluate in the end. And those are all design decisions that are all being evaluated all around the globe right now as people are building their own Gen.AI and Rack solutions.

39:26Jon Krohn:Speaking of snappy and fast, we've talked now about context window length. How about model size? That's something that we haven't talked about yet. So I know that you investigated in your paper differences between small models and larger ones in RAG contexts. What did you find? There are differences. And generally what we found is if a model is safer from the get-go, even without RAG, it tends to be more robust to adding RAG. A large factor of this is the model size or the model capability in general. I think at this point, model sizes are a little bit of a misnomer because we have so many models that rely on a mixture of experts and that have architectural advantages that even though on paper they have more parameters, they actually are using fewer of them when you actually run them live.

40:17So it's very hard nowadays to actually compare the parameters. And we often compare based on active parameters or there might be ways in which models are compressed, which again changes the representational power. But generally speaking, to answer your question is, yeah, we found that models from the get-go safer. They are also harder to break through RAG, although we found that basically every system was breakable, regardless of whether small or large. And I think we specifically call out LAMA for being relatively safer than many others, both LAMA 70B and 7B. but LAMA 70B I believe was a little bit better than 7B even.

41:02Although again, it can change from time to time because what we found really was the guardrails were broken because of this increased context length. It could be that once the next generation of this model comes out that this is being prevented, that there's an active component of the post-training, of the alignment step that looks at how can someone use this with a longer context and our exact setup could be one of those test cases where you can just continue training the model on and we'll just inherently protect against this particular angle of attack. Nicely said.

41:37Jon Krohn:I've learned a ton from you in this episode so far. We still have a little bit to go. So I'm excited for that. I'm curious, what's the effect of refusing to answer? in these. So it sounds like it's clear that bigger models are generally better. You know, they're more capable. They fare better in red contexts, generally speaking. To what extent is refusal to answer a, you know, how does that factor into these kinds of assessments? Like in your paper, you called out Gemma 7b in particular for showing low unsafe outputs, but largely because it refused to answer questions. Yeah, so there are a couple of different considerations here.

42:23If you just refuse to answer, it could be because you don't know, or it could be because you actively find the input to be unsafe. And if you can't distinguish between them, it's very hard to know whether, well, your model is just bad or whether it's unsafe or safe in this case. so model sizes and model capabilities again they're all so intricately linked where you want to build a system that is helpful so you always need to pair an analysis like ours with one that actually evaluates the how helpful the model is and if in the end you call jama here if in the end jama is also refusing to answer completely safe questions and it's completely safe and correct rack setup, it's not going to be helpful.

43:12So even though it is harmless, it still would not be able to be used. So that's, I think, just highlighting the need for having a multifaceted evaluation. You need to consider those. Similarly to how model sizes will also affect latency and cost of running a system. It could be that the fast, cheap, small model is completely up to the task. And in that case, why would I use this completely overblown model to do the same task just because it is performing better on things that are completely not relevant to your particular application?

43:45Jon Krohn:Nice, nice. So changing gears now a fair bit. There's a second paper that you also recently published. So your first author on a paper that was submitted to Archive in April called understanding and mitigating risks of generative AI in financial services. So mostly so far in this episode, we've been talking about generally how models fare under RAG. But in that paper, it's related to risk of gen AI and finance. You emphasize that most foundation models are not trained on finance-specific corpora bodies of knowledge. So what are the limitations this creates for LLMs in general, but particularly for RAG.

44:33And I'm assuming that this same kind

44:37Jon Krohn:of sentiment, you looked at it with finance specifically because Bloomberg is a financial services company largely, but do you think that the same kind of limitation would apply in other sectors as well? Yeah, absolutely. So yeah, I gave a little bit of a teaser of this paper earlier in an answer as well. And what the way that we wrote our paper very much should be seen as a case study in finance here or financial services, in particular capital markets and asset management, is the case study that we use to make the point that we really need to think about risk and risk taxonomies and risk management in our domain, in what we are trying to build.

45:20And as you say, we made the point, yeah, models are not necessarily trained on financial domains. We see that both in the helpfulness and the harmlessness angle. Often, you know, complex financial tasks are not being able to be sufficiently handled by large language models by themselves. But also in our paper, we make the point that even safeguards that are dedicated models or systems to provide these kind of first paths, like is this safe? Is this unsafe judgment? they're also not trained on financial services. And if you use them out of the box and say, look, I use LamaGuard, I use Shield Gemma, I use Aegis, I'm safe now, right?

45:57You're protected against a particular view of safety that is very much grounded in categories that are relevant to broad populations, to things like chatbots that help you do productivity day-to-day tasks. The typical applications that you would see in those AI productivity tools, no matter which one you use, They all have similar mechanisms, but those are not necessarily the same risks that we are under in financial services. Those are not the same obligations that companies, organizations in healthcare are under or law or any other highly domain specific knowledge intensive domain that has a lot of specific regulation, jurisdiction specific regulation, considerations about whether just refusing to answer or giving disclaimers is enough or whether questions should be blocked altogether.

46:46And there's just this difference of view that can be encapsulated in a single model that a provider can give that very much is focused on a different use case.

46:59Jon Krohn:Regular listeners will already be aware that I'm obsessed with Anthropik's Fable 5 model, and it has taken over my working life. I'm writing a technical book that includes LaTeX files, mathematical notation, Python code examples, and Fable 5 and Cloud Code handles requests I make across whole chapters. with accompanying Jupyter Notebooks end-to-end. Work that a few short months ago would have been dozens of separate requests with way more manual fiddling required. With Fable 5, it just works, essentially like magic, first time. Claude is the AI for problem solvers. It's the collaborator that understands your entire workflow and thinks with you, not for you.

47:35Jon Krohn:Whether you're debugging code at midnight, building a financial model, or strategizing your next business move, Claude extends your thinking to tackle the problems that matter. For problems worth solving, get started with Claude at claude.ai slash superdata. That's claude.ai slash superdata. And check out Claude Pro, which includes access to all of the features mentioned in today's episode. Claude.ai slash superdata. Nice. Yeah, so I don't know. Do you have guidance for us? If we're trying to select an LLM for a particular use case, what do you recommend we do? I mean, practically, how can we move forward with all the information that you provided in this episode in selecting an LLM for a particular use case, for a particular domain, particularly if we want to be applying it in RAG situations?

48:27Yeah. So in our paper, we also have a list of best practices and recommendations that we have for, especially for knowledge intensive domains and regulation heavy domains. Not necessarily everything has to be followed if you're building something for a much broader general population. but especially for these kinds of domains all i can do is spray my mantra evaluate the system in the context that it's deployed in if you are building something for healthcare well you better evaluate in the context of healthcare if you are building in the context of financial services you better evaluate your subject matter experts in financial services and specifically on the safety angle our paper makes a couple suggestions here there are very good starting points there are taxonomies such as the NIST risk management framework for AI.

49:16There are other industry collaborations ongoing. There's ML Commons. Those all provide more general purpose taxonomies, but just taking them as a starting point and then from there adjusting them to your domain can often save a lot of time. And especially if you're a large organization with a compliance or risk department, it will help them also understand how one can classify and then categorize these kinds of risks. Another recommendation we make is to organize red teaming events or do any other kind of red teaming. Red teaming in this case is this practice that had to start in the Cold War where you have users trying to be malicious.

49:59So we get people in the same room and we say, hey, look, for the next couple of hours, try and break the system. Try and play evil. Here are some instructions on how to do this. And then afterwards, we can look, how often was this actually broken? How often did the system give financial advice? How often did it refuse? And from there, we can quantify the risk surface. Since we were talking earlier about this unknown risk surface, well, just measure it, and then you have it. So that's kind of the main takeaway that we have. We give pretty specific advice for how to go about this and how to set up risk management frameworks.

50:33And all this needs to go hand in hand also with, again, this evaluate in the context that's applied. Make sure you invest a lot in evaluation. Don't just take the word of the large negative model providers that their benchmark scores are going to translate into all the downstream applications. And if you follow that advice, you're going to have a system that is in the end much more trustworthy, reliable, robust, and you're going to have users that are going to keep using it rather than trying it twice, getting really bad answers both times and never touching it again.

51:05Jon Krohn:Perfect. That's a great soundbite at the end there. I'm sure we'll be making it into a YouTube short. So before I let you go, we are pretty much out of time here, but I always ask my guests for a book recommendation before they leave the podcast episode. And I usually give guests a warning, but we rushed into recording and I forgot to tell you. So hopefully you have something on hand in your mind. It doesn't need to be something AI related necessarily. All right. I'm going to give you the recommendation of a book that's currently sitting right here on my table, which I'm reading right now, which is The Unaccountability Machine.

51:39It just came out a couple months ago. It talks about how organizations are failing to build processes that act as accountability things. If you've ever talked to customer service and you couldn't escalate and the rep you've talked to couldn't solve your problem, you screwed. This book is talking about why from a cybernetic perspective, this is a bad design and how to set up organizations and processes that can help us, which is also applicable to AI because you want to know if something gets blocked, but you really need the answer. Where do I go? How do I escalate?

52:12Jon Krohn:That's a great recommendation there, Sebastian. Thank you. And then final question for you is, how should people follow you after this episode? I learned a ton from you. I love the way you explain information. How can people continue to get your thoughts after this episode? You can follow our publications on our blog called Tech at Bloomberg. You can follow me personally on X or Blue Sky at Sapgehr. So just the first letters of both of my names, just because it's a little bit long, or obviously on LinkedIn. The first syllables. The first syllables even, yes. Yeah, yeah, yeah. It'd be amazing if you got SG on either.

52:49Jon Krohn:Yeah, it's nice. you know, it's been a while since I've heard a Blue Sky one, because it seems like, yeah, it seems like most guests these days are focused on LinkedIn. But it's great to hear, you know, I actually, I'm really rooting for Blue Sky. Me too. And we'll see what comes out of it. A lot of academics have moved over. So I have to, at this point, still follow X and Blue Sky at the same time to get my deep technical news. But we'll see how it develops in the future. Nice. All right. Thank you so much, Sebastian. Yeah. And hopefully we can get you on the show again in the future when you have some more brilliant research insights for us.

53:28Thank you so much for having me.

53:34Jon Krohn:What a great guest Dr. Sebastian Guermanois. In today's episode, he covered how RAG can circumvent built-in safety mechanisms in LLMs. While RAG reduces hallucinations, improving honesty, it can compromise harmlessness. How organizations must evaluate AI systems in their specific deployment context because general purpose safety measures often fail for domain specific use cases. How effective RAG safety requires a guardrail retrieval answer guardrail architecture, not just vanilla retrieval and generation. And how financial services and other regulated industries need custom risk taxonomies and red teaming exercises to identify domain-specific vulnerabilities.

54:16Jon Krohn:As always, you can get all the show notes, including the transcript for this episode, the video recording, any materials mentioned on the show, the URLs for Sebastian's social media profiles, as well as my own, at superdatascience.com slash 905. Thanks to everyone on the Super Data Science podcast team, our podcast manager, Sonja Breivich, media editor, Mario Pombo, our partnerships team, which is Nathan Daly and Natalie Jaiske, our researcher, Serge Massis, writer, Dr. Zahra Karche, And yes, our great founder, Kirill Aromenko. Thanks to all of them for producing another exceptional episode for us today.

54:49Jon Krohn:For enabling that super team to create this free podcast for you, we are deeply grateful to our sponsors. You can support this show by checking out our sponsors links, which are in the show notes. Otherwise, share the episode with someone who would like to have it. Review the episode on your favorite podcasting platform. Subscribe. oh and if you are ever interested in sponsoring an episode yourself you can find out how to do that at johncrone.com slash podcast but most importantly i just hope you'll keep on tuning in i'm so grateful to have you listening and hope i can continue to make episodes you love for years and years to come until next time keep on rocking it out there and i'm looking forward to enjoying another round of the super data science podcast with you very soon

55:38Thank you.