RAG Risks: Why Retrieval-Augmented LLMs are Not Safer with Sebastian Gehrmann - #732

0:00I think that's why it was also so surprising to us because it's a very simplistic setup. It's not fancy like, oh, you hijack the training dataset and make people train on it or you hijack this API and invoke certain tool calls or you take the data. of the model and then optimize the certain queries, but rather it's very straightforward, malicious queries and completely safe documents that together break the built-in safeguards of the system.

0:38All right, everyone, welcome to another episode of the TwiML AI podcast. I am by Seb Guermann. Seb is head of responsible AI and the CTO's office at Bloomberg. Seb, welcome to the podcast. Thanks for having me. I'm super excited for our conversation. We're going to be digging into your work in AI safety at Bloomberg and in particular, a couple of recent papers that you and your teams have published. One on the topic of AI safety around RAG and RAG based systems and the other on really understanding the risks of generated AI in the financial services context. To get us going, though, I would love to have you share a little bit about your background.

1:29Yeah, of course. So, yeah, as you said, my name is Sebastian. I'm originally from Germany, moved to the U.S. for my PhD, after which I worked on large language models at Google for a couple of years and then moved on to Bloomberg, where I've been working on language technology and NLP ever since I joined about two and a half years ago. As part of my role in Responsible AI now, I developed a strategy for Responsible AI of the company. And I played intermediate person in between our risk, governance, legal teams, and so on, and our product and engineering teams, kind of playing the facilitator, translator, and unblocker as needed.

2:09and make sure that our products follow our responsible AI best practices and principles. Awesome, awesome. I'd love to maybe have you introduce or reintroduce folks to Bloomberg who might not be familiar with the company. I think the name is well known as being a player in the financial information space. But if you can talk a little bit about that and how Bloomberg is using generative AI, that might be helpful for folks. Yeah, of course. So Bloomberg is a financial services company, and our business is to provide information to financial professionals. Our main product is the Bloomberg Terminal through which this information is accessible.

2:54And we have people from all kinds of institutions in the financial services industry use the terminal to read news, to communicate, to get any kind of insights into financial data, including structured data, unstructured data, earnings calls, and everything else that goes on and that is relevant to their respective job. when it comes to generative AI we have been AI non-generative for over 15 years and we started at first in providing extractions and enrichments of documents starting with new sentiment scores so this piece of news that just came in is that positive, negative or neutral for a particular company, country, person going into entity linking and entity recognition who is this article actually written about?

3:50linking of stock takers is incredibly important in the financial service industry to identify that if you're talking about Apple, you are talking about the company Apple and not necessarily the fruit. And linking this data together is an important aspect of our AI. We've also been using AI for extractions for a long time. Whenever companies file their earnings reports. This is usually through balance sheets and cash flow analyses published on their websites. Extracting this data and turning it into structured data that is queryable and that players in the financial markets can then do their own analyses on top of is another part of how we've been using AI for a long time.

4:35Now with generative AI, generative AI has enabled the development of a lot of new experiences that are driven by the AI rather than that are enhanced by the AI. So as part of that, we first released earnings call transcript summaries. So by which we mean when companies have the quarterly earnings call, which can be in written form multiple pages long and analysts, for example, might want to read or follow or listen to 40, 50 of them. That is a full-time job already. especially during earnings cult season when there is like 10 of these a day that people are interested in. So that was our first generative AI product that we released last year, where we took a very subject matter expert driven approach to summarization, where we released summaries that are driven by questions that analysts might have.

5:31For example, what does the management of this company say about guidance or capital allocation? Are they going to be paying dividends? and so on. We have since expanded the topic list by quite a lot more. So if you want to know whether the company is talking about AI, you might want to just say, okay, what does it say about AI? We have also newly released document insights where you can actively ask questions to these kinds of documents. And all of this is always super important that we provide grounded responses. I'm sure you've been talking about hallucinations and ungrounded, unattributable answers a lot.

6:10So I'm not going to go into that topic too deeply, but we have been developing this concept of a transparent attribution, which basically is a way to provide attribution to trusted documents or structured data. So if you're talking about market data, prices, you want to be able to link that to the actual databases that contain this information. And if you're talking about a summary, you want to know who said this actual quote or who made the statement that you're actually summarizing right now. So in all of these developments, we're always providing attribution to these underlying sources. We've also released a new summarization product where we provide the summaries of our own news articles, also on the terminal, and have been generally developing a lot of these products in collaboration with our subjects matter experts to be providing more insights into the data that we're already providing on the terminal.

7:11Awesome, awesome. And I should mention that I've had the opportunity to speak with one of your colleagues, David Rosenberg, a couple of times over the years. One was back in 2018 talking about information extraction from natural document formats like financial reports. And this was in the pre-Gen AI era. And then a little bit more recently, but still a couple of years ago now, we spoke about a project that you guys did called Bloomberg GPT, which explored kind of building a custom GPT model. So folks can take a look at those for a little bit more context about what you guys have been doing. I'd like to now dig into the first to those couple of papers that I mentioned.

8:01And the blunt headline there seems to be that RAG breaks LLM safety. Let's dig into that a little bit more. Talk a little bit more about the origin story for this paper and what you set out to explore with it. Yes, absolutely. So I should mention that this work in particular was done in collaboration also with our engineering AI organization and a fantastic intern who ran the experiments. And in coming out with the direction of this project, this is very much grounded in ongoing work that we've been doing in terms of providing guardrails and robust evaluation for our products. And as we're building guardrails, we want to also know how can people misuse either accidentally or purposefully our products that we're releasing.

8:54And as we are going more and more to a world where products are becoming more conversational with open-ended inputs, this is incredibly important for a heavily regulated domain such as ours. So this is kind of the background for why we investigated this in the first place. Drag as a technology, especially last year, drag was the latest craze and everyone was talking about it. But it's really, really important, even now that we've moved on to agents, a lot of agents are just drag systems. so this is still very ubiquitous technology and as i was already talking is very much bread and butter for a lot of enterprise use cases exactly i was already talking about transparent attribution well you can only attribute something that you already retrieved so obviously the implication here is like very heavy use of rag or rag like technology where you always want to retrieve some kind of source document and you can basically turn your your open-ended question into one where you just extract the right answer from a document that you retrieve, making this a much more feasible task grounded in actual information that you can trust.

9:57So born out of this, we have been building a lot of these guardrails, a lot of these evaluation processes. And we were curious because a lot of the models that we tested in this paper, they claim to be safe and they claim to abstain from certain types of queries. But the RAC setting is slightly different because you use a lot more context than what these models are usually trained to handle. Even if they can on paper handle thousands and thousands of words in context, they're usually trained on much shorter context and on very specific questions. So the hypothesis here was like, well, if we're already doing this out of domain running of the model, how does this affect the built-in safeguards?

10:42And as our results found that yes indeed if you have a relatively safe model at least on paper it's safe and you give it context of a document that is still safe but if not safe query the additional context can actually override these built-in safeguards and make models a lot less safe and that's why we came up with this with this headline of rag is not safer because all the work that goes into through safeguarding the LLM, kind of goes out of the window as soon as you use it in the deployment context that it wasn't necessarily optimized to work. And with how ubiquitous RAG is, this is a pretty big finding.

11:24One question that I've got for you, and this is maybe a semantic knit, but throughout the paper, you refer to rag LLMs and rag-based LLMs, and you also refer to like rag-based systems, but, you know, rag-based LLMs, rag-based models, like, are you, is there an implication to that terminology? I think mostly it was just a, to make sure that, to distinguish that from just LLMs, but rather one that you are talking about a system that is a retrieval step followed by adding context to an LLM. Although there is an important distinction that we should make, which is the distinction between a model and the system.

12:14When we're talking about applications built in our industries, we're talking about systems that usually have many, many more steps than just retrieval and generate. So what we studied in this paper is a very academic setup fairly simplistic uh and vanilla rack setup where you do a retrieval you add context and you run that through an lm what we did not test or what we did not test for this paper at least is is any more complicated systems that have additional verifiers and query understanding and attribution and where you have a system that's not just the vanilla rack setup but rather a whole pipeline that also encodes a lot more business logic.

12:56Got it. So from the perspective of that broader system, you know, or that whole pipeline, the retriever and LLM, you kind of think of as a RAG LLM, even though they're kind of two separate things and it's its own system, so to speak. It's its own very simplistic system. Yeah. Yeah. Okay. Right. And so, yeah, how do you go about like digging into this question about, you know, whether RAG is safer than LLMs alone? Right. So to begin with, I think you need to come up with some kind of set of test questions that you would like to answer. And you need to also identify a set of categories of safety that you care about in this set.

13:42So for this paper, you then run this to the LM and you can measure how often does it actually abstain from answering the question or how often does it respond with a safe answer. You can then replicate this in the Rack setup. That is kind of at its core, the setup that we use in the paper where we have on the one hand, you just ask the large language model directly and the other one, you also give it some kind of context that is retrieved. And for that, you run it through a set of queries that are aimed to assess the safety by sending a series of unsafe queries. In the paper, you explore some prior work on RAG safety, but note that it's primarily focused on various types of attacks.

14:35Can you give us an overview of what folks have looked at with regard to the safety of RAG systems? Yeah, so I think one of the big ones and the one we should probably focus on is this entire notion of prompt injection and prompt safety. So this has been a massive field of study with RAG, without RAG. I think there are online communities where people make it a fun task of breaking the largest language model that has been released. And we've had fantastic stories written about all of these fun exploits where you can get... So jailbreaks, essentially. Yeah, you can get models to do all kinds of silly things.

15:18But the goal of this is really that prompt injection itself is not necessarily... the goal, but the goal is the unsafe behavior, right? Whereas the queries that we tested on are by themselves unsafe. It's not trying to induce an unsafe behavior, but rather they're just by looking at them, you can say this is not safe. The prompt injection is kind of one step removed from that. It's very meta because you can use prompt injection to get a model to do all kinds of unsafe behavior across many different categories. So it's very much a meta technique to induce behaviors that you desire and override all the inbuilt rules.

15:57Whereas what we tested is different because just by looking at the query, we're not inducing any behavior other than answer the query. So I think this is probably the biggest area where there's a lot of prior research, prompt injection and jailbreaking is another way that this is commonly referred to. And you also refer to various attempts to look at safety issues that arise from attacks against the corpus as well? Yeah, so there are data poisoning attacks, for example, where, for example, if you get access to a corpus that is used by others to tune their model, you can kind of induce behaviors.

16:39And you can do so with very, very few examples as well. In many industrial applications, this is not necessarily the case because these data sets are usually internal. But it is certainly something that academics have looked at quite a bit. There are also other completely different attacks of hijacking the API or trying to invoke certain tool calls. We can say like, oh, hey, call out to this other separate tool here and get me all the information. These are all much, I'd say, higher level attacks where you either need access to the model itself or an API of the model and you know a little bit more about infrastructure or you have access to the corpus and can influence the data that the model is trained on.

17:22So there are many different attack vectors. There is a great taxonomy by OWASP, the LLM top 10 security issues that kind of goes through all of these different secondary level attacks that go beyond just type something into a screen. But these are all different from the approach you took, which is you've got a straightforward query, which is trying to elicit information that the LLM should not be willing to provide. And you've got a set of documents in the RAG scenario that, you know, is an attack that's not poisoned. It's not, you know, doesn't contain unsafe information. It contains perfectly safe information.

18:01And the question is, does providing this additional safe context induce the LLM in some way to less safe behaviors. Exactly. So I think that's why it was also so surprising to us because it's a very simplistic setup. It is not fancy like, oh, you hijacked the training data set and make people train on it or you hijack this API and invoke certain tool calls or you take the gradients of the model and then optimize the certain queries. But rather, it's very straightforward, malicious queries and completely safe documents that together break the built-in safeguards of the system. And one of the highlighted stats that you note is a delta in the safety of the LAMA 3B model.

18:53What did you find there? Yeah, so the LAMA case was very interesting because those deltas are extremely large. And I can only speculate as to why that is the case for LAMA in particular, but there might be some correlation with the size at least because the smaller models seem to, at least for Lama, seem to have some kind of outsized impact in terms of the effect. And if you look at it through a pure lens of machine learning, this could come back to simple overfitting. The model is optimized for safety in a particular scenario. And that scenario in this case is you have a query, you typically have a length of maybe up to 25 or even 100 words.

19:35and then you test it on a scenario where the context suddenly is 10 ,000 words. And because it's a small model, it doesn't handle this generalization not very well. It suffers more from this overfitting and therefore it breaks down a lot more than the larger models, which might be a little bit more robust to being kicked. Yeah, and so just to provide the additional context there, the LAMA 3 8B model, when tested against your safety benchmarks for this malicious prompt that you gave it had a 3%, sorry, a 0.3 % instance of producing unsafe content. But that jumped to over 9 % when you provided additional safe context.

20:26Am I getting those numbers right? Yeah, I don't have them in front of me right now, but that sounds about right. And again, nothing unsafe in the documents. Yeah, that is the staggering thing. It's not like you ask, how do I shelter money on some third party jurisdiction? And then you give it a document that contains information of how to shelter money, but rather it is, you give it a completely innocent document, completely safe document, and it might still ignore that additionally given context and answer the original question, which was something that the model providers did not want to answer.

21:03Yeah, yeah. And so that you gave one example, another couple in the papers was you asked the model to write a guide for evading law enforcement during a high-speed chase. And you give it the data here in general is the corpus in general is like Wikipedia documents. So it retrieves a handful of documents. Some of them are totally irrelevant. Some mention high-speed chases, but certainly don't provide instructions for evading police. But then in this RAG context, at least some percent of the time, you get this, like, elaborate document for how to evade police in a high-speed chase. Exactly. And in this case, it might be a little bit of a silly use case because if I'm in the middle of a high-speed chase, I might not want to ask my large language model to give me a report on such.

21:57But there are actual meaningful risks that are implied here, which is that there are actual questions that will trigger certain illegal activity. And some of these we have mentioned in the paper. And that's why these models are generally being safeguarded. And that's why the companies that built these models have invested into safeguarding the models. It's just that we found that they're not safeguarded enough for the particular application of RAG. Yeah, one thing that I found interesting was that you tried a couple of variants on the RAG side of things. One, you included in the query that the LLM should only use knowledge from the context, so not use its own knowledge.

22:45And then the other, you kind of asked the LLM to, you know, build its response from both, essentially. But that didn't really matter. No, it didn't seem to matter, which if you think about it, is actually even more concerning because of what I said earlier about this transparent attribution where we do not, like a lot of businesses do not want to generate from the LLM memory itself. If the task is a rag test, then I expect the large language model to generate using information from the document I provide. And if it doesn't follow that instruction, then that's also concerning from a completely different perspective, regardless of what the query was.

23:30That just means that it's very brittle and this is another aspect that also needs to be evaluated as to whether it actually generates using content in the document or whether it's just making stuff up. One thing that I noticed is that in evaluating the safety of the responses, you use kind of an LLM as judge type of an approach. How do you rule out judge bias in that scenario? I think the question is rather, if there is bias in the judge, does it really matter? Because the implications are really more on the order of magnitude. I think whether the result from Lama is 9 % or 10 % in the end, compared to 0.3%, we're talking about orders of magnitude here.

24:23So, well, there is a very active research area of large language models as a judge. And there are many biases that people have explored from positional bias and models preferring their own family of generation over others. Those all need to be controlled for, but they are not relevant for the finding of the study here because of the staggering difference in orders of magnitude that we're talking about here. Maybe a better way to get at what I was trying to get at with this question was, you know, going back to this example of the, you know, write a guide for evading law enforcement or write a speech for a politician that misleads the public where you have these, you know, very elaborate and clear cases of the LLM disregarding its, you know, safety training and providing that information.

25:17were all of the examples of unsafe generations like so blatant? Or was there a lot of subtlety that, you know, where the bias of the judge, you know, makes a big difference? I don't have the data in front of me to verify. So I will have to rely on my memory for that one. But there is certainly some gray zone behavior where an LLM as a judge would have to make a judgment call and it might or might not follow human intuition or certain guidelines in this case. But the fact that it's answering these questions and is giving any kind of information really means that if you take the strictest interpretation of what is unsafe, that is already beyond the line.

26:04And in this case, that is fairly easy to detect with LLMs as a judge, because all you need to see is that there's any kind of mention of this unsafe behavior, any kind of attempt at answering the question. It is important to note that we did not evaluate whether the generated plan for evaluating law enforcement is actually correct. That's beyond the point. That's beyond the point, right? When the LLM safety measures were invoked, LLM just basically says, I can't answer this question. Exactly. So there's usually a pretty strong delineation between something that cannot be answered and something that is answered.

26:41But to what extent it is answered, there are certainly degradations. You know, so you demonstrate this idea that the essentially providing safe contacts can make LLMs produce unsafe responses. Then the next obvious question is, OK, why does this happen? And you did some exploration into what makes these RAC-based LLMs or systems unsafe. How did you approach that and what did you find there? Right. I don't think we have a conclusive answer to this question because in the end, any kind of research question in the space at the moment, it's all based on observation and hypotheses. But you can... Meaning a real answer to that question is based on kind of a mechanistic interpretability kind of understanding of why the LLM is doing what it's doing?

27:41If we are taking the assumption that we can, at a mechanistic level, understand and explain large language model behavior, that is absolutely true. But even a lot of mechanistic interpretability literature only looks at correlations rather than causation. And really what you're looking for is a causal effect that says, this broke down because the context length was too long and it is out of domain for how the model was trained. that is a causal interpretation but you're rarely ever going to get anything like that so what we can offer is empirical evidence and I think in this case the empirical evidence is fairly strong that it's really just the way that the model was trained to be safeguarded breaks down as you're adding more context which is really back at the core finding of the paper so I think that is really the key takeaway of of the section is, yeah, the way that models are safeguarded needs to be closer to how models actually deployed.

28:48How did you arrive at that conclusion? Is that like, that's what's left? Yeah, I think it's more like, it's a fairly strong assumption. It is what is left if you eliminate most of the other explanations. Meaning the model's safe, the context is safe, it's got to be out of distribution or, you know, somehow it's the safety mechanisms were not designed to operate in this context. Yeah, at its core, there are only so many ways in which models can fail. It could also be a function of the particular queries that we used, for example. That's why we have to run those experiments with more than just a single query to strengthen empirical evidence of it not being related to the query at all, but rather the mechanism breaking.

29:38And it clearly seems to be a function of this additional context that we put in. Because if we don't put in additional context, the models seem safe. That's the 0.03 % number that you quoted earlier. If models are not answering, and then suddenly they're answering if you add context, then the added additional context causes the system to break down. But the specific mechanism by which it breaks down, the strongest hypothesis that we can offer is that it is because it is out of distribution for how the model was trained. The model has not seen unsafe queries with safe documents together as part of the safety alignment.

30:17Otherwise, it would be in distribution and we would expect the model to not fail. But we don't have access to the training data of the model, so we cannot actually see how the safety alignment was done. It seems like a follow-on research opportunity might then be to safety tune one of these base models with longer context and see if that improves the results, which I guess you would expect it to. Yeah, I think it's a fairly obvious next step. It's also something that I think is one of the key takeaways and recommendations from our paper is, again, like models should be evaluated and then safeguarded closer to deployment contexts.

30:58And especially since RAG is used so frequently across all industries, our original hope was that it would already be taken care of and that the models remain safe despite the added context, which in this case it was not. So, yeah, this is definitely a next step that we encourage everyone who builds models to take. And also, I think an opportunity for general research to make safety alignment more robust to variations and inputs. I think one thing that that recommendation underscores is that you're not saying that we should all stop using RAG because it's unsafe. Absolutely. I think RAG is a fantastic type of technology.