#147 Yilun Du: AI Debates, Reinforcement Learning, & The Power of Generative Models

0:00One of the fundamental limitations of ROHF is that you don't actually know if the model has learned the thing you want to teach it right? You have no guarantees that the model has learned what you have taught it. Some of the most successful companies right now are all in charge of basically collecting data. And I guess they're not covered as much in the news, but I feel like that seems like a huge amount of people. If you have this AI feedback and it can give you answers for everything, right? You would hope that the coverage is actually a lot more than the coverage you could get from human feedback.

0:32Hi, I'm Craig Smith, and this is Eye on AI. Large language models are notoriously inaccurate, stringing together coherent sentences that often have little grounding in reality. To fix this, companies like OpenAI deploy armies of humans to try and nudge LLMs toward more accurate responses using reinforcement learning, a process called reinforcement learning with human feedback, or RLHF. This is obviously an inefficient and labor-intensive way to tune the world's most powerful AI models, and researchers are looking for ways to automate the task. In this episode, MIT PhD student Ilun Du talks about new techniques for improving large language models through reinforcement learning with AI feedback, or RLAIF.

1:32Ilun explains how RLAIF circumvents the need for expensive and inconsistent human ratings by having AI agents debate and critique each other's responses. He shares insights from his research using multi-agent debate to enhance reasoning and accuracy, and we discuss the challenges of accessing proprietary models. I hope you find the conversation as interesting as I did. Hi, this episode is sponsored by Salonis, the global leader in process mining. AI has landed and enterprises are adapting, giving customers slick experiences and the technology to deliver. The road feels long, but you're closer than you think.

2:22You see, your business processes run through many systems, creating data at every step. Solonis reconstructs this data to generate process intelligence, a common business language. With process intelligence, AI knows how your business flows across every department, every system, and every process. With AI solutions powered by Solonis, enterprises get faster, more accurate insights, a new level of automation, and a step change in productivity, performance, and customer satisfaction. Process intelligence is the missing piece in the AI-enabled tech stack. Search Celonis, C-E-L-O-N-I-S, to find out more.

3:12This episode is sponsored by Crusoe Cloud. High-performance cloud computing or low environmental impact, Crusoe Cloud was built because the innovations of the future need both. Crusoe Cloud is a scalable, clean, high-performance cloud optimized for AI and HPC workloads and powered by wasted, stranded, or clean energy. Crusoe offers virtualized compute and storage solutions for a range of applications, including generative AI, computational biology, and rendering. Visit CrusoeCloud.com, that's C-R-U-S-O-E-C-L-O-U-D dot C-O-M, to see what climate-aligned computing can do for your business. I'm currently a PhD student at the Computer Science and AI Laboratory.

4:10So I've been working on generative models for the last five years, especially on their application to construct intelligent physical agents or robots. So, I guess another name for generative models recently has been generative AI. So, I've been working in this space for the last five years. And yeah, mostly I've been working a lot on diffusion models for quite a few years. And then recently also a reasonable amount on language models. Yeah. So, I'm interested in RLAIF, reinforcement learning with AI feedback. And I guess my first question, the reason I'm interested is because that's how OpenAI and the other large language model providers are trying to nudge their models into good behavior to stop hallucinating and things like that, which to me seems like a very unsophisticated method to use armies of workers around the world sort of, you know, scoring responses.

5:29So RLAIF is a natural extension or solution. but am I wrong that reinforcement learning with human feedback really began as a strategy in robotics? So not quite. Actually, in 2018, I was working on OpenAI. So that's when they just started actually this ROHF. So there was a paper between OpenAI and DeepMind. I think this was in 2018. That was, which was, I guess RL itself is a field that has been going on for like many, many decades. Deep reinforcement learning is also a thing that actually has been going on for even like for like two decades. Actually, even before DeepMind's DQN, there were quite a few works on deep reinforcement learning.

6:26But anyways, this ROHF idea at the time in 2018 was that RO agents would often exploit your environment, and they would exploit parts of the environment that you really would not want it to. So essentially, maybe you wanted to complete this race course, but it finds that it gets a lot of points by looping around the ship. So instead of completing the race course, it will end up looping around the ship an infinite number of times. So initially, the goal was that by having humans give feedback to our agents when playing these interactive games, you can prevent the agents from doing prevent the agents from like doing very odd behaviors.

7:09And then this evolved, I think around 2018 was when the first GPT model came out. So then there is the safety team at OpenAI. And I remember there were several people who were interested in the idea that you could use this idea of RL supervision to train the language models to summarize or to have humor. Because it seemed like these are things that are very hard to teach the language model if you just gave it a bunch of internet text. So I guess that might have been the start of this idea. And then afterwards, there were a variety of papers. and then eventually, I guess, like, I guess ChatGPT was a version of this, like, RL from human feedback.

7:54Right. And just for those listeners who don't understand how it works, can you describe the RLHF, how it works in the sort of process? and when you were working on it in OpenAI, was that before the GPT models or was that on early iterations of the GPT models or were you using, what were you using RLHF for? So I personally was not using RLHF. At the time when I was at OpenAI, it was a very small company, I think around 40 people. And then maybe, so we knew, I knew about all the research that was going on at the time. So this was right when the first GPT-1 model came out or before that. So this was very early on.

8:48The focus of the company at the time was RL. And then there was focus on this game called Dota that was going on. And the idea of RLHF is that your model will generate different outputs, and then you'll have people rate them. People will say, this is very desirable. This is not desirable. uh and then you'll basically and then this gives the model supervision on uh on like if you should generate this output um more often or less often and and as i said uh since these language models have blown up and hallucination uh has is such an obvious problem the there people have been using rlhf and and it's actually uh to to guide the the models into better behavior uh two things about that fascinate me one is that uh you're only guiding you're only kind of nudging them the model uh and and uh is there some metric for how many times you need to push the model to stop it from doing something you're pushing it away from but also uh as this has as llms have exploded uh there's like this new industry of human reviewers uh around the world and do you have any sense of how many people are involved in rlhf for large language models at this point i mean the human reviewers?

10:24Yeah. So I guess going on the first question. So yeah, I think that's one of the fundamental limitations of ROHF is that you don't actually know if the model has learned the thing you want to teach it, right? You have no guarantees that the model has learned what you have taught it. And then there's a variety of papers that have shown that you can take any of these large language models and just by adding some additional prompts, it's very easy to make them elicit behavior that you would have hoped that you had made the models forget. So yeah, a big issue with ROHF is that the reward function just nudges the model towards doing something you want it to do.

11:04It doesn't necessarily guarantee that your model has learned this. On the other hand, though, you could say that human learning, also, as a teacher, you kind nudge a student with, this is the response you want, this is not the response you want, but you don't necessarily know if the person has actually learned the thing you wanted to learn. Maybe they just have copied the equations you've given it. But yeah, so along the first question, I think that is an issue with ROHF. Along the second question, how many people are rating these models? That is actually something I don't really know very well.

11:43My understanding is it's a lot. So I I think based off what I've heard, some of the most successful companies right now are all in charge of basically collecting data. And I guess they're not covered as much in the news, but I feel like that seems like a huge amount of people. And I think the job is very, I guess you could say boring and maybe scarring also because you see all this stuff on the internet and then your goal is just to label things. Yeah. And the, yeah, I guess that's right. I mean, I hadn't thought of it in that regard. It is just labeling, right? It's like supervised learning labeling.

12:29It's just for a different purpose. And when you said some of the most successful companies, you're talking about data prop companies or? Yeah, like data, like, yeah, like, yeah, uh data companies that are focused on gathering data yeah yeah um so uh this bringing ai into the picture seems like a natural solution uh and can you talk about how you do that how do you get an ai to rate uh a response uh from from a model yeah so So instead of, so like when you do ROHF, what happens is your model will generate different outputs and then people will just say, this is good. This is not good. So in a very similar vein, you can imagine that you have two language models.

13:29One language model generates the answer and then you give the other language model that may be the ground truth answer and the question. And now it can like basically assess how good the generating answer is. like first, does it match the ground truth answer? And then they can like check the reasoning, like does the reasoning make sense to the language model? So that's a very simplistic version, you can imagine of ROI for an AI feedback. And then there's like a variety of different ways to make it much more sophisticated. So one way that I've been working on a lot is this idea of using multi-agent debate.

14:05So what happens is now, let's say I give you a question. what happens is you have multiple instances of your language model generate different possible answers to the question. So it's kind of like when I give you, when you're trying to solve a math problem or you're trying to answer a question, you have different thoughts in your mind. And each of these voices in your mind kind of talk to each other. So each of these language models talk to each other, try to, is this answer consistent with my first thought? Is it consistent with my second thought? And then you can imagine that you have these language models debate with each other.

14:37And then the answer you get at the end of this debate should be a more accurate answer than the original answer you have. So you can imagine that now this gets you better data, right? And then you can again use this data to retrain your model. The core idea is basically you can use the model to get better data. You generate from your model and then you can impose some structure on this generation process to encourage the model to do a bit more reasoning or causal chain of events or stuff like this. And then And at the end, the answer you get is an improved version of what the model would originally generate.

15:12So it gives you some improvement operation. Yeah. And is there... So in the first, the more simplistic scenario where you're giving a model the ground truth in the prompt, is that... you do that in the prompt or you're you're you're connecting a vector database with with uh you know verifiable or ground truth data how do you yeah i think probably the simplest way is to just give it as a prompt uh so so you can ask the first language model here's a question generated answer and then the second language model uh you give the original question again and then you give the generated answer and then you say here's the ground truth answer and then and then you can instruct the language model uh reflect and like uh describe what is correct versus not correct about the answers so walk me through that let's say uh you ask a model what's two plus two and the model answer is five and then you have another model yeah so so yeah so let's say the first model generates two plus two is equal to five so what you do now is you give the second model you say here's uh so you give it the same question what is two plus two and you and then you give it round truth answer for and then you give it answer generated by agent uh two plus two equals five uh and then you can say uh uh can you check if this answer is correct and provide feedback on what portions were correct or what portions were incorrect.

16:56So one thing you could do is you could feed the intermediate generations of the language model because the language model generates the answer step by step. And you can give it the first sentence and say, is this consistent with the right answer? You can give it the second sentence and say, is that consistent? And you can get much richer annotations than just, this is correct, this is not correct, which is in principle something you could also do with people, but it's very tedious for people to do this. So in practice, most people, most of the time, the ROHF feedback that you get is just like holistic over the entire output.

17:30Yeah. So in these LLMs talking to each other, you don't have a human operator that's asking one LLM and then takes that answer and asks another LLM this is all happening within the software between the two LLMs. Is that right? Or is there some human oversight required? Yeah. So in this debate speech feature that I was talking about earlier, yeah, there is no human oversight. Well, I think one big thing about RO for an AI feedback is this idea that you do not want humans to be their bottleneck. Because LLMs or all of these models, they can be accelerated very quickly on hardware accelerators. But people operate at a specific speed.

18:28They're expensive. Maybe they're inaccurate. So yeah, the goal would be that completely automate this as much as possible. Right. But where does the LLM, the reviewing LLM, get the ground truth? data from a vector database or uh yeah well i think there are two two two ideas right so one idea is that you yeah you could give that uh you could give the uh like the ground troops llm the correct answer right like if you if there are a set of problems that you want to teach the llm to be more accurate on right you could uh you could just uh generate a bunch of questions get a bunch of answers right also and then like use the cell of them to like critique your answers.

19:13I think the thing that people a lot of people want is I assume that you do not have the ground truth answers. So even if you do not have the ground truth answers right like let's say let's say I want to like write a proof for like a mathematical theorem. I don't know the ground truth answer but I can like check to see if someone else's answer is incorrect. So I can can like clearfully look at the first step and say, well, this step is logical or this step is not logical. I can check the second step. So one thought about this, like ARL for AI feedback is in the same way to like how people, you can like prove mathematical proofs by sequentially like just checking and making sure everything is correct to your knowledge.

19:55You can imagine each of these LLMs like check each other and see is every single step the way correct? And if every single step is rational and you get the answer you want at the end. Right. And again, even if you don't know what the ground choice answer is, we still know this is a satisfactory answer, or this is not a satisfactory answer. So you can imagine it's a way to search through the space of possible solutions. So having one LLM generate many different things and having the other one prune the ones that are probably wrong. And if you prune all the bad ones, then you get the right answer.

20:29And then if you have a model that can always generate the right answer, right, it's a more intelligent model than the one that's not, that's just generating all possible answers. And this, I mean, I understand how that could work for a logical proof or mathematical equation. But how does that apply to a more general and less precise questions and answers? I mean, if I ask an LLM, what's the history of this town? And it comes back with a hallucination that's not correct. How does the other LLM know to check that first LLM's answer? Yeah, so I think this is where, so I guess I talked a bit about how, like, in some of my work on ROEIF, RL for an AI feedback, I use this idea of having multiple models.

21:35So I think if you only have a single answer, right? So like, if I ask you like, when was this person born? And then the LLM just makes up some number. There's no way to verify if it's correct or not. But the thing about LLMs is they can generate multiple answers for a question, right? So if I ask the LLM, when was this person A born? And then it gives me 1876, and then it tells me it's 1896, and then it tells me it's 1895. Now you have multiple different answers, right? And now your critic can say, well, clearly something's amiss here, because none of these answers are consistent with each other.

22:13So I think when you introduce this idea of not just having one model generate, but have multiple instances of the model generate different solutions, then you can kind of verify issues with the causality or the chain of reasoning of language models. You can identify things that are logically inconsistent by just having multiple generations come out. yeah uh so by having multiple generations of or multiple agents debating as you said at the beginning is that is that what you mean yeah that's what i was thinking and and i mean uh this is all happening in some vector data vector space right it's it's not happening in actual

23:08language on on a screen and that's then being fed to the other agent how does how mechanically how does it work yeah so actually uh so far for the work that i've been working on debate uh you actually have the language models generate language outputs uh explicitly and then each of the language models takes in these language outputs from other models and then uh it gives feedback on that so So it's purely in the language space. And probably it could be more efficient to be able to do it in some type of vector space. But actually there's some desirable properties having it in language. If you can have all these agents reasoning language, then we can kind of understand what the agents are doing also.

23:52It's a hard constraint that forces the model to be kind of reasonable in a sense, because if you do it in some abstract space, maybe you as a person, we have no idea what the model is doing. And it might also be finding all types of like cheating ways to like communicate. So like having it, having all of this self-improvement be directly in language, I think is a desirable thing also. And so in your research, are you getting printouts of these conversations between AI agents? How do you monitor what the conversation or the debate is between agents. Yeah, you can actually, yeah. So you can actually see the entire printout of the conversation.

24:38So like it starts with like, you give it a question and then model and then agent A will say, based off the order of operations, I think it's X, Y, Z. And then here's the final answer. And then agent T will say, based off this other thing, I think the answer is A, W, A, B, C. And then each of these agents will respond to each other. So like one agent will say, well, actually, I think the second agent is wrong. I think it should be this. And the second agent will be like, oh, no, I am correct. And then you can have this multiple rounds of conversation. And then essentially at the end of this conversation, each agent will be like, based off what I've seen of the other agents, I think the answer is ABC.

25:16And then you just take this final answer. And it will converge eventually to one answer? Or do you end up with LLMs that will never agree? Yeah, so most of the time, you end up converging to the same answer, or like converging to one answer. I think this might be due to the fact that the models... So the models are trained with... These models have been trained to converse with people, right? And they've been trained to converse with people in such a way that they are not very stubborn. If a person's constantly suggests an answer, they will respond to it. So what we notice is when you try to repurpose this mechanism to have multiple models talk to each other, right, then the models are very definite to other models.

26:04So it's almost all the time they will converge to one answer. And then you check that answer against ground truth or against sort of... Yeah, basically we just check against ground truth and we'll see how good it is. Yeah. And so what are the, presumably you have metrics of how different strategies have improved response accuracy or something like that. But how do you measure that? So, yeah, so there are a set of existing benchmarks. And then what you can do is, and roughly they're in the style of, here are a set of math problems, and I know the ground truth answer. So basically, you take the final generation from all the agents, and then just compare the accuracy in which that final answer matches the real answer is one thing you can do.

27:00And in a similar sense, for factually correct things, you can ask the model to generate all these facts, right? And then you can get ground truth facts about the subject of interest. And then you can basically check the accuracy in which the language model generated things matches the ground truth. So all of these problems, to evaluate this approach, you can always come up with a database of answers and check those. I mean, the hope would be that you should you would always have these benchmarks that you know the answers for. So you can like concretely monitor monitor monitor the progress. But then ideally, you'd want to do it on all types of subjects where you don't have very clear ways to monitor how good it is also.

27:43Yeah, because and again, is this in the in the fine tuning stage that you're doing this? where you're trying to improve the accuracy of one model by having it debate with these other models? Or is this an architecture for, as a consumer, you know, I interface with a language model on my screen, but in fact in the background there's this debate going on between multiple models that then converge on an answer before supplying me with the answer i mean is this a training strategy or is this actually a new way of of building large language models yeah i mean so so in the in the paper we wrote about this we primarily focused on this application where you finish training this model.

28:51And then this is a way to get better answers from the language model. But I mean, I think one thing that I've been very excited about is this idea that this is also an improvement operator, right? During training time also, this is a way to get like RL for AI feedback. And like I've been recently talking to several companies that I've also been like, I guess I've been training the large language models and they're also like thinking, like exploring one of these strategies. uh so yeah so i mean i i mean it's it's i think it could be either right in general i think anything that you can do for like any technique for ai for an uh like rl from ai feedback each of these each of these strategies gives you some mechanism for how good an answer is right so you can always use that as a test time thing also as well as a training thing i like there i think they're like two flips of the same coin like i don't think one is very different than the other yeah uh but but But if you're using this strategy to train the model, that's one cost.

29:53But if you're building it into the model so that the model is going through this process to produce every inference, it seems to me that would be extremely costly. actually on one on one side if it's it's if it's built into the inference every time you ask a question before you get the output there is this debate going on in the background that would increase the cost right uh yeah but but then on the on the on the other hand if you're using it for training and this is something about rlhf also or or you know trying to nudge these models you you do it for example on uh you know questions where there's a very clear ground ground truth answer but that doesn't mean the model will necessarily generalize to all questions right uh at training time uh well yeah yeah so so i think yeah so definitely it's a case if you use that at test time the computational cost is much more uh i feel like it's always it always seems like a thing though that if you have a thing that improves performance the actual computational cost is always like not the thing to care about the most because that cost always decreases year on year.

31:23So I feel like it's always been the case that every year on year, things get much, much, much cheaper and cheaper to do. So if you have something that can fundamentally improve the performance of your system, it seems like worrying about cost is not the most important thing because people always find ways to make things faster. But yeah, on the latter point, So, yeah, so if you, well, ideally, any type of RL from AI feedback, you would be able to apply on arbitrary prompts, hopefully, because ideally you construct this interface that all the models can just autonomously talk to each other and generate answers.

32:04And as long as the models can autonomously generate answers, you can give it all types of prompts. One issue with having people rate the accuracy of something is it's really hard to rate the accuracy of not so well-known people. Because you would have to search for ages on Google and maybe to verify one passage that's generated, maybe you need to spend two hours. So it's not very scalable. And then for lots of prompts, maybe there's just no easy way to do it. But then if you have this AI feedback and it can give you answers for everything, right you would hope that the coverage is actually a lot more than the coverage you could get from human feedback right in the training phase you mean uh yes in the training phase in the training phase because that that's an issue uh right when when i mean the hope in in rlhf is that the model will eventually, it'll learn improved behavior.

33:06It'll learn to reason more carefully or to learn to double check itself or something in the background so that it then can generalize that behavior to any question that it's asked. But that's not necessarily the case. Is that right? Yeah, I mean, yeah, I guess with all these things, you can never know, like you can never guarantee, like this model will like see every possible thing, right? These models have like billions or trillions of parameters. So it's hard to guarantee, but the hope would be that you get more and more coverage, like the more scalable your automated critique is, the more possible problems you could cover.

33:51And hopefully, if you cover all possible problems, maybe it will be very difficult to elicit odd behavior for the models yeah um and on the on the uh improvement on the measured uh improvement um the um is there have you i mean how do you measure that so against the benchmarks like uh before rlai f uh you know you get a 60 score score by, and that's another question, how do you score, I guess, on a benchmark? Yeah, so you have a 60 % accuracy. And then after RLEIF, you've moved it up to 80%. I mean, what are the numbers that you're working with in terms of the improvements that you're seeing with these strategies?