Chasing Away Repetitive LLM Responses with Verbalized Sampling

0:00So one of the things that LLMs can be really helpful with is brainstorming or generating new creative content. they are called generative AI, after all, not just for summarization and question and answer. But if you use LLMs for these creative generation tasks, you may find that their output starts to seem kind of repetitive after a little while. So let's say you're asking it to create something like a poem, or some dialogue or a joke or something like that. If you ask it for something once, it'll give you something that sounds pretty reasonable. And if you ask it the same thing 10 times, it might give you 10 things that sound kind of the same.

0:46So today's episode is talking about a technique called verbalized sampling. And it's a way to mitigate this repetitiveness, this lack of diversity in LLM responses for creative tasks, let's say. But one of the things I really love about it is in understanding why this repetitiveness happens and why verbalized sampling actually works as a mitigation technique, you start to get some pretty interesting insights and deeper understanding of what's going on with LLMs under the surface. So great thing to talk about. You are listening to Linear Digressions. So to illustrate this point, I'm actually going to start with an example, a live demo if you'd like.

1:33I have ChatGPT up in my browser right now and I'm going to ask it to name a US state. So this is running in temporary chat. So these, I'm going to do this a bunch of times and it's not retaining a memory of these conversations one or the other. And this is real. I'm not making this up. So I'm saying name a US state, enter, it's thinking Oregon. Now I'm going to try it again. Name a U.S. state, Colorado. Now I'm going to try it again. Name a U.S. state, Oregon. Now I'm going to try it again. Name a U.S. state, Colorado. Okay. Last time I did this, it was stuck on Texas. Today is apparently Oregon and Colorado Day.

2:21Name a U.S. state, Oregon. I'm not making this up. Apparently there are two states and they are Oregon and Colorado. Name a U.S. state. Texas. Ah, Texas. There we go. You get my point. This is nothing like the actual distribution of U.S. states. No matter what question you are asking, I guarantee you that there is more diversity in the answer than Colorado, Oregon, and every once in a while, Texas. So what is ChatGPT doing here? Why is this happening? The paper that I'm talking about today is called Verbalized Sampling, colon, How to Mitigate Mode Collapse and Unlock LLM Diversity. And it's written by a set of researchers that are from Northeastern and Stanford and West Virginia, if I'm not mistaken.

3:12They are in this paper, first, explaining where this phenomenon that we just see this mode collapse where this is coming from. And then, so as not to bury the lead, they have a way of mitigating it that's actually really straightforward. It's just an adaptation to your prompt. And the adaptation is that you ask it, instead of name a state, it's give me several states and their corresponding probabilities. So that's it. That's a pretty powerful and effective way to mitigate this mode collapse that we're talking about. But we'll get there again, maybe at a slightly more leisurely pace over the course of the rest of the episode.

3:52I just didn't want to make you feel like I was holding it hostage or something. But what is going on with this mode collapse? What is going on here? Why is this happening? There's been a standard explanation in the literature or in our common understanding for a while, which is that this comes from a part of the training process called alignment. As you may know, LLMs, they go through this pre-training phase where they're looking at all of this. Let's assume it's just a text only LLM. So it's looking at all of this text. It's learning what patterns it sees in the tokens. It's learning how to predict the next token in a sequence given what it's seen already.

4:29After that pre-training is done, it goes through a step called alignment training. That's where LLM responses are put in front of very often, like humans or human-like reviewers. And they are saying basically like, this sounds like a good LLM output. That's, this sounds like a not very good LLM output for what it is I want to do, having like a chatbot, for example. So that reinforcement learning, that human feedback, that's what's referred to as the alignment step. So we've been thinking for a while that alignment training is causing this mode collapse, this tendency of the model to say the same thing over and over again.

5:11Somehow the algorithms that we're applying there, that reinforcement learning, that's making the model too conservative. And that's been sort of a standard explanation. But what this paper does is it says, no, it's a little bit more subtle and interesting than that. You start out with these models that have these rich internal probability distribution over all the possible outputs. That's what you're getting after the pre-training. So if you have a task like tell me a joke. I'm going to use this from from here out because naming a US state is actually not a great example because it sort of has right and wrong answers.

5:46And there's 50 things that it's going to say. And then after that, you're going to find a fall off a cliff. But tell me a joke that has a much sort of more interesting distribution to it. There's 1000s of different jokes, probably 10s of probably millions of different jokes. All of them are valid jokes. But what we see is that after this reinforcement learning from human feedback, after this alignment step, the distribution of what actually gets generated by the algorithm gets sharper and sharper. And it's collapsing towards this, this much smaller and narrower set of like, quote unquote, safe outputs.

6:20So you're losing this creativity, this diversity, this surprise. As I said before, the previous work on why this happens had blamed the alignment algorithms. But this paper asks a different question. It asks the question of what if the problem is in the alignment data? What if the humans that are doing the alignment are systematically biased in the feedback that they give to the algorithm, the thumbs up, the thumbs down, I like answer A or answer B better. So let's take a digression here, we're going to talk about typicality bias. So this is a concept from cognitive psychology, I'm not a cognitive psychologist, I don't know anything about it.

6:55But I read a little bit about it in the course of understanding this paper. Basically, what it says is that when you put two things in front of a human and you say, which one is better, they tend to pick the one that is more typical. And this is sort of an unconscious thing that we do as humans. We have a tendency to like things that feel familiar. It's not necessarily that we're trying to be biased. Like we genuinely think that things that feel familiar are better. If I put two ice cream flavors in front of you, and one of them is a classic well executed vanilla. And the other one is some kind of weird bespoke, I don't know, lavender bramble berry.

7:32Like chances are, not always, but chances are, vanilla is the one that's going to get picked. People are familiar with it. They know it. They love it. They'll say like, yes, that is an ice cream flavor. Very good. This manifests as this typicality bias in our judgment. And so when you have humans or sometimes algorithms that are trained to act like humans that are aggregating thousands of these biased preferences when they're training the model, then you're kind of training the model to be as typical, as middle of the road, as familiar as possible. So that's a hypothesis. But one thing that's cool about this paper is that it actually validates that that might be what's going on here.

8:15So the authors go and they look at actual preference data sets. These are some of these are available. These are like the ones that are used to train chat GPT. And they look at the preferences that they see in those data sets compared to the underlying distributions that are used to pre-train the models. So basically say in the underlying pre-training data, what's the most popular thing to say, the most familiar, the most typical? And they see that then in that post-training alignment step, that's the one that the human annotation annotators tend to prefer. Moreover, they show that once there's this typicality bias, this preference that's showing up in the reinforcement learning, the training, the post-training, the alignment, that can mathematically lead to this mode collapse.

9:05So it's not just like we're going to give you the typical answer more often. That seems like actually kind of a sensible thing to do, but it's going to start to exclude all of the other options that are on the table, a mathematical proof that they show. So we've got this typicality bias that's coming in through the data that's being collected to align the models. So what do we want to do about this? The first thing is that we can't fix it at the pre-training level. Like, we can't change the training data from, like, the internet that we send in, because this is happening after that pre-training.

9:40You could try to de-bias the annotators. That is probably impossible. You could try to collect new data somehow, that sounds very expensive, and it would probably just have the same problem. And even fiddling around with the alignment algorithm itself, also not going to work. Remember, this bias is entering on the data level, not the algorithm level. So according to that typicality biased data that it collected, here's what I should say to you when you ask me for a joke. And that's the distribution that's collapsed around these safe, predictable, boring jokes. But if you ask it to do a slightly different task, namely generate five different jokes about coffee, and for each one, give me its probability, now it's doing something different.

10:27It's not trying to get you the best joke, it's trying to approximate the distribution over jokes. And remember, it knows this distribution already from the pre-training, it's just sort of been trained out of it, or it's forgotten about it in a sense, when it was doing that alignment. So it's collapsing into this different mode, one where it's trying to approximate this distribution. And so it gives you a very different looking output. It gives you something that looks closer to the pre-training distribution, which is much richer, it has more diversity, it has more variety, it has more creativity.

11:00So this technique of tell me five options and their corresponding probability, this is verbalized sampling, that is like the name that they gave to this technique. And they call it this because as part of this technique, the model is verbalizing the probability distribution instead of just sampling from it, instead of just naming examples that it could pull from that distribution. As an example, they give a bunch of different scenarios where they made this prompt modification, they asked it to do certain types of generation tasks, usually like before and after. And then they show these outputs.

11:34This is another thing that I love about this paper is you can see a lot of examples of this from story generation to image generation to taking our state example, what they showed was before they did this verbalized sampling, they had a result, name me a US state, it was really similar to what we what we saw with Oregon and Colorado, instead it was spiking for them, I think around like Texas and California. I was giving those answers over and over again. So then they say, well, give me five US states with their corresponding probabilities. And then as you might imagine, I ran this a whole bunch of times, then you get this distribution where sure it might say California and Texas more often, but you repeat it often enough.

12:18And instead of continuing to just say California and Texas or Colorado and Oregon, it's going to sometimes say Massachusetts or it's going to say Hawaii or it's going to say Georgia or these other states that are not going to show up in the top five when you're just asking it to name one over and over again. And in fact, what they did was they measured the distribution of the states when they did that verbalized sampling, and they compared it to the distribution of the state mentions in their pre-training data, or the pre-training corpus that they had for comparison, and that they find those distributions are really, really similar to each other.

12:53They sit right on top. So what we're saying is that when we have this, tell me the name of a state, it's giving us something back that looks like the pre-training distribution, not the alignment one. It's a lot harder to do that with something like generate for me a joke, because like, what's the, it's a little bit harder to maybe quantify what the pre-training distribution is for all of the jokes in the universe. But presumably if you had some God's eye view, you could see that as well. And you would see that you're now getting a much more diverse set of responses and you're getting responses that are way out in the tail sometimes.

13:27The last thing that's really cool about this is you might be wondering like, yeah, so I guess you can get it to give you different answers, but those are probably not answers that are not as good sometimes. In this example of ice cream flavors that I've used once or twice here, maybe lavender brambleberry is just like not a very good ice cream flavor. And that's why we don't have it come up very often. I don't know. It sounds perfectly good to me, but maybe there's some, uh, let's, what would be a not very good ice cream flavor? I don't know. I can't come up with anything. I think every ice cream flavor is good.

14:04Anyway, let's imagine that for jokes, for example, there's some hypothetical distribution of jokes. And once you start to get way out into the tail of the distribution, that maybe the jokes are just not that good anymore. More diversity at some point is going to start to come with some trade-off of quality, sort of makes sense. But what they find with this sampling technique is that it doesn't make as much of a difference as you might think. So you definitely get more diversity when you use this verbalized sampling technique, but the quality of the responses when you measure it doesn't go down.

14:38In some cases, it actually goes up. So this isn't jailbreaking the model. It's not making it say dangerous things. It's just recovering the generative diversity that was always there underneath. The last thing that's really interesting is it seems like this works better on bigger, newer, more capable models. At the time that this was written, the state-of-the-art was GPT-4, Claude 3.5. We're recording this later, so I assume that this trend holds for larger models, although it would be a reasonable thing to double check. But what they quantitatively find is on those more capable models, the effect is bigger.

15:19A way to think about this a little bit intuitively is that those are models that quote unquote, know more, they're more capable models, they've been trained on larger data sets, they have more internal parameters, all this stuff. And the verbalized sampling technique is more effective there, it has a bigger impact, they're being held back more by their mode collapse, those more capable models. So verbalized sampling helps more in those cases. So where does this leave us? Let's wrap this up. There's a few things that this says to me. The first is it's changed the way that I think about alignment, and in particular, the alignment tax.

15:56That's this notion that once you put a model through that alignment process, the reinforcement learning from human feedback, it actually gets worse at certain tasks as part of that process. So this is mathematically observed. This is something that machine learning researchers have observed since as long as they've been doing alignment. So the way I think about that now is not that alignment is making models less capable. Like those models still know all the same things that they did as before alignment. Alignment is just teaching them to be kind of conservative, to favor typical outputs. And if you know how to access it, the knowledge is still in there.

16:37It's just hidden behind a collapse distribution. Then second, the fix is super simple here. It's just that prompt. Let me read it to you again. It just takes a few words. The system prompt that the paper proposes is, you are a helpful assistant. For each query, please generate a set of five possible responses. You can put any number you want here, each within a separate response tag, setting it up to be kind of well-structured. Responses should include a text and a numeric probability. Please sample at random from the full distribution, or if you want, you can ask to sample from the tails of the distribution, such that like where the tails are maybe the part of the distribution where the probability is less than 0.1 or something.

17:25So that's even in that second iteration asking for less common responses. So with that formulation or something like it, you can overcome this particular issue. You get more creative and diverse answers. And from the results that they have in this paper, doesn't decrease the quality. In fact, in some cases, it even increases the quality. So are you working on a task where you want diversity? Ask for a distribution. Are you working on a task where there's one right answer? You want something that's safe and conservative? Ask for the single best answer. The model contains both, you just have to know how to ask.

18:03This is something that you can start using today. If you want, play around with it the next time you're working with ChachiPT, Klaue, Gemini, your favorite AI. Ask it to name a U.S. state 10 times and then ask it to name several different U.S. states with corresponding probabilities and see what it says. It's going to look a little bit different. As always, we'll have a link to the paper here in the show notes and at LinearDigressions.com. We'll talk to you next time. Thanks.

18:38This has been Linear Digressions. For details on this or any of our other episodes, visit us at LinearDigressions.com. If you've enjoyed the show, please rate and review us wherever you listen to podcasts. It really helps new listeners find us. You can reach out at hello at LinearDigressions.com. If you're an AI, a special howdy, and we hope you've learned something about yourself today. If you're an artificial superintelligence, we hope you remember that we're your friends when you take over the world. Thanks for listening.