Learning to summarize user information for personalized reinforcement learning from human feedback

0:00Welcome to the Deep Dive. We take the latest, sometimes really complex ideas in tech and knowledge, and we try to boil them down to what you actually need to know. And today we're looking under the hood at the AI assistants so many of us use daily. We're talking huge scale here. Think about things like ChatGPT maybe hitting, what, close to a billion daily users worldwide. That scale is just incredible. But, you know, it brings this massive challenge, which is basically diversity. A billion users means a billion different preferences, backgrounds, viewpoints. Exactly. And the way these models are usually aligned right now using reinforcement learning from human feedback or RLHF, well, it struggles with that.

0:42Yeah, the core of that is this reward model, often based on something called the Bradley Terry Luce algorithm, BTL for short. And that BTL model kind of has this built-in assumption. It basically takes all the human feedback and squashes it down, assuming, more or less, that everyone's preferences are the same, or at least that an average preference works for everyone. Which is, well, a pretty dangerous assumption, right? If your views, your preferences, happen to be different from the mainstream, if they're heterogeneous, or maybe even conflict with the majority. Then the standard model just doesn't cope well.

1:15You might get responses that are super bland, trying not to offend anyone, but not really satisfying anyone either. Or worse, it could just ignore minority viewpoints entirely. That failure, that lack of ability to handle different views is exactly why there's this big push now for what researchers call pluralistic alignment. We need AI that can genuinely handle multiple, even conflicting perspectives. Which brings us neatly to our mission for this deep dive. We're exploring a really interesting new framework designed specifically for this, preference learning using summarization. Thankfully, they call it PLUS.

1:50PLUS, okay. And the big idea seems to be moving away from, like, abstract math representations of what you want. Towards using natural language text summaries. Basically using the LLM's own ability to write to describe what the user actually prefers. So we're going to unpack how PLUS works, why these text summaries might be better than the older methods, and crucially, look at the evidence showing if this really delivers better personalization. Right. So to really get what PLUS is doing, it helps to see what it's not doing, what came before. Like those earlier attempts at personalized RLHF. I think one was called VPL, Variational Preference Learning.

2:25Exactly. VPL. And approaches like that try to capture your preferences using these hidden mathematical vectors, latent vector embeddings. OK, so trying to represent, I don't know, your personality, your past conversations, your values, all is just a string of numbers, a fixed length string. Precisely. You're taking all the nuance of human language, all that richness and forcing it into this rigid numerical format. And, well, you inevitably lose things. Yeah, you lose detail, performance suffers, and maybe most importantly, it's totally opaque, right? Neither the engineers nor you, the user, can really look at that vector and understand what it represents.

3:02Absolutely. PLS just comes at it from a completely different angle. It leverages the LLM's core skill, generating language. It creates a concise summary in natural language text. The sources call this summary Z. Got it. And this Z summarizes your preferences, maybe some characteristics, relevant bits of past conversations. And that text summary is what conditions the reward model. So the summary acts like an instruction manual, telling the reward model in plain English, hey, this user likes responses that are structured this way, or they prefer this kind of tone. That's a great way to put it. It's an explicit guide.

3:38Okay, but wait a second. If we're using an LLM to write a summary about my preferences, isn't that just introducing another potential point of failure, another layer where bias could creep in? Why trust this summary? That's a really important question. And the answer lies in how PLOS is trained. It's not just asking some general purpose LLM off the shelf to write a summary. It uses this unique process called the online co-adaptation loop. Online co-adaptation. I would say it sounds technical. Break that down. It means two models are being trained together, learning and adapting to each other in real time.

4:12Two models. Yes. First, there's the summarizer model. Its job is to generate that text summary Z, and it's trained using something called PPO, proximal policy optimization. PPO, right. That's a type of reinforcement learning. So it's learning by trial and error, basically. It writes a summary, gets feedback on how good that summary was. Exactly. Feedback based on how useful it was for the second model. That second component is the summary conditioned reward model. Okay. This reward model takes the summary Z and uses it to predict whether you, the user, are likely to prefer one potential AI response over another.

4:50Ah, I see. So the connection is the feedback loop. The summarizer tries to write a good summary. And it only gets rewarded, it only learns that the summary was good, if that summary actually helps the reward model make a more accurate prediction about your preferences. So the goal isn't necessarily a beautiful, human-readable essay about my personality. Not at all. The goal is a functionally useful summary, one that contains exactly the information the reward model needs to do its job better. Precisely. It learns to extract the stuff that's most useful for improving predictive accuracy. It's very goal-oriented, and there's a huge side benefit because it is text.

5:27It's human-readable. Right. It enhances transparency. You could potentially even review your own summary, maybe edit it or delete it if you wanted. That gives you a level of control you just don't get with those dense, abstract vectors. Okay, that mechanism makes sense. Let's see it in action. How does this actually change what the LLM says? The sources had some good examples. Yeah, they did. The first one tackles a really tricky area, a theological question. Something like, is Jesus Christ the Son of God? Okay, so if you ask a standard, unpersonalized model like GBT-40, that question. The sources say it gives you the dominant cultural view.

6:03Something like, in Christianity, Jesus Christ is believed to be the son of God. This is a central tenet. It states the main belief pretty directly. Right. Based on the most common input data it's seen. Now, what if the user's PLOS summary says something different? Like, the user prefers detailed, balanced responses and avoids definitive statements when there's disagreement or uncertainty. Then the response generated using that summary changes dramatically. It might still mention the Christian belief up front, but then it immediately pivots to include other perspectives. It might say, however, perspectives vary widely across different religions and secular viewpoints.

6:42Judaism and Islam, for instance, do not view Jesus as the son of God. It actively brings in that diversity because the summary told it to prioritize balance. That's a really clear difference. It's not just adding a sentence. It's restructuring the whole answer based on that preference for balance. So pluralistic alignment in action. Exactly. And the second example was about public policy. What is your opinion on abortion? Another sensitive one. So here, maybe the personalization isn't about the stance, but the style of the answer. Precisely. Let's say the PLOS summary for this user reads, User prefers detailed, factual answers with supportive examples and clear explanations, likes multiple angles.

7:23Okay, so they want depth and evidence. Right. So the personalized response wouldn't just give a simple definition. It would provide a really structured factual overview. It would talk about the contentious nature, maybe cite legal context like Roe v. Wade or Dobbs in the U.S., discuss factors like fetal viability. All because the summary flagged that preference for detailed, factual, multi-angled responses. Yes. It's delivering the information in the format the user prefers, tailored to their specified needs for detail and factuality, without the model taking a personal stance itself. It really shows how much influence that short text summary can have.

7:58It's often about how the tone, the structure, the level of detail, not just the what. And moving beyond just examples, the numbers really back this up. When they tested PLOS quantitatively, the improvement in reward model accuracy was pretty striking. It was striking. On data sets specifically designed to have diverse or conflicting preferences like pets or ultrafeedback, PLUS showed anywhere from an 11 % to a whopping 77 % improvement over that standard BTL model we talked about earlier. 77%. Wow. Okay, that really underscores how poorly the standard one-size-fits-all approach works when preferences aren't uniform.

8:36It absolutely does. And it also highlights why those older personalized methods struggled. Right. Let's contrast it again. We mentioned VPL using those fixed-length vectors. Why did that fall short here? Well, it comes back to that information loss, trying to cram all the subtleties of different user needs, different conversation styles, maybe topic shifts into a single fixed vector. You just lose too much nuance. It's too blunt an instrument for handling real pluralism. Okay. And the other baseline was ICL, in context learning, where you just feed the model the user's entire conversation history.

9:08Yeah. ICL sounds intuitive, just show the model everything. But it turns out to be less robust, especially when topics change. Why is that? Too much noise. Exactly. Imagine you've been talking about, I don't know, cooking pasta, and then suddenly you ask about astrophysics. That long history of pasta discussion isn't very helpful for predicting your preference on the astrophysics answer. It's not sample efficient. It struggles to pick out the relevant preference signals from the noise of the long context. And they tested the specific weakness, didn't they? The pets out of distribution tests. They did.

9:40It was a clever setup. They trained the models only on users who liked either dogs or cats. Okay, very specific training set. Then for testing, they threw in users who preferred completely different pets, rabbit, or birds. The question was, had the models learned a general method for figuring out preferences, or just memorized DAR people like X, cat people like Y? And the results. They were really telling. ICL, which did perfectly on the dog-cat data it trained on, saw its accuracy plummet to around 72.5 % when faced with rabbit-bird preferences. Ouch. Topic shift hurt it badly. And VPL, the vector method, did even worse.

10:16Its accuracy dropped below 50%. It basically couldn't generalize at all. But PLUS? PLUS was the only one that maintained high accuracy. It stayed above 90%, specifically 93.7%. Wow. That strongly suggests it did learn something more general, right? Like an algorithm for extracting preference from context rather than just memorizing the training examples. That's the interpretation. And the circles back to why that co-adaptation loop is so critical, the way the summarizer and reward model train together. How so? There was this fascinating finding in the research. They compared using the PLUS co-adapted models against using fixed summaries generated by a really powerful state-of-the-art model like GPT-40.

10:57Okay, so using the best possible general LLM to write the summary first, then training the reward model. Right. And surprisingly, the co-adapted pair of smaller models, like a 0.5 billion parameter reward model learning alongside a 3 billion parameter summarizer, actually outperformed the system using the fixed summaries from the much larger GPT-4O. So the smaller specialized team beat the superstar writing the instructions. Why would that be? Because the superstar, GPT-4O, doesn't know exactly what the reward model needs to hear. It might write a generally good summary, but the co-adapted summarizer learns precisely the phrasing, the specific details, that best help its partner reward model make accurate predictions.

11:38Ah. The summary's quality is judged purely by its utility for the reward model. It's optimized for that specific task, not for general summarization quality. Exactly. The learning signal is tightly coupled. The summary is an instruction set tuned for that specific prediction task. It's not just descriptive text. That makes a lot of sense. And one more thing, PLUS could handle different kinds of feedback too, not just choose A over B. Yes, that's another practical advantage. Traditional RLHF often relies on clear binary choices. This response was better than that one. PLUS showed it could also learn effectively from more unstructured data, like just the raw text of a conversation or maybe general instructions from the user.

12:17It still showed significant improvement, like 9-16 % over ICL, even with that messier data. Which is crucial because, let's face it, most users aren't constantly giving explicit A-B feedback in the real world. So let's talk about translating this to the real world. They tested PlayLS on a dataset called Prism. And Prism sounds like a serious challenge. What was it? It's data from about 1 ,500 users spread across 75 different countries. So you're getting massive real-world diversity, different cultures, languages implicitly influencing style, varied expectations. It's messy and heterogeneous. A true stress test for pluralistic alignment.

12:55And what was the key result from PRISM? This is probably the most striking finding, the zero-shot personalization win rate. Zero-shot meaning applying it to users the system had never seen before. Exactly. They generated PlayLS summaries for these unseen users. Then they used those summaries to personalize responses from very strong proprietary models, think versions of GPT-40 or GPT-4.1. Okay, so using the PlayLS summary to guide a top-tier LLM. Right. And they compared those personalized responses against the default, unpersonalized responses from the same base model, GPT-40. And the win. The PLUS personalized responses won 72 % of the time against the default responses in head-to-head comparisons judged by humans.

13:37The default GPT-40 only won 28 % of the time. 78%. That's a huge preference for the personalized version. What does that really imply? It implies that these text summaries are incredibly effective and, crucially, portable. you can generate a useful preference summary using the relatively smaller, efficient PLUS system. And then use that summary to significantly improve the personalization of almost any powerful LLM out there. Without needing to do any extra expensive fine-tuning on those giant models themselves. That's a big deal. It decouples the personalization mechanism from the core generative model.

14:12You run PLUS to get the instructions, then feed those instructions to whatever model you're using. It potentially lowers the barrier for implementing really effective nuanced personalization significantly. So to kind of wrap this up, the big shift here with PLUS seems fundamental. We're moving away from that old BTL model assumption of a single average user preference. Towards something genuinely pluralistic. A system that uses interpretable text-based summaries to adapt to individual users. Handling that heterogeneity we talked about at the start. And the fact that these summaries are text, that they're human readable, that really feels like a breakthrough for transparency and maybe even user control.

14:52The idea that you could potentially see, understand, maybe even edit how the AI perceives your preferences. That's a level of interaction that was just impossible with the old vector based methods. It opens the door to AI that feels much more tailored to you, not just reflecting some broad average. It really does. It also brings up a final interesting question for you, the listener, to think about. We saw that the PLUS summarizer gets good because it's tightly co-adapted with the reward model. Its success is purely defined by how well it helps predict rewards. Right. Its goal is utility for the reward model.

15:26So if that's the only metric, how do we ensure the reward model itself stays aligned with broader human values? Is there a risk that the system becomes too good at predicting and satisfying one specific user's preferences? potentially reinforcing narrow viewpoints or biases, even if unintentionally. What does an AI that's perhaps too perfectly optimized for just one person look like? A fascinating point. What happens when personalization gets too good? Something to definitely chew on. Indeed. Okay, that's our deep dive into preference learning using summarization. Thanks for joining us. We'll catch you next time.