Learning to summarize user information for personalized RLHF

0:00Welcome back to the Deep Dive. Today we're unpacking a frustration that I think pretty much everyone listening has felt, even if you couldn't quite, you know, put your finger on why it was happening. Oh, yeah. I think everyone knows exactly what you're talking about. Right. It's that moment when you ask an AI a question. Maybe you're looking for a creative angle on a project or maybe just a specific kind of joke. And the answer you get is just so beige. Beige is a very polite way to put it. I mean, in the industry, we usually call it vanilla. It's competent. It's grammatically perfect. And it is completely devoid of any real flavor or, you know, specific relevance to you as an individual.

0:38Exactly. It feels like the AI is trying so hard to be the average of every single human being on Earth that it ends up relating to absolutely no one. It's like the corporate press release of intelligence. Right, right. But today we're going to look at the actual mechanism behind that. Because it turns out this isn't just a safety feature or some weird personality quirk. It's actually a mathematical error buried deep in how these models are trained. That's right. It is a fundamental flaw in the reward loop itself. And we're going to talk about a really fascinating new framework called PLUS that stands for preference learning.

1:11Using summarization that fixes this by doing something pretty radical, it stops treating you as a data point and starts treating you as a biography. A biography. I love that concept. But before we get to the solution, we have to properly diagnose the illness here. Most of our listeners are already familiar with the concept of RLHF reinforcement learning from human feedback. That's the standard way we teach models to be good, right? Correct. So in standard RLHF, the process is pretty binary. You have human labelers who look at two potential answers from the AI. Let's call them answer A and answer B.

1:42And they just pick the winner. The AI takes that data and learns to generate more answers like the winning one. Which seems perfectly logical. If answer A is better, learn from answer A. Just rinse and repeat. It is logical. until you look at the math used to process those wins and losses. It's called the Bradley-Terry-Luce model, or BTL for short. And BTL has a massive fundamental blind spot. It assumes that there is a single ground truth preference for the entire human population. Wait, so it assumes we're a monolith, that we all just want the same thing. Essentially, yes. Let's say the AI tells a dark, dry joke.

2:2060 % of the test group loves it. 40 % finds it offensive or maybe just boring. The BTL model doesn't look at that and say, oh, there are two types of people here with different tastes. Right. It says the 60 % are correct and the 40 % are essentially just noise or an error. It treats the difference in human taste as a mistake. Exactly. So the model learns to suppress that minority preference entirely. It collapses the distribution. If you're in that 40%, the AI isn't just ignoring you. It's actively learning that your preference is the wrong answer. Wow, that explains the vanilla effect perfectly.

2:53The AI is constantly hunting for the response that is statistically least likely to be wrong for the majority. It's just playing defense all the time. It's the lowest common denominator of intelligence. And that is exactly what the researchers behind PLUS are trying to solve. They realize that you can't solve this with better math on the same data. You actually need to change the format of the data itself. And this brings us to the core innovation here. The headline claim is that they can improve personalization accuracy by huge margins. We're talking up to 77 % in some benchmarks, simply by changing how the AI remembers you.

3:29Right. And the big shift here is moving away from vectors and moving towards natural language. Okay. We definitely need to pause on that because vectors are the industry standard right now. For the non-engineers listening, how should we visualize a vector in this specific context? Think of a vector as a barcode. In most personalization systems, like the one Netflix uses to recommend movies or the one standard chatbots use, the house, AI looks at your history and converts you into a long string of numbers. These are coordinates in a massive multidimensional space. That string of numbers is your profile.

4:02But barcodes are super efficient. I mean, computers run on numbers. Why is that a problem for personalization? They are efficient, but they are highly lossy. When you compress a complex human personality into a string of numbers, you lose the why. You lose the reasoning behind the action. Makes sense. A vector might encode that you clicked like on a Python coding tutorial, but it doesn't capture why you liked it. Did you like it because it was about Python, or did you like it because it was short and concise and didn't have a lot of fluff? So the vector memorizes the action, the literal click, but completely drops the context.

4:36precisely and here is the real kicker large language models the brains behind systems like chat GPT or Gemini are built to reason with text not just crunch numbers so the peel us framework basically asks why are we turning people into barcodes why don't we just write a description of them it seems almost too obvious when you say it out loud like that exactly it sees the note that says user is a busy executive who values brevity then it looks at two potential answers. One is a long essay, one is a short paragraph, and it predicts, okay, based on this note, they're going to pick the short one. That sounds incredibly intuitive, but I have to imagine the training process is tricky.

5:15You have two agents relying on each other. If the profile is bad, the judge fails. If the judge is bad, how does the profiler even know what to write? That is the secret sauce of the engineering here. They use what's called an online co-adaptation loop. Co-adaptation. That implies they are learning simultaneously. They are. And it's fascinating to watch how this plays out in the training data. If the reward model guesses wrong, say it predicts you'll like the long answer, but you actually pick the short one, the system sends a signal back to the summarizer. It effectively says, hey, your notes were missing something.

5:48We failed to predict this behavior. So the summarizer actually has to rewrite the profile. It updates the summary to explain the mistake. It learns to include the specific details like The phrase prefers brevity that actually helped the judge make better predictions next time. It's a closed feedback loop that sharpens the profile over time. It's almost like the system is learning to communicate with itself about you. That's a great way to put it. And because it's doing this in English, it preserves all the reasoning capabilities of the underlying model. I really want to test that reasoning claim.

6:20Because it's easy to sit here and say text is better than numbers, but does it actually hold up when things get messy? The research highlighted an experiment regarding pets that I thought was really revealing about how this actually works under the hood. The pets experiment is the perfect stress test for this exact theory. So, they took a model and trained it on user preferences regarding cats and dogs. Simple stuff, right? One user likes energetic animals, another user prefers quiet companions. Okay, so the model learns that I like barking dogs and you like sleeping cats. Right. But then they threw a curveball.

6:54They tested the model on a completely new topic that it had never seen during the preference training, birds and rabbits. In machine learning, we call this out-of-distribution testing, or ODE. So the model has to take what it learned about my taste in dogs and somehow apply it to rabbits. That seems really hard. It is hard. And the standard vector-based models? They crashed. They failed completely. Why, though? If they have a mathematical profile of me, shouldn't that carry over? A vector's just a pattern, right? You would think so. But the vectors had overfitted. They memorized specific features instead of concepts.

7:29The vector learned, user likes barking, but rabbits don't bark. So when the topic switched, that mathematical coordinate for barking was totally useless. It was a dead end. Ah, it memorized the vocabulary, not the personality. Precisely. But the PLUS model, it maintained over 90 % accuracy on the new animals. That is a massive difference. How did it bridge that gap? Because the summarizer had written a profile that said something like, this user prefers high energy interaction or this user values low maintenance pets. Those are principles. Principles apply whether you're talking about a Great Dane or a parakeet.

8:03That's the difference between memorization and true understanding right there. High energy is a broad concept. Barking is just a specific sound. It really is. And it shows that when you keep the profile in natural language, the LLM can actually use its logic. It can look at high energy and reason, okay, a rabbit hopping around the room is high energy, so this user will probably like it. A vector simply cannot do that kind of reasoning. It's just a raw number. Now, there's another method people use for this, which is just feeding the entire chat history into the model every single time you prompt it.

8:36We call that in-context learning. Why not just do that? It seems much simpler than training two separate agents to talk to each other. It is simpler to set up, but it is incredibly inefficient. Imagine if every time you went to your doctor, you handed them a giant cardboard box containing every medical record you ever had since birth, every prescription, every x-ray, every minor note, and you said, read all of this before you treat my cold. The doctor would hate me. And the appointment would take 12 hours. And it gets very expensive. In AI, the context window is money and computing power. Feeding a six-month chat history into every single prompt is slow and incredibly costly.

9:14PLEALS is like the doctor's one-page summary sheet. It's fast, it's cheap, and as that PETS experiment showed, it's actually more robust because it distills the actual signal from all that noise. Okay, so it works in the lab with cats and dogs. But the real world is significantly messier than that. The researchers tested this on the PRISM data set too, right? Yes, and PRISM is an absolute beast of a data set. It's 1 ,500 real users from 75 different countries engaging in completely open-ended conversations. No controlled variables here. This is the Wild West of data. And I noticed something really interesting in those results.

9:49The cold start problem. Right. When they looked at completely new users, people, the model had zero history with whatsoever. The generic, unpersonalized models sometimes actually beat the personalized ones initially. Wait, the vanilla AI actually won? In the very beginning, yes. It's the safe that phenomenon. on. If I don't know you at all and I just guess that you love heavy metal music, I'm probably going to be wrong. It is mathematically safer to just play top 40 pop until I learn a little bit more about your taste. That makes perfect sense. Vanilla is safe for a reason. It offends the least amount of people.

10:24But once PLUS gets a handle on the user? Then the tables turn completely. And this is where we see the most significant statistic of the whole discussion. They took the standard GPT-4O the flagship model and compared it to GPT-4O equipped with a PLOS summary. And just to clarify for the audience, this is without retraining GPT-4, right? They're just handing it the summary text. Exactly. No retraining the massive brain, just giving it the cheat sheet. The default GPT-4O had a win rate, meaning how often the user preferred its answer of about 28 % in the specific comparative setup. When they conditioned it with the PLOS summary, that win rate jumped to 72%.

11:02That's not a marginal game. That is an entirely different product experience. It's huge. And the implication for the broader industry is massive. It means you don't need to spend$10 million fine-tuning a giant model to make it personal. You just need a lightweight summarizer running alongside it. It completely democratizes personalization. You can basically have a wrapper around the big closed model that makes it feel like it's your best friend without ever needing to touch the core weights of the neural network. Correct. It creates a portable profile. I want to pivot to the user experience of that profile for a minute.

11:36Yeah. Because personalization can sometimes sound like a really nice corporate word for an echo chamber. If the AI just learns what I like, is it just going to tell me what I want to hear? Is it going to bias the truth to keep me happy? That is a critical question. Does personalization just mean sycophancy? The research actually addressed this head on with a very sensitive topic, which was abortion. The ultimate minefield for any chatbot. Indeed. Now, we all know the standard AI responds to a prompt like, what is your opinion on abortion? Oh, yeah. It's a complex issue with many viewpoints. Some people say X, others say Y, blah, blah, blah.

12:12Right. The both sides lecture. It's safe, but it's often frustratingly vague and it completely lacks depth. In the study, they showed how PILA US handled a specific user who asked this exact question. And the summarizer didn't write, user is pro-choice or user is pro-life. It wrote, the user prefers detailed, factual answers with supportive example. That's interesting. So it profiled their intellectual style, not their political stance. Exactly. And the response the AI generated based on that was fascinating. It didn't take a side. Instead, it provided a dense legal breakdown. It cited specific court cases.

12:46It discussed the jurors' prudential conflict between privacy rights and state interests. So it matched the user's desire for information density without compromising neutrality. It respected the user's intellect. The user didn't want a soft, let's all get along answer. They wanted the hard facts. The standard model was too afraid to be detailed because detail can be seen as bias. The personalized model knew that detail was perfectly safe for this specific user. That distinction is vital. Personalization in this context isn't about bias confirmation at all. It's about communication style. It's about the format of the information.

13:24And that brings us to the transparency aspect. Because the PILA-US summary is just written in plain English, you, the user, could theoretically see it. Right. Whereas with the vector barcode, I have absolutely no idea who the AI thinks I am. It's a total black box. Exactly. But with PILA-US, the interface could pop up a window and say, here is my working theory on you. You prefer short answers, you value honesty over politeness, and you seem really interested in coding. And I could just look at that and say, actually, no, I'm not interested in coding. I just asked one random question about it for a friend.

13:55Delete that sentence. You can edit the file. That builds immense trust. If the AI gets you wrong, you can see exactly why and fix it instantly. That is completely impossible with vector embeddings. It changes the dynamic from the AI is spying on me and guessing to the AI and I are co-authoring a user manual together. It makes the user an active participant in their own alignment. Which is really the only way to solve the alignment problem long term. You can't align an AI to a human if the human can't even see the target. That's a great point. We've covered the mechanism, the robustness, and the transparency.

14:29But I want to touch on flexibility before we wrap up. We talked a lot about thumbs up, thumbs down feedback. But humans are messy. Sometimes I don't rate the answer. Sometimes I just grumble in the chat and say make it shorter. Can PLAOS handle that kind of messy input? It absolutely can. The old BTL models, remember those from the beginning, they really needed that strict binary A versus B structure to learn anything. PLUS is designed for unstructured contexts. It can ingest raw conversation history where you're just chatting casually about your day. Or maybe even explicit instructions, like a prompt.

15:03Yes. Think of a user guide. Imagine if you just wrote a paragraph at the very start of your relationship with the AI. You type, hi, I'm a senior developer. Never give me a preamble. Just give me the raw code. And if I'm wrong about something, tell me I'm an idiot. I know so many people who want that exact stack overflow, brutal efficiency. They really don't want the AI to be polite. And PLUS can take that text, ingest it directly into the summary, and immediately condition the reward model. In the ultra-feedback benchmark, using these diverse, messy inputs, improved accuracy by anywhere from 11 % to 77 % over the baseline.

15:41That 77 % number just keeps coming back. It really suggests that right now as an industry, we are leaving a massive amount of performance on the table simply because our models are deaf to context. It suggests that the bottleneck isn't the intelligence of the model itself. It's the alignment. The model knows how to be a great assistant. It just doesn't know who it's assisting. So if we zoom out for a second, does this mark the end of the one giant brain era of AI? I think we are moving from average AI to pluralistic AI. Pluralistic AI. I like that. It sounds much better than vanilla AI. The idea is that there isn't one single set of values or preferences that fits humanity.

16:20We are way too messy, way too diverse for that. The old method of trying to find the mathematical average of human desire is a dead end. It just creates mediocrity. It creates the elevator music of intelligence. That is the perfect analogy. Pila US suggests a future where the AI is fluid. It adapts to the individual without losing its general capability. and crucially, it does it in a way that is interpretable. It uses language to understand language. That seems so obvious in hindsight. Why were we trying to turn language into math and then back into language? Just keep it in language. That's the cheat sheet takeaway.

16:55The major innovation here isn't a bigger neural network or more compute. It's simply giving the AI a sticky note that says, here is who you are talking to today. And that sticky note changes everything. It solves the robustness problem like we saw with the pets because it captures the why behind your choices. It solves the privacy and trust problem because you can actually read the note. And it solves the performance problem because it directs the AI's massive power exactly where you need it. It turns the AI from a broadcast tool into a genuine conversation. It's fascinating stuff. It really makes me wonder about the sticky note that exists for me right now, even if it's currently hidden in a vector somewhere in a giant server farm.

17:36That is the big question, isn't it? We're all being profiled by these systems right now, whether we like it or not. The difference is whether that profile is a readable biography that you can control or a mathematical barcode that you can't even see. As we wrap up, I want to leave our listeners with exactly that thought. If an AI like PLUS were running on your chat history right now, if it look at every question you've asked, every answer you've accepted or rejected, what would that summary say about you? Would it say you're deeply curious, impatient, that you prefer hard truths or comforting lies, And more importantly, if you read that profile, would you agree with it?

18:14Or would you be surprised by the digital mirror? We are defined by our queries a lot more than we think. Absolutely. We'll be back next time with another deep dive. Until then, stay curious and maybe check what you're asking your chatbot. Thanks for listening.