Inverse Reinforcement Learning Meets Large Language Model Post-Training: Basics, Advances, and Opportunities

0:28Welcome to the Deep Dive. it all, how do they keep getting better? What's the secret sauce? How does that continuous improvement actually happen? And that right there is the puzzle we're tackling today. This deep dive, it's all about reinforcement learning, RL, and specifically a really clever angle called inverse reinforcement learning, IRO. IRO, okay. Yeah, and these aren't just, you know, fancy acronyms. They're fundamental. They're how we align these huge LLMs, make them more reliable, more controllable, just frankly more capable. So our mission today is to really unpack this key paper, Inverse Reinforcement Learning Beats Large Language Model Post-Training.

1:07Basics, Advances, and Opportunities. It's by Hausson and Mihaela Van Der Schaar from Cambridge. We want to explore, you know, what makes these methods tick? Why are these things called neural reward models suddenly so vital? And what does it all mean practically? Think of it as maybe a shortcut to getting your head around the real cutting edge of LLM improvement. Exactly. You're fast track. So let's get into it. All right, let's maybe start with the basics. Why even combine LLMs and RL? I mean, LLMs alone are already pretty mind-blowing, right? They are. We've seen this massive success just by scaling things up more compute, more data, bigger models, and not just in language, but images, audio, video, even complex decision-making.

1:49Yeah, the progress is wild. And LLMs specifically, they stand out, partly because you interact with them using natural language, right? Makes them feel transparent. They're becoming these general purpose assistants, even getting kind of agentic, doing deep analysis. They understand instructions. They adapt. They're powerful tools. There's always a but. There's a but, yeah. For all that brilliance, they can't always self-correct. They don't automatically refine their own behavior if they make a mistake or generate something, you know, slightly off. There's no built-in, oops, let me fix that. Right.

2:19And that's where, I guess, the RL side comes in. We've seen RL do incredible things, superhuman performance in Atari, AlphaGo, chess, StarCraft. Even chip design, complex stuff. RL's core strength is that it can continually improve. It learns by interacting through trial and error. But RL has its own challenges, doesn't it? It can be a bit of a black box. That's exactly it. While RL can discover these really novel creative solutions, the systems themselves often lack transparency. It's genuinely hard sometimes to understand why an RL agent did what it did. How did it come up with that strategy?

2:54So putting them together, it's like they solve each other's problems. Kind of, yeah. It's a real synergy. For RL, LLMs offer this natural language interface. We can understand and maybe even get inspired by RL's creativity through the LLM. Okay. And for the LLM, RL provides that missing piece. The ability to keep getting better at a task as long as you can define better with some kind of reward signal. And that whole process, optimizing the pre-trained LLM for specific things like safety or helpfulness, that's what we mean by LLM alignment or post-training. Exactly. Those terms are pretty much interchangeable.

3:29It's about fine-tuning the LLM to gain specific capabilities, quantified by a reward. And we're already seeing this play out. Oh, definitely. Big time. Look at conversational AI. Reinforcement learning from human feedback, RLHF, that's become like the standard approach. Right, the preference thing. Yeah, you enhance the LLM just by having users say, I prefer response A over response B. It's super valuable where you can't easily define a perfect metric for, say, what makes a good conversation. OpenAI's feedback system is a great example. And it's not just chatbots being nicer. You mentioned math.

4:02Yeah, mathematical reasoning. RL is behind things like alpha proof, alpha geometry too. These systems got silver medals at the International Mathematical Olympiad. Wow. And Deep Seek R1, too. What's fascinating is they seem to learn this kind of deep thinking or self-reflection. They improve just by generating more tokens, exploring more reasoning steps out loud almost. Like thinking on paper. Pretty much. But despite all this progress, there are still some big hurdles. Okay, like what? Well, first, for many real-world tasks, you just don't have those clear rule-based reward signals. There's no obvious win condition like in a game.

4:41So figuring out how to model the reward, how to teach the LLM what good looks like is absolutely crucial. That's reward modeling. And it's hard. It's hard. Second, the compute costs are enormous. Training these things takes serious resources, which, you know, makes it tough for open source efforts. Right. Lem is who can play. Exactly. And finally, and this is really key, there's no single silver bullet RL algorithm. The best one totally depends on the specifics of the LLM alignment task. And that's kind of what this deep dive is about, bridging that gap, showing how inverse RL helps tackle these issues.

5:15Which brings us nicely to the technical foundation, the Markov decision process, the MDP. This is the formal framework for RL, right? Agent, environment, actions, states, try to get the most long-term reward. Right. You've got your state space, action space, how actions change, the state transitions, the reward function, where it starts, and that discount factor, gamma, balancing short versus long-term gain. Explore versus exploit. Find good stuff, then do it more. Exactly. But remember that point about no silver bullet? The algorithm you pick depends so much on the environment. State space size, action space size.

5:48Is the reward dense or sparse? Like, DQN works for Atari because you get points constantly. Dense reward. Right. But go. Huge state space. Reward only at the end that needed self-play, planning, totally different methods. Okay, so how does generating text with an LLM fit this MDP picture? It maps surprisingly well, actually, but with a big twist. The state is just the text generated so far. You know, the quick brown fox jumps over the Mac. The action is picking the next word, say lazy. The transition is just adding that word. It's deterministic. The initial state is the prompt you give it. And the discount factor.

6:23You said that affects conciseness. Yeah, exactly. If gamma is 1, a 300 ,000 word correct answer is theoretically as good as a 300 word 1. Less than 1, it prefers shorter, quicker rewards, so shorter answers. But here's the twist. The reward function. The missing piece. That's the crucial bit. Unlike a game, there's usually no external thing checking if the LM's response is good or correct. No automatic plus one point. Even for math problems, the reward for correct often isn't a simple rule. It has to be learned from data. Which is why the paper calls it MDPR. And MDP without the R, the reward function.

6:58Precisely. MDP minus R. So if you don't have that reward function built in, you have to learn from behavior, right? From data sets. Exactly. And there are two main reasons why this is necessary. First, sometimes specifying the reward is just really hard. Think about early autonomous driving. What is the exact formula for safe driving? Good point. Or helpful and harmless for an LLM. Very fuzzy. Super fuzzy. So instead of trying to write that impossible function, it's often more practical to just learn by mimicking good human behavior from examples. Makes sense. What's the second reason? It helps with exploration, especially when rewards are defined but really sparse.

7:38Like winning a game of Go, seeing human expert games gives the AI a starting point, shows it potentially good paths it might take ages to find on its own. AlphaGo did exactly this. Okay, so learning from behavior. And the two main ways are imitation learning, IL, and inverse reinforcement learning, IRL. Those are the big ones, yeah. Both try to make the learned policy behave like the expert, like the behavior data set. Is there a key difference? A couple. Most IL and IRL methods assume you can still interact with the environment, do rollouts, get new data. Offline versions don't. But the simplest IL is just behavior cloning, PC.

8:14Which is basically supervised learning. Pretty much. Given a state, predict the expert's action. Simple. But it has problems. Big problems. The main one is distributional shift. If your learned policy makes just one tiny mistake, it can end up in a state that wasn't in the training data. An out-of-distribution state. Right. And from there, since it doesn't know what to do, it's likely to make more mistakes. The errors compound, sometimes quadratically. It gets lost fast. Ouch. Yeah. That's why being able to interact with the environment to roll out the policy and get more expert data specifically for those states the policy actually visits is so important.

8:53Methods like Dagger do this. OK, so BC is simple but brittle. What about more advanced aisle? Things like jail, adversarial imitation learning. They use a discriminator network to tell the difference between expert behavior and the policy's behavior. The policy then learns to fool the discriminator. It's trying to match the overall distribution of behavior. Clever. But still, I.L. How is I.R.L. different? The core distinction. I.L. tries to directly copy the expert's actions. I.R.L. tries to figure out the reward function the expert was optimizing. It learns a reward model from the behavior and then uses RRL to train a policy that maximizes that learned reward.

9:31So IELA copies the what, RRL figures out the why, and then learns the what. That's a great way to put it. And a critical takeaway from the paper, accessing the environment dynamics, being able to interact, is really key for robust learning in both IELA and IRL. And crucially for IRL, those learned reward models, they're not unique. Different assumptions can lead to different valid reward models. Okay, so let's connect this back. LMs are already amazing language imitators, right? Yeah. Pre-training, supervised fine-tuning, SFT. That's basically next token prediction. It is behavior cloning on massive data sets.

10:05Absolutely. Figure 1 in the paper shows this nicely. Panel 1, the raw LLM, has a broad output distribution. Panels 2 and 3, prompting or SFT shifts that distribution towards better performance. And prompting techniques like chain of thought can be really effective. So if these imitation methods work, why the extra complexity of inverse RL and explicit reward models for alignment? Fantastic question. Because while prompting and SFT work to a degree, they have limits. Prompting can be expensive, you know, lots of tokens. It's often model-specific, query-specific. Yeah, prompt engineering is a whole field now.

10:42Right. And SFT requires really high-quality, perfect demonstrations, which are just hard and expensive to create. So why do we need reward models? What's the first big motivation? Practicality and scalability, especially for learning from human feedback and conversational AI. Think about interacting with a chatbot. Is it realistic to ask millions of users to write out the perfect response every time? No way. That's way too much effort. Exactly. But what are users good at? Choosing. They're much better at discriminative tasks looking at two responses and saying, this one's better. The aha moment.

11:15Yeah. Preference data is easier to get. Way easier and much more scalable. That's precisely why learning reward models from preference data is the core of large-scale RLHF pipelines. It gives you practical supervision without needing those perfect expensive demonstrations. So takeaway one, IRL makes data collection flexible and practical. Okay, practical data. What's motivation two? Generalizable reasoning skills, especially in complex domains like math. If you just fine-tune on a math dataset, SFT, the model might get good at the specific types of problems it saw. But struggles with new, complex reasoning patterns.

11:52Exactly. It's more like memorization than true understanding. This is where reward models really make a difference. They provide a flexible way to encourage generalization. Instead of just mimicking, the LLM uses the reward model as a guide to explore and discover good reasoning paths on its own. Like DeepSeek R1 you mentioned earlier. Perfect example. The reward model lets it do deep thinking, long chains of reasoning, self-correction stuff that's really hard to get just from imitation. So takeaway two, reward models empower generalization by letting RL discover powerful reasoning behaviors. And the third motivation, you said this is the coolest one.

12:27Test time optimization. So prompting and SFT are mostly offline things, right? The model is trained, then deployed. It doesn't really adapt its strategy during generation. But think about AlphaGo again. A huge part of its strength was search and planning at test time. It thought ahead during the game. Reward models let LLMs do something similar. Ow. That trained reward model, the golden vertical line in Figure 1, panels 4 to 6 can be used at inference time to evaluate different possible continuations or full responses and filter out the low-quality ones. Ah, so it can generate a few options and pick the best according to the reward model right there on the fly.

13:06Exactly. Especially in high stakes areas like math or following complex instructions, you can use it to improve performance dynamically. It's like planning. So takeaway three, reward models enable inference time optimization, letting LLMs adaptively choose better responses during deployment. Right. That makes a lot of sense. So we know why we need them. Let's get practical. How do we actually do IRL? How do we build these reward models from real world, often messy evidence? Yeah, the core idea of alignment is making the machine match our objectives, right? And we usually have to infer those objectives from signals like demonstrations, preferences, choices.

13:43And that real world data isn't clean. Never is. It's noisy, partial, biased. Humans aren't perfect optimizers either, but imperfect as it is, it's often the best signal we have. The trick is extracting the underlying intent. So starting with the most common approach, reward modeling from preference feedback, RLHF. RLHF. Right. Standard RLHF trains a reward model using that pairwise preference data. I like A better than B for query X. Often uses something like the Bradley Terry model, which basically learns to predict which response in a pair a human is more likely to prefer. And that learned reward model then guides the main LLMs training, often using PPO.

14:22Correct. PPO, proximal policy optimization, is very common there. But then came direct preference optimization, DPO. And DPO skips The explicit reward model training. It does. It's clever. It directly optimizes the LLM policy to satisfy the preference constraints, implicitly capturing the reward structure just through the relative probabilities it assigns to preferred versus dispreferred responses. So PPO versus DPO. Well, PPO with a really well-tuned reward model might edge out DPO in performance, but PPO can be notoriously unstable and hard to tune. DPO is generally simpler, more stable, often more robust to over-optimization, which we'll get to.

15:02So it's a trade-off. Stability versus potential peak performance. Pretty much. Depends on your task and resources. But fundamentally, both are IRL methods learning from preferences. And you mentioned Bradley Terry models. Anything more to unpack there? Just that modern methods often use BT regression on the LLM's internal representations, its embeddings. And an interesting insight. For some tasks, like picking the best out of N options, just getting the relative ranking right, order consistency is key. Simple classifiers can sometimes do this just as well or better than complex BT models, especially if the preference labels are noisy.

15:34Okay, so we need preference data. How do we get the best preference data efficiently? Active learning. Exactly. You don't want to waste human labeling effort on hairs that don't teach the model much. simple heuristics exist, like picking pairs the reward model is most uncertain about, or pairs where the predicted reward difference is largest. Makes sense. Explore uncertainty or exploit known differences? Right. But there are more principled ways using Fisher Information Theory. Basically, selecting pairs that provide the maximum information gain for refining the reward model's parameters. It's about choosing the data that will most efficiently improve the model.

16:11It really highlights that exploration-exploitation trade-off. What about when preferences aren't monolithic? People want different things. Personalization. Huge challenge. A single reward model struggles to capture diverse tastes. Solutions involve things like modeling user-specific latent variables, modeling reward distributions instead of single scores, or training to optimize for the worst-case outcome across different groups. Maxmin. And you mentioned decomposed reward models, DRMs. Yeah, this is a really cool work. DRMs break down complex preferences into interpretable sort of orthogonal components using TCA on the embeddings.

16:47You might get one component for general helpfulness, another for conciseness, another for a specific style. So you can see why preferences differ. Exactly. The first component often captures the majority view, but the others reveal these nuanced dimensions of preference. It helps understand the diversity instead of just averaging it out. Fascinating. Okay, let's switch context to mathematical reasoning. How has RL evolved there? It's been a journey. Started with prompt engineering, like chain of thought, just trying to unlock the reasoning already latent in the model. But that was model-dependent, black boxy.

17:22Right, and couldn't reliably fix errors. So the field moved towards search and planning, using things like Monte Carlo Tree Search, MCTS, guided by a learned reward or value function. Now the model was actively planning its reasoning path. More recently, you mentioned RLVR reinforcement learning with verifiable rewards. Yeah, systems like DeepSeek R1 use this. They rely on sparse but reliable signals, basically, is the final answer, right or wrong. And they show these emergent behaviors like self-reflection, backtracking, really impressive stuff. But you also had that surprising insight. Ah, yeah.

17:55The paper suggests that a lot of RLVR's success in math might actually come from the model learning effective response formats, like learning to output answers in a programmatic way or using very structured, long chains of thought. So it's not just exploring new math, it's learning how to present the math effectively. Kind of. It's almost like the RL is helping the model internalize good prompting or structuring techniques automatically. The format becomes part of the solution. Which loops back to prompting. There was that prompt RRL idea. Right. Automated prompt optimization is powerful but expensive.

18:30Lots of LLM calls. Prompt RRL is an IRL approach to make it cheaper. It learns from past prompt experiments, the trial and error history. To train a reward model that predicts prompt effectiveness. Exactly. A proxy verifier. Then, at inference time, you use this cheap reward model to pick the best prompt for a new query without needing more expensive LLM calls. It's learning from history to guide the future. Clever. Okay, what if you don't have preferences or verifiable answers, but you do have expert demonstrations, maybe for personalized tasks? That's where alignment from demonstration, or AFD, comes in.

19:07This is classic IRL territory learning reward models directly from expert examples. And this connects back to SFT and reward modeling. Yeah, the paper makes a nice theoretical connection. Both SFT, which is like behavior cloning, and reward modeling can be seen as special cases of AFD framed as trying to match the occupancy measure, how often states are visited, or minimizing certain divergences between the expert's behavior and the learned policy's behavior. So AFD is kind of a unifying view. In a way, yes. It shows that learning from demonstrations is a principled foundation, and methods like SFT and RLHF are different ways of leveraging that demonstration signal.

19:43It connects the dots. Okay, we have these reward models. How do we actually use them to make the LOM generate better stuff? Several ways. Broadly, you can use them during fine-tuning or estimate value functions or use them directly at inference time. Let's start simple. Best of n sampling. Simplest approach. Generate n candidate responses, score them all with your reward model, pick the highest scoring one. Done. Pros and cons. Pro don't. No extra LMM training needed. Con can be very slow and expensive at inference, especially if n is large or the responses are long. So more efficient ways. Iterative fine-tuning.

20:19Like RAFT. Right. RAFT basically takes the best responses identified by the reward model in a best event step and then fine-tunes the LLM on those winning responses. You repeat this. Eventually, the LLM learns to generate high reward responses directly. So you get the quality of best event without the inference cost. Exactly. You bake the quality into the model's parameters. What about direct policy optimization, like PPO? Still the workhorse for many. PPO uses the reward model, often alongside a learned value function, to estimate the advantage of taking certain actions, tokens, and updates the policy.

20:54But you mentioned the credit assignment problem. Yeah, especially with sparse rewards. If the whole math solution is marked correct at the end, which specific tokens or steps were responsible, it's hard to assign credit accurately. People try various tricks like reward shaping or distributing the reward back over the tokens. Are there alternatives to PPO that avoid this value estimation? Yes, Monte Carlo methods. Things like reinforce or more recent ones like GRPO, DPO. They work directly with the trajectory level rewards without trying to estimate intermediate values. They've shown really good results, especially in math and code generation, and can be simpler to implement.

21:32Okay, and the last category, reward guided decoding. Using the reward model during generation. Right. This modifies the sampling process at inference time. As the LLM generates token by token, the reward model provides feedback, And the decoding algorithm uses that feedback to steer towards higher reward sequences, essentially reweighting the probabilities of the next token. Examples. Methods like RA, ARGS, PAD for personalization, RSD for efficient decoding, lots of variations. The big challenge, again, is how well you can translate a potentially sparse trajectory-level reward signal into meaningful guidance at the token level in real time.

22:13So lots of options. PPO is common, but iterative fine-tuning and Monte Carlo methods look promising and maybe simpler. That seems to be a fair summary. The best choice really depends on your reward structure, task, and compute. There's no single winner yet. Okay. Brings us to the risks. Powerful tech always has downsides. Reward over optimization or reward hacking? Sounds like Goodhart's law. It's exactly Goodhart's law. When a measure becomes a target, it ceases to be a good measure. Reward models are learned from data. They're imperfect proxies for true goals. if you optimize too hard against that proxy.

22:46The model learns to exploit the proxy, not achieve the real goal. Precisely. It finds loopholes in the reward model. Figure 2 in the paper shows this clearly. The score according to the reward model keeps going up, solid line. But the score, according to some other perhaps better reference measure, dashed line, starts to drop. That gap is reward hacking. How do we fight that? Multiple ways. Using uncertainty estimates with the reward model. If the model is unsure, maybe don't trust its score as much. Ensembles help here. Regularizing the learning, adding auxiliary objectives, generative reward models, GRMs are another idea.

23:21And just careful analysis, like noticing if the model develops a length bias, just because users tend to prefer longer answers, even if they aren't better. That helps design better reward models or evaluation metrics up front. Makes sense. What about data issues? Offline versus online. Big challenge. We often train on data sets generated by older or different models off policy data. This distribution mismatch between the training data and the current policy being trained really hurts performance. So fresh data is better. Often, yes. Quality over quantity is key here. Studies show smaller, high-quality on-policy or nearly on-policy data sets can outperform huge, stale, off-policy ones.

24:02Which points back to active learning. Exactly. Active learning and online preference collection are vital for getting that high quality relevant data efficiently. There's also work on trying to adapt or reweight off policy data to make it more useful. So pulling it all together, what's the biggest remaining obstacle? Generalization, generalization, generalization. Getting these reward models to work well on new prompts, new types of responses, maybe even different base LLMs than they were trained with. That's the central challenge. And future opportunities. Definitely exploring richer feedback beyond just preferences, critiques, edits, comparisons on specific criteria, and developing methods that better bridge that gap between offline training, where most data lives now, and truly online continuous learning from interaction.