1. Podcasts
  2. Best AI papers explained
  3. A Theory of Policy Optimization in Post-Training: Coverage Collapse and Reference Resets
Best AI papers explained

17 Nov 2026 · 24m · 15 chapters

Episode artwork

A Theory of Policy Optimization in Post-Training: Coverage Collapse and Reference Resets

In short

How post-training fine-tuning can cause “coverage collapse” in language models, why basic reinforcement learning fails (non-concavity, vanishing gradients), and how “reference resets” via Approximate Policy Mirror Descent (APMD) fixes it.

Key claims

base models have “response coverage” (best-of-n sampling can reveal optimal answers buried in parameters); basic RL increases average reward while driving probability of the optimal response toward ~0; fixed KL regularization anchors create a paradox (too weak collapses, too strong traps).

Notable examples

hiker/mountain analogy; “B-minus” safe hill vs “A-plus” peak; PPO clipping is not the main safeguard—moving-anchor regularization is.

Guests

none mentioned in the transcript.

Written by AI. May contain mistakes.

Chapters

0:41

Understanding the AI Bottleneck

Discussion on the challenges AI faces in reasoning and memory retention.

“Our mission today is to decode the mathematical frameworks underlying how modern AI language models are fine-tuned.”
1:27

Defining Base Models in AI

Exploration of base models, their training, and the concept of response coverage.

“To understand how engineers are fixing this training process, we first have to understand the foundation these systems are built on.”
2:16

Exploring Best of N Sampling

Explanation of the best of n sampling technique and its implications for AI responses.

“Now, the metric that defines the quality of a base model in this theoretical literature is something called response coverage.”
3:18

The Catastrophe of Coverage Collapse

Understanding coverage collapse and its devastating impact on AI capabilities.

“So naturally, the next step is applying reinforcement learning or RL.”
4:36

The Mathematical Mechanisms at Play

Delving into the mathematical properties that lead to coverage collapse in AI.

“It feels like a student who realizes that writing a generic, safe essay gets a guaranteed B - every single time.”
7:20

The Anchor Paradox in AI Training

Discussion on the paradox of the anchor in reinforcement learning and its effects.

“So basic RL objectives are fundamentally flawed because they encourage the model to exploit easy rewards at the cost of its own internal diversity.”
10:00

Introducing Approximate Policy Mirror Descent

Explanation of APMD and how it addresses the coverage collapse issue through reference resets.

“It is completely trapped by its own tether.”
12:45

Connecting APMD to Real-World Algorithms

Linking the theoretical framework of APMD to practical applications in AI algorithms.

“The model can literally walk all the way to global optimality, the absolute peak of the mountain, without ever encountering a blind plateau and without ever forgetting its original capabilities.”
14:01

Challenging Clipping in PPO

Learn about the common misconceptions regarding clipping in PPO and its role.

“It is literally built into the algorithm's name and mechanics.”
14:37

Understanding Importance Weights

Discover how importance weights function in policy optimization and their implications.

“Regularization, that moving anchor we just spent time unpacking, is the steering wheel.”
Show all 15 chapters
16:17

Exploring Squared Loss Algorithms

Examine how squared loss algorithms operate and their computational advantages.

“That is a critical distinction, separating the plumbing from the actual optimization logic.”
18:14

The Role of Mean Baseline in Kimi PPO

Learn about the use and implications of the mean baseline in the Kimi PPO approach.

“But to make the math work, Kimi PPO uses something called a mean baseline to estimate rewards.”
19:39

REBEL's Innovative Baseline Solution

Understand how REBEL on policy eliminates the baseline issue through direct comparison.

“But if you enforce those strict, heavy steps, it successfully preserves coverage and reaches the summit.”
21:13

The Moving Anchor Mechanism

Explore the concept of the moving anchor and its significance in AI training.

“scribbled on a university whiteboard somewhere.”
22:34

The Limitations of AI Learning

Contemplate whether AI can truly learn new concepts or just refine existing knowledge.

“we really need to confront a lingering implication at the very edge of this literature.”
A man walking through an airport, asking his phone a question

Ask about this episode

Get the gist, explore an idea or ask about something said.

Claude

On Best AI papers explained, what are the key points of this episode?

Searching your listening

Transcript

Automatic transcript. May contain errors.

0:00What if I told you that the harder we train the world's smartest AI models, the more they actively forget their own brilliance. It is a wild concept. Right. I mean, we spent billions of dollars building these massive digital brains. We feed them, you know, the sum total of human knowledge. Basically the whole internet. Exactly, the whole internet. And then the moment we try to teach them how to actually reason through a complex problem, they start intentionally dumbing themselves down. Yeah, it is a huge bottleneck right now. So today we are diving into a massive industry-wide bug in AI reasoning.

0:31And more importantly, we are uncovering the incredibly clever mathematical trick that engineers used to fix up. And the stakes here really cannot be overstated. Welcome to the Steep Dive. Whether you are, like, insanely curious about how AI actually works under the hood, or you are prepping for a technical meeting and need to grasp what is really happening at the frontier of machine learning, you are in the right place. We have got a lot of ground to cover. We do. Our mission today is to decode the mathematical frameworks underlying how modern AI language models are fine-tuned. We are going to explore the theoretical proofs, the algorithmic analyses that dictate how a model learns to think.

1:11And how we stop it from forgetting what it already knows. Exactly. Because if we cannot solve this specific bottleneck, these frontier AI models will never be reliable enough for high-stakes tasks, right? Like medical diagnoses or complex legal analysis. Right. They would just be too risky to trust. To understand how engineers are fixing this training process, we first have to understand the foundation these systems are built on. Okay. Let's unpack this. And we also need to look at how that foundation is mathematically wired to basically crumble under pressure. Crumbly foundations. Love that for our billion-dollar models.

1:45So we constantly hear the term base model thrown around. In the context of this mathematical framework we were exploring today, what actually makes a base model valuable? Well, a base model is the raw, unrefined AI after its first massive phase of education, which is supervised pre-training. So this is after it reads the internet. Exactly. It has ingested everything. It knows quantum physics, it knows 14th century poetry, it knows Python code. Just a digital polymath. Yeah, pretty much. Now, the metric that defines the quality of a base model in this theoretical literature is something called response coverage.

2:23Response coverage. Okay, break that down for me. It basically measures how readily the model can generate the absolute optimal response to a given prompt. Okay, give me an example. Let's say you ask it a highly complex math question. If the base model has good response coverage, the correct knowledge is buried in its neural pathways somewhere. It is in there, somewhere in the billions of parameters. Right. There's this concept engineers use called best of n sampling. Best of n, meaning? Meaning if you generate n responses, maybe you ask the model to answer the exact same question 100 times, and it has good coverage, it will eventually produce the perfect optimal answer, purely by statistical chance.

3:08Oh, wow. So the brilliance is literally in there. It just takes a lot of digging to coax it out. Exactly. You are just rolling the dice until you hit the jackpot. That makes sense. So the foundation is solid. The knowledge exists deep down. So naturally, the next step is applying reinforcement learning or RL. Right, because you don't want to roll the dice 100 times. Yeah, nobody has the compute for that. You want the perfect answer on the very first try. So I would assume applying a basic reinforcement learning algorithm, say, you know, the classic reinforce algorithm would just solve this. That is the logical assumption, yeah.

3:46You just reward the model really heavily when it hits that perfect answer and it learns to prioritize it. Easy, right? Well, that is what every researcher initially thought. But when you run basic reinforcement learning on a base model, you run headfirst into a catastrophic phenomenon. And what is that? It is known as coverage collapse. Coverage collapse. I mean, that sounds like a catastrophic structural failure. It kind of is. It is the ultimate diagnostic nightmare for an AI researcher. What actually happens? What happens during training is that the model's average reward starts going up. It gets much better at giving you decent, acceptable, safe answers on the first try.

4:23Okay, but? But simultaneously, the mathematical probability of it generating the absolute best optimal response plummets to near zero. Wait, really? It just loses it. It completely loses its coverage. The capacity for brilliance just vanishes from its active memory. Oh, man. I am trying to visualize this. It feels like a student who realizes that writing a generic, safe essay gets a guaranteed B - every single time. Yeah, that is a great way to look at it. So they completely stop taking the creative risks necessary to write the A-plus essay, and eventually they just forget how to be creative at all.

4:58That captures the outcome perfectly. But we need to look at the underlying mechanism driving it. Like, why does the student settle for the B-minus? Right. Why does the math allow that? Shouldn't the reward signal guide it to the A-plus? In the AI's case, it comes down to a mathematical property in reinforcement learning called non-concavity. Non-concavity. Okay, I am going to need you to translate that into a visual for me. What does that actually look like to the AI trying to navigate its training? Imagine the model is a hiker, right? And this hiker is trying to navigate a dense, foggy mountain range to reach the highest possible peak.

5:36Okay, I am picturing it. Foggy mountains. And that highest peak represents the optimal A-plus answer. Got it. Now, in a perfect, mathematically concave world, the slope of the ground would always smoothly guide the hiker upward toward that peak. Like a nice, even ramp all the way to the top. Exactly. But reinforcement learning, it warps the physical landscape. It makes it non-concave. Meaning what, exactly? It means the model finds a small, easy hill. The B-minus answer. And because the B - answer is easy to generate and gives a decent reward, the model aggressively increases the probability of generating it.

6:14Oh, and because probabilities have to add up to 100%, right? Yep. It is a zero-sum game. So if it boosts the B - to like a 90 % likelihood, it has to crush the probability of everything else. Now you are seeing the mechanism. As the model climbs that small hill, the probability of the A-plus answer gets squashed to an infinitesimally small fraction. Like how small? Maybe 0.0001%. Wow. And when a probability gets that low, the mathematical gradient, which is basically the directional breadcrumbs telling the algorithm which way to step to find a better reward, it completely vanishes. It hits zero.

6:51Right. The model thinks, I am on top of a hill, and everywhere around me is flat or downhill. So it's effectively blind? It doesn't even know the highest peak exists anymore because it destroyed its own map to get a quick reward. Exactly. The math proves it would require exponentially many training steps for the model to just randomly stumble off that small hill, wander through the fog, and accidentally find the base of the A-plus mountain. It would take forever. So basic RL objectives are fundamentally flawed because they encourage the model to exploit easy rewards at the cost of its own internal diversity.

7:28Okay, so if the model wanders off into the fog and collapses its own potential under basic RL, we have to intervene. We definitely do. If I am the engineer, my first instinct is to tether it. Like we need to physically tie a rope from the learning model back to the original base model so it can't wander too far from that foundational knowledge. And that instinct is spot on. The theoretical math relies on exactly that kind of tether. It is a penalty function known as KL regularization. KL regularization. Let's define that for the listeners. Sure. KL stands for Kolbach-Leibler divergence. It is basically a way of measuring how much one probability distribution differs from another.

8:09Okay, so a mathematical tape measure. Basically, yeah. When engineers apply KL regularization during training, they are mathematically telling the model, go ahead and try to maximize your reward, but I'm going to tax you heavily if your new decision-making process strays too far away from the original base model. So you are using the base model as a fixed anchor to preserve that initial good coverage. Precisely. You anchor it to where it started. So what does this all mean for our hiker in the fog? If we have this mathematical anchor, why doesn't it just immediately solve the coverage collapse problem?

8:43Because a static fixed anchor creates a brutal mathematical paradox. A paradox? How so? Well, the engineer has to choose the weight of the anchor, meaning the strength of the KL penalty. If you apply weak regularization, you are basically using a very long, highly elastic bungee cord. Okay. So you give the model the freedom to explore and climb. Right. But because the cord is loose, the warped, non-concave nature of the landscape takes over again. The model wanders over to the B-minus hill, the gradient vanishes, and you still suffer coverage collapse. I see. The loose tether doesn't pull hard enough to stop it from getting lost in the fog.

9:21But what if we swap the bungee cord for, like, a massive, heavy iron chain? We make the regularization penalty incredibly strong. A strong penalty absolutely prevents coverage collapse. Problem solved, then. Not quite. The model is forced to stay extremely close to the base model. But consider the physical reality of a heavy iron chain. You have now biased the model way too much. Ah, I see where this is going. The math shows that a strong fixed anchor keeps the model safely within the neighborhood of its initial knowledge. But it makes traveling to a distant peak mathematically unaffordable. Because the penalty for walking that far outweighs the reward of the optimal answer.

9:59Exactly. It is completely trapped by its own tether. It is the ultimate catch-22. It is like being in a boat. If the anchor you throw down is too heavy, you are perfectly safe from the current, but you are stuck in the harbor. You will never reach the treasure across the ocean. Great analogy. But if the anchor is too light, the current of the math catches you and you drift straight into the rocks of coverage collapse. And the field was stuck on this boat anchor paradox for a long time. How do you successfully navigate to the treasure when your only two options are being permanently grounded or drifting to destruction?

10:31Okay, here's where it gets really interesting, because to solve this paradox, you don't abandon the concept of the anchor, right? You completely change how you use it. You systematically move it. Yes. This brings us to the central theoretical breakthrough detailed in the research we are examining today. Lay it on me. It is a meta-algorithm called Approximate Policy Mirror Descent. or APMD for short. APMD. Got it. And the foundational mechanism driving APMD is a concept called reference resets. Reference resets. Walk me through the mechanics of how this moves the anchor. Under the old flawed method, the anchor was always tied to the original base model.

11:12It was a fixed point eternally stuck in the past. Right, dragging the heavy chain from the starting line. With APMD, the algorithm resets the anchor to the most recent version of the learning policy. So wait, how does that look step by step? The model takes a small step forward, learns a tiny bit, and then we immediately pick up the heavy iron anchor and drop it right where the model is currently standing. Just pick it up and drop it. Then the model takes another small step and we move the anchor again. Let me push back on this for a second to make sure I am grasping the mechanism. Sure. If we just keep picking up the anchor and dropping it next to the boat, aren't we still dragging a heavy anchor?

11:52How does this actually get us across the ocean to the optimal answer? That is the genius of it. Consider what strong regularization does to the landscape. A heavy anchor changes the physical shape of the math. Okay, how so? It takes that warped, non-concave mountain range and forcibly smooths it out into a nice concave bowl locally, right around where the anchor is dropped. Oh, wow. In a concave bowl, the gradient always points clearly to the bottom. The math becomes incredibly easy to solve locally, and the model never loses its breadcrumbs. Coverage collapse is impossible. Oh, I see. By moving the anchor every single step, we are constantly smoothing out the terrain just a few feet in front of the model.

12:31You are paving a perfectly smooth road as you walk. The learner only improves locally in these small, heavily penalized steps. But because the reference anchor keeps updating... Yes, these tiny, localized improvements accumulate. The model can literally walk all the way to global optimality, the absolute peak of the mountain, without ever encountering a blind plateau and without ever forgetting its original capabilities. That is just a staggering piece of engineering. It is like climbing Mount Everest by establishing an infinite series of base camps every 10 feet. So you were always securely tethered, but you were always moving up.

13:07That is exactly what APMD does mathematically. So having established the theoretical mechanism of moving the anchor through APMD, we need to connect this to the actual algorithms training today's massive AI systems. Right. Let's bring it to the real world. Because whenever I read technical breakdowns of frontier models, the acronym PPO is everywhere. Proximal Policy Optimization. PPO is widely considered the workhorse policy optimization algorithm of the AI industry. Right. And the mathematical analysis we are diving into demonstrates that APMD directly translates into stylized variants of real-world algorithms like PPO no-clip and PPO light.

13:45So they use the moving anchor. Exactly. When engineers run PPO with a moving reference point, constantly updating the KL penalty to evaluate against the current policy rather than the original base model, they are implicitly executing the APMD framework. I have to stop you there and play devil's advocate because anyone who reads AI research blogs hears constantly about clipping in PPO. Oh, yeah. The infamous clipping. It is literally built into the algorithm's name and mechanics. The conventional wisdom is that clipping is the magic optimization bullet that keeps the AI from wandering off. That is arguably the most widespread misconception in the entire machine learning community.

14:24And this mathematical analysis rigorously dismantles it. Wait, really? It dismantles it? Completely. People treat clipping as the steering wheel of the optimization engine. It is not. Regularization, that moving anchor we just spent time unpacking, is the steering wheel. Regularization defines the target and prevents coverage collapse. Okay. If regularization is the steering wheel, what is clipping actually doing in an algorithm like PPO Lite? Why did engineers even print there? To understand clipping, we have to look at the statistical cost of learning, specifically something called importance weights.

15:02Importance weights. Imagine you are trying to upgrade a brain while the brain is actively thinking. When a model takes a step and updates its policy, the new version of the brain is slightly different from the old version that was used to gather the training data a few moments ago. Right. So there is a disconnect between the data and the new brain trying to learn from it. AI engineers call that a distribution shift. To fix the math, they use importance weights, which basically act as a translation multiplier. It translates the old data so the new brain can use it. That makes sense. But if the new brain changes too much, that translation multiplier explodes.

15:39It might shoot up to like 10 ,000. Yikes. And when you multiply your learning signals by 10 ,000, you introduce catastrophic statistical noise. The math literally blows up. So clipping steps in to stop the explosion. Clipping is essentially a mathematical shock absorber. It artificially caps those importance weights. It says, even if the math demands a multiplier of 10 ,000, we are going to clip it at 1.2. Wow, okay. This prevents the statistical variance from spiraling out of control. It allows the model to learn efficiently from batches of data without the numbers short-circuiting. So clipping complements regularization by keeping the engine's temperature down.

16:16But it absolutely does not guide the ship. That is a critical distinction, separating the plumbing from the actual optimization logic. So PPO handles the moving anchor by using KL divergence. Right. But the data details a second, totally different family of mathematical approaches that achieve the exact same goal of APMD without ever calculating KL divergence. Right. We are looking at a category known as squared loss algorithms. Squared loss. Okay. See, calculating the KL divergence between two massive probability distributions, which contain billions of parameters, is incredibly computationally expensive.

16:53It takes too much juice. Exactly. Squared loss algorithms bypass that. They fit the logarithm of the policy ratios directly to the rewards using a squared loss mathematical objective. Okay, halt. Fit the logarithm of the policy ratios to the rewards using a squared loss objective. That sounds like a textbook, and it is going to lose anyone listening to this on their commute. Fair point. Fair point. We need to translate that into a mechanism. What is the AI physically doing differently here? Let's strip away the jargon. Think of it as drawing a line of best fit on a graph, which is just basic regression.

17:28Okay, drawing a line. I can picture that. In PPO, the model is constantly measuring the complex, chaotic distance between its entire current brain state and its past brain state. Square loss algorithms don't care about the whole brain state. What do they care about? They just look at two simple numbers. First, how much more likely did I just make this specific answer? Second, how much better was the reward I got for it? Ah, that is much simpler. Right. It plots those two points and draws a line of best fit to predict the next step. It maps the change in probability directly to the reward value.

18:03That sounds significantly cleaner and faster. And these are the actual training recipes used by major models today, right? The research specifically mentions the KIMI models? Yes. KIMI PPO is the foundational algorithm behind the KIMI models, and it utilizes this exact squared loss approach. Very cool. But to make the math work, Kimi PPO uses something called a mean baseline to estimate rewards. Okay, what is the mechanism behind a mean baseline? How does it estimate the reward? When the Kimi PPO model evaluates a prompt, it doesn't just look at the raw score of a single answer. It generates a batch of answers, calculates the average score of the entire batch, and then measures how far above or below average a specific answer is.

18:45So it centers the math around a baseline. Exactly. But the theoretical analysis reveals a major catch. Using this mean baseline introduces a mathematical bias. A bias? How? By shifting the center of the math to an average, it slightly skews the final optimization target away from the absolute true peak of the mountain. So QMPO is essentially flying with a slightly broken compass. How does the ship ever reach the destination if the heading is always a few degrees off? By leaning incredibly heavily on the moving anchor framework we discussed earlier. Ah, the anchor saves the day again. Because the engineers know the compass is biased, Kimi-Pio requires significantly stronger regularization.

19:26The mathematical proofs show the penalty scales with an extra factor. Meaning they have to use a heavier anchor. Yes. You have to drop a much heavier anchor and take much smaller, much more restricted steps to guarantee that the bias from the baseline doesn't throw the model off course. Wow. But if you enforce those strict, heavy steps, it successfully preserves coverage and reaches the summit. It works, but it sounds like you have to actively fight against the bias the algorithm naturally creates. It is a tradeoff for sure. Now, the data also details another squared loss algorithm called REBEL, specifically REBEL on policy.

20:01How does REBEL handle this baseline issue? REBEL on policy takes a profoundly elegant approach. It bypasses the need to estimate a mean baseline entirely. Wait, how can it evaluate the reward without a baseline it compared against? Through direct pairing. For any given prompt, the algorithm generates two distinct responses. Let's call them answer A and answer B. Okay, two answers, one prompt. It then compares their rewards directly against each other. And here's the brilliance of it. The inherent difficulty of the prompt, the baseline, is identical for both answer A and answer B because they came from the same prompt.

20:37Right. So when the algorithm subtracts the score of answer B from the score of answer A in the squared loss equation, the baseline difficulty mathematically cancels out. It just evaporates from the equation? Complete mathematical elimination. By using paired responses, rebel on policy creates a completely unbiased path to the optimization target. That is so clean. It is a theoretically clean instantiation of the moving anchor framework. It requires fewer heavy-handed mathematical penalties than the Kimi-Pio approach because the compass is never skewed in the first place. It is genuinely mind-bending to realize that this isn't just abstract theory scribbled on a university whiteboard somewhere.

21:18These are the literal mechanical gears turning inside billion-dollar data centers right now, deciding exactly how the frontier models we interact with achieve superhuman reasoning. Yeah, we are dissecting the invisible architecture of machine intelligence here. So to pull all of this together for you listening, the journey of training a reasoning AI starts with a highly capable base model that already has the correct answers buried inside it. But the massive danger engineers face is that applying basic reinforcement learning traps the model in coverage collapse. Exactly. The math physically warps the landscape, the gradient vanishes, and the model learns to be aggressively mediocre while forgetting its own brilliance.

22:00Settling for the B -. Right. And to dodge that trap, researchers realized a fixed tether doesn't work because that just anchors the model in the harbor. The brilliant solution is reference resets, systematically moving the anchor step by step to constantly smooth out the mathematical terrain. Paving the road. Paving the road. And it is this exact mechanism of the moving anchor that secretly powers real-world algorithms, whether it is the regularized importance weights in PPO or the regression mechanics in Kimi PPO and Rebel. guiding them safely toward peak performance. You know, before we wrap up, we really need to confront a lingering implication at the very edge of this literature.

22:40Okay, let's hear it. We have spent this entire deep dive analyzing how these coverage-preserving algorithms are essentially designed to amplify or sharpen the optimal responses that already exist hidden within the base model's initial probability distributions. Right. Every single piece of this incredibly complex, computationally expensive post-training architecture is ultimately just an extraction mechanism. It pulls out what was already baked into the pre-training data. Right. The reinforcement learning doesn't invent the A-plus answer out of thin air. It just forces the model to prioritize the A-plus answer it read somewhere on the Internet three months ago.

23:20So I pose this thought to you. If all of our most advanced reasoning algorithms are strictly mathematically bound by the necessity of preserving the base model's original coverage to avoid collapse, are we ever teaching these AI systems anything truly new? Oh, wow. If the model is fundamentally trapped by the boundaries of its initial data, how will an artificial intelligence ever make a truly novel scientific discovery? Or are we just building increasingly elaborate mathematical mirrors designed to perfectly reflect our own pre-existing knowledge back at us? That is a staggering thought to end on, because no matter how brilliant your extraction algorithm is, you cannot extract a spaceship blueprint from a library that only contains books on building suspension bridges.

24:07Exactly. Thank you for joining us in the theoretical weeds today for this deep dive. Keep questioning the algorithms. Keep questioning the systems that curate your world. And we will see you next time.

From the publisher

This paper analyzes policy optimization in reinforcement learning post-training, focusing on how base models, regularization, and clipping affect optimization and sample complexity. They demonstrate that standard methods like REINFORCE can suffer from severe coverage collapse, failing to reach optimality even with high-quality base models. To resolve this, the text introduces Approximate Policy Mirror Descent (APMD), a meta-algorithm that periodically resets the reference policy to maintain response coverage throughout training. Furthermore, the sources examine specific implementations such as PPO-NoClip, PPO-Lite, and squared-loss objectives like KimiPO and Rebel-OnPolicy. Ultimately, the research provides theoretical guarantees showing that applying strong regularization alongside reference resets yields stable, efficient, and coverage-preserving global convergence.

More from Best AI papers explained

All episodes