1. Podcasts
  2. Best AI papers explained
  3. e3: Learning to Explore Enables Extrapolation of Test-Time Compute for LLMs
Best AI papers explained

17 Jun 2025 · 14m · 8 chapters

Episode artwork

e3: Learning to Explore Enables Extrapolation of Test-Time Compute for LLMs

In short

E3 (“Exploration Enables Extrapolation”) teaches LLMs to use more test-time compute by learning in-context exploration—chaining generation, verification, and refinement to improve beyond the training thinking budget.

Guests

No guest names or backgrounds are provided in the transcript; it’s a two-person discussion without identifiable guest bios.

Key claims

Extrapolation depends on internal asymmetries, especially the verification-generation (VG) gap; negative gradients (from mistakes) repurpose probability mass to encourage further exploration; a coupled curriculum is required (easy+8k tokens, then medium/hard+16k) to avoid instability and enable scaling.

Notable examples

C-D-O-W-N game shows natural VG gap and positive scaling with more budget; MULT vs MULTV (fine-tuned verification) induces VG gap; masking negative gradients (“GRPO mask”) causes performance to plateau. Results: best 1.7B model improves up to 40k tokens and beats larger 7B/32B models on AME25/HMMT25.

Written by AI. May contain mistakes.

Chapters

0:45

Introduction to E3 Concept

Understanding the E3 method and its significance in LLMs.

“It's essentially a method to teach LLMs how to manage their thinking time, their test time budget, by intelligently linking different operations together.”
1:24

Asymmetries and the VG Gap

Explaining the concept of asymmetries in LLMs and the VG gap.

“Simply put, it's just differences in how good a model is at various little steps within its reasoning process.”
2:24

Extrapolation through Verification

How the VG gap facilitates extrapolation in LLMs.

“It's like doing a little search inside its own context window.”
3:17

Role of Negative Gradients

The impact of negative gradients on LLM training and exploration.

“It starts chaining those steps, verification and generation, and improves with more compute.”
4:50

The Coupled Curriculum Approach

Understanding how structured training enhances LLM performance.

“The potential is there, but the amplifier, the thing that really encourages active exploration, seems to come from using negative gradients.”
7:37

E3 Recipe and Performance Results

Details on the E3 training recipe and impressive results achieved.

“Because just having the tools and motivation doesn't always guarantee success, right?”
11:30

Beyond Traditional Methods

How E3 surpasses traditional training methods in reasoning.

“That's what, two and a half times the longest budget it saw during training?”
12:20

Future of LLMs and Problem Solving

The transformative potential of LLMs in complex problem solving.

“They compared it to budget forcing, which is a prompting technique trying to make models just write longer chains of thought.”
A man walking through an airport, asking his phone a question

Ask about this episode

Get the gist, explore an idea or ask about something said.

Claude

On Best AI papers explained, what are the key points of this episode?

Searching your listening

Transcript

Automatic transcript. May contain errors.

0:00You know those times when you're facing a really complex problem and you just wish you had like an extra time button for your brain. Yeah, definitely. Well, what if large language models, LLMs, could actually do that? Hit a button, get more thinking time, and genuinely get better at solving the problem as they go. That's pretty much the dream for LLM reasoning, isn't it? Getting them to effectively use more compute time right when they need it at test time. Right, this test time scaling idea. Exactly. And the really big prize there is extrapolation, getting the model to keep improving even past the amount of thinking time it saw during training.

0:36That's been tough. Until now, maybe, because today we're diving into this really fascinating new recipe called E3, exploration enables extrapolation. Yeah, E3. It's essentially a method to teach LLMs how to manage their thinking time, their test time budget, by intelligently linking different operations together. Things like generating, then verifying, then maybe refine. Okay, so our mission for this deep dive, let's unpack the, what is it, three key ingredients that make E3 work. We want to explore how they actually change the way and LLM reasons. And why it's getting these, frankly, groundbreaking results, even with models that aren't the absolute biggest ones out there.

1:18Right. So what could this mean for AI reasoning down the line? Let's get into it. OK, so to really get E3, we first need to talk about something called asymmetries in the LLMs. Asymmetries. Yeah. Simply put, it's just differences in how good a model is at various little steps within its reasoning process. You know, the whole chain of thought it produces, the output trace. Gotcha. Like it might be good at step A, but not so great at step B. Precisely. And the prime example, the one that's really key for E3 is the verification generation gap, the VG gap. OK, VG gap. Now, this sounds a bit weird, doesn't it?

1:51The idea that an LLM is often much better at checking if an answer is right than it is at coming up with the right answer from scratch. It does sound counterintuitive, but it's a really crucial observation. Why is that gap so important then? Because these asymmetries, especially the VG gap, they basically act like an internal feedback system for the model. Ah. When that gap exists, the model can actually start chaining these steps together. It generates something, then it uses its better verification skill to check it that maybe generates again based on that check. It's like doing a little search inside its own context window.

2:28Okay. That makes sense. Almost like talking itself through the problem. And the source material actually shows this happening, right? Like with that countdown game. Exactly. C-D-O-W-N. That game naturally has this VG gap. And models trained on it, they just keep getting better, generating longer, more complex reasoning the more test budget you give them. They're using the asymmetry. Okay, but what about tasks where that gap isn't naturally present? Like a standard multiplication. Great point. So on a normal multiplication task, MULT, a base model that isn't good at verifying. It doesn't get better with more time.

3:03It hits a wall. But, and this is the cool part, if you fine-tune that same model specifically to get better at verification, they call this MULTV suddenly, it develops the VG gap. And then, boom, you see the same positive scaling again. It starts chaining those steps, verification and generation, and improves with more compute. So it's not just about inherent task structure. You can actually induce this useful asymmetry. Looks like it. And this ability, this chaining of asymmetric skills enabled by the VG gap, that seems to be what allows for real extrapolation. Meaning it can handle problems requiring way more thinking time than it ever saw in training.

3:41Precisely. Models with the gap keep climbing. Models without it just plateau. Okay. And there was something interesting about KL divergence, too. Yeah, this is quite insightful. When the VG asymmetry is there, training with more budget actually reduces the KL divergence from the base LLM. Remind us what KL divergence means here. Sure. It's basically a measure of how much the model's new way of thinking, its output distribution, differs from its original programming, the base model. So lower KL means it's sticking closer to its roots, even while thinking longer. Exactly. It suggests that even as it's generating longer, more complex chains of thought, it's not just like going off the rails.

4:21It's building on its core knowledge in a stable way, like adding floors to a building with a solid foundation, not just tacking on weird extensions. Okay, so asymmetries, particularly the VG gap, provide the potential, the sort of internal mechanism for self-improvement and exploration. Yep, the foundation. But is that foundation enough? Or does it need, I don't know, a push, something to really make it use that potential? That's the perfect lead in. The potential is there, but the amplifier, the thing that really encourages active exploration, seems to come from using negative gradients. Negative gradients.

4:58Okay, so we're talking reinforcement learning. RL now. Usually RL focuses on reinforcing the good stuff, right? The positive gradients from correct answers. That's the common approach, yes. Reinforcing correct traces. Yeah. But E3 puts a lot of emphasis on the negative gradients. Which come from the mistakes, the failed attempts. Exactly. The gradients that arise from incorrect or incomplete reasoning traces. So how does focusing on the bad help? Is it just punishment? It's more sophisticated than just punishment. Unlike, say, supervised fine-tuning, which is all positive, negative gradients actively push down the probability of those incorrect paths, especially the ones that just stop too early.

5:37Okay, suppresses the bad stuff. But here's the clever bit. the probability saved by suppressing the bad paths isn't just thrown away, it gets repurposed. Repurposed how? It's used to increase the probability of the model trying out other actions, specifically chaining more of those asymmetric skills we talked about, like adding another verification step or trying a different generation approach. Ah, so instead of just stopping after a mistake, the negative gradient encourages it to think, wait, that didn't work. What else could I try? Maybe I should double check. Precisely. It learns to value the act of exploring, of verifying, of refining, not just the final correct answer.

6:13And the results show this. Clearly. The source material shows that using negative gradients leads to longer responses overall. But crucially, it leads to more chained asymmetries, like more verification steps, particularly on the incorrect traces where it's trying to recover. So it tries harder when it messes up? Yeah, and it also leads to more diverse responses. Higher entropy, both within a single attempt and across multiple tries. It's exploring more options. Okay, so you combine the potential from asymmetries with the push from negative gradients. And you get what they call meta-exploration or in-context exploration.

6:50The LLM isn't just following a script. It's actively learning how to search for a solution within its own thought process. And that learn-searching skill is what lets it effectively use extra compute time at inference. That's the idea. It allows it to tackle harder problems and extrapolate better. And they showed this really clearly if you mask the negative gradients using a technique called GRPO mask. Performance just flatlines. Performance, response length, verification attempts, they all plateau. It really highlights how vital those negative gradients are for driving this learned exploration.

7:24It's teaching the model how to think deeper, not just what to think. Okay, this is making sense. Asymmetries provide the tools. Negative gradients provide the motivation to use them and explore. But you mentioned three ingredients. What's missing? Because just having the tools and motivation doesn't always guarantee success, right? Just giving it more training budget doesn't automatically work. Exactly right. Negative gradients are powerful, but they aren't the whole story. Just cranking up the training budget arbitrarily doesn't lead to the best extrapolation. Structure matters. Structure in the training.

7:56How so? Oh. Well, they found a couple of problems with, let's call it, unstructured training. If you only train the model with very short thinking budgets, say 4 ,000 tokens. It never learns to think long. Pretty much. It kind of kills the exploration tendency. So when you need it to think longer later at test time, it performs poorly. It hasn't learned that skill. Okay, so train longer then, like 16 ,000 tokens. You'd think so. But training only at very long budgets creates its own headaches. major optimization challenges, high gradient variance in the RL process. Basically, the training becomes unstable and less effective.

8:32Hmm. Tricky. And it wasn't just about the budget length, right? Didn't the mix of problem difficulty also matter in a weird way? Yeah, this was really counterintuitive. Training on a fixed mix, easy, medium, hard problems seems logical for getting good average performance on problems like the training ones. But not for extrapolation, for totally new, harder problems. Surprisingly, no. They found that for out-of-distribution extrapolation, training on only easy problems initially could actually lead to better results later on harder stuff. That is wild. Why would that be? It's complex, but it seems related to letting the model first solidify its exploration strategy on simpler problems before scaling it up.

9:12Which brings us to the third ingredient, the coupled curriculum. Coupled curriculum. Okay, sounds deliberate. It is. It's a specific strategy that carefully varies both the problem difficulty and the training token budget together. So what's the actual E3 recipe here? The E3 approach fine-tunes the base model first on easy problems, using an intermediate training budget, 8 ,000 tokens in their work. Okay, easy problems, 8K budget, then what? Then it continues training but shifts to medium and hard problems, and simultaneously increases the training budget to a larger one, like 16 ,000 tokens. Ah, I see.

9:49So it's a two-stage process linking difficulty and budget. Why does that specific combo work so well? It seems to hit a sweet spot. That initial 8K budget is long enough for the negative gradients to really encourage that chaining of asymmetries we talked about to get the exploration going. But not so long, it causes those training instability problems. Exactly. And then the second stage, moving to harder problems with a longer 16K budget, lets the model take the exploration skills it just learned and apply and scale them to more complex situations. So it learns the ropes on easy stuff, then proves it can handle the tough stuff with more time.

10:23That's the idea. And this specific coupled approach significantly outperformed just changing the data difficulty or just changing the budget alone. The coupling was key. Okay, so let's recap the full E3 picture. You need the LLM to have these internal asymmetries, like the VG gap. Check. Ingredient one. Then you use negative gradients during RL training to really push the model to use those asymmetries to explore, learn from mistakes, and try different paths. Check. Ingredient 2, the amplifier. And finally, you structure the whole training process using a coupled curriculum starting easier with a moderate budget, then getting harder with a longer budget to properly develop and scale that learned exploration skill.

11:04Check. Ingredient 3, the structure. Put them all together, and that's the E3 recipe. And the payoff. The results were pretty stunning, right? They really were. This recipe produced the best performing 1.7 billion parameter model. They tested on some notoriously difficult math benchmarks, AME25 and HMMT25. A 1.7 billion parameter model. That's relatively small these days. It is. And here's the really striking part. Its performance consistently improved as they gave it more thinking time at test, all the way up to 40 ,000 tokens. 40 ,000. That's what, two and a half times the longest budget it saw during training?

11:40Exactly. That serious extrapolation. It wasn't just getting slightly better. It was genuinely leveraging that extra time effectively, way beyond its training data. And you mentioned it outperformed bigger models. Yeah. Get this. This 1.7B model trained with E3 actually beats significantly larger open source models. We're talking 7B and even 32B parameter models on these math problems. Wow. Okay, so smarter training, this E3 recipe, can actually overcome a massive size disadvantage in some cases. It certainly demonstrates the efficiency and power of teaching the model how to reason and explore, not just feeding it more data or parameters.

12:19And it's not just some fancy prompting trick either. It's actually learn this behavior. That's another key point. They compared it to budget forcing, which is a prompting technique trying to make models just write longer chains of thought. E3 scaled much better. Meaning the model learned the exploration strategy internally rather than just being instructed externally. Precisely. The core takeaway is that E3 seems to provide this clear, effective mechanism for LLMs to genuinely learn in-context exploration, leading to these unprecedented extrapolation results and just more effective reasoning overall.

12:53all. Okay, so we've walked through how teaching LLMs to really explore leveraging those internal asymmetries, amplifying learning from errors with negative gradients, and using that smart coupled curriculum can lead to pretty amazing breakthroughs. It's a fundamental shift in how they tackle complex problems. And for you, listening to this, think about it. The very AI tools designed to help you get informed to understand complex topics are themselves learning to think deeper, to explore Or more thoroughly. Yeah, imagine an LLM that doesn't just give you the first answer, but can really dig in, connect disparate facts, verify its own reasoning, kind of like a superpowered research assistant.

13:31It does make you wonder, if LLMs can learn to self-explore and extrapolate their reasoning this effectively, what kinds of problems that seem impossible today might become tractable? That's the big question, isn't it? Yeah. Could this shift redefine problem solving, not just in, say, math or science, but across creativity, strategy, maybe even understanding human interaction? Definitely something to mull over as this technology continues its incredibly rapid evolution.

From the publisher

The provided text introduces "e3," a new training methodology for Large Language Models (LLMs) designed to improve their reasoning capabilities and enable extrapolation of test-time compute. This means LLMs can continue to enhance performance even when given more processing time than they were trained on. The core of e3 lies in three key components: leveraging asymmetries in LLM competence, where models are better at verifying answers than generating them; utilizing negative gradients in reinforcement learning to encourage exploration and chain these asymmetric operations; and employing a coupled curriculum that aligns task difficulty with training budget to structure this exploration effectively. Experiments demonstrate that e3 significantly boosts performance on complex mathematical reasoning tasks like AIME and HMMT, outperforming other models within its size class and showing robust scaling with increased test-time compute.

keepSave to notecopy_alldocsAdd noteaudio_magic_eraserAudio OverviewflowchartMind Map

More from Best AI papers explained

All episodes