Do We Need to Verify Step by Step? Rethinking Process Supervision from a Theoretical Perspective

0:00Welcome to the Deep Dive. Today we're plunging into something really core to how AI learns to think. reward signals we see these incredible leaps right especially with large language models LLMs they're reasoning solving complex stuff but how how do they actually get good at it yeah and it really boils down to feedback yeah how they learn what's good and what's not exactly and for these big open-ended AI systems getting that feedback right is a huge challenge it's this thing called reward specification right it's not like say a chess program or winning is super clear. How do you tell an LLM if its actual thinking process was any good along the way?

0:41Which brings us neatly to the main event today. The two big ways AI models get this feedback and reinforcement learning. We're talking process supervision and outcome supervision. Okay. So outcome supervision, that's like getting feedback only at the very end. Did the final answer work? Was the overall essay good? That kind of thing. Just the final grade, basically. Pretty much. No comments on the drafts. And process supervision. Ah, that's the detailed one. That's where you give feedback, often really fine-grained, at each step of the thinking process. Like solving a math problem. Exactly. If an LLM is doing multi-step reasoning, it gets feedback on, you know, line one, line two, not just whether the final number is correct.

1:18Okay. Now, for ages, let's call it the conventional wisdom has really leaned heavily towards process supervision. It just feels right, doesn't it? Totally intuitive, like a teacher guiding you. Correcting each step seems obvious it would be better for learning. And that feeling, that intuition, has led to, well, frankly, massive investments in collecting this super detailed step-by-step process data, which isn't cheap. Not at all. Hugely expensive and time-consuming to get humans to provide that level of feedback. Yeah. The thinking was, you know, for complex reasoning, you need that detail to find errors, assign credit properly, understand why the model did what it did.

1:59Right. But, and this is the core of our DEAM dive today, there's a new paper out for March 2025 that asks a really fundamental question. Is process supervision actually better from a statistical efficiency standpoint? Or it's something else going on? Yeah, so our mission today is basically to challenge that long-held belief. See what these researchers actually found. Okay, let's unpack this properly. Two ways to train. Final grade, or feedback on every single step. Why did that step-by-step process supervision seem like such a slam dunk for so long? Well, like you said, the intuitive appeal is just massive.

2:36It feels right because it kind of mirrors how we often teach humans complex things. Think about, I don't know, learning long division or a complex proof. Right. You want feedback as you go. Exactly. Get feedback on each step. Seems way easier to figure out where you messed up. Easier to give credit for the bits you got right, especially if it's a long chain of steps. Plus, there's the hope for better interpretability understanding the AI's thought process. And this drove huge efforts, huge costs to get that data. Absolutely. Significant efforts, really substantial costs in terms of human labeling time, just based on this perceived advantage.

3:09So what about the other side then? Outcome supervision. If process supervision is so great, outcome must seem, well, blunt, imprecise. That's definitely the perception. Less detailed feedback for sure. But, and this is key, it's way, way easier and cheaper to collect. Just did it work in the end? Yes or no? Much more scalable. Much more scalable. And interestingly, it's how humans learn lots of things in the real world, right? You play games, you win or lose. You don't usually get a point score for every single move you made. So there's this tension. Yeah. How much hand-holding does the AI really need?

3:45Can it figure things out from just that final signal? That's the million-dollar question. Can systems learn optimal behavior just from these sparse, maybe delayed rewards? Okay, now here's where it gets really interesting. This new paper, the title says it all. Do we need to verify step by step? It digs into the statistical side of learning only from outcomes. And what they found kind of shakes things up. It really does. The core finding, their main theorem, is pretty stunning. It says that basically, under standard assumptions about the data you have, Learning from only outcome supervision is statistically no harder than learning from process supervision.

4:20Wait, no harder? Even for complex, long tasks? Up to what they call polynomial factors in the horizon, which basically means, yeah, even for long sequences, the fundamental statistical difficulty isn't significantly worse. So if we do see that process, supervision works better in practice sometimes. Their argument is that it's likely due to the algorithms we're using. Maybe our current algorithms are better suited to handle process data, but it's not because the outcome data itself is inherently worse from a statistical learning perspective. Wow. That could change how people approach data collection entirely.

4:54Potentially, yes. It suggests the bottleneck might be algorithmic design, not data type. But how? How can that be? If you only know if you won the game, how do you figure out which moves were the crucial ones? It still feels like learning in the dark. Ah, okay. So this is where their key technical insight comes in. They call it the change of trajectory measure, lemma. It sounds a bit technical, but the idea is clever. Layman's terms. Okay, so traditionally the big fear with outcome supervision was this thing called trajectory concentrability. Imagine trying to estimate the probability of one very specific long sequence of moves in a game.

5:33There are billions of possible sequences, right? Right. Seems impossible. Exponentially hard. Exactly. That complexity could blow up, making learning seem intractable. That's the prohibitively large problem. But this lemma provides a way around that. How? Instead of focusing on the probability of entire trajectories, they show you can focus on something called state action concentrability. Think of it like this. Rather than tracking every possible pass a river could take, you just measure the flow at key decision points, like junctions or forks in the river. So you aggregate, look at specific moments rather than the whole sequence.

6:06Precisely. You look at the probability of being in a certain state and taking a certain action, averaged over all the trajections that pass through that point. This measure is much, much smaller, much more manageable. The lemma shows that if a certain condition holds related to the value functions, you can effectively convert your outcome data into data that looks like it has step-by-step rewards just by focusing on these state action pairs. Okay, so it avoids that exponential blowup. Exactly. It avoids the exponential dependence on the length of the task, the horizon, allowing them to get statistical guarantees similar to what you'd get with full process supervision.

6:45So they didn't just point out a problem. They actually built a way to, like, transform the data, to make outcome data useful for existing methods. Yes, exactly right. They lay out an algorithm for it. They call it an offline outcome-to-process transformation. Offline means you're working with data you've already collected. How's it work, roughly? Roughly. You take your outcome data trajectories labeled only with the final result. You split the data. On one chunk, you learn a function that tries to estimate what the reward should have been at each intermediate step, maybe using techniques like least squares.

7:17Okay, so you're guessing the missing step rewards. You're estimating them, yeah. Then you take that learn reward function and use it to fill in those estimated rewards on the second chunk of your data. Now, suddenly, your outcome-only data look like process supervised data. And then you can just use standard algorithms. Bango. You can then apply any of the many existing offline RL algorithms that were built assuming they had process supervision. You don't need entirely new algorithms just for outcome supervision. You can leverage the existing toolkit. That's a really elegant theoretical move. It basically says the statistical potential is there in the outcome data.

7:53We just needed the right way to unlock it. Now, the paper doesn't just stop at the theory of offline data. It also looks at the online setting. That's where the AI is actively learning, trying things out, generating its own data, right? Correct. Learning by doing, essentially. And here they look at a very common practical strategy, trying to estimate those step-by-step rewards using concepts from reinforcement learning called Q functions or advantage functions. Okay. Familiar terms. What do they find there? This is another really important practical insight. They provide the first sort of rigorous theoretical backing for using the advantage function in this context.

8:32They prove that the advantage function, which kind of measures how much better an action is compared to the average action in that state, can serve as an optimal process reward model. Optimal meaning. Meaning if you use the advantage function to guide the learning, you'll end up with the same best possible policy as if you had the true, perfect step-by-step rewards from the environment itself. Okay, so advantage functions get a theoretical thumbs up. What about Q functions? Those are also everywhere in RL. People use them all the time, right? They are, and this is where the paper sounds a note of caution.

9:01It's a really crucial distinction they make. They prove, using the lower bound, that simply using the Q function, which estimates the total future reward as a stand-in for the step-by-step reward, can actually fail. Fail how? It can lead the AI to learn suboptimal policies, policies that aren't the best possible or might even be undesirable in some way. And this is super significant because, as they point out, using approximate Q functions as reward models is a widely used approach in large language model training right now. Uh-oh. So a common practice might be theoretically flawed? According to this analysis, yes.

9:39It suggests that relying on Q functions for this specific purpose might not be theoretically sound, and that advantage functions are the more correct choice if you're trying to infer step-by-step rewards from outcomes. Okay, let's pull this all together. What are the big takeaways here? For someone working on LLMs, or even just for you, the listener, trying to understand where AI is heading. I think the biggest thing is it challenges this very deep-seated assumption about data, that fine-grained step-by-step feedback is always necessary or statistically superior. This paper says, well, maybe not.

10:10The massive cost of getting that process data might not always be justified? Potentially, yes. If we can design algorithms that properly leverage outcome supervision using insights like this transformation lemma or the advantage function guidance, we might be able to train powerful models much more efficiently. And they even connect this to preference-based reinforcement learning, which is, you know, super relevant today. It's how many of the big LLMs learn from human feedback, humans basically saying, I prefer output A over output B. Right, like the reinforcement learning from human feedback, RLHF pipeline.

10:45Exactly. Techniques like DPO, direct preference optimization, are really popular now because they streamline that. They combine learning the reward and optimizing the policy. This paper provides an improved analysis for DPO. They show that DPO's sample complexity, how much data it needs, can also be understood in terms of that more manageable state action concentrability we talked about earlier. This leads to, as they put it, an exponential improvement in the theoretical guarantees for DPO compared to previous ways of analyzing it. Better theory often leads to better methods down the line. Wow.

11:18So this isn't just abstract theory. It has direct implications for current popular training methods like DPO, potentially making them more efficient or better understood. This could really change things. It could streamline data collection, maybe open up new algorithmic possibilities. It definitely makes the prospect of training powerful reasoning models potentially more scalable. All right. So let's recap. We've dug into this fascinating debate today, really questioning that old assumption that detailed step-by-step feedback is the only way, or always the best way, to train smart AI. We saw how this new research suggests that, statistically speaking, just getting the final outcome might be surprisingly powerful if we're clever about how we use that data.

12:00Right. Thanks to ideas like that change of trajectory measure lemma, transforming sparse data, and understanding the critical difference between using advantage functions versus Q functions for estimating those intermediate rewards. A difference that has real teeth for current LLM training practices. So for you, our listener, what this points towards is a potential future where building powerful AI might become less dependent on incredibly expensive, fine-grained human feedback. It's about working smarter with the data we can get easily. Finding efficiency not by cutting corners in understanding, but by being more theoretically grounded in how we approach learning itself.

12:38Which leaves us with a final thought. As AI keeps accelerating, maybe the biggest leaps won't come just from collecting more data, but from finding more ingenious ways to squeeze intelligence out of the data we already have access to. And what other obvious truths about AI are just waiting for a deeper theoretical dive to challenge them?