Compute as Teacher: Turning Inference Compute Into Reference-Free Supervision

0:00Imagine you've poured, you know, just billions of tokens into training one of these huge language models. It's incredibly smart, right? Absolutely. It's due to the art stuff. But then you needed to do something really nuanced, like generating expert clinical guidance, maybe, or writing really complex code. Tasks needing real precision. Yeah. Things where close enough isn't good enough. Exactly. And the big hurdle isn't the model size anymore. It's the supervision gap. How do you actually teach it these specialized skills when you don't have, like, perfect gold standard answers to check against?

0:36That's the fundamental problem. Traditional post-training, it leans heavily on human experts labeling tons of examples. Which is super slow and expensive. And, honestly, often flawed, especially in areas where there just isn't one single correct answer. Think about it. Freeform conversations, creative writing. Or, like the source mentioned, clinical chat analysis. Precisely. Experts disagree all the time in these complex, non-verifiable domains. You can't just write a simple rule to check the answer. And getting consistent human labels. Almost impossible, or at least prohibitively expensive. Right.

1:10So you're kind of stuck. Okay, so let's unpack this, because the source material we're looking at introduces a really fascinating idea. It's called compute as teacher, or CAT. Cat T. And the core question it asks is pretty profound. Can the computation you're already using for inference just for generating answers, can that somehow replace the missing human supervision? So using the model's own work to teach itself, essentially. Basically, yeah. Our deep dive today is all about showing you how Cat takes the model's own exploration, its own attempts, and turns that into a, well, a rigorous teaching signal without needing external references.

1:48So at its heart, CAT is this really elegant feedback loop. It's all about self-correction. It converts the model's own inference time activity, its attempts, into a teacher signal. And it generates a single high quality estimated reference purely from looking at its own behavior. How does it actually do that? What's the process? It seems to have two main stages or maybe roles. First is exploration. The current model, let's call it the student policy, it generates a bunch of parallel answers to the same question. Like multiple shots at the target exactly the paper suggests using eight parallel rollouts so eight different tries eight exploration paths okay eight different answers generated by the current model then what and then comes the clever part synthesis they bring in what they call a frozen anchor policy let's call it$20 frozen anchor so like an older stable version of the model yeah typically the initial version before this specific CAT training started.

2:45And this anchor, this referee, it only looks at the set of eight answers the student just produced. Wait, so it doesn't see the original question again. It only sees the student's attempts. That's right. That seems to be the key insight. Okay, why? Why freeze the anchor and why limit its view like that? Well, what's really interesting is that separation of roles. Freezing the anchor, the teacher, guarantees stability. That's crucial. Right, because if the teacher was also changing constantly, the signal could become unstable. It might just amplify the student's latest errors. Exactly. You need a consistent standard.

3:17Okay. Makes sense. And forcing it to only look at the rollouts. That prevents the anchor from just, you know, giving its own best answer independently. It forces it to actually synthesize something from the student's attempts. Ah. So it has to reconcile the different answers it sees, integrate the good bits, maybe suppress the errors. Precisely. It looks for complementary evidence across the different attempts and tries to filter out the sort of unique idiosyncratic mistakes in any single one. The synthesized answer, the teaching signal, comes purely from analyzing the student's exploration. Okay, and this is where it gets really, really interesting for me.

3:54Because we've seen methods before, like best event, right? Generate 10 answers, pick the one that looks best as based on some score. Or majority voting or picking based on perplexity, PPO. Those are common baselines. But CAT is fundamentally different. The source emphasizes this strongly. It's synthesis, not selection. It's not just picking. It's creating. That is the critical distinction, absolutely. With those older selection methods, you're always limited by the best answer you happen to generate in your batch. Right. If all 10 of your attempts are flawed. You're just picking the least bad one.

4:29There's no mechanism to go beyond the quality of the initial rollouts. But with CAT, that frozen anchor model actually constructs a new answer, a synthesized reference. And this as doesn't have to be one of the original eight attempts. It can be genuinely novel derived from them, but not one of them. And they've got data to back this up. On the Math 500 benchmark, which is pretty quantitative, right? The synthesized answer disagreed with the majority vote of the eight rollouts, something like 14 % of the time. Wow. OK, so it's definitely not just following the crowd. Not at all. And even more striking, they found cases almost one percent on those math problems where the synthesized answer was correct.

5:08But it disagreed with all eight of the students attempts. It disagreed with all of them and was still correct. How? It implies it's identifying a common failure mode or integrating partial successes in a way none of the individual rollouts managed. It's like true error correction, spotting flaws across the board. That's, yeah, that's pretty remarkable. Synthesizing correctness out of collective failure. There's this great little example in the appendix, a math problem about finding a term like SISN$2 in some recursive sequence. Yeah. All eight student rollouts got it wrong. They made small calculation errors somewhere in the middle, like a wrong division or a missed factor, you know.

5:46Uh-huh. Easy mistakes to make in long calculations. But the synthesized answer, generated by the anchor looking only at those flawed rollouts, apparently identified the pattern of errors, corrected the underlying reasoning, and produced the right final number. So it didn't just fix a typo. It understood the logic errors implied by the set of answers. That seems to be the implication. It constructed a successful path by analyzing the failures. Okay, that's powerful, but let's talk practicality. Generating eight full answers, then running the anchor model for synthesis. That sounds computationally expensive, right?

6:22A lot more FLOPs per question just to get the supervision signal. That's the trade-off, yeah. It definitely requires more compute up front during this teaching phase. Is it worth it? Does it scale reasonably? According to the research, yes. They found performance scales pretty predictably, monotonically even, with the number of rollouts. The more exploration paths you give it, more compute, the better the synthesized reference generally gets. So it's a direct trade, more compute for better supervision. Exactly. You're essentially trading inference time FLOPs for a high-quality supervision signal that you couldn't easily get otherwise, especially not from humans in these complex domains.

7:02Okay, so now we have this high-quality synthesized reference maze. Great. But how do you actually use it to train the student model? That's the KTRL part, right? Reinforcement learning. Right. You need to turn ZAIDS into a reward signal. And CAN cleverly works in two different settings, which covers a lot of ground. What are the two settings? So regime one is for what they call verifiable tasks. Think math problems, coding tasks with unit tests, maybe structured data extraction where you know exactly what the output format should be. Things with a clear right or wrong answer, basically. Exactly.

7:33And here, the reward mechanism is super simple. You just use a basic programmatic checker. Does the student model's final answer string match the final answer in the synthesized reference, Iza? Ah, okay. So if Davin says the answer is 42 and the student outputs 42, it gets a positive reward. If that puts 43, it gets zero or negative. Pretty much. It's designed to be a drop-in replacement for standard RL reward functions in these kinds of tasks. No human labels needed, no complex verifiers beyond simple answer matching against the synthesized target. Okay, that covers the clean cases. But what about the messy stuff?

8:09The whole reason we need CAT in the first place, the non-verifiable tasks. Right, like the freeform dialogue, summarization, or that health bench medical guidance task they mentioned. No single ground truth answer exists. So you can't just check for string equality. How do you generate a reward there? This is where I think the really neat innovation comes in. Instead of just outputting the synthesized references, the frozen anchor model does something extra. It converts as into a self-proposed rubric. A rubric, like a grading sheet. Exactly. A short list, maybe five to ten points of binary checkable criteria that are specific to that synthesized reference answer is yes.

8:46Okay, give me an example. What would a criterion look like? So for a medical dialogue response A's, a criterion in his rubric might be, does the response ask about current medications? Or does the response mention potential side effects X and Y? Or does the response cover all key patient history points from the prompt? Very specific yes-no questions based on the content of S. And how is this rubric used for reward? Do you still need a human? No, that's the point. They use an independent LLM judge, say something powerful like GPT-4-Row, to apply the rubric. For each criterion on the list generated by the anchor, the judge simply gives a yes or no verdict for the student's actual response.

9:26Ah, so the judge isn't making a holistic quality judgment. It's just checking off the specific points from the self-proposed rubric. Precisely. And the final reward for the student's response is simply the normalized proportion of criteria satisfied. So if it meets 8 out of 10 criteria, it gets a reward of 0.8. Okay, but hold on. If you're using GPT-40 as a judge, aren't you just swapping expensive human labels for expensive API calls? How is that reference-free? That's a fair question. The key difference lies in what you're asking the judge to do. Instead of a complex subjective, is this good judgment, which is known to be noisy and expensive, you're asking for simple, fast, binary checks against highly specific criteria.

10:09So it's cheaper and more reliable because the task is decomposed. Exactly. It's much more efficient and auditable. And the rubric itself comes from the anchor model, derived from the synthesized reference, not from a human. That's the self-proposed part. The supervision signal originates internally. And why are rubrics better than just asking the LLM judge for an overall score? Because they break down a potentially fuzzy, subjective judgment, is this helpful medical advice, into concrete, verifiable pieces. It sidesteps a lot of known issues with LLM as a judge, like instability or biases. Like the verbosity bias, where models get rewarded just for being long-winded.

10:45Exactly. A rubric checks for a specific substance, not just length or style. Did it cover point A? Yes, no. Point B? Yes, no. Much cleaner, much less gameable, and much more aligned with producing actually useful output based on the synthesized ideals. Okay, so let's talk results. Did this CAD approach actually work? How much better did the models get? The results seem pretty impressive, actually. First, just using CAD at inference time, so no RL training yet, just generating the eight rollouts and the synthesis to get a better single answer, that alone gave significant boosts. How significant?

11:17Up to a plus 27 % accuracy jump on Math 500 and plus 12 % on HealthBench compared to just taking a single answer from the base model. Wow. Okay, so just the synthesis step itself is powerful. Very. And crucially, they showed it outperformed all the selection baselines they tested, picking the answer with the lowest perplexity, majority vote, etc. It really hammered home that synthesis is better than selection here. All right, but the real magic is supposed to be the CAT-TRL training using those rewards, right? Exactly. And when they use these CAT-T generated rewards, either the simple checker for verifiable tasks or the rubric system for non-verifiable ones, to actually fine-tune the student policy via RL, the gains compounded.

11:58Across different models. Yep, they tested it on three different model families, Gemma, Quinn, and Llama 3.1, all seeing solid improvements. The LAMA 3.18b model did particularly well. Oh, well. It achieved up to a plus 33 % relative improvement on Math 500 and plus 30 % on HealthBench compared to its own starting baseline performance before CAT TRL training. 30%. That's huge for these kinds of benchmarks. It really is. It suggests this process is genuinely teaching the model new, more reliable capabilities in these specialized domains. Which points towards that virtuous cycle idea they mentioned.

12:33Absolutely. The improved student model generates better, more consistent rollouts. The fixed anchor model synthesizes an even better reference as from these improved rollouts. This better as provides a stronger reward signal. Which further improves the student model. And round and round it goes. The student starts to outperform the original teacher. Exactly. The policy learns from its own improved behavior guided by the stable anchor synthesis. But you mentioned, or the source did, that this cycle doesn't go on forever. It plateaus. Why? Yeah, that's a known phenomenon in this kind of self-improvement RL.

13:07Essentially, the student model gets too good, or rather too consistent. How is that a problem? Because the synthesis step in cat relies on having some diversity, some disagreement among the rollouts it analyzes. That disagreement is the signal it uses to identify errors and find better path. Ah, okay. So if the student model converges and all eight rollouts become nearly identical. Then there's no disagreement for the anchor to resolve. The synthesis step offers less and less new information, the teaching signal weakens, and the improvement curve flattens out. The model needs a bit of uncertainty or exploration to keep learning this way.

13:42That makes sense. Sort of confident convergence trap. You could put it that way, yeah. And there was one other really interesting finding about the training method itself, wasn't there? Comparing the RRL approach to just supervised fine-tuning? Oh, right. That was critical. They found that KTRL training with the rewards derived from the self-proposed rubrics actually performed better than just doing standard supervised fine-tuning, SFT, directly on the synthesized reference answers. So how you teach matters, not just what you teach. Using the decomposed rubric reward was more effective than just showing the model the perfect synthesized answer.

14:18It seems so. It suggests that the step-by-step reward signal from the rubric might provide a better learning gradient, or perhaps avoids overfitting to the specific phrasing of ears compared to just trying to mimic the entire synthesized output perfectly. Hashtag tag tag outro. So pulling this all together, what's the big picture here? What does Kat really tell us? I think it's a powerful demonstration that when you hit that wall of scarce, expensive, or just plain non-existent human reference data, compute itself can step up. Right. It can generate meaningful, high-quality supervision signals using the model's own exploration.

14:55And it works across the board from tasks with clear answers like math to these really fuzzy, non-verifiable domains like clinical dialogue. So for you listening, what's the key takeaway? What should you remember from this? I'd say it's a sign that the main bottleneck in creating highly specialized AI might be shifting. It's maybe less about meeting infinite perfect human labels now. And more about what? And more about leveraging the model's own computational power for rigorous self-correction and synthesis. The limiting factor might become the quality of the model's ability to explore alternatives and then judge its own explorations effectively.

15:30That's a really interesting shift in perspective. It is. And KTRL specifically shows a path for how these models can systematically improve themselves, potentially pushing beyond the limits of the human data they were initially trained on. Which leads to that final, maybe slightly provocative thought. If better models generate better explorations, which create better synthesized teachers, which lead to even better models. Yeah, where does that cycle ultimately lead? Can this kind of computational self-improvement create capabilities that genuinely surpass what's directly present in human knowledge or examples?

16:04A question to definitely mull over. Can machines bootstrap their way to truly novel expertise using just computation as their guide? It's certainly the direction Kat seems to be pointing. A fascinating possibility.