ExpRL: Using Reference Solutions as Rewards for LLM Mid-Training

0:00So imagine you're tasked with teaching someone how to ride a bicycle. But the rules of this teaching arrangement are just absurdly strict. Okay. Like you can't give them instructions. You can't offer any encouragement. And most importantly, you cannot give any partial credit at all. Right. You're literally only allowed to speak when they attempt this grueling, you know, 10-mile race. Yeah. If they perfectly complete the entire 10 miles without a single mistake, you say yes. Wow. Okay. But if they, like, wobble or put a foot down or crash at mile nine, you just say no. Under those conditions, I mean, how long do you think it would take them to learn?

0:39They probably wouldn't. I mean, they'd just be completely paralyzed. Because the only feedback you ever get is no. You have no way of knowing if, you know, your balance was right for the first three seconds or if your pedaling technique was just completely wrong from the very start. Right. It's a totally impossible learning environment. But as wild as that sounds, that is essentially the exact environment AI researchers have been using to train large language models to solve complex math problems. Yeah, it really is. So today we're doing a deep dive into this really fascinating new research paper.

1:10It's called XPRL, Exploratory RL for LLM Mid-Training. It's from a joint team at Stanford, CMU, and OpenAI. Yep. And we are going to explore how these researchers are trying to fix this broken teaching method, like how they're moving away from the strict binary pass-fail system and trying to build a framework that actually issues partial credit. It's a huge shift. And what's fascinating here is that this paper attacks what the industry calls the initialization bottleneck. The initialization bottleneck. Okay, let's unpack that. Yeah. So if you want an AI model to actually reason its way through a novel problem rather than just regurgitating memorized textbook answers, you kind of have to understand why the current standard methods are hitting a brick wall.

1:54Right. And let's define that wall for a second. Yeah. Because the current standard for this kind of problem solving is sparse reward reinforcement learning or RL. Exactly. And sparse really is the key word there, right? It relies entirely on a binary zero or one. Yeah. Just pass or fail. So think about a complex multi-step math problem. If an AI uses standard RL, it generates an entire page of work. And at the very end, an algorithm checks the final number. If the answer is exactly right, boom, the model gets a reward of one. But if it misses by a fraction or, you know, makes a single arithmetic error on step seven, the reward is zero.

2:32Which brings us back to that initialization bottleneck. Because, like, if I'm a base model, meaning I have general language skills, but I'm not a math genius yet, and you give me an extremely hard competition level math problem. Right. The statistical probability of me just randomly guessing the correct sequence of logic to arrive at the perfect answer is basically zero. That is the core issue. In reinforcement learning, the algorithm learns by having its successful behaviors reinforced. Yeah. But you cannot reinforce a behavior the model never naturally performs. If the model is so raw that it assigns new zero probability to the correct reasoning path, it's just never going to stumble upon the correct final answer.

3:10It's like trying to hit a half-court basketball shot while totally blindfolded. And like wearing noise-canceling headphones. That's actually a great analogy. Right. Because if you shoot a thousand times and miss a thousand times, your feedback is identical every time. It's just nothing. You don't know if you missed by an inch or threw the ball into the parking lot. Exactly. You learn absolutely nothing about your shooting mechanics. So if zero feedback reinforcement is a dead end for hard problems, I mean, the obvious human instinct is to just intervene. Why not just hand the AI a massive textbook of correct human solutions and say, hey, memorize and copy these exact steps?

3:50Well, and it is the most intuitive next step. And honestly, it's what the industry has relied on really heavily. We call it supervised fine tuning or SFT. GEC-SFT. Yeah. You take a data set of thousands of human written reference solutions and you force the model to imitate them. There's also a variant of this called self distillation. Where they try to get the model to imitate a slightly smarter version of itself, right? Exactly. But both of these approaches suffer from a really fatal flaw when applied to advanced reasoning. They create what's known as an off policy mismatch. Off policy mismatch.

4:20Yeah. That sounds like. Like you're trying to force the model to operate completely outside of its own internal logic. That is precisely what it is, because human reasoning and AI reasoning under the hood, they just do not look the same. Right. The cognitive path a human takes to solve a theorem might involve, you know, leaps of intuition or spatial reasoning or structural setups that a specific neural network simply doesn't possess the weights and activations to naturally understand. Oh, wow. So when you force the AI to blindly copy a human's exact steps, you are actually dragging it away from the logical paths it actually knows how to navigate.

4:57It would be like forcing me to memorize and recite a philosophical debate in a language I don't even speak. Yes. Like, I might be able to parrot the sounds perfectly for the test, but I haven't actually learned how to debate. And if you change the topic even slightly, I'll completely freeze up because I don't understand the underlying vocabulary. Exactly. You disrupt its underlying capabilities. by forcing imitation, you might actually make the model worse at reasoning on its own. That's wild. And that realization is really the jumping off point for the XPRL framework. XPRL stands for Exploratory Reinforcement Learning.

5:30They realize you need the human reference solutions, but you cannot use them as strict paths to mimic. Okay, so if we aren't imitating the human solutions, how are we actually using them? We use them as reward scaffolds. Under the XPRL framework, the AI is allowed to attempt the math problem completely on its own terms. Like doing it its own weird way? Yeah, it uses its own natural logic, its own weird detours. This is called on-policy exploration. The model is driving the car. But we introduce a separate entity into the system, an LLM judge. Okay, an LLM judge. So a secondary AI model acting as the greater.

6:06Yes. As the main AI explores the problem, the judge model sits in the background holding the human written reference solution. And its job is to look at the AI's messy natural logic, compare it to the human reference, and then hand out dense partial credit rewards based on whether the AI's logic is actually productive. Wait, okay, let's unpack this for a second because it seems like there could be a massive loophole here. Oh. If the judge is comparing the AI's work to the reference solution, does the AI taking the test ever get to actually see that reference? Because if it does, couldn't it just reverse engineer the answer?

6:42That is a critical point. No, the human reference solution is strictly hidden from the policy model of the AI solving the problem. OK, good. The reference is essentially locked in a vault that only the judge can access. The judge uses the human solution to construct a problem-specific grading rubric dynamically. So the student never sees the answer key, they just get the partial credit scores. Okay, that makes sense. But grading logic is notoriously subjective, right? How exactly is a language model reliably outputting numerical grades? The paper breaks this down into two different variants they tested, XPRL outcome and XPRL process.

7:17us. Let's look at outcome first. How does the judge grade the overall outcome without just reverting to a binary pass fail? So in XPRL outcome, the model generates its entire multi-step attempt at the problem, and the judge then reads the whole output and assigns it a score on a 1 to 5 Likert scale. Just 1 to 5? Yeah, but it doesn't just guess a number. The researchers engineered a specific prompt that forces the judge to explicitly map the logical links between the AI's attempt and the hidden reference solution before it generates that final digit. Ah, okay. So a 1 means the reasoning is totally flawed.

7:54A 3 means it has the right core idea but maybe failed to execute. And a 5 means the logic is perfectly sound even if it took a weird path. That 1 to 5 score is then mathematically converted into a dense 0 to 1 reward scale for the model to learn from. Got it. So even a totally failed attempt might earn like a 0.6 reward if the core logic was solid. Exactly. That solves the blindness problem right there. The model finally knows when it's getting warm. Yes, exactly. But as intuitive as that is, grading the entire essay at the very end is still a bit broad. Which brings us to the second variant, XPRL process, where the judge evaluates the AI step by step.

8:33Yeah, this is where it gets really interesting. And I found a detail in the methodology here incredibly clever. because the grade step by step, you have to define what a step even is. You have to slice the text up. Yes. Segmenting reasoning is historically just a huge headache in machine learning. Right. But the researchers didn't use some elaborate syntactic parser. They just sliced the text using a specific delimiter. Three hashtags in a row. Yeah. And they didn't even invent that. They noticed that the base model they were working with, which was Quen34B instruct, naturally defaults to using those three hashtags to separate its own thoughts.

9:10It's brilliant in its simplicity, really. You don't force a structure onto the model. You exploit the structure it already natively uses. By slicing the reasoning at those natural breakpoints, the judge can look at a single logical step in isolation. It can look at step four and ask, does this specific action advance the state of the problem toward the reference solution? Which introduces a really interesting mathematical dilemma, right? Let's say I write three brilliant steps and the judge gives me high scores for all of them. Okay. Then for step four, I write another brilliant step that just restates the exact same logic without actually advancing the problem at all.

9:49If you just add up the raw scores, my model gets a massive reward for essentially stalling. How do they prevent the AI from gaming the system like that? That is exactly why they couldn't just use raw scores. For the process rewards, they implemented a mathematical concept called centered segment level advantages, which they refer to as delta norm. Okay, delta norm. Let's translate that out of math speak. Sure. Think of it like playing the game hot and cold. If I hide a set of keys and you take a step toward them, I say warmer, you get a positive reward. But if you just stand there in that same spot and take a step in place, I don't keep shouting warmer.

10:25Right, because I haven't actually improved my position. Exactly. In fact, if you take a step sideways, I might say colder, delta norm looks at the change the delta between the current step and the previous step. Oh, I see. So you are rewarding the momentum of the logic, not just the absolute position. Precisely. A step only receives a positive reward if it improves the alignment with the reference relative to the step right before it. If your score goes from a three to a four, you get a positive reward. If your score stays a four, your reward is neutral. If your score drops to a two, you receive a negative reward.

11:01It punishes regression and rewards forward momentum. That completely changes the incentives for the model. It's no longer trying to just write a single perfect answer. It's trying to build a chain of upward progress. Yes, exactly. But here's the ultimate test. We have spent this entire time talking about mid-training. Like, we are giving the AI this safe space with partial credit and supportive grading. Eventually, though, this model has to graduate. It has to face what the paper calls downstream RL. Yeah, reality kicks in. Downstream RL is the final optimization phase. It's the real world. You take the training wheels off, you strip away the LLM judge, and you subject the model to standard, brutal, binary pass-fail reinforcement learning.

11:46So does the partial credit actually prepare it for that binary final exam? Spectularly. And to quantify that, we really should look at a metric called PASIC-K. Okay, let's define PASIC-K for everyone. Imagine I give you an incredibly difficult math problem. Instead of one piece of paper, I give you 16 blank sheets of paper. I tell you to try 16 completely different independent methods to solve it. Pass at 16 measures the probability that at least one of those 16 sheets contains the correct final answer. So if a model's pass at K goes up, it means the model has developed a really diverse toolkit.

12:18It isn't just a one-trick pony that memorized a single path. It actually knows at a search of problem space from multiple angles. Exactly. Now, let's look at the actual results on the AIA 2026 benchmark. This is competition-level mathematics. Very hard stuff. When the researchers used the traditional imitation method, forcing the model to copy human steps via SFT, and then sent it to downstream RL, the model achieved a 30.26 % success rate. About 30%. Okay, pretty low. Yeah. When they skipped the scaffolding entirely and just used standard GRPO, which is a popular optimization method that relies purely on sparse binary rewards, the model hit 58.75%.

12:57Oh wow, a huge jump just by letting it explore, even with the bad pass-fail feedback. Right. But when they prepped the model using XPRL process, giving it step-by-step partial credit before sending it to the final downstream RL phase, it achieved a 63.41 % success rate. Pushing past 63 % on the A benchmark is a massive leap. It really is. But honestly, the raw score isn't even what caught my attention. Here's where it gets really interesting. It's how the models were actually arriving at those answers. The paper details these behavioral shifts in the AI's reasoning traces. The XPRL model started exhibiting search-oriented behaviors completely naturally, like they started self-correcting, backtracking, and verifying.

13:38This is arguably the most important finding in the entire paper. When you look at the text the model is generating, it literally reads like a human thinking out loud. It's spooky. Yeah. Self-correction happens when the model generates a line of math, stops, and outputs something to the effect of, wait, this contradicts my earlier assumption, let me rethink this. Backtracking is when it realizes a path is just a dead end, abandons it entirely, and explicitly returns to step two to try a new branch. And verification is when it finds a solution, but before finalizing it, it plugs the number back into the original equation to prove it actually works.

14:13It learned how to double check its own work. And what's wild to me is that the traditional methods supervised fine tuning actively destroys that behavior. It does. Because if you think about it, a human written reference solution in a textbook is perfect. Right. It doesn't include the human's mistakes or their scratch pad math or their moments of doubt. It's just the clean final path. If you force an AI to blindly imitate that clean path, it never learns how to recover from an error. It forgets how to verify because a perfect path doesn't need verification. Exactly. XPRL teaches the AI that making a mistake and then recovering from it is actually a highly rewarded behavior.

14:55You're teaching the habit of critical thinking rather than just the product of it. Yes. But I do have to challenge the central pillar of this whole framework. Oh. Because the entire XPRL system lives or dies on the reliability of the LLM judge. That's true. If the judge is handing out dense rewards, how do we know it actually knows what it's doing? Language models hallucinate all the time. They get tricked by articulate-sounding gibberish. If the judge gives a high score to a hallucinated math step, the policy model learns the exact wrong lesson. The researchers were acutely aware of this vulnerability because you're right.

15:28A flawed judge collapses the entire training pipeline. Right. So they designed a series of tests, starting with a mixed-domain scale study. They expanded beyond just math and tested the judge on science question answering and coding. Furthermore, to make this computationally feasible, they didn't use a massive trillion parameter model as the judge. What did they use? They used a lightweight 4 billion parameter model to judge the homework of a much larger 8 billion parameter policy model. Let's contextualize that because a 4 billion parameter model is small enough to run locally on a high-end laptop.

16:04top. You have a relatively lightweight system grading a much more computationally heavy capable system. That seems really counterintuitive. It does, yeah, but it actually worked across the STEM subjects, and it proves that the judge doesn't need to possess innate genius. It just needs to be competent enough to read the human reference solution it's holding and compare it to the AI's output. The reference solution is doing the heavy lifting. It acts as an absolute anchor. But how do they prove the judge is actually using that anchor? How do we know the 4B model isn't just getting lazy and handing out high scores because the 8B model uses sophisticated formatting and big words.

16:40Ah. If we connect this to the bigger picture to prove that, they ran what they called the calibration stress test, and it is beautifully devious. I love devious tests. They took the judge and they deliberately gave it the wrong reference solution. They just handed the grader a fake answer key. Oh. So if the judge is just grading on vibes and formatting, it shouldn't matter that the answer key is wrong. It would keep handing out good grades. Exactly. But that's not what happened. When they swapped the reference solution, the judge's grading completely fell apart. The misplacement rates, meaning the false positives and false negatives, just skyrocketed.

17:17It started punishing brilliant logic and rewarding absolute nonsense. Which is actually a massive success for the researchers. It is the ultimate validation. It proves that the dense reward signal is genuinely, causally driven by the reference scaffold. the judge is actively performing the hard work of cross-referencing the logic against the specific answer key it was provided. It isn't just relying on surface-level plausibility. I buy the mechanics of it. But I did notice one domain in that mixed study where XPRL did not dominate the standard methods. Yes. On the coding tests, specifically the live code bench, the XPRL partial credit system, actually underperformed compared to standard sparse reward RL.

17:59If this system is so good at teaching reasoning, why does it falter when the model is writing code? It comes down to the fundamental difference between mathematical theory and executable code. In math, evaluating whether a transitional step is logically sound requires a lot of nuance. But code can literally be compiled and executed. You can just run the code against a suite of test cases. A compiler gives you a perfectly objective, immediate, binary, sparse reward. Does the code run without throwing an error, and does it produce the expected output? There's no partial credit in a compiler. If you miss a single semicolon, the entire program crashes.

18:34Exactly. Because execution provides such an incredibly strong and accurate sparse reward natively, introducing a human reference judge actually muddies the waters. Oh, interesting. Think about it. There are a thousand different ways to write a Python function that executes perfectly. If an LLM judge is trying to force the AI to align with one specific human's coding style, it might penalize a highly efficient, brilliant piece of code simply because it takes a different structural approach than the reference. Oh, I see. So in the domain of coding, the compiler is already the ultimate judge. So when the environment provides perfect, instantaneous feedback on its own, just let the AI figure it out.

19:15But when the environment is opaque, like a 10-page math proof, bring in the scaffold. Precisely. So what does this all mean? Bringing this out of the realm of neural networks and back to you, the listener. The core philosophy of XPRL is fundamentally about how we approach deeply complex problem solving. If you are navigating a massive, ambiguous challenge in your own work, brute forcing attempts from start to finish and only valuing the final success is wildly inefficient. You need to build frameworks that reward your own partial progress. Absolutely. You have to recognize the value of the delta, the momentum, And most importantly, you have to realize that self-correction and having the courage to backtrack aren't signs of failure.

19:55They are the literal mechanics of advanced reasoning. The capacity to explore a space, hit a wall, recognize the wall, and pivot is the hallmark of true intelligence. It is how we move from memorization to real discovery. But I do want to leave you with one final, unresolved puzzle that this paper kind of highlights. Okay, what is it? The brilliance of XPRL rests entirely on having a human-written reference solution to act as a scaffold. We are using the known to help the AI explore the boundaries of the unknown. Right. The judge needs the answer key to build the rubric. But what happens in a year or two?

20:29The goal of AI isn't just to solve the math problems humans have already solved. We want to train AI to solve the deepest scientific mysteries of our time problems in quantum physics or biology or climate modeling that no human has ever cracked. Right, the unsolved problems. If there is no human reference solution to give the judge, we have no answer key. So how do we build a rubric to guide an AI's exploration when the destination is truly unknown to everyone? It's the ultimate frontier. Because once you leave the confines of the 10-mile track, you have to figure out how to build the road as you ride it.