Success Conditioning as Policy Improvement: The Optimization Problem Solved by Imitating Success

0:00You know that old advice, fake it till you make it. Oh, yeah. I've always had a bit of a, I don't know, a complicated relationship with that phrase. It implies you're just pretending, right? Right. Like you're a fraud just waiting to be caught. It definitely has a connotation of deception or at least, you know, unsubstantiated confidence. Exactly. But there's a slightly different version of that idea that I think we all actually use. And it's much more honest. It's more like, look at your past. Look at the few times you accidentally did something right. Right. Maybe you nailed a presentation or you made the perfect omelet and then you just you ignore all the times you fell on your face and you try to do that one successful thing again.

0:39Right. It's less faking it and more like selective imitation. You're curating your own history to find the gold nuggets and discarding the rocks. Selective imitation. I like that. And while that sounds like just a life hack for, you know, stumbling through adulthood, it turns out this is actually a fundamental technique in how we are training some of the most advanced AI systems. right now. It is. In the field, we call it success conditioning. Success conditioning. It sounds almost like self-help-y. Condition yourself for success. It does. But today, we are going to unpack why this isn't just a slogan.

1:15It's a technique that's powering everything from language models like DeepSeek or Llama to robots learning to grab a coffee cup without crushing it. Yeah. And the mission for this deep dive is to reveal that this shortcut, because it really does feel like a shortcut, is actually doing something mathematically profound. And that is a very important distinction to make right up front. For a long time, researchers, myself included, honestly, we looked at this technique as a heuristic. Right. A hack. Right. Something you do because it's computationally easy, not because it's theoretically perfect.

1:48It feels like cheating, doesn't it? You aren't doing the heavy calculus, you're just copying the winners. Exactly. But the reality we're discussing today flips that assumption on its head. We're going to walk through how this hack is actually a mathematically precise solution to a very specific, very elegant optimization problem. Okay, so we're going to get into the geometry of trust, why being conservative is sometimes the boldest move you can make, and why trying to be too perfect, setting your standards too high, can actually turn an AI into a gambling addict. It's a really fascinating story of how a simple intuition turns out to be backed by some really rigorous geometry.

2:29So let's start with the basics just to grind everyone. We mentioned success conditioning shows up everywhere. If I'm just a user of AI, where have I seen this logic at play? You see it most directly in how we align large language models. Think about the concept of rejection sampling or best event. Walk me through that. Okay, so you have a model. You give it a prompt. You ask it to generate, say, 10 different answers. Got it. Now, the model is imperfect, so some answers might be toxic, some might be hallucinations, and maybe one is perfect. Okay. You or a grading algorithm, you act as the filter.

3:04You throw away the nine bad ones. You keep the one good answer. And then, and this is the key, you retrain the model on just that good answer. You say, do more of that. So that's success conditioning. That is success conditioning. It's basically natural selection for data, survival of the fittest output. Exactly. It's also the secret sauce in those reasoning models we keep hearing about, like STAR or DeepSeek R1. These models generate their own, you know, chains of thought. They keep the chains that lead to the right answer, and then they learn from them. It's in robotics, too, right? A robot arm cries a bunch of random twitchy movements.

3:39It finds the one trajectory that actually opens the door and says, OK, that's the one. Let's encode that. It's the core logic behind decision transformers. Now, decision transformers always sounded like magic to me. The original concept was basically, we don't need to calculate how good every move is. Right. We just ask the model, please output the actions that lead to a high score. And that's where the skepticism comes in. That's where the math nerds get uncomfortable. Why? Right. Because traditional reinforcement learning, or RL, is usually very explicit. It tries to build a value function. It wants to know exactly how many points this specific action is worth in this specific situation.

4:17It's like a chess player explicitly calculating, if I move my knight to F3, my probability of winning the game goes up by exactly 2.4%. Exactly. It's predictive. Right. It's looking at the future. Whereas success conditioning is just looking at a replay of a winning game and saying, hey, last time we moved the knight to F3, we won, so do that. Right. One is predictive calculation. The other is conditional generation. And for a long time, the assumption was, is this actually optimizing anything? Yeah. Or is it just mimicry? Because if you're just copying, you might be copying luck. You might be copying noise.

4:53Exactly. But the insights we're looking at today prove that it is optimizing. It says success conditioning isn't a sloppy approximation. It's an exact solution to a trust region optimization problem. Now, trust region sounds like something from a geopolitical treaty, but I know in math it's about navigation. It is. I love the mountain climbing analogy here. It really clarifies things. Imagine you're trying to climb a mountain in a thick, thick fog. You want to get higher, but that's optimization. You want to maximize your elevation. But you can only see a few feet in front of you. Okay, so I have a map, which is my current policy, my current way of doing things.

5:33But I know my map is imperfect. It might show a trail where there's actually a cliff. Exactly. Yeah. So you defined a trust region, a circle around where you are right now. You say, I'm willing to move to a higher spot, but only within this circle where I trust my map. Because if you step outside that circle into the fog, you might fall off a cliff. You might. Or you might find a shortcut, but it's just too risky. So practically, almost all modern reinforcement learning algorithms like PPO or TRPO, which are the industry standards, they use this trust region concept. They try to find the steepest path up the hill, constrained by staying inside that circle of safety.

6:10Yes. But here is the plot twist. The difference between standard RL and success conditioning isn't that one uses a trust region and the other doesn't. They both use trust regions. The difference is the shape of the circle. The geometry. The geometry. Standard RL uses a metric called KL divergence to measure that distance. KL divergence. Okay. We hear this term all the time in machine learning. It's the standard yardstick for how different two probability distributions are. But without getting into the formula, what is the personality of KL divergence? The personality of KL divergence is fear of missing out.

6:44FOMO. Basically, yes. KL divergence hates it when you stop doing things you used to do. It measures the distance between your old strategy and your new one, and it penalizes you heavily if you drop options. So if my old strategy is to try option A and option B, kale divergence gets really mad if I suddenly decide to strictly do option A and ignore option B completely. Yes. It forces you to cover your bases. It wants the new distribution to cover the old one. It prevents you from committing too hard, too fast to just one path. It says keep exploring a little bit just in case. Okay, so standard RL is the anxious friend who wants to keep all options on the table, but success conditioning, that works differently.

7:23It works completely differently. Success conditioning doesn't use KL divergence. The math proves that it minimizes a completely different metric, chi-squared divergence. Chi-squared. Okay, if KL is FOMO, what is the personality of chi-squared? The personality of chi-squared is stranger danger. Stranger danger. That's a very different vibe. It is entirely different. Chi-squared doesn't care if you drop old options. you can forget option B completely. It won't penalize you for that. What Chi-Squared hates is if you go somewhere you have never been before. So it hates exploration. It hates exploration outside of your known data.

8:02This feels like a huge shift in mindset. KL says don't stop doing what we used to do. Chi-Squared says don't do anything we haven't seen. Precisely. And this geometric difference explains why success conditioning works the way it does. Let's go back to a sports analogy. Okay. Imagine you have a data set of yourself trying to play basketball and 90 % of the time you miss the shot. You're just, you know, throwing bricks. But 10 % of the time you do this specific gentle layup and you score. Okay. So I have a history of mostly failure and a little bit of success. If I used standard RL with KL Divergence to train myself.

8:36The math would try to improve your game, sure. But because of that FOMO, that need to cover the old distribution, it would force you to still take some of those bad shots. He would resist letting me just become the layup guy. Yes. Because that would mean deviating too much from your historical distribution of guy who throws bricks. It holds me back because it thinks the real me involves missing shots. Exactly. But Chai squared. Chai squared looks at that data. Chai Square looks at that data and says, hey, I see a layup in your history. It works. I don't care about the other 90 % of shots. We are allowed to drop them completely.

9:13Just do the layup. It permits concentration. It permits concentration. It allows specialization. As long as that layup was in the history, even if it was rare, Chai Square says, go for it. Zero in on that. That makes so much sense for something like a labage model. You don't want it to cover the bases of all the gibberish it produced during training. You don't want it to say, well, I used to hallucinate 20 % of the time, so I better keep hallucinating 10 % of the time just to stay close to my roots. Exactly. You want it to zero in on the coherent stuff. You want it to forget the failures. It turns out this shortcut of just copying the successes is actually the mathematically optimal way to improve if your constraint is don't do anything wild that we haven't seen.

9:55So it's a conservative optimizer. It's not trying to discover a new continent. It's trying to find the best house in the neighborhood you already live in. Perfectly put. It shifts the philosophy from explore everything to exploit what works. And this leads to another discovery in the field, which is called the action influence identity. This sounded almost mystical when I was reading up on it. The magic equation. It is beautiful. And it's strictly true for this type of conditioning. The math proves that for success conditioning, three very different things are mathematically identical. They're equal at every single state.

10:29Okay, walk us through them. What are the three things? Number one, the relative improvement in performance. Basically, how much better do we get? Number two, the magnitude of the change in the policy. How different is our behavior from before? Okay. And number three, action influence. Action influence. This is a term we really need to nail down. What exactly is action influence? It's a measure of agency. It asks a simple question. Does my choice actually change the outcome? Give me an example of low action influence so I can visualize this. Imagine you're playing a game of war with a deck of cards.

11:06It's pure luck. You flip a card, I flip a card. High card wins. Right. Now imagine you have a choice. You can choose to flip the card with your left hand or your right hand. It makes absolutely no difference to the game. Right. Your action has zero influence on the success rate. The outcome is purely a random distribution of the cards. In that state, action influence is zero. And high action influence. You're driving a car. You turn the wheel left, you go left, you turn it right, you go right. Your action dictates the outcome. Okay, so back to the identity. You said improvement equals policy change equals action influence.

11:39Yes, and this gives us a surprisingly powerful diagnostic tool. Usually in AI, it's really hard to know if your training worked until you deploy the model and test it on a benchmark. It's expensive and slow. But here, because of this equality, you can just look at how much the policy changed. That's it. So if the model is behaving differently, if it has shifted its strategy significantly, we know it got better. Mathematically, yes. Yeah. If the policy changed a lot in terms of chi-squared, it means the action influence was high, which means the improvement must be high. And conversely, if the policy didn't change.

12:17It means you didn't improve. You don't need to run a benchmark to know that. If the model looks at the successful data and says, it looks the same as the random data, it means your choices didn't matter. You were playing the card game with your left hand versus your right hand. This actually leads perfectly into the failure mode because sometimes success conditioning doesn't work. We've all seen AI that seems to get stuck. And based on what you just said, it must fail when choices don't matter. Exactly. There is a classic example used in probability called the two-armed bandit. Okay. Imagine two slot machines sitting side by side.

12:49Okay, I'm at the casino. Arm number one pays out 49.5 % of the time. Arm number two pays out 50.5 % of the time. So they're almost identical, but arm two is slightly better. A tiny edge. Right. Now suppose you start out playing randomly, 50-50. You collect a bunch of data, you lose some, you win some, your overall win rate is 50%. Now I apply success conditioning, I filter out all the losses, I look at the pile of winning tickets. And you ask, in the winning pile, which arm did I pull more often? Well, logically, since ARM2 wins barely more often, I guess I pulled it slightly more often in the winning pile.

13:25Slightly. Maybe 50.005 % of the winners are from ARM2. It's barely noticeable. So when you retrain the model to imitate the winners. It learns to pull ARM2 50.0005 % of the time. So it basically didn't learn anything. It stays random. It didn't figure out that ARM2 is the better ARM. Correct. Because the action influence was low. The difference between the choices was drowned out by the noise of the probability. The signal was just too weak. That sounds like a failure, but, and this is the silver lining, what happens when standard reinforcement learning fails in this scenario? Oh, standard RL can be a disaster here.

14:04If standard RL gets confused by noise, it tries to chase it. It might convince itself that ARM1 is actually a magic jackpot machine because of a statistical fluke where it won three times in a row. it can induce distribution shift. It starts doing crazy things, hallucinating patterns that aren't there. Exactly. So standard RL fails by going crazy. Success conditioning fails by doing nothing. It's a safe fail. It is over conservative. If the signal isn't clear, success conditioning just shrugs and stays where it is. It won't degrade performance. It just won't improve it much. That feels surprisingly reassuring for AI safety.

14:40It's like a doctor who follows the Hippocratic oath. First, do no harm. If I don't know what to do, I'll just stick to the baseline. Exactly. It makes it very stable. But, and there is always a but, there is a trap. And this is where it gets really interesting for anyone building these systems or even managing a team where you are measuring success. The trap of dense rewards. Right. So far we talked about win-loss binary outcomes. But usually we have scores, We have returns. And a common technique is something called return thresholding. This is where you say, I don't just want successful runs.

15:15I want runs where the score was over a thousand. Or I only want to hire employees who sold over a million dollars. Precisely. You set a bar. You filter for elite performance. And this seems logical. Of course. If I want to be great, I should only study the greats. But the logic breaks down if you aren't careful. It creates a proxy reward. Yeah. You think you're optimizing for score, but you're actually optimizing for probability of score X. And those aren't always the same thing. Not at all. Let's look at the lucky arm example. This blew my mind because it feels so counterintuitive. Yeah. Okay.

15:48Imagine you have 100 slot machines. 99 of them are moderate arms. They usually give you nothing, but occasionally, very occasionally, they spike and give you a huge jackpot. Highly volatile. Very volatile. High risk, high reward, or mostly low reward. occasionally giant reward. Right. And then you have one special arm. It is consistent. It gives you a good solid return every single time. Let's say it gives you a 0.9 out of 1.0 constantly. So in the long run, the special arm is the best choice. It's the skilled choice. It's the student who gets an A on every test. Yes. Now imagine you set your filter threshold to 0.6.

16:24The special arm at 0.9 clears that bar easily every time. The volatile arms usually fail. So success conditioning works? It works perfectly. It sees the special arm is reliable and learns to pick it. Okay, so far so good. The system is working. But now, let's get greedy. Let's set the threshold to 0.95. Okay, super elite. We only want the absolute best. We want A pluses only. Well, the special arm only gives 0.9. It never reaches 0.95. It is consistent, but it has a ceiling. So your filter throws away every single instance of the best arm. Oh no, it filters out the competence because it wasn't perfect.

16:59Exactly. But the volatile garbage arms, once in a blue moon, they spike to 1.0. So the only data left in your success pile are the lucky spikes from the bad arms. So the model looks at the data and says, wow, these volatile arms are amazing. They're the only ones that win. And there you go. You have success conditioned your AI to be a gambling addict. You taught it to count on luck rather than skill. You broke the alignment. You aren't optimizing for the best average score anymore, you're optimizing for the highest variance. And this ties back to that action influence concept. By raising the threshold, you artificially inflated the influence.

17:35You made the distinction between success and failure incredibly sharp. But in doing so, you disconnected it from reality. This feels like a massive lesson for corporate strategy too. If you set your promotion metrics so high that only a lucky break can reach them, you lose your consistent performers. That is a perfect analogy. If you only promote the sales people who land the one in a million giant deal and fire the ones who consistently brine steady revenue, you end up with a company full of gamblers who eventually go bust. So success conditioning works best when the definition of success is faithful to what you actually want.

18:10Yes. If you start manipulating the threshold to force improvement, you might just be amplifying noise. You're teaching the model to play the lottery. You want to learn from skill, not from outliers. Precisely. So let's bring this all together. We started with the idea that copying success feels like a cheap hack. You know, fake it till you make it. And we found out it's actually a rigorous conservative optimization method. It uses that chi-square geometry to say, find the best version of what we already know how to do. Don't wander off into the unknown. It fundamentally shifts the philosophy from explore everything to exploit what works.

18:46Yes. It's about mining your own history for gold rather than digging new mines. And we learned about the action influence identity. That improvement is just, it's inextricably linked to how much your choices actually change the outcome. If your choices don't change the probability of success, you can imitate the winners all day long and you won't learn a thing. That's the takeaway that's going to stick with me. We all want to improve. We all look at successful people or successful companies and try to copy them. But you have to ask yourself, are you copying the things that actually cause the success?

19:20Or are you just copying the random noise, the left-handed card flip that happened along the way? That is the ultimate question. Success conditioning is powerful, but only if you know which actions actually influence the result. Otherwise, you're just wearing a turtleneck because Steve Jobs did. Exactly. And that won't make you an iPhone. A lot to think about. Next time you're looking at your own successes, maybe check your action influence before you start patting yourself on the back. Thanks for diving in with us. Always a pleasure.