1. Podcasts
  2. Best AI papers explained
  3. Training Foundation Models with a Single Algorithm
Best AI papers explained

8 Nov 2026 · 23m · 12 chapters

Episode artwork

Training Foundation Models with a Single Algorithm

In short

The episode explains “unified training” for large foundation models, replacing staged training (pre-training, mid-training, supervised fine-tuning, RL) with one continuous algorithm. Key claim: keep the same learning math throughout; only the data distribution changes over time via a “data diet.” It uses MaxRL (maximum likelihood reinforcement learning) as a bridge so RL rewards can be translated into the same likelihood-loss format as static text, while filtering to keep only high-scoring RL generations.

Notable examples

experiments with a 124M-parameter model on 12-person logic puzzles (reducing catastrophic forgetting and improving generalization under the same compute); scaling to 0.76B on Math1.2K and HumanEvalPlus, showing a trade-off where more mid-training data improves math/science but can hurt coding unless schedules like half-annealing are used.

Guests

No specific guest names are provided; the episode is a host-led discussion.

Written by AI. May contain mistakes.

Chapters

0:45

The Illusion of AI Progress

Discussion on the public's perception of AI versus its actual development process.

“It's this rigid, clunky, highly compartmentalized assembly line.”
1:58

Understanding Staged Training

Explaining the conventional staged training approach for AI models.

“So in the conventional setup, what researchers call staged training, models learn in very distinct phases, right?”
3:19

Limitations of the Assembly Line

Critique of the limitations and flaws of the assembly line training model.

“Each of these stages uses entirely different training recipes and completely different mathematical objectives.”
5:07

Introducing Unified Training

Overview of the unified training approach and its significance.

“before locking the biology door forever, It turns the entire life cycle of an AI into one fluid optimization problem.”
7:11

The Role of MaxRL

Explaining the breakthrough of Maximum Likelihood Reinforcement Learning and its implications.

“Maximum likelihood reinforcement learning.”
8:43

Experimenting with Unified Training

Details on the experiment conducted to test unified training with logic puzzles.

“You have pristine static text documents and fresh verified trial and error successes sitting in the exact same digital waiting room.”
11:32

The Success of Unified Training

Results of the unified training model showing better retention and generalization.

“And in doing so, it actually overwrites the neural pathways that store its broader general knowledge.”
14:02

Understanding KL Penalty in AI Training

Explore how KL penalty affects model learning and knowledge retention.

“Meaning a mathematical rule that penalizes the model if its internal weights drift too far from its original pre-trained state.”
14:48

Unified Training's Approach to Data Diet

Learn about the role of data diet in improving AI training outcomes.

“It proved that a proper data diet is far superior to arbitrary mathematical penalties.”
16:44

Balancing Data for Optimal AI Training

Discover the delicate balance needed in AI training schedules for performance.

“It is a very delicate balancing act, and this tension led the researchers to experiment with different schedules.”
Show all 12 chapters
19:08

The Philosophical Shift in AI Development

Understand the significant change in how AI models are built and trained.

“Instead of a giant algorithmic assembly line headache, we now have a giant scheduling headache for the engineers.”
21:13

The Future of Continuous Learning in AI

Consider the implications of perpetual learning and adaptability in AI systems.

“Instead, we are moving toward a single, elegant learning engine.”
A man walking through an airport, asking his phone a question

Ask about this episode

Get the gist, explore an idea or ask about something said.

Claude

On Best AI papers explained, what are the key points of this episode?

Searching your listening

Transcript

Automatic transcript. May contain errors.

0:00You know, when you picture the cutting edge of artificial intelligence, you probably imagine something, I don't know, incredibly fluid. Right. Yeah. Like the seamless digital brain. Exactly. Just absorbing information from the ether, constantly evolving, drinking in the universe in real time. That is definitely the popular imagination. I mean, we're so conditioned by science fiction to think of machine intelligence as this, you know, organic, continuous awakening. Right. But then you actually look under the hood at how these massive AI models are built today. And that beautiful illusion just completely shatters.

0:36Yeah, it really does. Because it doesn't look like a sci-fi brain at all. It looks more like a 1920s car factory. That is a very grounded way to view it, actually. It's this rigid, clunky, highly compartmentalized assembly line. Right. The industry standard right now is deeply segmented. We build these systems in very isolated stages. And once a model leaves one stage, the door basically locks behind it. And that assembly line approach is exactly what we are dismantling today. We are doing a deep dive into some cutting-edge research coming out of institutions like Cardinal D. Mellon and Johns Hopkins.

1:11And they are proposing a radical, singular solution to this rigid factory model. Yes, they call it unified training. Unified training. And our mission for this deep dive is to understand exactly how breaking down those factory walls might be the key to creating smarter, far more adaptable AI. It's a massive philosophical shift, really. A complete structural shift in the underlying algorithms. And more importantly, we want to talk about why you, listening to this right now, should care about this, because this is going to fundamentally change how the tools you use every single day are engineered.

1:43Absolutely. But to really appreciate how radical this new unified approach is, we first have to understand the limitations of the assembly line we're currently stuck with. Okay, yeah, let's unpack this properly, because if we're tearing down the assembly line, we need to know what the stations are. So in the conventional setup, what researchers call staged training, models learn in very distinct phases, right? Like moving down a conveyor belt. Let's walk through it. So phase one is pre-training. Right. This is the brute force phase where the model reads unfathomable amounts of raw internet text.

2:16I mean, it's not learning facts per se. It's absorbing the statistical patterns of human language. So it's just learning grammar and syntax by reading like a trillion web pages. Exactly. Then the conveyor belt moves to mid-training. Now, mid-training is a bit of a blanket term, but here the data is highly curated. Like textbooks and stuff? Yeah, think high-quality educational textbooks, advanced mathematics, very specific domain knowledge. We want the AI to learn actual facts and logical structures here, not just internet chatter. Okay, got it. Then we hit supervised fine-tuning, or SFT, where human engineers provide explicit step-by-step demonstrations of tasks, like showing it how to summarize an email.

2:56Right. And finally, the model reaches the end of the line, which is reinforcement learning, or RL. The trial and error phase. Yes. That last stage is crucial. The model is given a prompt, it generates answers on its own, and a verifier rewards it for good answers and penalizes it for bad ones. But, and this is the big but, here is the critical flaw in the assembly line. Right. Each of these stages uses entirely different training recipes and completely different mathematical objectives. Exactly. And this is where unified training steps in and just flips the table entirely. This new paradigm argues that this algorithmic separation isn't actually necessary.

3:36Wait, not necessary at all? Not at all. It proposes using a single learning algorithm from the very first second of training to the very last. So the math doesn't change? In a unified framework, no, the math doesn't change. The algorithm doesn't change. The only thing that changes over time is the evolving distribution of the data. You know, what the AI learns from and in what proportions. Okay, let me try an analogy here. The staged assembly line feels like a highly rigid high school curriculum. Okay, I'm tracking. First you take biology for a year. Then you drop it completely, never look at a cell again, and take chemistry for a year.

4:11Right. But unified training sounds like dropping a kid into a giant interactive museum where they can absorb the biology exhibit, practice a trial and error chemistry experiment, and read a history book all concurrently. You're on the right track, yeah. But I'd add that it's an interactive museum where the curator is constantly adjusting which exhibits you spend time in based on how well you're learning. Oh, interesting. But I have to push back here because that sounds overwhelming. Doesn't mixing it all up just cause total chaos inside the neural network? It sounds like it would, yeah. Right, because if you're trying to passively read a textbook while simultaneously doing stressful trial and error chemistry experiments, doesn't the digital brain just overload?

4:52It's a very logical concern. But what the mathematics reveal is that it doesn't cause chaos at all. In fact, it dramatically simplifies the science of training. Simplifies it? How? Well, instead of obsessing over stage-specific metrics, like trying to figure out if it memorized enough biology before locking the biology door forever, It turns the entire life cycle of an AI into one fluid optimization problem. Okay. The core challenge becomes entirely about balancing what we can call the data diet. The data diet. That makes total sense. But if we are going to understand how this diet is digested, we really need to look at the underlying math.

5:30Because passively reading static text from a data set and actively learning via trial and error, those usually require completely incompatible mathematical languages. They absolutely do. Traditionally, pre-training and mid-training rely on a mathematical objective called Maximum Likelihood Estimation, or MLE. Okay, let's break MLE down for the listener so it doesn't just sound like alphabet soup. Good idea. Maximum Likelihood Estimation is basically a giant game of fill in the blank, right? Essentially, yes. The AI looks at a sentence, let's say the cat sat on the, and has to guess the next word.

6:06If a guess is Matt and the actual text says Matt, it strengthens the internal neural weights that led to that guess. That's a perfect summary. It's fitting itself to a static, unchanging target. It's learning the probability distribution of static data. Right. But reinforcement learning, RL, is an entirely different beast. In RL, there is no static target. The AI generates a completely novel response to a prompt. It's just making something up from scratch. Exactly. And then a verifier grades it, like did it solve the math problem correctly? Yes or no? The math used to update the model based on that binary reward is called policy optimization.

6:48Got it. It is about maximizing expected rewards over time, not filling in blanks. Historically, trying to run both of those math functions, MLE and policy optimization, at the exact same time inside a model is what crashes the system. Yes. It's like trying to run an engine on both gasoline and diesel simultaneously. But this new research introduces a mathematical bridge called MaxRL. Right. Maximum likelihood reinforcement learning. This is the big breakthrough. MaxRL is basically the mathematical translation layer that allows RL, that reward-based trial and error, to be expressed in the exact same mathematical format as MLE.

7:27So wait, instead of the AI physically having to switch its underlying architecture from reading mode to practice mode, Max RL acts like a universal translator. That's a great way to put it, yeah. It takes the successful practice mode trial and error generations and translates them into the exact same fill-in-the-blank format as reading a book. You've got it perfectly. When the model generates a successful answer during RL, MaxRL essentially takes a snapshot of that generated answer and treats it as if it were a brand new piece of static text. Oh, wow. Yeah, it calculates the likelihood loss for that generated text, just like it would for a line from a Wikipedia article.

8:06But wait, I'm stuck on this single buffer idea. If you throw high-quality textbook data into the exact same blender as random trial-and-error AI guesses, doesn't the AI end up learning from its own garbage? Well, it would if it translated everything, but the crucial mechanism here is the filter. Okay. Max RL only takes the successful trial and error generations, the ones that scored highly with the verifier, and translates those into the buffer. The garbage is discarded. Oh, wow. Okay, and if we connect this to the bigger picture, that is a massive engineering win. This means you can have a single shared data buffer.

8:44Exactly. You have pristine static text documents and fresh verified trial and error successes sitting in the exact same digital waiting room. And because they are in the same waiting room, generation and optimization can run entirely independently. Which is huge. It is. You have generators off to the side, asynchronously spitting out new responses, grading them, and throwing the successes into the buffer. Meanwhile, the trainer, the core learning engine, is just constantly sampling from this mixed buffet. A little bit of static mid-training data, a little bit of fresh RL generation. Right. All processed through one single mathematical function.

9:22It's an all-you-can-eat buffet where everything, whether it's a history book or a newly generated math proof, is instantly translated into the exact same nutritional format for the AI's brain. It's incredibly elegant. But as you know, elegant theory and slick mathematics are only half the story in AI development. Definitely. A theoretical universal translator is great, but what happens when you actually torture test it in a lab setting? If this unified diet is going to replace the assembly line, it needs to prove it can build a better brain. The researchers knew this, so they set up a highly controlled experiment.

9:56They trained a relatively small 124 million parameter model to solve logical reasoning puzzles. Okay, these are the classic Einstein-style logic puzzles, right? The ones where you get those convoluted clues like Zoe lives three floors above Ella and Ethan lives 10 floors above his best friend. Yes. And you have to figure out exactly who lives on what floor in a massive tower. Exactly. And solving them requires complex multi-step deduction. Now, in the traditional stage training approach, the modeled first reads thousands of what are called oracle-generated reasoning traces during the mid-training phase.

10:31Okay. Let me stop you right there. Oracle-generated reasoning traces. For the listener, what exactly is that? Are we just talking about an answer key? Essentially, yes. An oracle in this context is just a perfect solver, usually an algorithmic script or human expert, that provides the flawless step-by-step logical deductions required to solve the puzzle. Got it. So during mid-training, the AI is just passively reading these perfect answer keys. It learns the basic patterns of deduction. But then the assembly line moves. The door locks. Exactly. It moves entirely to the RL phase where it has to solve novel puzzles purely through trial and error, and it is completely cut off from those static answer keys.

11:11And here is where the assembly line fundamentally breaks down. Because when the traditional AI focuses entirely on those RL logic tasks, it suffers from catastrophic forgetting. Yes. Catastrophic forgetting is one of the biggest bottlenecks in AI right now. It actually starts losing the broader capabilities and general knowledge it acquired during its earlier training. Because traditional RL is aggressively optimizing for the immediate reward, solving the specific puzzle in front of it. And in doing so, it actually overwrites the neural pathways that store its broader general knowledge. The model becomes hyper-specialized, but incredibly brittle outside of that specialty.

11:47If you are listening to this right now and thinking, why do I care about neural pathways getting overwritten? Think about the last time you gave an AI a really long, complex prompt with a bunch of specific rules. and halfway through its response, it just completely forgot the instructions you gave it at the start. That is a very practical manifestation of catastrophic forgetting. The AI hyper-focuses on the current sentence and loses the broader context. But here's where it gets really interesting to me. If I, personally, spend a month doing nothing but Sudoku puzzles, I don't suddenly forget how to speak English.

12:20Right. And I don't forget who the president is. Why does the AI forget? Because your brain has billions of years of evolution prioritizing stability. An AI model doesn't. Its weights are entirely fluid. If you don't reinforce a piece of knowledge, the math will happily cannibalize that space for whatever task is currently yielding rewards. Wow. But unified training prevents that cannibalization? It does. By mixing the mid-training data, those static reasoning traces, directly into the buffet during the RL phase, The unified model didn't just retain its general pre-training knowledge better. It actually improved.

12:57Yes. It achieved noticeably higher accuracy on the logic puzzles themselves. It's like keeping a dictionary and an encyclopedia on the desk while doing the Sudoku. It keeps those general pathways active and firing. Exactly. And the research shows it goes beyond just memory retention. They tested these models on 12-person logic puzzles, puzzles that were significantly larger and structurally more complex than anything the models had seen in either the mid-training data or the RL data. Which is the ultimate test in AI, right? True generalization. Can the model extrapolate its reasoning skills to a scale it has never encountered?

13:31Yes. And the unified model crushed it. It generalized far better than the staged model. And crucially, it did this within the exact same compute budget. That's a key point. We're talking about the same number of flops, floating point operations. It didn't use a massive supercomputer. It just used the data more efficiently by mixing it. I should mention, too, that the industry has tried to fix catastrophic forgetting before using traditional mathematical band-aids. In this experiment, they tested something called a KL penalty, Kuhlbach-Leibler regularization. KL penalty. Meaning a mathematical rule that penalizes the model if its internal weights drift too far from its original pre-trained state.

14:10Like a teacher slapping its wrist and saying, don't change too much or you'll get a penalty. That's a very good way to put it. But what they found was that while the KL penalty successfully prevented the model from forgetting its general knowledge, it severely degraded the model's ability to actually solve the logic puzzles. Oh, really? Why? Because you are mathematically punishing the model for changing its mind. You essentially paralyze the learning process. So forcing the math to stay static completely failed. But Unified Training's approach, managing the data diet by constantly feeding it a mix of mid-training examples and fresh RL, that succeeded beautifully.

14:47It did. It proved that a proper data diet is far superior to arbitrary mathematical penalties. It is a fundamentally more organic solution. Right. However, you know, logic puzzles in a controlled lab are perfect, pristine environments. Yeah. A sterile lab is basically a parlor trick. If this unified diet is actually going to replace the multi-billion dollar assembly lines at major tech companies, how does it handle the absolute chaos of real world data? Right. To test this, they scaled up significantly. They used a larger 0.76 billion parameter model. And instead of pristine logic puzzles, they trained it on massive, messy real world data sets, specifically advanced mathematics from a data set called Math 1.2K.

15:33and complex computer coding benchmarks like Human EvalPlus. Math and code. Two things that require extreme precision, but they use very different types of logic. Exactly. And this is where they discovered the fascinating and frankly somewhat frustrating trade-offs of the data mix. Uh-oh. They found that increasing the proportion of mid-training data in the unified buffet predictably improved the model's mathematical coverage. It also preserved its general knowledge beautifully. It scored much higher on general science and trivia benchmarks. Awesome. Win-win. You feed it more textbooks, it gets better at math and science.

16:07Well, not quite a total win. Because while the math and general knowledge went up, that exact same heavy mid-training diet visibly dragged down the model's coding accuracy. Wait, why does that happen? Why does reading a science textbook mean the AI worse at writing Python? Because coding isn't about memorizing facts. It's about executing a strict, highly sequential logic. If the AI's neural pathways are too saturated with factual text and trivia from the mid-training diet, it dilutes the strict, rule-based reasoning it needs to write clean, executable code. Ah, I see. It's like a seesaw. The more it focuses on memorizing the encyclopedia, the less mental bandwidth it has for logical execution.

16:50It is a very delicate balancing act, and this tension led the researchers to experiment with different schedules. Remember, in unified training, you control what data the model sees, in what proportions, and most importantly, when. Right, the timeline of the diet. They tested several schedules. A constant mix, where the proportion of static data and RL data never changes. A linear decay, where the static textbook data slowly tapers off. A linear growth, where it slowly increases. And finally, something they called half-annealing. Half-annealing. Okay, I know annealing is a blacksmithing term. What does heating and cooling metal have to do with an AI's data schedule?

17:31The metaphor is actually quite literal here. In metallurgy, you heat metal to make it malleable so you can shape it, and then you let it cool down to lock in the crystal structure and give it strength. Got it. In AI, a highly mixed, diverse data buffet is the heat. It makes the neural pathways highly malleable and adaptable. Cooling it down means reducing that diversity so the model can lock in specific reasoning skills. Okay, so how does half-annealing apply to this diet? In half-annealing, they start with a healthy mix of mid-training data in the buffet alongside the RL generations. The model is heated and malleable, learning general facts.

18:10But exactly halfway through the total training time, they completely cut off the static mid-training data. Just cut it off completely? Completely. For the second half of training, the model only gets fresh RL trial and error data. It cools down and locks in its logical execution skills. Wow. Okay, so what does this all mean for the final result? I'm picturing an athlete training for a marathon. Oh, that's a good analogy. It sounds exactly like formulating the perfect carb loading schedule. You want to eat a lot of heavy pasta, the static mid-training data, early on to build base strength and general knowledge.

18:48But if you have too much heavy pasta right before the finish line, it slows down your final coding sprints. That is a brilliant way to conceptualize it, yeah. The half-annealing schedule allowed the coding accuracy to completely recover while retaining all the general knowledge gains from the first half of training. But wait, doesn't this mean unified training is just replacing one problem with another? Instead of a giant algorithmic assembly line headache, we now have a giant scheduling headache for the engineers. They have to figure out the exact perfect day to cut the carbs for every single model they build?

19:21That is the essential tradeoff, yeah. Finding the optimal data mix over time, the perfect schedule, is absolutely the new central challenge in this paradigm. But the ultimate payoff makes that scheduling headache entirely worth it. because the ultimate payoff is something called continual learning. Continual learning. As opposed to the model freezing, locking its weights, and never learning a single new thing once it leaves the factory floor. Exactly. In the traditional assembly line, once a model finishes RL, it is incredibly brittle. If you try to teach it a brand new task six months later, it completely falls apart.

19:59It forgets its old training. Right. But the research data shows that unified models proved significantly better at adapting to entirely new tasks later on. They tested this specifically, right? They took the models trained on math and code and suddenly threw them into a completely different general reasoning domain. They did. And the models that were raised on the unified data diet adapted to those new tasks far more successfully than the models built on the staged assembly line. Their foundational architecture was simply more robust, more plastic. Because they never had their brains partitioned into rigid stages.

20:34They were used to a dynamic, chaotic environment from day one. Exactly. The unified training provided a much stronger, more resilient starting point for continuous adaptation. So bringing all of this advanced math and blacksmithing metaphors together, what is the ultimate takeaway for you, the listener? Why does this matter when you open up your laptop or use an AI app tomorrow? It matters because it represents a massive philosophical shift in how machine intelligence is developed. We are moving away from building complex, multistage algorithmic pipelines that resemble factories. We are abandoning the idea that AI has to be built in pieces.

21:13Instead, we are moving toward a single, elegant learning engine. An engine where progress is dictated purely by the quality, the proportion, and the timing of the data diet. It means the AI powering the tools you use every day, whether it's helping you write a sensitive email, debug a massive piece of code, or analyze a complex spreadsheet, is going to become fundamentally less brittle. Absolutely. It will be far less prone to randomly forgetting the instructions you just gave it and far more adaptable to your specific evolving needs over time. It's the difference between a tool that is manufactured and a tool that is cultivated.

21:52Cultivated is the perfect word for this shift. We are finally learning how to cultivate intelligence by feeding it the right things at the right time, rather than trying to hammer it into shape across different assembly stations. Which leaves me with a brand new concept to chew on. A slightly crazy thought, actually, that wasn't explicitly tested in the documentation we looked at. But it seems like the logical next step. I'm intrigued. What's the thought? Well, if AI training is no longer this rigid sequence of stages that has a definitive end, if it's now just a matter of managing a continuous data diet through a single unified algorithm, what happens when we just leave it on?

22:29Leave the training on. Yeah. What happens if an AI is left on unified training indefinitely, constantly connected to real-time world data, constantly generating RL trial and error in a live environment, and constantly reading new static text as it's published on the Internet? Could we see a future where an AI model essentially never stops growing up? That is the ultimate profound implication of this research, a model that is in a state of perpetual, stable learning. It completely dissolves the boundary between training an AI and using an AI. It would just be a living system. It's completely wild to think about.

23:05I mean, we started this deep dive by looking at a rigid 1920s factory floor with strict stages and locked doors. But thanks to unified training, it looks like we might be tearing the roof off that factory completely, leaving us with an intelligence that just keeps feeding, adapting, and growing long after the assembly line has been dismantled.

From the publisher

Foundation models are traditionally trained through a sequential pipeline of distinct phases, including pretraining, fine-tuning, and reinforcement learning. Researchers demonstrate that this algorithmic separation is unnecessary by framing these diverse phases as maximum-likelihood fitting to evolving data targets under a unified training system. This approach integrates static corpora and model-generated responses into a single objective, transforming training design into a strategic problem of data composition, timing, and mixing proportions. Extensive experiments reveal that unified training improves task accuracy, compute efficiency, and generalization while providing a superior starting point for continual learning.

More from Best AI papers explained

All episodes