


25 Jun 2025 · 17m · 12 chapters

The episode discusses a research paper arguing that today’s large language models have fragile, non-generalizable reasoning because supervised pretraining entangles “knowledge” with “how to reason.” It claims general intelligence (AGI-like) requires reward-based pretraining (RPT) that disentangles reasoning from knowledge, inspired by AlphaGo vs AlphaZero. It highlights overfitting/memorization in SFT and proposes RL on synthetic tasks plus an architecture with small “working memory” and learned external memory access (more active than typical retrieval-augmented generation).
No guests are named; only authors are mentioned: Sungwakan, Joethja Perry, Samuel Gershman, and Polkett Agrawal.
SFT teaches token correlation shortcuts; in unfamiliar environments models fail. RL-trained models generalize better. Starting RL from scratch can outperform SPT→RFT.
Brainfruc and Befunge tests (low accuracy for LLaMA/GPT-4o; O1 much higher). Go experiments (pure RPT 100% vs SPT; 66–92% vs SPT→RFT). Synthetic vector orthogonality task (RPT ~93% test vs SFT→RFT 80% test). Proposed memory/working-memory decoupling to reduce spurious correlations and “lost-in-the-middle.”
Written by AI. May contain mistakes.
Discuss the limitations of large language models in reasoning and knowledge transfer.
“So, starting with the problem they identify.”
Examine how LLMs perform on unconventional programming languages to test their reasoning abilities.
“It shows the limits of relying on those learned correlations.”
Introduce the concept of reward-based pre-training and its potential benefits over traditional methods.
“Stop mushing them together during pre-training.”
Present experimental results that favor reward-based learning for better generalization.
“And did they run experiments to back up this RPT idea?”
Discuss strategies to implement reward-based pre-training effectively in AI.
“The SF key part encouraged memorizing the specific examples, and it didn't generate those intermediate reasoning steps needed to apply the concept broadly.”
Explore proposed changes in AI architecture for better reasoning and memory management.
“It suggests that, yes, meaningful circuits already exist within the architecture, potentially ready to be activated by the right kind of training, like these synthetic tasks.”
Explore how reinforcement learning stabilizes memory access and fosters efficient strategies.
“And using RL to train those memory access decisions keeps it stable.”
Discuss the significance of moving towards general intelligence in AI systems.
“There must be counter arguments or challenges, right?”
Examine the intertwined nature of knowledge and reasoning in AI and the challenges it presents.
“It's often much slower and more computationally intensive than supervised training.”
Identify the hurdles in RL training, including computational costs and task design.
“Picking the right pre-training tasks is crucial and, as the paper says, an important subject of future research.”
Consider how reward-based pre-training may redefine intelligence in AI.
“And then architecturally separating knowledge and reasoning using a small working memory paired with a learned interaction with an external knowledge base.”
Contemplate the new forms of intelligence that could emerge from AI learning to reason independently.

Get the gist, explore an idea or ask about something said.
On Best AI papers explained, what are the key points of this episode?
Automatic transcript. May contain errors.
0:00Welcome to this deep dive. We're jumping into something that's, well, everywhere lately. AI, specifically large language models. Right. They're incredibly impressive. Totally. They've shown some amazing real world uses, haven't they? Like they've become this sort of artificial useful intelligence, you know, helping with reasoning, math, coding, everyday stuff. Definitely useful AUI as some call it. That raises the big question. Exactly. Is it real intelligence? Is it like general intelligence, AGI? That's the heart of it. That gap between the useful AI we have now, the AUI, and true artificial general intelligence, AGI.
0:38Which would be systems that can reason adaptively, robustly, across all sorts of different areas, kind of like humans do. Precisely. And that brings us to this new research paper we're looking at today. It's titled, General Intelligence Requires Reward-Based Pre-Training. The authors are Sungwakan, Joethja Perry, Samuel Gershman, and Polkett Agrawal. OK, and their core idea sounds pretty radical. They're basically saying to get from AUI to AGI, we need a, quote, fundamental overhaul of the pre-training paradigm for these models. It's a bold claim, yeah. So that's our mission for this deep dive.
1:09We're going to unpack why today's LLMs, even the really smart seeming ones, can sometimes, you know, falter when it comes to genuine reasoning. We'll look at why they tend to overfit, just memorize the training data instead of truly learning. And explore how this proposed shift in training, plus some interesting architectural ideas, might be the key to unlocking AI that can actually generalize. Ready to get into it? Let's do it. So, starting with the problem they identify. Okay. While these LLMs are great at many things, their ability to reason in a flexible, adaptive way is still, well, the paper calls it fragile.
1:47Fragile. And the root cause, they argue, is this tight coupling of reasoning and knowledge inside the models. Meaning the what it knows and the how it thinks are kind of tangled together. Exactly. And because they're tangled, the reasoning part doesn't transfer well. It struggles when faced with something genuinely new or different from its training. Right. And that ties back to how they're mostly trained now, the supervised pre-training or SPT. Yes, predicting the next token, the next word, based on trillions of words scraped from the internet. Just passively absorbing text. Like reading the whole internet and learning to guess what comes next.
2:23But that passive data, it often doesn't show the steps of reasoning, does it? Just the final answer. Precisely. That's a huge part of the problem. The intermediate steps are often missing. So the risk is, what, that they just learn statistical tricks? Yeah, they learn to exploit correlations between tokens, seeing patterns, just to get better at predicting the next word without actually understanding the underlying logic or algorithm. Finding the shortcut, not the actual path. You got it. The paper had some great examples of this, like basic math. Five plus three will say it. The model might just learn that number plus number often leads to an equal sign and another number, not what addition means.
3:03Right, or the Python indexing example you mentioned. Oh, yeah, the one-based versus zero-based indexing. I remember struggling with that. It's a classic beginner hurdle. But if you tell a model, use one-based indexing for this task, which is unnatural for Python, many just revert to the memorized zero-based patterns. So it's just pattern matching, not really implying a rule flexibly. Exactly. It shows the limits of relying on those learned correlations. Okay, so to really test reasoning without relying on memorized knowledge, the researchers got creative. They did. They designed tests using these esoteric programming languages.
3:36Esoteric. Love that. BrainFruc and Befunge, what's special about them? Well, they're designed to be really unconventional. Their rules have almost no connection to languages like Python or Java that the models would have seen in their training data. Ah, so no pre-existing knowledge to fall back on. None. BrainFruc is super minimal, just eight commands operating on a tape. Befunge is even stranger. It's like 2D code executes up, down, left, right. Whoa, okay, so totally alien environments for these LLMs. They have to reason from the ground up using just the rules provided. That's the idea. Pure reasoning test.
4:11So how did the big names do? LAMA, GPT-4-0. The results were, well, pretty sobering. Even on simple tasks like printing two letters or sorting a tiny list, the performance was poor. How poor? We're talking around 12 % accuracy on average in brain foam. Maybe 29 % in Befunge. Ouch. That's low. Really low. And what's striking is even giving them examples within the prompt in-context examples barely move the needle. Performance only marginally improves. Wow. Okay, so just seeing an example wasn't enough for them to grasp the underlying logic in these unfamiliar systems. It really highlights that fragility you mentioned.
4:49It does. It suggests that when you strip away the familiar patterns and knowledge, the core reasoning ability is weaker than it might appear. But there was one model that did way better, right? The O1 model. Yes. That's a really important piece of the puzzle. O1 stood out big time. What were its scores like? Much higher. Like 71 % on one brain fur task, 95 % on another. It vastly outperforms the others. And the key difference for O1 was? It had gone through a lot of post-training using reinforcement learning, specifically for solving reasoning tasks. Ah, okay. So it wasn't just pre-trained on text.
5:22It was explicitly trained to reason using RL. That feels like a major hint. It's exactly the hint the paper leans into. It points towards their proposed solution. Reward-based training. Which brings us to their main argument. To get to AGI, we need to deliberately disentangle knowledge from reasoning. That's the core hypothesis. Stop mushing them together during pre-training. And the way to do that, they propose, is this fundamental shift towards reward-based pre-training or RPT. Right. Learning reasoning skills through rewards, not just predicting the next word. Okay, this idea of disentangling is crucial.
5:59The paper uses an analogy I found really helpful, AlphaGo versus AlphaZero. Can you walk us through that? Yeah, it's a great analogy. So think about the standard LLM training pipeline. Supervised pre-training SPT first, then maybe some reinforcement learning, fine-tuning, RFT. Yeah, learn from text, then refine with RL. That STT then RFT approach is very similar to how the original AlphaGo was built. AlphaGo was first trained by studying millions of games played by human experts. That's the supervised part. Then it improved by playing against itself using RL. Right. So it learned from human examples first.
6:33Its foundation was human play. Exactly. But then DeepMind created AlphaZero. And AlphaZero was different. How was it different? It learned by purely through self-play reinforcement learning, right from scratch. No human games were used in its initial training. Okay, just learned by playing itself, figuring out the rules and strategies. And the result. AlphaZero didn't just match AlphaGo. It completely surpassed it. It discovered entirely new, sometimes more creative, definitely more effective strategies that humans hadn't thought of. Wow. Okay. So the parallel the paper draws is this. Maybe relying heavily on supervised tree training on human text data actually limits the LLM.
7:12It traps it in a kind of local minimum of reasoning ability based on human patterns. Like it learns our ways of thinking, or at least the patterns in our text, and then the later RL fine-tuning can't fully break out of that mold to find potentially better, more general reasoning strategies. That's exactly the hypothesis. The initial SPT constrains what the later RFT can achieve. It hinders that deep exploration needed for truly general reasoning. That makes a lot of sense conceptually. But, you know, humans do learn a lot from observation and examples initially. How does the paper counter that?
7:46And did they run experiments to back up this RPT idea? Good question. They're not saying supervised learning is useless, just that making it the primary pre-training step might be the bottleneck. And yes, they ran experiments. Like what? They tested it directly in Go on a smaller 9x9 board to make it feasible. They trained one model purely with RPT, just reinforcement learning from the start. Okay. That RPT model achieved a 100 % win rate against a model trained only with supervised learning, SPT. Total domination. And interestingly, against models trained the standard way SPT than RFT, the pure RPT model still won most of the time, like 66 % to 92 % of the games.
8:23So even adding RL fine-tuning after supervised learning wasn't as good as just starting with RL. Correct. It strongly suggests that the initial supervised phase really does, as they say, constrain the RL fine-tuning and hinder exploration. The model gets stuck in patterns learned from the passive data. Okay, that's compelling evidence from Go. What about that synthetic math task they mentioned, the vector orthogonality thing? Right, identifying perpendicular vectors. That also showed a fascinating difference. Oh, so? A model trained with pure reinforcement learning fine-tuning, starting from scratch, essentially pure RPT for that specific task, got around 94 % accuracy on the training data and, crucially, 93 % on the test data.
9:07Very good generalization. Okay. Learned the concept well. What about the standard approach? The model trained with supervised fine-tuning first, then RL fine-tuning, SFT, then RFT, actually hit 100 % on the training data. Perfect memorization. Exactly. But it only achieved 80 % on the test data. Ah, there it is. The classic overfitting signature. It memorized the examples but didn't grasp the underlying rule as well. Precisely. It confirms their idea. SFT memorizes while RL generalizes. The SF key part encouraged memorizing the specific examples, and it didn't generate those intermediate reasoning steps needed to apply the concept broadly.
9:45Okay, so the evidence seems to point towards reward-based learning being better for generalization. But how do you actually do reward-based pre-training for something as complex as language? It sounds incredibly hard. It is hard. Training RL from scratch in the massive open-ended space of language tokens is, well, the paper calls it potentially infeasible. Right. Where would you even start? So they proposed two main directions to make RPT practical. The first is don't start with full natural language. Okay, start where? Start by pre-training on synthetic tasks or games in constrained environments.
10:19Think simpler worlds with fewer possible actions, clearer rules, a smaller token space. Like learning the basic building blocks of reasoning first. Exactly. Using these synthetic tasks lets models acquire core skills, hierarchical thinking, cause and effect, putting concepts together without getting bogged down by the messiness and ambiguity of natural language. So you isolate the reasoning process itself. Right. You remove the ambiguities and semantic dependencies, letting the model focus purely on applying rules and logic. Is there evidence this kind of pre-training helps? Yes, actually. Think about code.
10:53Pre-training models on computer code is known to boost their logical reasoning and problem-solving skills. That makes sense. Code is like language, but with very strict rules. It's a great example of an artificial environment that parallels language. And it's like how learning programming helps kids develop thinking skills. Plus, there's even research showing that randomly initialized transformers, before any real training, already have some capacity to solve algorithmic tasks. Really? Like the basic architecture has some innate logical structure? It suggests that, yes, meaningful circuits already exist within the architecture, potentially ready to be activated by the right kind of training, like these synthetic tasks.
11:35Okay, so direction one is pre-training on simpler synthetic tasks to build a reasoning prior. What's direction two? Direction two involves rethinking the model's architecture to specifically decouple knowledge and reasoning. How? Well, current models often use huge context windows, trying to cram in as much information as possible. Yeah, that seems like a good thing, right? More context, better understanding. Not always. It can lead to problems. Models might learn spurious correlations just because things appear together in a long text window. Or that lost-in-the-middle issue where they ignore info buried in the middle of a long prompt.
12:12Right, I've heard of that. Too much information becomes noise. Exactly. And this is where cognitive science offers some clues. Think about human working memory. Miller's magical number, like four plus or minus. Wait, used to be seven. Now maybe closer to four chunks of information we can actively juggle. Something like that, yeah. The idea of a limited working memory being efficient. Or how we use chunking. The paper suggests maybe AI could benefit from a similar constraint. So restricting the amount of information the reasoning part focuses on at any one time. Precisely. Restricting the model to reason over only a limited set of tokens reduces the chances of exploiting spurious correlations.
12:52Okay, so what does the proposed architecture look like? They suggest separating the model into distinct parts, a core reasoning network that operates with a deliberately small context window, like our working memory. And the knowledge, where does that live? In a separate external memory bank. The key is that the reasoning network learns strategies for actively reading from and writing to this external memory. Okay, so it pulls in only the knowledge it needs, when it needs it, into its small work Exactly. How is that different from retrieval augmented generation or ARGO, which lots of systems use now?
13:28ARGO also pulls in external knowledge? It's a good question. The distinction is important. Traditional ARGO often does a one-time retrieval, stuffs that retrieved info into the LLM's potentially very long context and lets the LLM figure it out. This proposal is more dynamic. It involves potentially multi-round interactions with memory. The reasoning network learns, the RL, when and what to read or write. It's making active, discrete decisions about memory access, managing its small working memory. So it's a much more active, learned memory management process, not just passive retrieval. And using RL to train those memory access decisions keeps it stable.
14:07That's the idea. Avoid some instability issues seen in older attempts at differentiable memory. It's about learning efficient memory interaction strategies. Okay, this is fascinating. So stepping back, why does this matter to, you know, us, the listener? What's the big picture here? Well, it's really about the path forward for AI. Are we just building ever larger pattern matching machines? Or are we trying to build systems that can genuinely think, adapt, and reason, especially when faced with something totally new? Moving beyond just being incredibly fluent, Mimics. Exactly. Towards something more like general intelligence.
14:41Of course, this is a research proposal. There must be counter arguments or challenges, right? The paper acknowledges some. Absolutely. A major one is that some people believe knowledge and reasoning are just too fundamentally intertwined in humans to be effectively separated in AI. Like that coffee example. The coffee was unbearably hot, so he poured it into it. Yeah. You need knowledge about heat, cups, pouring, and reasoning to complete that sentence sensibly. Right. It feels seamless. But the paper's counter is that even if they can be tightly coupled for efficiency in familiar scenarios, an agent needs the ability to separate them, to apply reasoning generally, in unfamiliar contexts.
15:21It needs that flexible mechanism. So the capability needs to be there even if it's not always used explicitly. Makes sense. What about practical hurdles? A big one is just the sheer cost and difficulty of RL training. It's often much slower and more computationally intensive than supervised training. It needs to gather data through interaction, which takes time. Yeah, RL can be notoriously sample inefficient. And then there's the challenge with those synthetic tasks we talked about. Right. How do you design the right tasks? Exactly. There's a risk of creating tasks that are too narrow, too specific, and the skills learned just don't transfer well to the complexity of natural language or the real world.
16:01Picking the right pre-training tasks is crucial and, as the paper says, an important subject of future research. Definitely an open question. For sure. Okay, so let's bring this all together. We started with the amazing usefulness of current LLMs, this AUI. But quickly saw their limitations, that fragile reasoning and tendency to just memorize or overfit, especially highlighted by those esoteric language tests. Which led us to the core proposal, this fundamental shift to reward-based pre-training, RPT, inspired by that alpha-go-to-alpha-zero jump, the idea of learning reasoning from scratch through rewards.
16:37And we looked at the two key ways to make that feasible, using synthetic tasks first to build a solid reasoning foundation. And then architecturally separating knowledge and reasoning using a small working memory paired with a learned interaction with an external knowledge base. Right. It paints a picture where the future of AI might depend less on sheer scale and more on smarter, different training paradigms. So maybe the path to truly intelligent AI isn't just about making models bigger, but making them think differently. That seems to be the core message. And it leaves us with a really provocative thought to chew on.
17:11Yeah. If we succeed in building AI that learns to reason largely from scratch through its own trial and error and reward, perhaps less biased by the specific patterns in human language and data, what entirely new forms of intelligence, what novel ways of thinking might actually emerge?
This position paper argues that Large Language Models (LLMs), despite their current utility as Artificial Useful Intelligence (AUI), often struggle with robust and adaptive reasoning required for Artificial General Intelligence (AGI) because their training methods overfit to specific data patterns. The authors propose a shift from the current supervised pretraining (SPT) paradigm to reward-based pretraining (RPT), similar to how AlphaZero surpassed AlphaGo by learning purely through reinforcement. To achieve this, they suggest training on synthetic tasks with reduced token spaces to foster generalizable reasoning skills and decoupling knowledge from reasoning through an external memory system. This proposed architecture would allow the reasoning module to operate with a smaller context, relying on learned retrieval mechanisms for information, thereby promoting more robust generalization across novel domains.