Emergent temporal abstractions in autoregressive models enable hierarchical reinforcement learning

0:00We are, I think it's safe to say, in the middle of a revolution in artificial intelligence. And it's really all being fueled by these massive autoregressive models. The ones that just predict the next token, the next word, the next action. Exactly. They are so powerful, so general. And they've given us this incredibly rich foundation for AI behavior. And that foundation is everything. Because once you have a model with that kind of deep understanding, the next logical step is to use reinforcement learning, or RL, to fine tune them. Right. That's how they learn genuinely new intelligent behaviors that weren't just, you know, copied from the training data.

0:39The pre-training gives them world knowledge, but RL, RLs gives them ambition. But there's this huge efficiency wall that we just keep running into. It's especially bad for tasks that need long-term strategy, where the reward is, well, SMARSE. SMARSE, meaning you only get that little good job signal maybe at the very, very end of a long chain of actions. And standard RL on these models, it forces them to explore token by token, just one tiny action at a time. Which is incredibly inefficient. I mean, think about it. If you need to string together 50, 100 precise microactions just to get one intermediate step right, the chance of random exploration stumbling on that exact sequence is, well, it's astronomically small.

1:19Statistically almost zero. People have quoted numbers like one in a million, or even worse for some of these compositional tasks. It's the classic bottleneck. It really is the difference between trying to navigate a city by optimizing the exact pressure on the gas pedal for the next millimeter versus just thinking, okay, I need to get on the freeway. Right. Humans and really any intelligent system, we operate using these high-level concepts, these subroutines. We use temporal abstraction. We think in chunks. Exactly. We use options like drive to work or make coffee. And our deep dive today is into a really radical new approach called internal RL.

1:54OK. The mission of this deep dive is to unpack research that seems to confirm a really powerful hypothesis that these giant autoregressive models, they already implicitly learn these abstract subroutines in their internal structures. And that we can leverage that for, I assume, massive efficiency gains. Exponential gains. That's the claim. OK, that is a huge claim. So let's unpack the foundation first. We're talking about models pre-trained from scratch on huge amounts of behavioral data. What were the models they were working with here? So the researchers looked at two pretty different architectures.

2:28And both were pre-trained just on sequences of observations and very low-level actions. Okay. For discrete tasks like a complex grid world. You can think of it like a 2D navigation puzzle. They use standard causal transformer models. Makes sense. But for continuous control, which is much harder in some ways, like controlling a four-legged ant robot in a physics simulator, they used a state space model, an SSM, specifically one called Hawk. And using an SSM for that continuous control seems smart. They're often better at really long sequences. But the key thing, I guess, is the training objective, right?

3:06These base models were only ever trained to predict the next token. That's it. Just the next raw, low-level action and observation, a pure sequence predictor. So no explicit instructions, no reward, no label telling it, hey, you're currently trying to go to the blue region. None of that. That high-level intent was completely missing from the loss function. And yet, and this is the major discovery, despite that low-level training, the models implicitly learn these robust representations of temporally abstract actions, of sub-goals. So where do these hidden plans, where do they actually live inside the network?

3:42They're in the model's residual stream activations. If you picture the model, the residual streams like this, the central information highway that runs straight up through all the layers. I see. All the information flows through it, and each layer modifies it a bit. And the hypothesis was that as the information gets deeper, the high-level plans sort of emerges. It crystallizes. Okay, so how did they prove that? How did they show it wasn't just noise, but an actual meaningful structure representing these subgoals? They use some really cool mechanistic interpretability techniques, specifically linear probing.

4:15Linear probing. They train these simple linear classifiers, think of them like tiny specialized translators, to try and decode the agent's current high-level sub-goal, like go red, directly from the activation vectors in that stream. And what did that tell them about where the plans lived? It was striking. The accuracy of decoding the correct sub-goal got better and better the deeper they went. Really? Yeah. It peaked right around the middle layers of the network, which tells us something really profound, I think. What's that? The early layers are just dealing with raw sensory data, immediate actions.

4:49But the abstract intent, the aha, I'm trying to get to the blue thing, that only really becomes a clean, stable signal deep inside the model. That's the knowledge piece of it. The model knows what it's trying to do, even if it wasn't explicitly told. But for RL, just knowing isn't enough. You have to be able to control it. Exactly. You need to be able to scare it. And this was the second absolutely critical proof they showed these representations were easily controllable. They introduced a simple low-rank linear controller. If you've heard of LoRa fine-tuning, a very similar idea, just a tiny lightweight adapter.

5:23Okay. And this little controller was designed to just nudge the activations in the residual stream at a specific layer. And by injecting a command, say the signal for go red, they could reliably steer the whole pre-trained model to execute the 50, 60 micro actions needed to get there. Wow. So you inject this high level command into the middle of the network's highway and the downstream layers just automatically translate that into the perfect sequence of low level actions. Precisely. And just like with the probing, they found that this steering was most effective when they put the controller in the middle layers.

5:57It's the point of maximum leverage. Too early and the signal gets garbled too late and the actions are already set in stone. You've got it. The middle layers are that sweet spot where the abstract plan is fully formed, but it hasn't been fully compiled down into micro actions yet. Okay, let's unpack this next part because we've confirmed the abstract actions exist and that they're controllable if we manually feed in the right label. But in a real world problem, we don't have those labels. The whole point is for the system to discover and use these things on its own. Correct. You need an autonomous mechanism that can look at the state of the world and decide which steering signal to generate.

6:35And that is where the main architectural innovation comes in, the metacontroller. The metacontroller. Think of it like the foreman. or the high-level strategy module. Its only job is to read the model's internal state and generate those steering instructions. The parameter updates for that little linear controller completely on its own. And there's a finding here that seems just fundamentally counterintuitive, especially for anyone who does deep learning. They found that the parameters of the huge base model have to be frozen. Yes, absolutely frozen after pre-training. Why is that so important?

7:06Usually you fine-tune everything together. I know, it's a profound finding. Because if they tried to co-train the base model and the metacontroller at the same time, the whole system just collapsed into what they call degenerate solutions. It breaks. It breaks. It basically loses that valuable, robust alignment with the abstract actions that the pre-training works so hard to establish. It's like the moment you let the foreman start messing with the building's foundation, the whole structure just loses its integrity. That's a perfect analogy. The pre-training builds this incredible foundation that aligns with abstract, generalizable actions.

7:43Freezing it preserves that. It lets the tiny metacontroller just exploit what's already there. The base model becomes a frozen, perfect, high-level action executor. So frozen base model is the muscle. The metacontroller is the brain generating the steering signals. Tell us about the mechanism inside the metacontroller that actually enables that temporal abstraction. Right. So the metacontroller itself is this sophisticated generative stochastic recurrent neural network, a recurrent hyper network. It doesn't output actions. Remember, it outputs the parameters for the steering controller. But its real genius is in this recurrent switching unit that it has inside.

8:21The part that figures out when to change its mind? Precisely. This unit outputs a continuous gate, beta t, that goes from 0 to 1. And this little number dynamically determines the time scale of the abstract action. How so? If beta t is close to 0, the system basically understands, I'm not done with my current task yet. And it sticks with the current abstract action. It holds onto the previous controller code. It's holding the high-level goal steady. Yes. But if beta T spikes up towards one, it's a signal. Subroutine complete. Time to pick a new abstract action. And that's the mechanism that lets it discover these meaningful subroutines with learned endpoints, all without any supervision.

9:02So it's not on a timer. It decides when the task is logically done. Exactly. And they trained this metacontroller with a self-supervised goal, right? Just trying to predict the next raw action. Yeah. But with that switching gate inside, what did the analysis of that gate actually show? The analysis showed it worked beautifully. It successfully recovered the underlying discrete hierarchical structure of the task. And crucially, that switching gate learned to behave in a quasi-binary, sparsely switching way. Meaning? Meaning it didn't just jitter around. It would hold steady, close to zero for many, many time steps, and then bam, it would spike to nearly one.

9:39Almost perfectly aligned with the moment the agent actually achieved a sub-goal and needed to think about what to do next. It validated the whole idea. Which brings us to the actual application. Internal reinforcement learning. We've built the machine. We know how to switch between abstract actions. How do we now use this to learn new hard tasks? Internal RL is a real paradigm shift in how you even frame the RL problem. The idea is to treat the frozen base model plus the controller as part of the environment. Oh, interesting. And the agent's learning problem gets dramatically smaller. Instead of having to explore this huge, messy space of raw output tokens, you know, foot pressures, steering angles, its actions are now abstract interventions, Z, inside the clean, compact controller code space.

10:23So instead of optimizing over, say, a million possible low-level action sequences, the RL agent is now just picking between maybe 100 high-level subroutines. You've just reduced the search space exponentially. And that's the whole game. That massive contraction is why this works. They tested it on tasks that require compositional generalization. For that ant robot, for instance, the task was to visit a sequence of colored goals, red, then blue, then green, but in new orders it had never seen before. And these were pure sparse reward tasks. You only got a reward at the very end if you did the whole sequence perfectly.

11:00Okay, so here's where it gets really interesting. Because sparse reward, compositional tasks, that's the definition of the efficiency bottleneck we talked about. So how did the standard methods do? They failed spectacularly. Standard RL fine-tuning, what they called raw action RL. It achieved success rates that were basically zero. The odds of randomly stumbling on the right high-level plan by just tweaking individual token probabilities were, as we said, effectively impossible. And even more sophisticated hierarchical RL methods, like one called Compile, or even versions of their own internal RL where they didn't freeze the base model, they all failed to learn.

11:38So token-level exploration just can't see the high-level strategic forest for the trees. It can't. And even other hierarchical methods couldn't recover the necessary structure to generalize. But the full internal RL approach, with the frozen model and that self-supervised switching unit, it achieved high success rates, and it did it quickly. Wow. By optimizing only on that contracted, temporally abstract timescale, it made the credit assignment problem, figuring out which high-level decision led to success so much simpler. It made exploration tractable. It was just orders of magnitude more capable than any of the baselines.

12:13This is really compelling evidence that the latent representations in these big models, even when they're only trained for next token prediction, are just so much richer than we thought. It's a fundamental counterpoint to the whole stochastic parrots argument, I think. They're not just regurgitating. They are forming consistent, robust, temporal abstractions and internal plans deep in their hidden layers. Internal RL doesn't teach the agent how to walk to the blue target. The frozen base model already knows that perfectly. Exactly. It just teaches the agent the much, much easier problem of when to sequence those known subroutines.

12:49It shifts the entire optimization problem from the raw, noisy, high-dimensional action space to the clean, compact, low-dimensional abstract action space that the metacontroller discovers. The complexity of the low-level physics is just handled by the pre-trained model. It leaves the metacontroller to focus only on high-level strategy. It's an incredibly elegant solution. So what does this all mean for the future of building agents? I mean, if these frozen pre-trained models are already encoding these highly structured generalizable subroutines, just waiting in their residual streams, that raises a pretty fascinating and provocative thought for you to consider.

13:25What other complex capabilities might be lying dormant inside these large autoregressive models? Capabilities we haven't even thought to look for, that we haven't even named yet. Just waiting for the right metacontroller, the right internal steering mechanism, to be built and trained to unlock them.