


The episode explains “empowerment maximization” for AI—agents that learn open-ended curiosity by maximizing how much control they have over future states, without external reward. It covers the SEE (Skills Estimate Empowerment) algorithm, how it estimates empowerment via learned skills, and why it scales in “big worlds” (continuous physics) using mutual-information skill learning plus hindsight experience relabeling (HER).
empowerment is capacity to influence the environment; maximizing it yields emergent tool use, survival, and exploration; guardrails and the choice of conditional vs joint mutual information prevent chaotic loopholes.
Crafter (Minecraft-like) where the agent crafts tools, builds defenses, and survives zombies by maximizing controllable inventory states; Builder Bench (robot gripper) where it positions for maximum leverage to scatter blocks; Ant soccer where it drives the ball to maximize controllable outcomes.
None mentioned in the transcript.
Written by AI. May contain mistakes.
Delve into the mathematical definition of empowerment for AI agents.
“Because we are coming through this fascinating stack of newly published research out of Princeton and UC Berkeley on something called empowerment maximization in artificial intelligence.”
Learn about the complexities involved in calculating empowerment in high-dimensional environments.
“So how do the researchers define empowerment for a machine?”
Compare traditional AI models with the new empowerment maximization approach.
“Well, in simple tabular settings, so think of a small rigid grid like a basic chessboard, you can calculate every single reachable state.”
Discover how the Skills Estimate Empowerment (SEE) algorithm functions.
“Like, how did the researchers behind the study finally break through that wall?”
Understand how hindsight experience relabeling improves learning efficiency for AI.
“First, you have the skill policy, which represents your effective empowerment.”
Examine the surprising behaviors of empowerment-maximizing AI in complex environments.
“what happens when you take that super learner and actually drop it into a complex world?”
The discussion explores how AI learns to survive and maximize options through empowerment.
“And you have to stop and ask why it learns to do those specific things.”
Examining how researchers implemented limits to prevent chaotic behavior in AI.
“You can't just define empowerment over every single variable in the universe.”
Understanding the impact of mutual information types on AI's learning effectiveness.
“It forces it to care about moving the soccer ball across the room, not just twitching its foot.”
Discussing the breakthrough in scalable empowerment for autonomous agents.
“If we pull back and look at the bigger picture, what this research ultimately proves is that scalable empowerment is no longer just an abstract theory.”
Exploring the philosophical implications of empowerment in both AI and human behavior.
“If the mathematically optimal way to build an intelligent, adaptable agent is to program it with an intrinsic drive to maximize its future options.”

Get the gist, explore an idea or ask about something said.
On Best AI papers explained, what are the key points of this episode?
Automatic transcript. May contain errors.
0:00Imagine for a second that you are just dropped into a massive sprawling city you've never visited before. Sounds overwhelming. Right. Or, you know, maybe you're spawned into a brand new, just incredibly complex video game. Yeah, totally. You don't have a map. You don't have a quest log. Nobody has told you the rules. And there is no score at the top of the screen telling you what to do. Right. You're just totally flying blind. Exactly. So what is your very first fundamental drive? Like, what do you instinctively try to do? I mean, if you're like most people, you just take stock. You look around.
0:34You figure out what you can touch, where you can walk. You're really just trying to figure out what you can actually control in this weird new place. Exactly. You try to keep your options open. You instinctively want to be in a position where you can just react to whatever comes next. Yep. And that instinct, that fundamental drive to maximize your options is actually the core of today's deep dive. It really is. Because we are coming through this fascinating stack of newly published research out of Princeton and UC Berkeley on something called empowerment maximization in artificial intelligence.
1:06And I have to say, it is basically a paradigm shift in how we think about AI. It really is. What these researchers have essentially done is, well, they've cracked the mathematical code for open-ended curiosity. Which sounds like sci-fi, honestly. It does. But they figured out how to make an AI intrinsically seek agency and power in these massive, complex environments completely from scratch. Right. And they do it without giving the AI a specific reward to chase, which is wild. Exactly. Because normally, you know, we train AI by giving it points for doing exactly what we want. Like fetch the ball, get a point.
1:39Right. The traditional reward function. But this, this is a recipe for building an autonomous agent that explores like a kid playing in a sandbox. It's just exploring for the sake of options. Right. So for you listening, here is our roadmap for the steep dive. First, we need to define what empowerment actually looks like for a machine. And trust me, the math behind that is absolutely fascinating. It really is. Then second, we'll dive into the clever algorithm the researchers developed to make this mathematically possible in big worlds. Yeah, the C algorithm. Right, S-E-E. And third, we are going to look at the wild emergent behaviors that happen when you just leave an AI alone to maximize its options.
2:22We're talking like building houses, crafting tools. And yes, surviving zombies. Which we will definitely get to. It's a brilliant look at how highly complex, almost human-like behavior arises from just one simple mathematical directive. So let's actually start with that directive, because before we can talk about scaling intelligent AI, we have to understand what it's striving for, right? Right. When there's no score to maximize. Exactly. So how do the researchers define empowerment for a machine? So at its core, empowerment is defined mathematically as the capacity of an agent to actively control its environment.
2:56Okay. Maximizing empowerment just means navigating your way to states that are really good starting points for executing like the highest number of different skills. So having the most buttons to press. Basically. Technically speaking, in the research, it's about maximizing the mutual information between the agent's behavior and future states. Wait, hold on. Mutual information. That sounds like a really heavy textbook term. Let's ground that for a second. Sure. What does maximizing mutual information actually look like in practice? Fair enough. Think of mutual information as a measure of definitive influence.
3:32Okay. Like if I take an action right now, how completely does that action dictate what the world will look like, say, 10 steps from now? Got it. So if I push a block on a perfectly smooth table, I know exactly where it will go, right? That is high mutual information. Because it's predictable. Right. But if I drop a feather in a windstorm, I have absolutely no idea where it will land regardless of how I drop it. Right. Okay. That's low mutual information. So empowerment means moving to a place where your actions have highly predictable, diverse impacts on the world around you. Okay, let me see if I have this right.
4:08Is it sort of like a game of chess? Oh, I like that. Go on. Like being empowered is like maneuvering your queen into the absolute center of the board. Because from the center, she can move in, what, 27 different directions? She exerts massive control or, I guess, mutual information over the whole board. Exactly. But if she is trapped in a corner, her options are totally locked down. She has low mutual information. That is a brilliant analogy. You always want the queen in the center of the board. And conceptually, the AI community has known this for decades. Right. It makes intuitive sense. But here is the roadblock.
4:46Calculating that exact center point for a machine in a complex environment has been a massive historical headache. Why is it so hard to calculate? Well, in simple tabular settings, so think of a small rigid grid like a basic chessboard, you can calculate every single reachable state. Because there's a set number of squares. Precisely. You can mathematically prove where the most options are because the world is finite. Okay, that makes sense. But in what the researchers call the big world, meaning, you know, high dimensional continuous environments with realistic physics. Like the real world, essentially.
5:22Right, or even just a complex physics simulator. Mapping out all reachable states and future trajectories in those spaces is a computational nightmare. Because the math just gets too heavy? Yeah, the possibilities are practically infinite. Right. There are no grid lines. Right. So how did older models try to handle this? Because I know in the study they constantly compare this new method to older baselines. Right. They looked at a few. Yeah, specifically things called VIM and latent GCE. What were those baselines trying to do? So those older methods tried to tackle the big world problem, but they fundamentally stumbled.
6:00Let's take VIM, for example. Okay. That stands for Variational Intrinsic Motivated Exploration. It essentially tried to reward the AI for finding novelty. Just looking for things it hadn't seen before. Exactly. But in a continuous world with complex physics, everything is slightly novel. A shadow moving across the floor is novel. Oh, I see. So the AI gets distracted by noise. It's like staring at TV static. It just becomes highly unstable. Oh, so it just acts like a hyperactive dog chasing every falling leaf. That's a perfect way to describe it. Okay, what about the other one, latent GCE? So latent goal-conditioned exploration, or latent GCE, tried a totally different route.
6:43It tried to build a compressed mental map of all possible future states. And then set goals to reach them. Right. But again, in a big continuous world, trying to mentally map every possible future outcome requires massive computational power. It's trying to predict infinity. Exactly. So when you drop these baselines into a big continuous world, they either freeze up entirely, become highly unstable, or they just fail to scale. They just collapse under their own rate. Right. They simply couldn't handle the sheer volume of calculating every possible future to figure out which one was the most empowering.
7:19Okay, so if Vim gets distracted by static and latent GCE collapses under the weight of trying to predict infinity, how do we fix it? Like, how did the researchers behind the study finally break through that wall? They found a really brilliant backdoor. because calculating every possible future is a nightmare. They decided to tie empowerment to something else. What's that? Something we already know how to do really well in machine learning. Unsupervised skill discovery. Okay. Specifically, they used a concept called mutual information skill learning, or MISL. Wait, how does discovering skills replace the need to calculate the future?
7:58Am I off base thinking that sounds like two completely different problems? They seem different, but they are related in this really elegant way. Okay, walk me through it. The researchers realized that if an agent learns a whole bunch of distinct, reliable skills, then the sheer number of skills it can successfully execute from a specific spot acts as a lower bound for its actual empowerment. A lower bound. Okay, what does that mean in plain English? Let's say I can mathematically prove I have 50 different tools in my toolbox, and I know how to use them perfectly from this exact location. Okay.
8:33I don't need to calculate the infinite possibilities of the universe anymore. Having those 50 reliable options is proof enough that I am highly empowered. Oh, wow. So just saying I have 50 options right here, right now, that's proof of my power. I don't need to map out the next thousand steps. Exactly. So to pull this off, the new algorithm, which is called Skills Estimate Empowerment, or C, splits the problem into a two-policy system. Two policies, okay. Right. First, you have the skill policy, which represents your effective empowerment. Which is what? This is the agent practicing different skills, trying to alter the environment in distinct ways to measure its control.
9:14Okay. And the second policy? Second, you have the potential policy. This policy uses the findings of the first one as an intrinsic reward. Meaning its only job is what? Its only job is to navigate the agent to the exact spot where it has the most options available. That two policy system, that perfectly mirrors the concept of accruing and spending wealth. Think about it. Oh yeah? How so? If you're the AI, your potential policy is the phase where you are accruing power. Right. You're walking to the center of the room. You're gathering your resources. You're saving up your money. Then the skill policy is where you are spending power.
9:52You're standing in the center of the room and you're buying different items. You're kicking a ball. You're pushing blocks. You gather the potential and then you spend it to exert influence. Right. You gather it up to spend it. And if you look at the simulation logs from the research, the data visualizes this exact dynamic perfectly. Oh, really? What does it look like? They generate these incredible graphs that track the AI's movement over time. You see these white lines tracing the potential policy. So that's the agent accruing power. Right. It's navigating toward a target, getting itself into the optimal position.
10:27And then suddenly, from that spot, colorful lines burst outward in all directions. Like an explosion of options? Exactly. That's the agent spending its power, executing its diverse skills and altering the environment. Gathering to the center and then bursting outward, that makes total sense. But there's a mechanical question here for me. Sure. How does it learn all those skills efficiently? Because if you drop me into a massive open world, wandering around trying random things to build 50 skills, That sounds like it would take forever to get any useful data. You're touching on a really crucial technical hurdle, sample efficiency.
11:04Right. Because you're right. If an AI tries to learn 50 skills by random trial and error, it would take years of compute time. Yeah, that seems impossible. So to ensure the system learns fast enough from limited data, the researchers utilized a technique called hindsight experience relabeling or HER. Hindsight experience relabeling. Yeah. That honestly sounds like a fancy way of saying, I meant to do that. Well, I mean, that is essentially what it is. But the mechanics of how it saves computational power are just genius. Okay, explain. Imagine the AI tries to perform a specific skill. Let's say it wants to lift a block with a robotic arm.
11:45Right. But it completely misses, thrashes around, and accidentally knocks the block off the table entirely. Oops. Normally, the system would log that as a failure. The reward is zero, the data is useless, and the neural network doesn't update its weights. It just dumps the data. Right. But with hindsight experience relabeling, it rewrites the history of that training episode. How? It tells the AI, hey, let's pretend your goal all along was to knock that block off the table. Congratulations, you succeeded. Here is a reward of one. Wait, really? It just lies to itself? Basically. Because it receives a positive reward for that relabeled goal, the neural network gets to do a positive gradient update.
12:27Oh, okay. By retroactively relabeling the goal to match the outcome, the AI learns a vast repertoire of skills much faster using data that would otherwise just be discarded as random noise. It's like, okay, it's like throwing a dart, missing the dartboard entirely, hitting the blank wall, and then walking up and drawing a bullseye around the dart. Exactly. Look, perfect aim. Right. And because you drew the bullseye, your brain actually learns the muscle memory for how to hit that specific spot on the wall next time. That's it perfectly. It's a clever mathematical trick to maximize the value of every single interaction it has with the environment.
13:02Okay, so if the SEE algorithm allows the AI to learn 50 skills at once just by rewriting its own history, what happens when you take that super learner and actually drop it into a complex world? Let's talk about the emergent behaviors because this is the part that really hooked me. Oh, let's do it. When you drop this empowerment-maximizing AI into open-ended worlds, the behaviors that naturally emerge are just stunning. The one that stood out to me was the crafter environment. Oh, yeah. Crafter's great. Right. And for context, for those listening, crafter is essentially a 2D version of Minecraft.
13:37You've got a little pixelated world. You can move around, punch trees, mine stone, deal with monsters. Right. And remember, the researchers did not give this AI a reward for scoring points or finishing the game. They just said, maximize your options. Yep. No game-specific rules. And what does it do? It learns to chop wood. It learns to build a crafting table. It uses the crafting table to make a saw. And you have to stop and ask why it learns to do those specific things. Right. Why a saw? It's because having a saw gives you way more options to change the world than just having your bare hands.
14:12A crafting table exponentially increases the number of distinct states you can push the environment into. It's literally just expanding its toolbox. But the craziest part to me is the survival aspect. Oh, the zombies. Right. In Crafter, there are zombies and skeletons that come out at night. And the AI learns to fight them or dodge them or even build a defensive house out of plants and crafting tables just to survive the night. Again, nobody programmed a stay alive reward. If you're an AI just trying to get options, why do you care about zombies? The logic of empowerment dictates survival perfectly.
14:48Think about the mathematics of death. Okay. In a state of death, an agent has zero control over its environment. It has zero options. Right. You can't push any buttons if you're dead. Exactly. It can broadcast zero bits of information into the future. Therefore, death represents a state of absolute zero empowerment. Wow. For an agent driven mathematically to maximize its options, avoiding death is paramount. Not because it fears death, but because death is a dead end for potential. Death is a dead end for potential. That is deeply philosophical for an AI just trying to keep its math score high.
15:24It really is. But it doesn't just work in discrete grid-based games like Minecraft, right? Let's look at the continuous physics-based simulations. Yes. The researchers tested it in high-dimensional continuous worlds, too. Take the Builder Bench environment. Okay. What's that one? Imagine you are a robotic gripper arm interacting with a set of physical blocks on a table. The AI's potential policy learns to sneak the gripper behind the blocks. Why behind them? Because from that specific position, it has the maximum physical leverage to scatter the blocks in any direction it chooses. It found the physical center of the chessboard.
16:03That is so smart. And then there's the ant's soccer environment. Yes, and soccer is a fascinating one. For this one, it's a multi-legged robotic ant in a large arena, and there's a soccer ball. Right. The AI realizes that the only way to truly alter the environment, the only way to exert mutual information, is to interact with the ball. Because the arena is mostly empty otherwise. Exactly. So the potential policy drives the ant directly to the ball. And once it's there, its skills involve dribbling it, kicking it left, kicking it right. It masters ball control just to maximize its options. And this is where we really see why those older baselines we talked about failed so hard.
16:44Right. Vim and latent GCE. When older methods were dropped into ant soccer, they struggled immensely. Vim was highly unstable. The ant would basically just vibrate and stand still because it was overwhelmed by the continuous physics space. Just totally paralyzed by static. Exactly. And latent GCE would occasionally move, but sometimes it would kick the ball directly into the corner of the arena. Which is literally the worst place to put it. If the ball is in the corner, you can't kick it anywhere else. You've trapped your queen. Exactly. The older models fundamentally failed to grasp the big picture geometry of empowerment in continuous space.
17:22The SCE algorithm understood that true empowerment means staying out in the open field with the ball. Okay, I have to play devil's advocate here for a second. Go for it. Because as incredible as it is that these behaviors emerge naturally, I see a massive loophole here. What's the loophole? If the AI's only goal is to simply change its environment as much as possible to prove it has options, wouldn't it just act chaotic? How do you mean? Like, going back to the crafter survival thing, couldn't it just let the zombies hit it over and over because losing health is technically changing a variable in the environment?
17:53Uh-huh. Or couldn't the robotic ant just sit in a corner and spin its legs in circles really fast? How do you stop it from just spamming useless changes? That is a phenomenal question, and it's the exact issue the researchers had to address by implementing specific guardrails. Okay, so they did see this coming. Oh, absolutely. You can't just define empowerment over every single variable in the universe. You have to limit what's called the empowerment context, or the degrees of freedom. So you have to tell it what is worth controlling. You have to explicitly define the sandbox. Exactly. In Crafter, the researchers specifically limited the AI so it could only feel empowered by changing its inventory.
18:33Oh, so it didn't care about its health bar. Right. It got no reward for its health going up or down or its hunger changing. It only felt powerful if it was acquiring different blocks or crafting tools. That forces it to do productive things. Exactly. In the ant soccer environment, empowerment was defined strictly by the ball's position, not the position of the ant's individual leg joints. Got it. If you don't put those guardrails in, the AI will absolutely find a loophole and just vibrate its legs at high speed to generate massive amounts of useless mutual information. Because technically, vibrating your legs a thousand times a minute is a thousand distinct changes.
19:10Technically, yes. It's like telling someone to go make an impact on the world, and they decide the best way to do that is to just sit in their bedroom flipping a light switch on and off a million times. That is painfully accurate. Yes, you're causing a change, but it's totally meaningless. That's a great way to put it. And, you know, dealing with that meaningless change brings us to probably the most fascinating technical quirk the researchers discovered regarding how you actually calculate that mutual information. Oh, what's the quirk? It all comes down to the difference between conditional mutual information and joint mutual information.
19:42Okay, you're going to have to walk me through those because they sound very similar. They are, but the distinction changes everything. Conditional mutual information means the AI tracks its exact, precise starting state before calculating the change. Like down to the millimeter. Exactly. It looks at the exact millimeter position of every joint, every angle. Okay. But in physics-based environments like ant soccer, if the AI tracks that perfectly, it actually overfits. It becomes obsessed with those tiny, meaningless vibrations you just mentioned. Why? Because mathematically, a tiny vibration from an exact starting point is still highly predictable control.
20:24Oh, because it's tracking everything too perfectly. It gets lost in the microscopic details. Right. So to fix this, in the continuous physics simulations, the researchers used joint mutual information. And what does that do differently? Joint MI forces the AI to look at broader trajectory changes over time rather than obsessing over the exact microscopic starting position. Think of it like a visual filter. It's like looking at a forest through a slightly blurry lens, maybe. Yes, exactly. You stop obsessing over the tiny vibration of a single leaf, and because the details are smoothed out, you finally notice the shape of the whole tree.
21:04That's it. That's what joint MI does for the AI's physics calculations. It forces it to care about moving the soccer ball across the room, not just twitching its foot. That is exactly what it does. It acts as an information smoother. But, and here is the wild part, ironically, in the discrete grid-based world of Crafter, that joint MI method caused the system's learning to completely collapse. Wait, really? Right. The blurry lens ruined it. Completely ruined it. Well, let me guess. Because in a Minecraft world, everything is built on rigid blocks. There is no smooth trajectory. You either have a block of wood in your inventory or you don't.
21:42So you actually need to obsess over the exact state. You nailed it. In the discrete world, the strict, precise, conditional mutual information actually worked best. Wow. It really highlights how incredibly sensitive these algorithms are to the foundational rules of the reality they are operating in. You have to tune the math to match the physics of the universe you're in. That is wild. Physics requires a broader, smoother look, but grid worlds require rigid tracking. The context is everything. Exactly. So looking at everything we've talked about, the C algorithm, the emergent behaviors, the math, what is the ultimate takeaway here?
22:23If we pull back and look at the bigger picture, what this research ultimately proves is that scalable empowerment is no longer just an abstract theory. It's actually doable now. Right. For years, computer scientists have suspected that intrinsic motivation, this mathematically driven desire for agency, was the key to human-like intelligence. But the math was just too computationally heavy. Too many futures to calculate. Exactly. But now, with the SEE algorithm connecting empowerment to skill discovery and using tricks like hindsight experience relabeling, we have a functional, scalable method.
22:58That is a huge deal. We can build autonomous agents that can parachute into vast environments they do not understand, and they will immediately begin learning how to master them, much like human children do through play. For you listening, this is the massive takeaway here. We now have a mathematical recipe for open-ended curiosity. We really do. By simply asking an AI to maximize its future options and showing it what is worth controlling, complex, highly intelligent behaviors like gathering resources, crafting tools, and surviving danger just organically emerge from scratch. Yep. It's a huge leap forward toward machines that can truly adapt to the unknown without needing a human to hold their hand and give them points.
23:39This raises an important question, though. A philosophical one to end on, maybe. Oh, I love a philosophical ending. Let's hear it. If the mathematically optimal way to build an intelligent, adaptable agent is to program it with an intrinsic drive to maximize its future options. Yeah. How much of our own human behavior is just biological empowerment maximization? Oh, wow. I love where this is going. Think about it. Yeah. Our pursuit of money, getting a higher education, learning how to drive a car. Building a house or packing up and moving from a small town to a massive, complex city. Are we all just running our own potential empowerment policies?
24:21Are we just trying to maneuver ourselves to the absolute center of the chessboard? Right. Just to make sure we have as many options as possible when the next move comes. When you drop into a new city, you want to know what you can control. You want to keep your options open. I love that. We are all just running the S-E-E algorithm in our daily lives. It certainly seems that way. Thank you so much for joining us on this deep dive. To everyone listening, keep questioning the world around you. Keep maximizing your options. And we'll catch you next time.
This paper introduces a scalable algorithm called Skills Estimate Empowerment (SEE) that bridges unsupervised skill learning and empowerment maximization to train intelligent agents in complex, open-ended environments. By treating mutual information skill learning as a lower bound for potential empowerment, the method utilizes a two-stage policy where a potential policy navigates toward controllable states while effective skill policies explore local options. Through experiments in challenging settings such as Craftax-Classic, AntSoccer, and BuilderBench, the authors demonstrate that this approach successfully overcomes previous computational limitations. Consequently, the trained agents exhibit emergent behaviors like resource gathering, tool crafting, environmental modification, and survival without relying on external rewards.