Embodied AI Agents: Modeling the World

0:00Imagine AI that doesn't just live on your screen, you know, responding in a chat window. Right. But AI that can literally see what you see, hear what you hear and interact with the physical world right alongside you. Yeah. We're really on the cusp of a huge shift in how we live with technology. And while this isn't science fiction anymore. No, it's really not. Today, we're taking a deep dive into embodied AI agents. And it's a field that's moving incredibly fast, like changing by the day, it feels like. It does. Our insights today, they come from this fascinating, really current paper, an overview of research on these agents.

0:37It was actually published just today, July 4th, 2025. Wow. Hot off the press. Pretty much. Yeah. With correspondence from Pascal Fung at Meta. And this paper, it really unpacks the current state and where things might be headed. So our mission for you today is to explore what these agents truly are, why this whole embodiment thing is such a game changer. the incredible applications they're already enabling, how they actually think and perceive the world around them, and really importantly, the ethical side of things, the considerations we all need to be aware of as they get more, well, integrated into our lives.

1:12Absolutely. So let's unpack that. What exactly is an embodied AI agent? What makes them different from the AI we're already kind of used to? Okay, so at their core, embodied AI agents are AI systems that exist in some kind of form, visual, virtual, or physical. And this form lets them learn and interact both with the user and their surroundings, whether that's physical or digital. Okay. The key distinction here, I think, is unlike, say, a web-based AI with no visual form or even a robot just following remote commands. Right, like telepresence. Exactly. True embodied AI agents have this crucial mix of autonomy and adaptability.

1:52They don't just follow orders. They perceive and act within their environment in a meaningful way. And that really demands a deep understanding of the world around them. That distinction really strikes me between just being a robot and being a truly embodied AI agent. That autonomy, that perception action loop, it feels like it changes everything. It does. But what does that actually unlock? Why is giving AI a body, so to speak, so powerful beyond just a physical shell? Well, the paper highlights a couple of main purposes. First, there's physical interaction. Embodiment lets AI systems act directly in the physical world.

2:27Think robotic agents or gain this really intimate awareness of their surroundings, like wearable agents that share your viewpoint. Okay, acting or sensing. Right. And second, and this is crucial, it's about enhanced human-machine interaction. Studies keep showing that embodied agents, well, people tend to trust them more. Interesting. And if you connect that to the bigger picture, there's this growing idea, the potential for these agents to learn and develop through rich sensory experience, kind of like how we humans understand and adapt. That human-like learning idea through senses, it's really fascinating.

3:01It reminds me of that philosopher Maurice Merleau-Ponty, his idea, I am not in my body, I am my body. You know, the body isn't just a vessel. It's fundamental to how we exist, learn, relate to the world. That's a great parallel. And it seems like that same kind of thinking is emerging for embodied AI. So how do we get here? How did we evolve from, say, early chatbots to these new, really autonomous agents that leverage this embodiment? It's been a long road, really a persistent quest in AI. Think back. We started with simple rule-based chatbots. Yeah, ELISA and things like that. Then AI call center assistants, the virtual assistants like Siri and Alexa, and more recently, of course, these powerful online AI agents like ChatGPT.

3:46Right. The huge difference now is that today's AI agents are just way more autonomous. They can handle complex multi-step tasks. They figure out the steps themselves, what resources they need, even which other agents to maybe collaborate with. Wow. And they do all this while ideally implicitly understanding what you need just from the context. And it feels like the rapid advances in the large language models, LLMs, and these vision language models, VLMs, they've really put the rocket fuel in this, haven't they? Oh, absolutely. Enabling these agents to run on smart glasses, VR, robots, all sorts of hardware.

4:19Precisely. And for these embodied agents to really work effectively, not just react, but genuinely understand and adapt, there's this critical concept, world modeling. It's absolutely essential for them to make sense of and interact meaningfully with their environment. Okay. World modeling. So that's basically building an internal representation of their surroundings, like a dynamic map of reality. I think the paper mentioned two kinds of models. That's right. Almost like two lenses. First, you have physical world models. These capture the relevant stuff about the environment objects, their properties like shape, size, color.

4:52The nuts and bolts. Exactly. Spatial relationships, how things move or change over time, environmental dynamics, and crucially, causal relationships, things grounded in physics. It's their understanding of how the world literally works. Okay, so that's the physical map. But what about the human side? That seems critical for interaction. I think the paper called that a mental world model. Spot on. The mental world model is the agent's internal representation of the human context. This is where it tries to capture your goals, intentions, motivations, preferences, values, even emotions. Wow, that's ambitious.

5:25It is. And it also includes understanding complex social dynamics, relationships, cultural norms, and even the subtle stuff in communication like tone of voice or body language. It's basically their attempt to model you and your social world. So having both a physical and a mental world model, that sounds like it gives them a really comprehensive understanding, not just what's there, but what people want or intend. Exactly. It feels like this is where they really start moving from just tools to maybe intelligent companions. What does having that dual understanding actually enable them to do? Well, it enables several really critical capabilities.

6:00It allows for much more advanced reasoning and planning so the agent can make informed decisions and carry out tasks effectively. It supports zero-shot task completion. That means the agent can adapt to totally new tasks that it wasn't explicitly trained on because it understands the underlying principles of the world, not just patterns and data. So it's not just pattern matching. Right. It's moving towards a more adaptable intelligence. It also facilitates human-in-the-loop active learning so it can keep getting better based on real-time feedback from you. Okay. And finally, it leads to efficient exploration.

6:34The agent can focus on what's relevant, avoid pointless actions, and gather information much more quickly. That's a really powerful combination. Now let's dig into the inner workings. How do these embodied AI agents actually, you know, think and perceive? How do they bridge all these different models? It sounds like it has to start with multimodal perception, kind of like how we use all our senses. Indeed. Multimodal perception is absolutely fundamental for image and video understanding. It's all about these general purpose vision language models and something called perception encoders or PEs.

7:06Yeah, think of them as super sophisticated digital eyes. They're state-of-the-art vision encoders. When you train them on huge amounts of video data, they can produce these general visual embeddings. Mm-hmm. Basically, a rich understanding of what's being seen. And that lets them do things like? Things like zero-shock classification, recognizing things they haven't been specifically trained on, and spatial reasoning, understanding where things are in relation to each other. Okay. And what about audio and speech? For something like wearable agents, that sounds incredibly complicated. They're not just listening for Hey Siri.

7:41Oh, far from it. Wearable agents need to interpret everything. Ambient sounds, traffic, dishes clattering, ongoing conversations happening around the user, not even directed at them, and speech specifically directed to them. Plus, they need to be able to synthesize speech to respond naturally. That sounds like a nightmare of challenges. It's significant. There's noise robustness, filtering out all that background junk. There's speech variability, understanding different accents, dialects, languages, and telling your voice apart from someone standing nearby. Yeah. And critically, doing all this with the limited computational resources you have on a small wearable device.

8:20So what are the breakthroughs needed there? What are the future directions for tackling audio and speech? Well, there's a big push towards Edge AI. That means doing the processing locally right on the device. For privacy and speed. Exactly. It reduces latency and keeps sensitive audio data off the cloud. There's also a huge focus on personalization, getting the agent to adapt to your specific voice and way of speaking. Right. And multilingual expansion, supporting more languages, especially ones that are mostly spoken, not written. Beyond sight and sound, the paper also talks about touch. Why is touch so important for embodied AI, especially for, say, a robot hand?

8:59Touch provides this absolutely vital extra layer of sensing. It's especially crucial when you're manipulating objects and maybe your vision is blocked. like reaching into a bag. Exactly. Or finding something in a dark drawer. Touch also helps estimate force, detect if something's slipping from a grasp, recognize textures. There are general purpose touch encoders, like one called Sparsh, being developed specifically for this to give AI that sense of feel. So once the agent perceives the world through all these input, sight, sound, touch, how does it turn that perception into actually doing something purposeful?

9:36How does it handle actions and control going from data to, well, action? Right. So embodied agents act in different spaces. They might operate in a 2D or 3D virtual world, or they might advise a human in the real world. That's what rural agents do. Or they might directly control robotic hardware. And within that, there are different levels of planning involved. Think micro level versus macro level. Okay. Can you break that down? Low level versus high level planning. Sounds like a big difference. It is. Low-level motion planning is about the really fine-grained mechanics, like the specific joint torques a robot needs to move its arm just right happening in milliseconds.

10:10It's the precise control for delicate tasks. High-level action planning is more abstract. It's generating sequences of actions over seconds or minutes. Think prepare a meal or clean the living room. Ah, the bigger picture. Exactly. And the big challenge there is simulating all the different ways you could do that, predicting the consequences dynamically, and then how do you even objectively evaluate if it did a good job in a messy, open-ended, real world? It's not just about moving. It's the why and how as part of a larger plan. And how do those world models we talked about fit into this planning process?

10:46They're fundamental. Learned predictive models, which are, you know, at the heart of these world models, are essential for planning. The agent uses them to predict what will happen if it takes a certain action. This lets it plan towards a goal or minimize some kind of predicted cost. And importantly, this allows for planning that can generalize even to new situations leading to that adaptive behavior we mentioned. Then there's memory. That sounds like the bedrock for understanding and adapting over time. How does memory fit into this whole complex picture? Yeah, memory is absolutely a core ingredient of world models.

11:21It's how the agent captures its interactions and consolidates them into its internal, evolving picture of the world. Like our own memories. Kind of. We can think about two main types. There's working memory, which is like the AI's short-term thought process, similar to how a transformer model uses its KV cache to hold immediate context in a conversation. Okay, the here and now. Right. And then there's external memory. Think retrieval augmented generation, or RAG. That's basically giving the AI access to a huge external library of knowledge can pull from instantly, like us looking something up online mid-conversation.

11:57So what are the unique memory challenges for embodied AI out there in the real world? It sounds way harder than just recalling facts. Oh, absolutely. The paper highlights this crucial need for episodic memories. The agent needs to recall specific events and experiences from its own past interactions. Like remembering where it left something. Exactly. The challenges include personalization, efficiently storing your unique history and preferences without needing constantly growing resources. And then there's the massive challenge of lifelong training. Meaning the agent can learn continuously day after day, adding new knowledge without its resource needs ballooning and critically without forgetting what it learned before.

12:36Ah, the catastrophic forgetting problem. Precisely. It's like trying to raise a child that learns constantly and never forgets anything. A huge, fascinating challenge. It really is. Okay. It's incredible thinking about how broad this field is from virtual beings to physical robots. Let's move into what the paper calls the three flavors of embodied AI, virtual, wearable, and robotic. First up, virtual embodied agents, or VEAs. What do these look like and where are we seeing them? Right. VEAs. These can range quite a bit from the 2D or 3D avatars you see on screen all the way to highly realistic robotic androids with very human-like features and expressions.

13:14And applications. They're really making waves in areas like AI therapy, systems like Wobot or WISA, providing emotional support. They're also key in the metaverse and mixed reality spaces, acting as guides, mentors, friends, even really sophisticated NPCs in virtual worlds like Second Life or Horizon Worlds. NPCs, non-player characters. Exactly. And another big area is AI studio avatars, creating incredibly realistic, emotionally intelligent characters for films, TV, video games, almost like digital actors. So the future here sounds like personalized learning, super efficient customer service, maybe even helping patients in healthcare.

13:52What are the key capabilities that make these VEAs so believable? Well, they need to be able to emote convincingly facial expressions, proper lip sync. They need real social intelligence, understanding interpersonal dynamics. Like reading the room. Sort of. Active listening, visual synchrony, natural turn-taking in conversation, and advanced models like Meta's Modivo can even control physics-based humanoid avatars for complex full-body action. Yeah. Plus, projects like Fairer's Seamless are pushing for emotionally accurate avatars that really resonate. Okay, next flavor. Wearable agents. These seem really unique because of that egocentric perception.

14:28They literally see the world through your eyes. They really are unique. Unlike other smart devices, these agents have cameras, mics, sensors worn by you. Meta's AI glasses are a good example. They see and hear what you see and hear. That shared viewpoint seems powerful. It creates this unprecedented shared perceptual field. The agent gets direct insight into your immediate environment and what you're doing. So what roles are they playing now? What's the potential? Right now, they're often functioning as coaching agents, assisting with physical tasks like cooking, assembling furniture, sports, sharing your perspective.

15:02Like having an expert looking over your shoulder. Exactly. They also act as tutoring agents, guiding you through cognitive tasks like math problems or learning a new skill. What do these wearable agents need to be truly effective, not just, you know, reactive? They need that robust multimodal perception we talked about, the ability to respond quickly to changing surroundings, and crucially, long-horizon action planning, not just simple reactions, but understanding a whole sequence of steps. And anticipating needs. Yes, exhibiting machine initiative, intelligently anticipating what you might need next instead of just waiting for commands.

15:38The future impact could be huge. Enhancing human performance, personalizing learning like never before, revolutionizing healthcare assistance, creating truly immersive entertainment, even speeding up scientific research. Okay, finally, robotic agents. These are maybe what most people first think of with embodied AI. What defines them and what's their potential in the physical world? Right, robotic agents. These are AI systems operating robots in the physical world, doing tasks independently or collaborating with humans. And we're really talking about general purpose robots here, not single task factory arms.

16:12Okay. Adaptable robots. How do they perceive? Through a whole suite of sensors. Standard RGB cameras, tactile sensors for touch, IMUs. Those are inertial measurement units for orientation and movement force torque sensors, audio sensors. A lot of inputs. Definitely. And their transformative potential is immense. They could address major labor shortages and demanding jobs, be deployed in dangerous situations like disaster relief, provide vital support and care for the elderly or help overwork medical staff and, you know, potentially free people from a lot of strenuous or just tedious household chores.

16:48There's also that concept, the embodiment hypothesis, specifically with robots. That sounds pretty profound. What's the idea there regarding intelligence? Yeah, the embodiment hypothesis. It basically suggests that interacting with the real world is actually key for robots to learn general intelligence, maybe even reach artificial general intelligence, AGI. So intelligence from interaction. The idea is that through direct sensory input, physical manipulation, real world trial and error, they can learn in a much deeper, more grounded way than AI just processing data in the cloud. That's a fascinating idea.

17:21Intelligence emerging from getting your hands dirty, basically. What are the key capability categories for these robots? Sounds like a physical side and a brain side. Yeah, that's a good way to put it. On the physical side, you have locomotion, which is just moving around, maybe legged robots on rough terrain, navigation, planning paths to a goal without crashing, and dexterous manipulation. That's the really fine-grained control for grasping, placing objects, using tools. Often involves full body control. Okay, the physical skills. What about the brain side? That includes things like generalization, adapting to new tasks, environments, objects they haven't seen before, efficient and lifelong adaptation, learning new skills without forgetting old ones, and personalizing behavior, spatial and temporal memory, knowing where things are and how events unfold over time.

18:09Understanding language instructions, too. Crucially, yes. Language instructions and planning, breaking down complex commands like clean the kitchen into actual steps and sophisticated interaction with humans and other agents taking instructions, asking clarifying questions, physically helping, collaborating and inferring intent using those mental world models. This all sounds incredibly complex, which raises a big question. How do we actually evaluate these agents? How do we know if they genuinely understand the world or if they're just good mimics? That's a critical challenge, and good benchmarks are key.

18:44The paper goes into several, especially for testing their world models. One is called Minimal Video Pairs, or MVP. It's designed to test physical and spadio-temporal reasoning. It uses pairs of videos with tiny changes that require really fine-grained physical understanding, like should this object fall or not. And how do the models do? The results are pretty stark. Current AI models get around 40.2 % accuracy. Humans, 92.9%. So a huge gap. Wow. That really suggests they're missing something fundamental about basic physics. What about intuitive physics, like common sense physics? For that, there's Infys II.

19:19It assesses understanding of principles like object permanence, solidity stuff a toddler gets. Again, models really struggle with the deeper reasoning, often performing near chance level while humans are near perfect. So these benchmarks really highlight a gap. A profound gap. Our best AI still struggles with the intuitive physics a young child masters. It's humbling. Shows how complex common sense really is and where the next big breakthroughs need to happen. And what about causal reasoning? That seems vital for planning and understanding consequences. It is. There's causal VQA. It evaluates causal reasoning in real videos, asks questions about counterfactuals, what if, hypotheticals, predicting what happens next, planning.

20:02Again, state-of-the-art models are way behind humans, especially in predicting the future and hypotheticals. They struggle with grasping cause and effect and temporal structure. Okay, and there was one more benchmark mentioned, world prediction, particularly for wearable agents. Right. World prediction focuses on high-level procedural planning and understanding sequences over time. Super relevant if an agent is guiding you through assembling furniture, for example. It tests inferring high-level actions and sequences when the agent doesn't have perfect information, partial observability. And surprise, surprise, models are significantly behind humans.

20:37Like 57 % on world modeling, 38 % on procedural planning versus near perfect for people. So the benchmarks clearly show AI is advancing fast, but there's a long road ahead to match human understanding of the physical and causal world. So what does this all mean for the future? Where is embodied AI heading in terms of how these agents learn and interact, maybe even with each other? Well, major future direction is embodied AI learning. The really big shift is moving towards continuous, interactive, goal-directed learning. Much more like how animals and humans learn from the moment they're born, constantly engaging with the world.

21:14Instead of separate training and deployment phases. Exactly. Current AI often has these distinct phases. The vision is to integrate them fully. Can you explain that integration idea using System A and System B? That sounds like a really interesting concept in the paper. Yeah, it's a powerful way to think about it. Think of System A as learning through observation. It finds structure and patterns in passively collected data like watching tons of videos. Self-supervised learning, essentially. It builds abstract representation, sort of an internal map. Okay. Learning by watching. Then system B is learning through action.

21:48It actively interacts with the environment to learn, driven by goals. Like reinforcement learning, try something, see what happens, learn from it. Learning by doing. Exactly. The really exciting idea is integrating them. System A gives system B useful structure and compressed representations to make the action-based learning much more efficient. And System B helps System A by actively gathering better, more relevant data and grounding those abstract representations in actual, real-world behavior. Ah, so it becomes a feedback loop. Perception informs action. Action fuels perception. Constantly refining each other.

22:23Precisely. That kind of bidirectional pathway could lead to much faster adaptation, maybe even agents discovering new tasks for themselves. Self-supervised task discovery. Wow. Yeah. And another crucial future direction is multi-agent interactions. This is where you unlock the power of collaboration. Multiple embodied AI agents working together can tackle complex tasks way beyond what a single agent could do. Can you give some examples of what that collaboration might look like? Sure. You could have multi-robot teams and disaster relief. One searches, one clears debris, one delivers supplies. Or think about your own devices, smart glasses, smartwatch, maybe a haptic vest.

23:02All collaborating as wearable agents for a really integrated experience. In VR, multiple virtual agents could engage in complex social simulations or teamwork. And even in daily life, you could have a family of agents helping out. One suggests recipes, another orders groceries online, a robot helps cook, all coordinating. That sounds potentially amazing, but also complicated. What are the big challenges to making multi-agent collaboration work smoothly? The paper points to three main hurdles. First, communication. Agents need to efficiently share info, intentions, what they're perceiving. They need robust protocols for that.

23:38Okay. Second, coordination. Synchronizing actions, allocating resources intelligently, managing control when there's no single boss, decentralized control. Right. And third, conflict resolution. What happens when their goals or planned actions clash? They need ways to negotiate. Researchers are leveraging existing work in multi-agent systems, emerging communication, and multi-agent reinforcement learning to tackle these. Okay, so as these agents become more integrated into our lives, seeing, hearing, acting alongside us, the ethical side becomes really critical. What are the main concerns the paper highlights?