#303 Fei-Fei Li: Spatial Intelligence, World Models & the Future of AI

0:00The spatial intelligence work I've been thinking about in the past few years is truly a continuation of my entire career's focus in computer vision and visual intelligence. Why did I emphasize on spatial is because we've come to a point in our technology that the level of sophistication and profound capabilities of this technology is no longer at the level of staring at an image or even simple understanding, simple videos. It is deeply, deeply perceptual, spatial, and also connects to robotics, connects to embodied AI, as well as ambient AI. Welcome to the podcast. In this episode, I have the honor of talking again to Fei-Fei Li, a pioneer in artificial intelligence and computer vision.

0:51I had Fei-Fei on the podcast a few years ago, and I invite you all to go listen to that episode. I'll put a link at the end of this video. We'll explore her insights on world models and the importance of spatial intelligence, crucial elements for creating AI that truly understands and interacts with the world around us. While large language models are amazing, much, if not most, of human knowledge is not captured in text. And to reach a more general artificial intelligence, models need to experience the world firsthand, or at least through video. We talk about her startup World Labs and their first product, Marble, which generates incredible complex 3D spaces from the model's internal representations of the world.

1:44Plus, she tells us her guilty pleasure watching her favorite TV show on airplanes. and you probably won't be surprised what that show is. Stay tuned for an enlightening conversation. I'm Fei-Fei Li. I'll be joining the podcast of Eye on AI and discussing spatial intelligence and world models. Come and join us. Build the future of multi-agent software with Agency. That's A-G-N-T-C-Y. Now an open source Linux foundation project, agency is building the Internet of Agents, a collaborative layer where AI agents can discover, connect, and work across any framework. All the pieces engineers need to deploy multi-agent systems now belong to everyone who builds on agency, including robust identity and access management that ensures every agent is authenticated and trusted before interacting.

2:51Agency also provides open, standardized tools for agent discovery, seamless protocols for agent-to-agent communication, and modular components for scalable workflows. Collaborate with developers from Cisco, Dell Technologies, Google Cloud, Oracle, Red Hat, and more than 75 other supporting companies to build next generation AI infrastructure together. Agency is dropping code, specs, and services, no strings attached. Visit agency.org to contribute. That's A-G-N-T-C-Y dot O-R-G. I wanted to talk not so much about Marble, your new model, which is amazing, that generates a sort of consistent and persistent 3D worlds that the viewer can move through.

3:56I want to talk more about why you're focusing on world models on spatial intelligence, why that is necessary to go beyond learning in language, and how your approach differs from the Kuhn's. So can you talk first of all about, did the world model work emerge from your work on ambient intelligence, or has this been a parallel track? The spatial intelligence work I've been thinking about in the past few years is truly a continuation of my entire career's focus in computer vision and visual intelligence, right? And why did I emphasize on spatial is because we've come to a point in our technology that the level of sophistication and profound capabilities of this technology is no longer at the level of staring at an image or even simple understanding, simple videos.

5:05It is deeply, deeply perceptual, spatial, and also connects to robotics, connects to embodied AI, as well as ambient AI. So from this point of view, it's a continuation, really, of just my entire career in computer vision and AI. And the importance of spatial intelligence, and I've talked about this, I've had this podcast for a while, human knowledge, language models learn from human knowledge encoded in text. but that's a very finite subset of human knowledge and as you pointed out many people pointed out humans learn a lot from interacting in the world without language so it's important we're going to move beyond the current LLMs as amazing as they are that we develop models that have more direct experience of the world, learn more directly from the world.

6:13Your approach has been, well, certainly with Marble, to take the internal representations of the world that the model learns and create an external visual reality with those. Lacoon's approach is to create internal representations from direct experience or video input that allows a model to learn physical laws of motion and things like that. Is there a parallel between that? Are the two approaches complementary or are they overlapping? So yeah, so first of all, I actually would not pit myself against Young or vice versa, because I think we are on this continuum of intellectual

7:17approaches to spatial intelligence and world modeling. So you probably read my recent long essay that I called Manifesto of Spatial Intelligence, and there I was very clear. I actually think both implicit representation and eventually some level of explicit representation, especially in the output layer, are both probably needed if we are considering a universal, omnipotent world model eventually. And they both play roles. For example, our current world model in WarLabs, Marble, does explicitly output 3D representation. But within the model, there is actually also implicit representation, you know, along with the explicit output.

8:13So it is, eventually, I think we need both, to be honest. And also, in terms of the input modality, yes, learning from videos is very important. You know, the whole world is a continuous input of lots of frames. But the whole world to intelligent or just to animals is not just watching passively. It's also, you know, an embodied experience of movements, of interaction, of tactile experience, of sound, of, you know, smell and also just forces, you know, physical forces and temperature and all this, right? So I think it's deeply multimodal. And if you even look at, of course, Marble as a model is just the first step.

9:12But in our tech piece that we released just a few days ago, we were pretty clear that we believe in multimodality as a learning paradigm, as well as an input paradigm. So, yeah, so I think, you know, there's a lot of intellectual discussions on this. It also shows the excitement in the early days of this model. So I would not say that we have finished exploring exactly the model architecture, the representation and all that. In your world model, the inputs are primarily videos, and then the model builds an internal representation of the world? Not quite. If you experience Marble, our world model, our inputs are generally pretty multimodal.

10:14You can do a pure text. You can do one to multiple images. You can do videos. You can also input coarse 3D layouts, you know, like boxes or voxels. So it's multimodal, and I think we're going to deepen that as we go. And is the ambition beyond, you know, it's a fantastic product with many applications, is the ambition to build a system, and when I said the input is video, but to build a system that can learn from direct experience, whether it's through video or some other modality, but it's not learning through a secondary medium like text. Yes, I think world model is about learning about the world, and the world is very multi-model.

11:13And whether it's machines or animals, we are multi-sensory as well. So the learning is through sensing, and the sensing has different modalities. So text is one form. Again, this is where we depart from, you know, animals, because most animals don't learn through sophisticated language, but humans do. But, you know, our today's AI's world model will learn from a lot of language input as well as other modality. But it's not squeezed through only language. Yeah, and one of the limitations of LLMs is that the models are, the model parameters are fixed after training. So the models don't learn continuously.

12:07There's a certain amount of learning at test time inference. Is that something that you're tackling with world models? because presumably a world model is learning as it encounters new environments. Yeah, the continuous learning paradigm is definitely a very, very important one, especially for, you know, living beings. That's how we do it. And even there, there is, you know, online learning versus offline learning. So in the current form of our world model, we are still more in the batch or offline learning mode. But we're definitely open to, you know, continuous, especially eventually online learning modality.

13:02Yeah. And how would that, I mean, it wouldn't be a very different architecture. It would simply be a matter of engineering. Is that right? Well, I would be open-minded. You know, yeah. Yes, I think it's going to be a mixture of both. You know, obviously, good engineering, good, you know, fine-tuning online, you know, learning can already happen, but maybe there will be new architecture. Yeah, yeah. Can you talk about the real-time frame model, which underlies Marble and your work with world models? Yeah, so you're referring to a tech blog that we also put out a couple of weeks ago that specifically double clicks on our real-time frame model.

13:56So WorldLab is a very research-heavy organization. We also care about product, but so much of our work right now in this stage is model first. So we're definitely looking at how we can push forward spatial intelligence. And this particular line of work, which definitely connects to marble, is really focusing on how we can achieve frame-based generation with as much geometric consistency and permanency as we can. Because some of the early work in frame-based generation, you lose that permanency as you move forward. But in this particular case, we try to really balance and also do it in a compute efficient way during inference, which in this case we achieve through a single H100 in inference time.

15:00We don't really know. Some other frame-based models, they don't tell us how many chips they use during their inference time. So we have a hypothesis. It's quite a number. But we don't know that information to compare ourselves. Yeah, and in your manifesto, I think you called it, you talk about the need for a universal task function. A universal what function? Universal task function. Oh, yes, yeah. Analogous to the next prediction, next token prediction in language models. Does RTFM does have a prediction element to it? What are you talking about when you say universal task function beyond what the predictive element of RTFM?

16:03Well, so one of the biggest breakthrough of Gen.AI is really this discovery of the objective function of next token prediction, right? Because it really is such a beautiful formulation because language is in this kind of sequential, you can tokenize language in this kind of sequential representation. and your learning function of next token prediction is precisely what you need during inference, which is as you generate language, whether humans generate or computers generate, it really is putting token after token forward. So having an objective function that is 100 % aligned with the actual eventual task that is supposed to achieve or carry out is just great because it makes the optimization just on target.

17:12In computer vision or in world modeling, it is not as simple because if you look at our relationship with language, it really is just to speak it or to generate it. There's no language in nature that you're staring at it. I mean, eventually you learn to read, but that's because it's already generated. So, but there is, you know, your relationship with the language is deeply generative. It literally is a generated thing by humans. But the relationship with the world is so much more multimodal, I would say, right? There is a world out there for you to observe, for you to interpret, for you to reason with it, for you to eventually interact with it.

18:08But there is also a mind's eye that can formulate different versions of reality as well as imagination and also allow you to also generate the story, the imaginal world. or so it's much more complex. So what is the task that defines or the objective function that defines a universal function that we can use that is as powerful as next token prediction? It's actually a really profound question because is it, let's say, is it 3D reconstruction? You know, just some people actually would argue that the universal task for world modeling could be that it's just be able to 3D reconstruct the world. Because if that is the objective function and if we achieve that, a lot of things will fall through, just fall naturally out of it.

19:16But that is, you could also argue, I don't think so, because most animal brain don't necessarily do precise 3D reconstruction. You know, like, it's not clear a tiger or a person actually can reconstruct the world, yet we're such powerful, visually intelligent, spatially intelligent beings. So maybe that's not necessarily the right task. Then is it the next frame prediction as if it's the next token prediction? Well, there's some power to that, right? First of all, there's a lot of training data for this. Second is that in order to predict the next frame, you have to learn the structure of the world because worlds are not white noise.

20:11So there's a lot of structure that connects one frame to the other. And if you do it well, maybe that is the right universal task or objective function. But maybe it is also unsatisfying because you treat the world as 2D. And the world is not 2D. It's so you do you really force the representation, collapse it in a very unsatisfying way. And also, even if you managed, you can say if you managed to do it perfectly, 3D is implicit. That's true. But it's also very wasteful because with the 3D structure, there's actually a lot more information that you don't have to lose in the way that frame-based prediction might.

21:07So there is still a lot of exploration on that. RTFM. And I have to ask you, was that a joke, naming it RTFM? It was a brilliant play with the, I did not invent this. One of our researchers is really brilliant in naming, right? Like, as you know, you know, I don't know how much we're allowed to say those words. I'll say it. Yeah, read the fucking manual. That's the acronym, yeah. Every computer scientist knows that. So we find that really fun to just play with that name. Yeah, but RTFM is predicting the next frame, right? That's, that's, and. With 3D consistency. Yes, and that's what's interesting about the internal representation that, that the model learns.

22:06I mean, sitting here, I'm looking at my computer screen. I know what the other side of the computer screen looks like, even though I can't see it. And there's an internal representation in my mind of that. And your model does that. I mean, that's why you can move around objects, even though it's a 3D representation on a 2D screen, but you can move around, see the other side of things. So the model has an internal representation of the 3D object even if its current view can't see the backside of something. And that really interests me. That's essentially what you're talking about when you say spatial intelligence, understanding that 3D world.

23:07So does that learning include things like the physical laws of nature? I mean, does it understand that you can't walk through a solid object? or that I think on one of the podcasts I saw you on, somebody was talking about using this for people with fear of heights. So you could look over the edge of something. But if you create a physical representation, I mean an explicit representation of a cliff, let's say, and you move the point of view of the agent or the viewer over that cliff, will it know that it's no longer standing on solid ground or will it float in space? Yeah, so what you are describing is both physical and semantic, to be honest.

24:18Like, you know, of course, falling off a cliff is very much dependent on the law of gravity and all that. But the fact that you, you know, going through a wall, it's very much material-based and semantics-based, right? Solid object versus non-solid object. So RTFM as a current model right now is not focusing on the physics yet. But most of the physics, to be honest, coming out of the Gen AI models are statistics. If you look at, for example, you know, a lot of these Gen video models where you do see water running and trees moving, that is not based on a Newtonian law of forces and masses. It's really based on I've seen plenty of movements of water and leaves in this particular way, and I'm just going to follow that statistical pattern.

25:24So we have to be a little careful. Right now, World Labs is still focusing on generating and exploring static worlds. We are going to explore dynamic. and there a lot of that will be you know statistical learning um i don't think any of today's ai language ai or pixel ai has the capability to abstract at a different level and deduce physics like at the level of a newtonian law out of out of it everything we've seen is statistics-based, statistical-based physics and dynamic, physical and dynamical learning. We could, on the other hand, put these worlds into physics-based, physics engines. And these engines have the laws of physics.

26:28And they eventually, these physics engines, game engines, and also world generation, eventually they're going to combine into neural engines. I don't even know. Maybe we should call them neural spatial engines or something like that. I think that we are moving towards that direction, but it's still early days. Yeah. Yeah. And I didn't mean to pit you against Jan. What I was driving at is it seems like you're focused on explicit representations coming out of an abstract internal representation. Jan is just focused on the internal representation and learning, and that's where the learning takes place.

27:18And it just seemed to me that they would marry beautifully. That's a possibility. Like I said, like you already repeated, we are, you know, We are exploring both. The explicit output is actually a very deliberate approach because we want to be useful for people. We want to be useful for people who are creating, who are simulating, who are designing. And if you look at today's industry, whether you're creating VFX effects or you're creating games or you're designing interiors or you're simulating for robots or autonomous vehicles or industry digital twins or whatever you're doing, it's very 3D, right?

28:06So there's entire industry after industry that's very much 3D in the workflow of 3D. And we want to be absolutely useful for people and businesses to use these models. And the reason I brought up continuous learning, right now, the model understands, you know, depth of field and other sort of spatial properties. What is the learning that is implicit in the model or that the model has that allows it to generate that explicit representation? I mean, because the ultimate goal is to build a model that presumably can learn over time. I mean, I'm sure everybody, you and everybody else, I mean, I imagine it having a model that maybe it's on a robot or maybe it's attached to a, it's getting data from a video camera that moves around in the world.

29:30but that eventually learns not only the scene that it's seeing, but understands the physicality of the space. And eventually then you marry that with language and you've got a really powerful intelligence. Is that something that you think about? And that would require continual learning, that it's not a finite data set that you're feeding the model, that the model is continuously learning from its interaction in the world. Yeah, definitely, especially as one comes close to a use case, especially if the use case requires continuous learning. You know, there are many ways of continuous learning, right?

30:20Like in language model, we see that taking to context itself is continuous learning, right? like having the context as memory. So it helps inference. And then, of course, there is other methods of online learning, of fine tuning. So continuous learning is a term that can encompass multiple ways of doing it. And I think in spatial intelligence, especially like you said, some of these use cases, whether it's robots in personal or in customized situations, or or artists with particular styles, creators with particular styles, these will all eventually push the technology to be able to be more responsive in whichever time horizon that the use case requires it to be.

31:15Some are real-time, some are maybe just more segmented in the time horizon point of view. So it depends. Build the future of multi-agent software with Agency. That's A-G-N-T-C-Y. Now an open source Linux Foundation project, Agency is building the Internet of Agents, a collaborative layer where AI agents can discover, connect, and work across any framework. All the pieces engineers need to deploy multi-agent systems now belong to everyone who builds on agency, including robust identity and access management that ensures every agent is authenticated and trusted before interacting. Agency also provides open standardized tools for agent discovery, seamless protocols for agent-to-agent communication, and modular components for scalable workflows.

32:24collaborate with developers from cisco dell technologies google cloud oracle red hat and more than 75 other supporting companies to build next generation ai infrastructure together agency is dropping code specs and services no strings attached visit agency.org to contribute that's That's A-G-N-T-C-Y dot O-R-G. Yeah. And, you know, you've come a tremendous way since you're running a dry cleaning shop, as I recall, in New Jersey. And in a very short period of time. And you're moving very quickly now. do you have any sense of where you'll be in five years with this technology? Will you have, you know, some physics engine built into the model, or will you have, you know, the ability to learn in longer time frames and build up richer internal representations that the model can begin to understand the physical world.

33:53Yes. I actually think, you know, as a scientist, it's very hard for me to give a prediction in terms of precise time because some part of the technology has moved much faster than I thought, some are much slower than I thought. But I think that is very much a very good goal. And also, five years is kind of a fair guesstimate. I don't know if we'll be faster, but it's a better guesstimate than 50 years, in my mind, or five months. Can you talk a little bit about why you think spatial intelligence is the next frontier? I mean, we've already said that human knowledge is contained in text is only a subset of all human knowledge.

34:51And while it's very rich, you can't expect an AI model to understand the world simply through text. Can you talk about why that's important and how Marble and World Labs relates to that larger goal? Yeah, so I think fundamentally technology should help people. In the meantime, understanding the science of intelligence itself is the most fascinating, audacious, ambitious scientific quest I can think of myself, and it's a quest of 21st century. Both of these, whether you're fascinated by the curiosity of the science or you're motivated by using technology to help people, it points to one fact is that so much of our intelligence and so much of our intelligence at work goes beyond language, right?

35:56It's, I kind of lightheartedly say that you cannot use language to put down the fire. In my manifesto, I described examples, whether it's the spatial reasoning deduction of DNA double helix structure or a first responder putting down fire in a rapidly evolving situation with a team of coworkers. a lot of this goes beyond language so it is obvious both in terms of use case point of view as well as in terms of a scientific quest that we should try our best to unlock how to develop spatial intelligence technology that you know takes us to the different to the next level so So that's really the 30 ,000 feet view that describes how I'm motivated by this dual purpose of scientific discovery, as well as making useful tools for people.

37:09And then we can dive deeper into, for example, being useful, like we alluded to a little bit earlier, whether you're talking about creativity or simulation or design or immersive experiences or education or healthcare or, you know, manufacturing. There's just so much that you could do with spatial intelligence. I actually, I'm very excited that so many people who are thinking about education and immersive learning and immersive experiences are telling me that Marble, our first release of the model, is inspiring them to think how to use it for immersive experiences, you know, to make learning more interactive and fun.

38:01And that's just so natural because, you know, pre-verbal children learn entirely through immersive experience. And even today as grown-ups, our life, right, like so much of our life is immersed in this world and involves, Of course it involves speaking and writing and reading, but it involves doing and interacting and enjoying and all that. So it's so natural. Yeah. I mean, one of the things that has struck everybody is that this marble, I'm not sure it's marble or the itself or the RTFM, the generation of next frame in a moving through a space, is that it's running on one H100 GPU. And I've heard you in other talks refer to experiencing the multiverse, which, you know, everyone was very excited about until they realized how much computation it was going to take and how expensive it is.

39:29do you really think that this is a step toward creating worlds for education, for example, because you've been able to reduce the computational load? Not just that. First of all, I really believe at the inference front, we will be speeding up, we will be more efficient, and we'll be also better, bigger, higher quality, um longer you know um experiences all this i that is the trend of technology i i really do things are moving fast i also do believe in multiverse a multiverse experience you know humanity um as far as we know the entire history of humanity our world experience is in one world And it's literally physically this earth, right?

40:26Okay, a handful of human beings have gone on to the moon, but that's about it. And that is the one shared 3D space. And we build civilization. We live our lives. We do everything in it. But with the digital revolution, digital explosion, We are moving part of our lives in digital world. And there is also a lot of crossover. This is not, I don't want to paint a dystopian picture of we have abandoned our physical world. nor am I going to paint a utopian, total hyperbolic utopian world of everybody wear a headset and never even enjoy and look at our beautiful real world. And that's the fullest of life.