Fei-Fei Li: World Models and the Multiverse

0:00That space, the 3D space, the space out there, the space in your mind's eye, the spatial intelligence that enable people to do so many things that's beyond language is a critical part of intelligence. If you think of an image, you're like, you know what we're missing? I said, what do I miss? You said, we're missing a world model. I'm like, yes! We can actually create infinite universes. Summer for robots, summer for creativity, summer for socialization, summer for travel, summer for storytelling, it suddenly will enable us to live in a motive where the imagination is boundless. When we talk about AI today, the conversation is dominated by language.

0:43LLMs, tokens, prompts, but whatever missing something more fundamental, not words but space. The physical world we move through in shape. My guess today, I think we are. Faye Le, a pioneer in modern AI, helped usher in the deep learning era by putting data at the center of machine learning. Now she's co -founder and CEO of WorldLapse, building world models. AI systems that perceive and act in 3D space. She's joined by A16Z General Partner, Marquis Casado, computer scientist, repeat founder, and one of the first people Faye -Faye called when forming the company. Today, they explain why spatial intelligence is core to general intelligence and why it's time to go beyond language.

1:25Let's get into it. As a reminder, the content here is for informational purposes only. Should not be taken as legal business, tax, or investment advice, or be used to evaluate any investment or security and is not directed at any investors or potential investors in any A16Z fund. Please note that A16Z and its affiliates may also maintain investments in the company's discussed in this for more details, including a link to our investments. Please see a16z .com forward slash disclosures.

1:58I think you so much for joining us here today. Martine, why don't you briefly brag on behalf of Faye a little bit and how would you summarize your contributions to AI for people unfamiliar? Yeah, someone that doesn't need a lot of introduction and she's done so many things that I can't fill in, so maybe I'll just do the ones that are appropriate to this. Of course, she's on the Twitter board, she's a Google -exec. Found her CEO of World Labs. But very, very importantly, like we all know AI, and we all talk about kind of neural networks, and there's a number of people that focused on making those effective.

2:23But Feifei really singularly brought in data to the equation, which now we're recognizing is actually probably the bigger problem the more interesting one. And so she truly is the godmother of AI as everybody calls her. And Feifei, why did you have to have a Martinez the first investor? Well, first of all, I knew Martin for more than a decade. Oh, no. I joined Stanford in 2009 as a Yanis system professor, and Martin was finishing his PhD there. So I always know, and of course, Martin's advisor, Nick McHugh, was a good friend, and I always know Martin One Town to become a very successful entrepreneur and very successful investor.

2:59So we see each other, we talk about things, but as I was formulating the idea of world labs, I was looking for what I would call my unicorn investor. I don't know if that's a word, but that's how I think about this. Who is not only obviously a very established and successful investor who can be with entrepreneurs on this journey through the ups and downs, who can be very insightful, who can bring the kind of knowledge, advice, resource, but I was also particularly looking for an intellectual partner. Because what we are doing at World Labs is very deep -peck. We are trying to do something no one else has done.

3:40We know with a lot of conviction it will change the world literally. But I need someone who is a computer scientist, who is a student of AI, understand product, market, market, market, market, market. It just can't be on the phone or in person with me every moment of the day as an intellectual partner. And here we are. We talk almost every day. It is true. It's cool. It's actually the origin story of us first connecting. It's actually pretty interesting. So Vefi has clearly been thinking about this idea for a very long time, like well before, sorry. So maybe years even. Yeah. She's a very deep intuition of what AI needs in order to basically navigate the world.

4:21Right? But we were at one of Mark's fancied lunches, and there's a bunch of AI people, and everybody was so excited about LLMs, right? And it was talking about language. And I'd come to this independent conclusion just because I've actually done a lot of image investing That like that wasn't the end of the story and so if they were in the end of this table all these people talking about it Faithy links over to me. She's like, you know what we're missing. I said what are we missing? She said we're missing a world model and I'm like, yes And it fell into place then because I'd been like thinking about stuff at a high level But as she does she's kind of perfectly articulated this so she had a year's worth of thinking about this had talked to people Etc.

4:54So in some way we kind of in our own crooked paths had arrived at a very similar intuition which is hurtful, way more filled out. Mine was just this fancy thing. But then after that, we actually had a number of conversations. We both agreed that we were in line on this idea. Actually, I don't know if you know this. So of course, during that lunch, we hit it off on this world model idea. But I was at that point already talking to various people, not just computer scientists, technologists, but also investors potentially being this partners. And to be honest, most people didn't get it. You know, when I say world model, they nod, but I can just tell that was just a polite nod.

5:33So I call Martin. I'm like, do you mind coming over to Stanford campus? I have coffee with me. And yeah, cool. And then as soon as Martin came and said, Dan, I said, Martin, can you define your world model to me? I really wanted to hear if Martin actually meant it. And the way he defined it about an AI model that truly understand the 3D structure shape and the compositionality of the world was exactly what I was talking about. And I was like, well, he's the only person so far I've talked to who actually meant it. It's not just nodding. Well, okay, so we're going to get to World Labs and the specifics of this.

6:13But first, I want to take you back both to your PhD days, your Professor Days, and reflect on if you could go back in time and sort of have knowledge of what's happened that were sitting ten years. What do you think would have been the biggest surprises? Or was the thing that you didn't see coming that would have shocked your younger self? Or did you have a good sense of how this feeling will play out? Yeah, it's ironic to say because as Martin said, I was the person who brought data into the AI world. But I still continue to be so surprised. not surprised intellectually but surprised emotionally that the data hungry models, the data -driven AI can come this far and genuinely have incredible emergent behaviors of thinking machine, right?

7:01Yeah. Let's get into specifics. Why start another foundation model company? Why aren't LMS? My intellectual journey is not about company or papers, is about finding the North Star problem. So it's not like I woke up and say I have to do a company. I woke up every day after day for the past few years thinking that there is so much more than language. The language is an incredibly powerful encoding of thoughts and information, but it's actually not a powerful encoding of what the 3D physical world, that all animals and living things, living, and if you look at human intelligence so much is beyond the realm of language, language is a lossy way to capture the world and also one subtlety of language is purely generative.

7:54Language doesn't exist in nature. We look around, there's not a syllabus or word, whereas the entire physical, perceptual, visual world is there, and animals' entire evolutionary history is built about so much perceptual and eventually embodied intelligence. Humans not only survive live work, but we build civilization beyond language, constructing the world and changing the world. So that's the problem I want to And in order to tackle that problem, obviously research was important and I spend years doing that as an academic. And it's still fun, but I do realize, and especially talking to Martin, that the time has come that concentrated industry -grade effort, focus effort in terms of compute data talent is really the answer to bringing this to life.

8:56And that's why I wanted to start where laps. Amazing. Eric, you can do a very simple thought experiment that kind of highlights the difference between language and space. So if I put you in a room and I blindfolded you and I just described the room and then I asked you to do a task, the chances of you being able to do it are very little. I'm like, oh, 10 foot in front of you is like a cop. You know, like this is just, it's this very inaccurate way to convey reality because reality is so complex and it's so exact, right? on the other hand, if I took off the blindfold, and you can see the actual space, right?

9:30And what your brain is doing is actually reconstructing the 3D, right? Then you can actually go and manipulate things and touch things, right? And so one way to think about it is we do a lot of language processing and we use that to communicate in the high level ideas at session. But when it comes to navigating the actual world, we really, really rely on the world itself and our ability to reconstruct that. And how and when did you realize that language might be worn enough? Because it seems like it's not super widely known. I don't hear about this all the time. Well, so Davey asked me, like what does this surprising breakthrough?

10:00It's that language went first because we've worked so hard on robotics, right? I mean, I feel like even a look at autonomous vehicles as an industry, we've invested like $100 billion in it. I remember when Sebastian Thrun like actually won like the DARPA Grand Challenge in 2006. 2006, so we're like, hooray! The AV is done, right? And then 20 years later, like we're finally there, $100 billion in it, this is like a 2D problem. And so that was the path we were going on, is do you actually solve world navigation? And it's harder than out of nowhere comes these LLMs. And they are unit economic positive.

10:34They solve all of these language problems like basically immediately. And so it just took me a moment, actually, faithfully said it beautifully early on when we were talking, which is the part of our brain that actually deals with language is actually pretty recent. And so we're actually pretty inefficient at it, right? And so the fact that a computer does it better is not super surprising, but the part of the brain that actually does the navigation You know the spatial has been around. It's a million brains maybe the reptilian brain. We're about four million years Even more than that. It's a trial by break.

11:04Yeah, right? It's trial by had break right 500 million years Yeah, so it's almost like we're unrolling evolution right so the language part is actually very very important for like high -level concepts and like The laptop class type work, which is what it's impacting right now but when it comes to space and this is everything from robotics or anything where you're trying to construct something physical you have to solve this problem and then we know from AV that is a very tough problem and then maybe this is what is we're talking about like the generative wave gave us some insight on how you might want to do it so it really felt like that was the time my journey is very different because I've Always been vision right so I feel like I Didn't need lllm to come in speak lwm is important And I do want to say we're not here bashing language.

11:45I'm just so excited. In fact, seeing chat GPD and LLMs and these foundation models having such breaks through success inspires us to realize the moment is closer for world models. But Martin said it so beautifully. It's that space, the 3D space, the space out there, the space in your mind's eye, the spatial intelligence that enable people to do so many things that's beyond language is a critical part of intelligence. It goes from ancient animals all the way to humanity's most innovative findings, such as the structure of DNA, right? That double helix in 3D space. There's no way you can use language alone to reason that out.

12:32So that's just one example, another one of my favorite scientific examples, was Bucky Ball, carbon molecule structure that is so beautifully constructed. That kind of example shows how incredibly profound space in 3D world is. This paint even more of a picture when World Labs has achieved its vision or language model has achieved their vision. What are some applications or use cases that we can present to the audience to help make it concrete? Yeah, there is a lot, right? For example, creativity is very visual. We have creators from design to to movie, to architecture, to industry design, creativity is not just only for entertainment, it could be for productivity, for machinery, for many things.

13:16That alone is a highly visual, perceptual, spatial area or areas of work. Of course, we mention robotics. Robotics, to me, is any body machines. It's not just humanoid or cars. There's so much in between, but all of them have to somehow figure out the 3D space it lives in have to be trained to understand the 3D space and have to do things, sometimes even collaboratively with humans. And that needs spatial intelligence. And of course, I think one thing that's very exciting for me is that for the entirety of humans' civilization, we all collectively, as people, lived in one 3D world, and that is the physical Earth 3D world.

14:09A few of us went to the moon, but you know, very small number. But that's one world. But that's what makes the digital virtual world incredible, with this technology, which we should talk about, it's the combination of generation and reconstruction. Suddenly, we can actually create infinite universes. Some are for robots, some are for creativity, some are for socialization, some are for travel, some are for storytelling. It suddenly will enable us to live in a multiverse way. The imagination is boundless. I think it's very important because these conversations can sound abstract, but they're actually not.

14:51But the reason they sound abstract is because it's truly horizontal, just like LLM's art, right? So like we've got say like what are LLM's good at the same LLM we use for like an emotional conversation We use to write code Yeah, we use to do lists. We use it for self -actualization, right? And so I think we can get actually pretty concrete about what these models do, right? And so let me just give it a shot and then five is the expert of course So with these models you can take a view of the world like a 2D view of the world and then you could actually Create a 3D full representation including what you're not seeing like the back of the table for example within the computer.

15:25So given just a 2D view, you have the full thing and then you ask, well, what can you do with that thing, for example? Well, you can manipulate it, you can move it, you can measure it, you can stack it. So anything that you would do a space you could do, right? That means you could do architecture, you could design. But it turns out the ability to fill out the back of the table means that you can fill out stuff that was never there to begin with, right? So let's say that I just had a 2D picture of this, I could create a 360 of everything, right? And so now you have fully generative, and so what does that mean?

15:51That means that video games It's creativity. And so it's a super horizontal piece that takes basically a computer with a single view in the world or maybe multiple views in the world and creates a full 3D representation that that computer then can act on. And so you can see that that's a very concrete pivotal thing from everything from robotics to video games to art and design. Yeah. It seems like we haven't fully been appreciating sort of 3D components until now. Is that fair to say? It is fair to say. In fact, I think took evolution a long time. 3D is not a easy problem, but I always come back to the fact that I had a conversation with my six -year -old years ago about why trees don't have eyes.

16:35And the fundamental thing is trees don't move. They don't need eyes. So the fact that the entire basis of animal life is moving and doing things and interacting gives life to perception and spatial intelligence. And in turn, spatial intelligence is going to reinvent horizontally, as Martin said, so many of the way of work and life that humans are doing. Yeah, fascinating. But it is definitely worth asking the question, why can't you just use 2D video for this? 3D is very, very fundamental to this. Maybe you suggested let's get deeper into the technology. What can we share more about how it works or what the breakthrough is or what's worth commenting on the technology?

17:18Two more teams point. Does it need to be 3D or why can't you just use 2D? I think you could do a lot of things using 2D. But the fact is that 2D will get you very far. In fact, today's multimodal LLMs is already making a big difference in the robotic learning world, helping guiding you to know what's next, the state of the world, but fundamentally, physics happens in 3D. And interaction happens in 3D. Navigating behind the back of the table needs to happen in 3D, composing the world, whether physically, digitally needs to happen in 3D. So, fundamentally, the problem is it's 3D problem. One way to think about it is, if it's a human being looking at, say, a 2D video, The human being can reconstruct the 3D in their head.

18:10But let's say I've got a robot that has the output of the model. If that's 2D and then you ask the robot to do, I don't know, distance. So, aren't you proud of something? That information's missing. You've got the XYZ, you played the Z -plane, this isn't there at all. And so, for many things that are spatial, you need to provide that information to the computer so that you can actually navigate in 3D space. And so, 2D video is great if it's a human because we already can turn it into 3D, but like for any computer program, it'll need to be 3D. Actually, I want to tell you a personal story about five years ago.

18:41Ironically, I lost my stereo vision for a few months because I had a cornea injury. And that means I was literally seen with one eye. And like Martin said, my whole life has been trained with stereo vision. So even if I was seen with one eye, I kind of know what the 3D world looked like. But it was a fascinating period as a computer vision scientist and this brought me to experiment, what the world is. And one thing that truly drove home literally is I was frightened to drive. First of all, I couldn't get on highway. That speed I could not. But I was just driving in my own neighborhood. And I realized I don't have a good distance measure between my car and the parked car on a local small road.

19:27Even though I have perfect understanding of how big is my car almost, how big is the neighbors, the park cars I know the roads for years and years, but just driving there, I had to be so slow, like almost 10 miles an hour, so that I don't scratch the cars. And that was exactly why we needed stereo vision. That's actually a great articulation of why 3D is just actually key if you're doing some processing, right? Yeah, so I don't recommend it, but if you're staring, park your car one and drive your car two with one eye and feel it, that's your own car. On the tech side, with LLAMs, a lot of research was done at the big companies.

20:05What's the state of the research here? This is definitely a newer area of research compared to LLAM. It's not totally fair to say new because in computer vision as a field, we have been doing bits and pieces. For example, one important revolution that has happened in 3D, computer vision was a a new radiant field or nerf, and that was done by our co -founder Ben Mildenhock and his colleagues at Berkeley. And that was a way to do 3D reconstruction using deep learning that was really taking the world by storm about four years ago. We've also got a co -founder Christoph Laster, whose pioneering work was part of the reason Gautian's flat representation started to again become really popular as a way to represent volumetric 3D.

20:57And of course, Justin Johnson, who was my former student, also co -founder of World Labs, were among the first generation of deep learning computer vision student who did so much foundational work in image generation when before transformer were out, we were using gas to do image generation and then style transfer, which was really popularized some of the components or ingredients of what we're doing here. So things were happening in academia, things were happening in industry, but I agree what is exciting now is that at World Lab, we just have the conviction that we're gonna be all in on this one singular big North Star problem, concentrating on the world's smartest people in computer vision, in diffusion models, in computer graphics, in optimization, in AI, in data, all of them coming to this one team and try to make this work and to productize this.

22:03I will say from an outsider standpoint, and so I'm not an expert in any of these spaces, but it really feels like to solve this problem, you need experts both in AI, and that's like the data and the models, like the actual model architecture and graphics. which is like how do you actually represent these things in memory in a computer and then on the screen? So it's a very special team to actually practice probably much faith has managed to put together. Well that's an inspiring note to wrap on. Faye, thank you so much for joining us. Thank you, thank you, Eric. Thanks for listening to the A16z podcast.

22:37If you enjoyed the episode, let us know by leaving a review at ratethispodcast .com slash A16z. We've got more great conversations coming your way. See you next time.