The Frontier of Spatial Intelligence with Fei-Fei Li

0:00This is fundamentally philosophically to be a different problem. The previous decade had mostly been about understanding data that already exists. But the next decade was going to be about understanding new data. Visual spatial intelligence is so fundamental, it's as fundamental as language. It is like unwrapping presence on Christmas that every day you know there's going to be some amazing new discovery, some amazing new application or algorithm somewhere. If we see something or if we imagine something, both can converge towards generating it. I think we're in the middle of a Cambrian explosion.

0:40To many, the last two years of AI have felt like a light switch. Pre and post GPT -3. Pre and post being able to generate an image with natural language. And even pre and post translating any video with the click of a button. But to some, like Dr. Fefei Lee, often referred to as the quote, Godmother of AI and long -time professor of computer science at Stanford, who by the way taught some very well -known researchers like Andrei Karpathi, to people like Fefei, artificial intelligence unlocks have existed on a multi -decade long continuum. And that continuum is destined to proceed into the physical spatial world.

1:17At least, that's what Fefei and her co -founders of new company, World Labs, beliefs. And these four founders pioneered the ecosystem in so many ways, from Fei -Fei's image net to Justin Johnson's work on scene graphs, Ben Mildenhall's work on nerfs, or even Kristoff Lassener's work on the precursor to the Gaussian Splat. And in today's episode, you'll get to hear from Fei -Fei and Justin as they explore this evolution with A16Z General Partner, our team Casado. From the very earliest seeds to the recent explosion of consumer grade AI applications and the key watershed moments along the way. Will of course dive into the why now behind world labs, but also their choice to focus on spatial intelligence and what it might really take to build at that frontier, from algorithmic unlocks to hardware.

2:05All right, let's get started. As a reminder, the content here is for informational purposes only, should not be taken as legal, business, tax, or investment advice, or be used to evaluate any investment or security, and is not directed at any investors or potential investors in any A16Z fund. Please note that A16Z and its affiliates may also maintain investments in the company's discussed in this podcast. For more details including a link to our investments, please see A16Z .com slash Disclosures.

2:38Over the last two years, we've seen this kind of massive rush of consumer AI companies and technology and it's been quite wild, but you've been doing this now for decades. And so maybe we'll talk to a little bit about how we got here, kind of like your key contributions and insights along the way. So it is a very exciting moment, right? Just moving back, I is in a very exciting moment. I personally have been doing this for two decades plus, and we've come out of the last AI winter. We have seen the birth of modern AI, Then we have seen deep learning taking off showing us possibilities like playing chess, but then we're starting to see the deepening of the technology and the industry adoption of some of the earlier possibilities like language models.

3:27And now I think we're in the middle of a Cambrian explosion in almost a literal sense because Now in addition to texts, you're seeing pixels, videos, audios, all coming with possible AI applications and models. So it's very exciting moment. I know you both so well. And many people know you both so well because you're so prominent in the field. But not everybody thrupping AI. So maybe it's kind of worth just going through like your quick backgrounds, just the kind of levels at the audience. Yeah, sure. So I first got into AI at the end of my undergrad. I did math and computer science for undergrad at KelTak.

3:59It was awesome. But then towards the end of that, There was this paper that came out that was at the time, a very famous paper, the cat paper, from Hong like Lee and Andrew Aang and others that were at Google Brain at the time. And that was like the first time that I came across this concept of deep learning. And to me, it just felt like this amazing technology. And that was the first time that I came across this recipe that would come to define the next more than decade of my life, which is that you can get these amazingly powerful learning algorithms that are very generic, coupled them with very large amounts of compute, coupled them with very large amounts of data, and magic things started to happen when you compile those ingredients.

4:32So I first came across that idea around 2011, 2012 -ish, and I just thought, oh my god, this is going to be what I want to do. It was obvious you got to go to grad school to do this stuff, and then saw that Feifei was at Stanford, one of the few people in the world at the time who was on that train. And that was just an amazing time to be in deep learning and computer vision specifically. Because that was really the era when this went from these first nascent bits of technology that were just starting to work, and really got developed and spread across a ton of different applications. So then over that time, we saw the beginnings of language modeling, we saw the beginnings of discriminative computer vision where you could take pictures and understand what's in them in a lot of different ways.

5:07We also saw some of the early bits of what we would now call Genai, generative modeling, generating images, generating text. A lot of those core algorithmic pieces actually got figured out by the academic community during my PhD years. It was a time I would just wake up every morning and check the new papers on archive and just be ready. It's like unwrapping presents on Christmas. Every day you know there's going to be some amazing new discovery, some amazing new application or algorithm somewhere in the world. In the next two years, everyone else in the world kind of came to the same realization of using AI to get new Christmas presents every day.

5:36But I think for those of us that have been in the field for a decade or more, we've sort of had that experience for a very long time. I come to AI through a different angle, which is from physics, because my undergraduate background was physics, but physics is the kind of discipline that teaches you to think audacious questions and think about what is the still remaining mystery of the world, of course, in physics, in the atomic world, you know, universe and all that. But somehow that kind of training, thinking got me into the audacious question that really captured my own imagination, which is intelligence.

6:12So I did my PhD in AI and computational neuroscience at Caltech. So Justin and I actually didn't overlap, but we share the same alma mater at Caltech. And the same advisor. Yes. same advisor, your undergraduate advisor, my PhD advisor, Pietro Perona. And my PhD time, which is similar to your PhD time, was when AI was still in the winter in the public eye. But it was not in the winter in my eye because it's that free spring hibernation. There's so much life, machine learning, statistical modeling was really getting power. I think I was one of the native generation in machine learning and AI, whereas I look at Justin's generation is the native deep learning generation.

7:01So machine learning was the precursor of deep learning and we were experimenting with all kinds of models. But one thing came out at the end of my PhD and the beginning of my assistant professor time. There was a overlooked elements of AI that is mathematically important to drive generalization, but the whole field was not thinking that way and it was data. Because we were thinking about the intricacy of Bayesian models or kernel methods and all that, but what was fundamental that my students and my lab realized probably earlier, the most people is that if you let data drive models, you can unleash the kind of power that we haven't seen before.

7:49And that was really the reason we went on a pretty crazy bet on image that, which is, you know, just forget about any scale we're seeing now, which is thousands of data points. At that point, an LP community has their own data sets. I remember you see Irvine data set or some data set in NLP was, it was small conversion community has their data sets, but all the order of thousands or tens of thousands were like, we need to drive it to Internet scale. And luckily, it was also the coming of age of Internet. So we were riding that wave, and that's when I came to Stanford. So these epochs are what we often talk about.

8:34ImageNet is clearly the epoch that created, or at least maybe made popular and viable computer vision. In the Transformers paper, which is attention, we talk about stable diffusion. Is that a fair way to think about this, which is there's these two algorithms that come locks that came from academia or Google, and that's where everything comes from, or has it been more deliberate? Or have there been other kind of big unlocks that kind of brought us here that we don't talk as much about? I think the big unlock is compute. I know the story of AI is often the story of compute, but no matter how much people talk about it, I think people underestimate it.

9:06And the amount of growth that we've seen in computational power over the last decade is astounding. The first paper that's really credited with the breakthrough through moment in computer vision for deep learning was Alex Net, which was a 2012 paper where a deep neural network did really well on the image net challenge and just blew away all the other algorithms that they've been working on, the types of algorithms that you'd been working on more in grad school. That Alex Net was a 60 million parameter deep neural network and it was trained for six days on two GTX 580s, which was the top consumer card at the time which came out in 2010.

9:37So I was looking at some numbers last night just to put these in perspective and the newest the latest and greatest run video is the GB200. Do either of you want to guess how much raw compute factor we have between the GTX 580 and the GB200? Shoot, no, what? Go for it. It's in the thousands. So I ran the numbers last night, that two week training run, that of six days on two GTX 580s. If you scale, it comes out to just under five minutes on a single GB200. Justin is making a really good point. The 2012 Alex that paper on ImageNet Challenge is literally a very classic model. And that is the convolutional neural network model.

10:18And that was published in 1980s, the first paper, I remember as a graduate student learning that. And it more or less also has six, seven layers. Practically the only difference between Alex, that and the confnet, the difference is the two GPUs and the deluge of data. Yeah. So I think most people now are familiar with quote the bitter lesson. And the bitter lesson says, if you make an algorithm, don't be cute, just make sure you can take advantage of available compute because the available compute will show up. On the other hand, there's another narrative, which seems to me to be just as credible, which is it's actually new data sources that unlock deep learning, right?

10:56Like ImageNet is a great example. Self -attention is great from Transformers, but they'll also say this is a way you can exploit human labeling of data because it's the humans that put the structure in the sentences. And if you look at Clip, let's say, well, I could pre -using the internet to like actually have humans use the Alt tag to label images, right? And so that's a story of data, that's not a story of compute. And so is the answer just both? Or is one more than the other? I think it's both, but you're hitting another really good point. So I think there's actually two epochs that to me feel quite distinct in the algorithmics here.

11:24So the ImageNet era is actually the era of supervised learning. So in the era of supervised learning, you have a lot of data, but you don't know how to use data on its own. Like the expectation of ImageNet and other datasets of that time period was that we're gonna get a lot of images, but we need people to label everyone. And all of the training data that we're going to train on, a human labeler has looked at every one and said something about that image. And the big algorithmic unlocks, we know how to train on things that don't require human label data. As the naive person in the room that doesn't have an AI background, it seems to me if you're training on human data, the humans have labeled it, it's just not explicit.

11:58I knew you were going to say that, Martin, I knew that. Yes, well, so typically that's a really important question. But that actually is more true in language than pixels. Fair enough, yeah. I do think it's an important distinction because Flip really is human -labeled. I think attention is humans have figured out the relationships of things and then you learn them. So it is human -label just more implicit than explicit. Yeah, it's still human -labeled. The distinction is that for this supervised learning era, our learning tasks were much more constrained. So you would have to come up with this oncology of concepts that we want to discover.

12:30right? If you're doing image net, faith A and your students at the time spent a lot of time thinking about which thousand categories should be in the ImageNet challenge. Other data sets of that time, like the Cocoa Data Set for Object Detection, they thought really hard about which 80 categories we put in there. So let's walk to Genai. So when I was doing my PhD before that you came, so I took machine learning from Andering and then I took the Asian something very complicated from Deaf and Colour and it was very complicated for me. A lot of that was just predictive modeling and then I remember the whole kind and a vision stuff that you unlock.

12:59But then the generative stuff is shown up, like I would say in the last four years, which is to me, very different. You're not identifying objects, you're not predicting something, you're generating something. And so maybe kind of walk through like the key unlocks that got us there and then why it's different. And if we should think about it differently and is it part of a continuum, is it not? It is so interesting. Even during my graduate time, generative model was there. We wanted to do generation, nobody remembers, even with letters and numbers, we were trying to do some. Jeff Hinden has had generated papers.

13:33We were thinking about how to generate. And in fact, if you think about probability distribution point of view, you can mathematically generate. It's just nothing we generate would ever impress anybody, right? So this concept of generation mathematically, so radically is there. But nothing worked. Just as PhD, his entire PhD is a story, almost a mini story of the trajectory of the field. He started his first project in data. I forced them to. He didn't like it. So, in retrospect, I learned a lot of really useful things. I'm glad you say that now. So actually, my first paper, both of my PhD and like ever, my first academic publication ever, was the image retrieval with scene graphs.

14:18And then we went into taking pixels generating words and just then and Andre really worked on that. But that was still a very very lossy way of generating and getting information out of the pixel world. And then in the middle, just the one to off and did a very famous piece of work. And it was the first time that someone made it real time, right? Yeah, yeah. So the story there is there was this paper that came out in 2015, a neural algorithm of artistic style led by Leon Gattis. And the paper came out and they showed these real world photographs that they had converted into a Van Gogh style. And we are kind of used to seeing things like this in 2024, but this was in 2015.

14:59So this paper just popped up on archive one day and it blew my mind. I just got this geni -i brainworm in my brain in 2015 and it did something to me and I thought, oh my God, I need to understand this algorithm. I need to play with it. I need to make my own images into Van Gogh. So then I like read the paper and then over a long weekend, and I re -implemented the thing and got it to work. It was actually very simple algorithm. So like my implementation was like 300 lines of Lua because at the time it was pre -... Lua. It was Lua. This was pre -py torch. So we were using Lua torch. But it was like very simple algorithm, but it was slow, right?

15:30So it was an optimization -based thing. Every image you want to generate, you need to run this optimization loop, run this gradient -distant loop for every image that you generate. The images were beautiful, but I just wanted to be faster. And Justin just did it. It was actually, I think, your first taste of an academic work having an industry impact. A bunch of people had seen this artistic style transfer stuff at the time. And me and a couple others at the same time came up with different ways to speed this up. But mine was the one that got a lot of traction. Before the world understand Gen AI, Justin's last piece of working PhD was actually inputting language and getting the whole picture out.

16:08It's one of the first Gen AI workets using GEM, which was so hard to use the problem is that we are not ready to use a natural piece of language. So Justin, you heard he worked on scene graph. So we have to input a scene graph language structure. So the sheep, the grass, the sky in the graph way. It literally was one of our photos. And then he and another very good master student, Grim, they got that again to work. So, you can see from data to matching, to style transfer, to generative images, we're starting to see you ask if this is a drop change for people like us. It's already happening in a continuum.

16:53But for the world, the results are more abrupt. So I read your book and for those that are listening to a phenomenal book, I really recommend you read it. And it seems for a long time, like a lot of you, and I'll talk to you faith, like a lot of your research has been and your direction has been towards kind of spatial stuff and pixel stuff and intelligence and now you're doing world labs and it's around spatial intelligence. And so maybe talk through, is this been part of a long journey for you? Like, why did you decide to do it now? Is it a technical unlock? Is it a personal unlock? Move us from that may lieu of AI research to world labs.

17:28For me, it is both personal and intellectual, right? my entire intellectual journey is really this passion to seek North Stars, but also believing that those North Stars are critically important for the advancement of our field. So at the beginning, I remembered after graduate school, I thought my North Star was telling stories of images, because for me, that's such an important piece of visual intelligence. That's part of what you call AI or AGI, but when Justin and Andre did that, I was like, oh my god, that's that was my livestream. What do I do next? So it came a lot faster. I thought it would take a hundred years to do that.

18:13But visual intelligence is my passion because I do believe for every intelligent being like people or robots or some other form knowing how to see the world reason about it, interact in it, whether you're navigating or manipulating or making things, you can even build civilization upon it. Visual spatial intelligence is so fundamental, it's as fundamental as language, possibly more ancient and more fundamental in certain ways. So it's very natural for me that our Our North Star is to unlock spatial intelligence. The moment to me is right. We've got these ingredients. We've got compute. We've got much deeper understanding of data.

19:06Way deeper than image that days, compared to those days where so much more sophisticated. And we've got some advancement of algorithms, including co -founders in World Lab like Ben the Mildenhorn and Christoph Laster, they were at the cutting edge of nerve that we are in the right moment to really make a bet and to focus and just unlock that. So I just want to clarify for folks that are listening to this, you're starting this company world lab, Spatial Intelligence is kind of higher generally described in the problem you're solving. Can you maybe try to crisply describe what that means? Yeah, so Spatial Intelligence is about machines ability to perceive, reason, and act in 3D space and time.

19:50To understand how objects and events are positioned in 3D space and time, how interactions in the world can affect those 40 positions over space time, and both perceive, reason about, generate, interact with, really take the machine out of the mainframe or out of the data center and putting it out into the world, and understanding the 3D 40 world with all of its richness. So to be very clear, are we talking about the physical world or are we just talking about an abstract notion of world? I think it can be both. I think it can be both and that encompasses our vision long -term. Even if you're generating worlds, even if you're generating content positioned in 3D has a lot of benefits.

20:24Or if you're recognizing the real world, being able to put 3D understanding into the real world as well is part of it. Just for everybody listening, like the two other co -founders Ben Noltholakristoff Lastner are absolute legends in the field at the same level. These four decided to come out and do this company now, and so I'm trying to dig to like why now is the right time. Yeah, I mean, this is again part of a longer evolution for me, but post -PhD, when I was really wanting to develop into my own independent researcher, both for my later career, I was just thinking, what are the big problems in AI and computer vision?

20:53And the conclusion that I came to about that time was that the previous decade had mostly been about understanding data that already exists. But the next decade was going to be about understanding new data. And if we think about that, the data that already exists was all of the images and videos that maybe existed on the web already. And the next decade was going to be about understanding new data, right? People have smartphones, smartphones are collecting cameras. Those cameras have new sensors. Those cameras are positioned in the 3D world. It's not just you're going to get a bag of pixels from the internet and know nothing about it and try to say if it's a cat or a dog.

21:24We want to treat these images as universal sensors to the physical world. And how can we use that to understand the 3D and 40 structure of the world, either in physical spaces or generative spaces? So I made a pretty big pivot post PhD into 3D computer vision, predicting 3D shapes of objects with some of my colleagues fair at the time. Then later, I got really enamored by this idea of learning 3D structure through 2D. Because we talk about data a lot, 3D data is hard to get on its own. But because there's a very strong mathematical connection here, our 2D images are projections of a 3D world. And there's a lot of mathematical structure here we can take advantage of.

21:59So even if you have a lot of 2D data, there's a lot of people who've done amazing work to figure out how can you back out the 3D structure of the world from large quantities of 2D observations. And then in 2020, you asked about very breakthrough moments. There was a really big breakthrough moment from our co -founder Ben Nildenhall at the time with his paper Nerf and the Royal Radiance Fields. And that was a very simple, very clear way of backing out 3D structure from 2D observations. That just lit a fire under this whole space of 3D computer vision. I think there's another aspect here that maybe people outside the field don't quite understand.

22:30That was also a time when large language models were starting to take off. So a lot of this stuff with language modeling actually had gotten developed in academia even during my PhD I did some swerly work with Andre Carpati on language modeling in 2014. LSTM, I remember. LSTM, Tarnand, GRU, like this is pre -transformer, but then at some point like around the GPT2 time like you couldn't really do those kind of models anymore in academia because they took a way more resourcing, but there was one really interesting thing. The Nerf approach that Ben came up with, like you could train these in a couple hours on the single GPU.

23:02So I think at that time there was a dynamic here that happened, which is that I think a lot of academic researchers ended up focusing a lot of these problems because there was core algorithmic stuff to figure out. And because you could actually do a lot without a ton of compute and you could get state of the art results on a single GPU. Because of those dynamics, there was a lot of research, a lot of researchers in academia were moving to think about what are the core algorithmic ways that we can advance this area as well. Then I ended up chatting with Faith Aymor and I realized that we were actually, she's very convincing.

23:30She's very convincing. Well, there's that. But you were talking about trying to figure out your own independent research trajectory from your advisor. Well, it turns out we had ended up kind of concluding on similar things. Thank you. Well, for my end, I want to talk to the smartest person. I can't just that there's no question about it. I do want to talk about a very interesting technical story of pixels that most people working language don't realize is that pre -genii era in the field of computer vision, those of us who work on pixels, we actually have a long history in an area of research called reconstruction, 3D reconstruction.

24:08It dates back from the 70s. You can take photos because humans have two eyes, right? So in general, it starts with stereo photos and then you try to triangulate the geometry and make a 3D shape out of it. It is a really, really hard problem. To this day, it's not fundamentally solved because there's correspondence and all that. So this whole field, which is an older way of thinking about 3D, has been going around and it has been making really good progress. But when there's happened in the context of generative methods, in the context of diffusion models, suddenly reconstruction and generation start to really merge.