Teaching AI to Understand the Physical World, with Dr. Fei-Fei Li of World Labs

0:05Hi, listeners, and welcome back to KnowPriors. Today's guest is Dr. Feifei Li, a pioneer in computer vision and deep learning. She created ImageNet, the groundbreaking data set that helped spark the deep learning revolution. Fei-Fei is a Stanford professor and the co-director of the Stanford Institute for Human-Centered AI. She's also led AI Google Cloud, advised international policymakers, and recently co-founded World Labs, a company dedicated to developing spatially intelligent AI. Fei-Fei, thank you for joining us today. Well, thanks for inviting me. This is going to be fun. So you have made extraordinary contributions to science and policy over the past two decades.

0:42I'll start with the biggest question, like why start a company now? Because in my heart, I want to build. I see this as such a critical and fun and exciting moment to build some extraordinary technology that everybody can use. And I believe so much in spatial intelligence and the kind of 3D world models that can empower so many people as well as so many use cases. And I think that's just, it's going to be really exciting. And I can do that with an extraordinarily, extraordinarily brilliant group of young technologists. I want to come back to, you know, the people you're working with, because I know some of your co-founders and was, you know, trying to convince them desperately to start a company a while back.

1:34And then they were like, oh no, we have a bigger mission now with Fei Fei. What is spatial intelligence? Can you define it for a broader audience? Spatial intelligence to me is the ability to understand reason and interact and generate 3D worlds. Because our world fundamentally, no matter how you say we can project it, fundamentally is 3D. And it's 3D because physically it's 3D. And digitally, if there is a true 3D representation, then we can make a lot of things happen more easily, whether it's designing or curation or navigation or simulation or the experiencing of AR, VR. All this, to me, is part of spatial intelligence.

2:24And again, I think it's what really excites me is humans have spatial intelligence. We are, it's part of our core intelligent capabilities. Animals have a spatial intelligence. The entire journey of evolution also is deeply intertwined with the evolution of spatial intelligence. So it's so fundamental. Without spatial intelligence, AI would be incomplete. How does that translate into what you're doing with your company? Or is there anything you can share in terms of what that means relative to what you're building? Yeah, so we're cracking one of the hardest problems in AI, which is actually making world models that are fundamentally 3D.

3:10Because once you can crack that problem, you can unlock a lot of spatial intelligence problems. So we are the first company we know of that is solving the 3D generation foundation model problem. I have many questions, but since you are, you know, describing this first as, you know, 3D's criticality to just sort of understanding the world, does that imply you feel that the world models that, you know, world labs will create or others in academia or in companies will create will someday be like, you know, realistically accurate, like represent physics and understand the world that we can do? many more things with?

3:59Yeah, it should. It should be realistically accurate or plausible. So you can create a fantastical world, but it should be plausible because the geometry and the physics of it need to be plausible. And that is fundamental to spatial intelligence. Does that imply you have a particular point of view from like a neuroscience perspective of like, you know, how fundamental visual, I mean, you've always been a leader in computer vision, right? But in how important visual intelligence is versus let's say like large language models and textual intelligence. I actually do. I think from a neural and cognitive science point of view that spatial intelligence is a really hard problem that evolution has to solve for animals.

4:52And what's really interesting is I think animals have solved it to an extent, but not fully solved it. It's one of the hardest problems because what is the problem animal has to solve? Animals have to evolve the capability of collecting lights in something, which we call eyes mostly. And then with that collection of eyes, it has to reconstruct a 3D world in their mind. somehow so that they can navigate and they can do things. And of course, they can interact. For humans, we're the most capable animal in terms of manipulation. We can do a lot of things. And all this is spatial intelligence. To me, that's just rooted in our intelligence.

5:44What is interesting is it's not a fully solved problem, even in animals. We, for example, for humans, right? If I ask you to close your eyes right now and draw out or build a 3D model of the environment around you, it's not that easy. We don't have that much capability to generate extremely complicated 3D model till we get trained. You know, there are some of us, whether they're architects or designers or just people with a lot of training and a lot of talent. And that's a hard thing to do. And imagine you do it at your fingertip much more easily and allow much more fluid interactivity and editability.

6:37That would just be a whole different world for people, no pun intended. Are there other big areas like spatial intelligence that you feel haven't been as developed as it could be from a model perspective or other sort of missing gaps that you think, in general, as we think, as we build this sort of AI future, we should focus on over time or people should build out. I was just wondering, in addition to sort of 3D and world generation and other big problems like that, because it feels like there are a few big things that we've solved for over time and other things we're working on. We're short of solving language.

7:10I would say language is solved to a huge extent. And 3D to me is as critical and difficult as language. So what else does that solve? I mean, the entire space of emotional intelligence is something that I don't even know how to begin to solve. I know a lot of people who haven't solved it. That's when AGI is achieved. Yeah, so that's another one. And I can tell you the training data for that is not going to come from Silicon Valley people. don't underestimate the Silicon Valley yeah so I'll put myself in this bucket but I think we probably need a broader set of people yeah no that I agree but these are the three three big buckets to be honest that's I don't know what do you think Ilan and Sarah I think it depends a lot on um what you encapsulate in each model so I agree with your framework in terms of those three.

8:13And then certain things like, you know, the spatial intelligence, I'm assuming also delves into different types of physics simulation and simulations of the world. And that, you know, like those are big areas that I think a lot of people aren't working on that I think are really interesting or important. So, and there's sort of the macro and the micro scale of that. The micro scale eventually becomes material sciences and other very different types of things from what you're talking about, where it's more molecular modeling or, yeah. Right. And also somewhat goes out the current definition of AI, which I do think they'll be empowered by it.

8:42Of course, there's robotics, but robotics is very much a system integration problem as much as a, you know, even if you look at animals, it's not just the compute in the brain per se, right? Yeah, a lot of these things seem much more distributed in terms of spatial intelligence relative to specific systems that animals have. And in some cases, it's to your point, not as centralized as one would think. So it's very interesting to start thinking in terms of those models of more distributed intelligence across an organism versus the CNS. But yeah, I think it's very interesting stuff. You've also done work in this field, Feifei, of robotics and like physical intelligence.

9:21I think of the data hierarchy for, you know, robotics foundation models and actuation as, you know, people want to, of course, use video, right? Because that is what is available to us. There's a big question on like simulation and how much you can get from that today. Perhaps people do not see the future of like the quality and the physics that are going to be available to us. And then there's, you know, close to embodied, like different forms of tele-op and then like embodied data collection. Is that the hierarchy you have in your mind or do you think people underestimate simulation and world models for the future?

9:56Yeah, great question. First of all, I, like you said, I do work in robotics, especially in my lab at Stanford. I have no doubt that humanity will move into an age where we cohabit with robots. And also the world robot is not humanoid per se. Robots taking all kinds of forms and shapes. Actually, a few years ago, my lab wrote a really fun paper about morphological intelligence is where the morphology of an agent actually can change by optimizing the tasks they're trying to achieve. So we should be a little more imaginative than just humanoids. Having said that, how to train robot, you mentioned this whole data, some people call it data pyramids or data cakes or whatever.

10:49I agree. I think it's going to be a hybrid of many different forms of data. I also think simulation is underrated. Actually, it's not underrated by a lot of experts and people in the field. If you look at a lot of robotics companies, they are working on simulation and synthetic data. I also think we have to be also aware that unlike language models or even unlike spatial intelligence foundation models, robotics is a highly multimodal system that I think what is truly underappreciated, in my opinion, is haptics. is there's so much, especially if we want to do manipulation, not just navigation. I think haptics data and the ability to really integrate haptics into vision and perception and spatial data is absolutely critical.

11:53One thing that you said that I thought was really interesting is how many different, what are the different morphological forms that a robot may adopt? And there's sort of two counter arguments people make in terms of the potential future. One argument is that from a supply chain perspective and managing builds and scale of manufacturing, you're going to have many fewer form factors. And the other argument is the economic value of specialization is very high. And therefore, there'll be, you know, thousands and thousands of different form factors as we move to sort of a robot-driven future. Do you have a point of view on sort of where we're likely to land between those two viewpoints?

12:27I think we're going to gradient descending to optimization of productivity and efficiency. My hypothesis is that the requirements of different tasks are so vast that having very few form or sticking with one form is energy inefficient. and a lot of tasks can be done and should be done by much more energy efficient form factors. Just an extreme and trivial example. If we put robots underwater, they should not be in the shape of humans. They better be in the shape of fish, right? Just think about energy efficiency And the same with flying. I don't think human form is our airplanes are becoming more and more robots.

13:20And so I do think there's going to be diversity. Robotics is one potential application for the future. You're a scientist first, but also, you know, did the Twitter board involved in startups. What are the near term commercial applications that you can imagine for generating 3D worlds? I believe creativity is a vastly exciting area where humans can be superpowered by AI and by spatial intelligence. And here I draw an analogy with software engineering. If you look at today's success of LLMs in software engineering, including applications like Cursor and Winsurf and all that, What you see is a lot of collaboration between AI and humans.

14:16And then the collaboration comes in different levels of skill sets and all that. And I think creativity will be similar, is that whether we're talking about designers, 3D artists, VFX artists, or even marketing talents and game developers, There's so much need in designing and creating 3D space. And this is fundamentally such a hard problem, even for the trained, skilled people, that having a collaborator will be extremely fun if we do it right. And so I see creativity as an area that is really exciting. I also do think that a lot of what we're waiting for for metaverse or XR, AR, VR is content creation.

15:12I understand hardware itself needs to continue to evolve. But I also think software, we're looking for content creation and that lends itself so naturally to 3D modeling and 3D or generative spatial models. And that's another interesting area to look into. Do you have a strong point of view on whether or not world models are like an interesting answer to scalable RL for like more generalizable agents? I actually do think this is, like I said, AI is not complete without spatial intelligence because humans interact in 3D worlds. And in the digital world, we need all kinds of interaction. You know, take design as an example.

16:03It's a deeply, you know, it has, when we are thinking about design, there's so much we are optimizing for in our mind's eye, whether it's beauty or efficiency or optimization or whatever it is. And that lends itself pretty naturally to RL settings. What are the biggest challenges in, I guess, trying to go down this path of designing and training world models? I imagine one is like you worked on images, you worked on video, but we have images and we have video and we don't have lots of 3D worlds in a format I assume you're building. Yeah, data is absolutely a challenge. You're totally right about that.

16:45You know, to create world models, 3D foundation models, we require more and more sophisticated data engineering, data acquisition, data processing and data synthesis. So I am envious of my NLP LLM colleagues that the data is so abundant on the internet, and we don't necessarily have that luxury. So that's definitely one challenge. Another one challenge is that 3D, this is kind of ironic, right? Every one of us use 3D every day, like in so many settings. Basically, you open your eye and the whole life that you experience is 3D. Even when we type on the computer or stare at a screen all the time, yet it's still not as easy a form factor to deliver in the hands of people compared to language.

17:47The language is just so easy. and it's also a very active form of it's not a passive consumption of viewing nobody wakes up and say i'm just gonna sit here and watch 3d you know so um that creates challenges for for productization and how to do it in the right way were you ever a like a second life player or anything? I'm not a gamer, but my kids love Minecraft. I was going to ask you if there was like a world that you want to experience or imagine. That's a great question, Sarah. You know, I would love to see worlds. I love seeing worlds I don't see, for example, like zooming in and in and into like microscopic worlds or, you know, go into the inside of an engine, you know, knowing how the actual engine is, I know, of course, I know theoretically how it works, but seeing it with my own eyes, experiencing it, or even you might laugh at this.

18:55I want to be inside a dishwasher and just experience what that is. All this can be done in a virtual way if we manage to create, you know, world models of anything. OK, I think a lot both a lot and I both want to talk a little bit about your past career and maybe some insights for anyone doing research or trying to have an impact within AI. Right before this, I asked Andrej Karpathy what I should ask you. And he said, you know, Fei-Fei is really magic about ambition and thinking about data. You should ask her about her Ph.D. like and the creation of that one on one data set with Pietro because it's instructive.

19:37So I have to ask you about that. You know, first of all, I have to say it's always really the greatest thing when your student is more well-known and achieving so much more than you can. It makes me so proud. So very proud of Andre. I'm surprised he remembers my PhD work. So, yes, it's true. Well, gosh, it goes back to 2003-ish, and the world was just barely scratching the surface of internet, and data was not much of a thing, but doing computer vision. My PhD work was really trying to get object recognition to work. That's the problem of calling out cats and dogs and microwaves and chairs and all that when you're presented with a picture.

20:29And we were beginning to hypothesize that data matters, but we had no idea. There's no scaling law. We had no idea, you know, how far data can go. All we wanted is if we have a machine learning algorithm, whether it's a neural network or base net at that time was very popular or support vector machine, we need some data to train. And there was no data to train. And as a PhD student, you want to, you know, graduate. And Pietro was like, well, if I curate a data set. And, you know, I was thinking, yeah, I do need to curate a data set because every data set out there is so tiny. I'm just not convinced.

21:12And Pietro and I were just talking, you know, is it 15 different things or 30 different things? And then God forbid the PhD advisor said the three digit number 100. And I was like, you know, that's a lot of work. But I deep in my heart, I know he's right from a mathematical point of view is pushing the model to generalize. We need enough data at least. So, you know, I did write about this process in my book, The Worlds I See, that I stumbled upon a dictionary somehow and it really was for my own English study that the dictionary I think it's the Webster dictionary if I'm not wrong it just kind of randomly has depiction of a visual depiction of some words I don't even know what rule they follow to be honest to be honest some are flowers some of bicycles some of dogs I was like okay this is actually you can call it a cheat or a tool.

22:15I grabbed 101 of those words. And that really made my PhD advisor kind of chuckle because he's like, ah, yeah, you just want to do one more than I asked for to, you know, dare me. So that's what I did. And I gotta say that I still remember I downloaded or, you know, tried, you know, from Google and Google was so new at that point. And the Google image search were so terrible at that point, you know, compared to today. And I had to do so much cleaning. At some point, I got so desperate. I just asked my mom to do the image cleaning because I wrote a little interface on the computer. She doesn't know computer, but at least she knows click, click.

23:02So she helped me to do some of that. I mean, you've had one of the most storied careers in AI. And to your point, many of your students have similarly gone on to do really great things across the field, across industry, across the world. What are two or three moments that you think of when you think back on your career to date? And obviously, there's still a lot of career to come, but I'm just sort of curious. I mean, obviously, there's a lot of things that you did in terms of sort of image and visual recognition related systems. But I'm just sort of curious, like when you think of the last 20 years, what stands out the most, just given everything that you've done?

23:36Oh, thank you for asking that question. Of course, ImageNet is one of those, ImageNet consists of multiple moments from the early struggles and being told I will not get tenure to actually realizing Amazon Mechanical Turk comes to rescue to the moment of AlexNet winning. And also to a couple of years ago, I was at an event in Toronto with Jeff Hinton, and he said publicly like how that was so defining. And he was almost a little bit apologetic, that image that was not as recognized as neural networks. So that journey is very validating. And for scientists, the validation is not about recognition or awards.

24:28It's that you made a difference, like that conjecture that no one believed in, that hypothesis that no one believed in. We were able to make it happen. So that's one thread. Just to make sure for any, like, you know, people from the business world that are not familiar with it, ImageNet was a large scale, is a large scale data set with millions of labeled images across thousands of categories, not just 101, right? 15 million labeled images. 15 million labeled images. Thank you, Fei-Fei, that, you know, led to amazing breakthroughs in deep learning, in particular AlexNet and lots of progress in the field of computer vision overall.

25:06Yeah, driven a lot of machine vision forward. And I actually remember in 2016 or 2017, I used to show a slide which was the history of AI or, you know, back then it was CNNs and RNNs and just GANs were, you know, kind of going. And I had ImageNet and AlexNet as like one of the seminal moments of, you know, this very small number of events that really defined AI progress. And obviously now we have transformers as part of that and maybe diffusion models or something. But it was such a big breakthrough. Yeah, thank you. Another moment I'm very proud of was actually Andre and also Justin Johnson and their dissertations.

25:41It's where, in my opinion, the first time that language and images converged by captioning and writing stories of the visual world, it was significant for me for two reasons. One is that I literally thought, I kid you not, at the end of my PhD, I thought if I can live to a 100-year-old, that was the problem we might be able to solve, which is storytelling of pictures. So I entered my career, like my first year assistant professor, thinking, okay, I'm going to do image that to solve object recognition. And then I'm going to spend the rest of my entire career solving this problem of storytelling.

26:32And then by the time Andre and then a little later, Justin Johnson entered my lab, that was around 2013, 2014, the beginning of deep learning. And then suddenly the combination of sequential model at that point is LSTM. It's not Transformer models, but LSTM and CNN just had this lasted open the image captioning work. And Andrea and my work were the first together with Google's that was out of the door. And that was really to me, I almost had it was made me so proud. I almost had a crisis, which is like, what am I going to do for the rest of my 70 years? or 65 years. So that was really exciting how fast the field has evolved.

27:32Can I ask you one more question about this just because you have made this amazing progress like very efficiently, right? Like you and I have offline talked before about how you feel it's really important for there to be moonshots and creativity in AI research beyond like very large funded corporate labs, let's say. And, you know, you pointed to several moments that they come from like creativity and research in academia. What advice do you have for people about whether or not there's still opportunity for that or, you know, it's all just$10 billion training runs from here? My singular advice, and I still say that in my company, in my lab, is be fearless.

28:19I think scientists and technologists and entrepreneurs have to be fearless. You know, eventually you have to figure out, do you need$10 billion runs? Or then you come to Sarah and ask for funding. Probably a lot, but both. Yeah. Or you have to figure out, you know, I don't know, data. Sometimes fearless is this very interesting position where you're somewhat delusional and crazy, but somewhat just rationally bold. And it kind of is in between because if you're too rational, it's not courageous enough. You're not identifying problems that are big enough. But if you're completely crazy, then I don't know.

29:12There's many things that can go wrong. So be fearless, be courageous. To me, that is, you know, even as old as I am, that's how I feel. I started my startup world labs is I want to be fearless and solve this problem of spatial intelligence. As part of problem solving, you've worked with some of the best AI researchers in the world over time and best engineers. How do you think about that in the context of your company? Like what sorts of people are you trying to hire? Are there open roles currently? And Dadily, it's an amazing team. I'm just curious, like, what sorts of folks you want to add and how you're thinking about that over time?

29:51Yes, we have open roles and we would love to hire the best engineers as well as product thinkers at this point for our company. So if you're an engineer or AI researcher or product talent out there passionate about joining the most talented team and solving this problem, please join us. So who do we hire? First of all, we really do hire in diversity of thinking. And this is where, you know, you call us an AI company, but if you look under the hood, we've got computer graphics experts. We've got computer vision experts. We've got data experts. We've got, you know, generative AI experts. We've got machine learning infra experts.

30:36We've got optimization. So it's actually really important to hire a diverse group of really talented people because the problem as hard as spatial intelligence is not a homogenous problem. Like it takes talents of all kinds of background to solve it. And then I also just, like, I look for fearlessness. Like, you know, we all have. How do you do that? Like, how do you identify if somebody has fearlessness in their background or in their thinking processes? It's in their background. You talk to them. You can sense someone is fearless. You know, you can sense what drives them. You know, you can sense the questions they ask.

31:23If they are, if they start to asking you a lot of things about, I don't know how to get this done. I mean, of course you have to ask those questions because you want to get it done. But if you sense that it comes from the point of view of being scared of solving that, then that's not fearlessness. But those fearless people, they are creative, they're ambitious. they they they they can they're not afraid of the uncertainty or the unknown and I really love that well I think a lot and I you know we try to make make a business of doing business with fearless people and hopefully those that are technically creative one one last broader question for you because I think an important part of your work has been also thinking how to bring more people into AI, you know, co-directing the Stanford Center for Human Centered and Artificial Intelligence.