#331 Sergey Levine: The Robot Revolution Nobody Is Talking About

26:38Yeah. You know, I had Fei-Fei Li on recently talking about world models. And in the past, I've had Yan Le Koon on several times talking about world models. Where do world models fit into the stock? I mean, are you, because world models, as I understand it, certainly from Yan Le Koon's point of view, It's building an internal representation of the world that then can be used to predict futures or reason through problems. If you don't have that, it seems that it would be much harder to build a foundation, robot foundation model that can generalize and operate in the wild. I mean, right now in your lab, in a lot of the industrial deployments, they're very controlled environments and the robots training is very task specific.

27:47But if you want a more generalized model, where do world models fit in? Yeah, it's a really interesting question. I think that some folks tend to present world, like, I guess we should nail down what we mean by world models. Typically, what people mean when they say world models is some kind of predictive model that operates at the level of raw observations. It doesn't mean that it predicts raw observations. It may be that it is, you know, like Jan LeCun, for example, I know he advocates for essentially a latent space world model, which predicts a sufficient statistic of observations. But roughly speaking, it's something that predicts something about your future observations.

28:25It's a very reasonable idea, but I think that something that we should keep in mind is that for human behavior, like prediction definitely plays a role. But there are also things that we do that are not grounded entirely in prediction. There's a place where prediction is easy and there's a place where prediction is difficult. And the abstractions that we use are really critical to intelligent behavior. So to give you an example, if I want to figure out how to get from where I am now in San Francisco to New York City, maybe I'm going to imagine something about it. Maybe I imagine how I get my car keys and I get in my car.

29:01I might imagine that I'm taking an airplane. But the further out that I think about this, the more abstract that imagination becomes. I'm not imagining exactly what my seat in the airplane is going to look like. So abstractions are really key to actual effective world model. And I think at some level, what language models do, what visual language models do, and what video prediction models do and other kinds of world models is not actually that different. They're just operating with different abstractions. And I think for a real capable embodied intelligence system, like a robot, we'll need many different abstractions.

29:36And I suspect that once we figure out how to use abstractions in general, like that general part of the question is actually the important one. And then we'll use kind of the right kind of thing at each level and that'll be fine. So I guess what I would say is that like, definitely I'm very sympathetic to the world models view, but I also suspect that the dichotomy between language models and what people today call world models is not as large as some people might see. Yeah. So, so with a good world model, there isn't anything that a robot couldn't do with being trained through with a vision language action model.

30:11It's just, yeah, go ahead. I suspect the key is to have a system that correctly uses the right abstraction for the jaw. So if you are, you know, folding a piece of clothing, probably, you know, that's clearly not a semantic task. You're not thinking in words about like, oh, this fold is here, this fold is there. But you're probably also not imagining literally how all the particles of clothing move. At some level, especially if you're good at the skill, you're mostly kind of using your muscle memory a little bit, but it's, it's reactive. It's using perception, but it's, it's not as model based as just like imagining exactly how every particle will move.

30:48So the really cool thing about a proficient human motor skills is that they blend prediction and this kind of model free behavior and semantic reasoning. It all kind of comes together with the right tool for the job at each level. And I think that kind of blending is really, really important. So do your models actually imagine possible futures before acting? Right. So the way that our models work is they actually perform inferences at different levels of abstraction. I think there's still a lot of research to be done to figure out exactly how that should be done and what abstraction should be used and where.

31:20Like, you know, right now, I would say that our choice of abstraction is quite naive, but you can reason semantically about higher level things. Like if you're cleaning a kitchen, do you pick up the plate or do you pick up the towel first? You can figure out where things go spatially and then you can produce the actions. So there's already a little bit of that multiple abstractions, but I think there's a lot more research that needs to be done to get the abstractions to emerge automatically and to automatically figure out the right abstraction for any given stage of task. Yeah. The, the, the, the, so, so world models in your view are not necessary for a generalized robots that can operate in messy environments.

32:00I guess what I would say is not that it's necessary or not necessary, but that it's not as much of a dichotomy as I think that some folks tend to, tend to present. Like, I think that the view that I would disagree with is that there is like model free policies, reinforcement learning, VLAs and world models. And these are like totally separate things. I actually don't think they're separate things. And I think that, uh, the right answer will be a model that can do all of those things together that can use world model like prediction when it's necessary and can do the model free stuff when that's more appropriate and smartly decides what kind of abstraction is the right obstruction to use for a given problem.

32:32Yeah. Your models, the AI brain, so to speak, is not on the robots. It's communicating with them remotely. But when you send robots out into the world, you really can't afford, as with cars, autonomous vehicles, you can't afford for them to be waiting for instructions from the cloud. So how much of the models that you're building can be on device? Yeah, that's a really interesting question. So, you know, so far we're still very much in like the research and development phase. We haven't had to worry about this very much. And you're completely right that currently the models actually live on the cloud.

33:25They've actually run through an inference API that looks very similar to what someone might imagine using for an LLM. But in the long run, I think you're right that there needs to be an on-device component that is reliable, that is not vulnerable to connectivity issues. generally, I think that the way to move towards this, which I think is already reflected in current models that we and others have been developing, is to have a system that performs multiple types of inferences in parallel and at different levels, where the highest levels are maybe more appropriate to offload to a remote inference server, and the lowest levels, the ones that are really doing motor control and closing the loop very tightly in perception, function run locally.

34:09Now, the good news is that the lowest levels are probably also going to be the smallest ones in terms of the number of parameters, because they're not, you know, you can sort of think of these as like, uh, instinctual reactions, reflexes, that sort of thing. Like these are things that are very important. They need to be very fast, but they also are not as cognitively demanding and not as complex. So they can run locally. Now the trick of course is figuring out all that communication so that you still preserve the benefit of end-to-end training. So this whole thing is trained together to active concern, but at inference time can be partitioned in this way with different size components running in different places.

34:44And it's kind of cool to imagine like, you know, you have good internet connectivity, you get, you get a lot of intelligence. Your internet connectivity starts to degrade. Okay. Maybe the robot gets a little dumber. Like it has to, you know, uh, maybe pull down some nice inferences from the cloud, keep them locally and do some stuff. And then, okay, now let's, let's stop and think again. Let's ask the cloud to think some more. So right now you're not concerned about shrinking models to fit on device. The first step is to get a generalized model and then you'll do the distillation or whatever is required to get it on device.

35:21Is that fair? So that's right. But even separately from any inference concerns, we do actually, we've sort of, I guess, conversion evolution in some sense, we already have models that have this kind of multi-scale property where the lowest level motor control components are already smaller. So while we're not so worried yet about running things on device, the natural trajectory of that technology is leading to a place that makes that actually reasonably straightforward. Yeah. So I have some questions about generalization, but before I get to that, I get very confused about all the different kinds of ways that people are training robots.

36:04So there's vision language action models, but there is one shot, two shot, few shot, vision language action, world models. Can you sort of give me a little primer on the different ways that people are training robots and why you've settled on VLAs and RL? Yeah. Yeah. Let me think about how to best describe this. So I think it's, it's often hard to get like a complete picture of the robotic learning world. Just by looking at the kind of results that people present, there's kind of one general truism of the robotics demo, which is robotics demos can be set up in such a way that shows something really cool, but doesn't actually provide like a general solution to a problem.

37:00because if you want to just like stage a robot demo, you kind of make things work in that one setting. So because of that, it's like a little hard to figure out what are like the big clusters of major effective techniques. But if I were to like very soberly look at the current robotic learning environment, I would say that there are actually like two big things that work very well. One thing which we've been discussing is vision language action models. And more generally this idea of like learning manipulational skills and things like that from data. Typically it's with imitational learning, but it can also be with reinforcement learning insofar as it can leverage that data.

37:31And oftentimes actually the recipe is like use imitation learning to initialize and then maybe like fine tune it with RL. And then the other big cluster is sim to real transfer using simulation to learn motor skills in a sufficiently randomized way and then run them in the real world. And when you see demos of like, you know, robots doing like, you know, dancing and acrobatics and all that stuff, that's typically doing sim to real. And it's kind of a funny, unstable equilibrium of the current research world that these two types of techniques are very different, and they are also used to attack very different domains.

38:04So the vision language action models, they're basically currently the dominant paradigm for robotic manipulation. Problems where robots need to interact with diverse environments and diverse options. The simporeal stuff is the method of choice for highly acrobatic and athletic movements, typically for human points. And these are situations where the task is physically extremely demanding, but the diversity of the environment is very low. So if you put the robot on stage and it's dancing, you're not really worried about the shape of the stage, you're just worried about the robot's body. So this maybe tells us actually a lot about the strengths and weaknesses of these two types of approaches, that the simptorial stuff is great for really understanding the physics of the robot, but not so great for generalization.

38:48The vision language action models are great for generalization, but of course, because they're dealing with the physical world, the real world data, they can't run these like giant RL loops that practice for billions of trials and really overfit to the particular body of the robot. Now, of course, there's a lot more to do in the future, but this hopefully gives you some sense for the layout. And VLA, just walk me through the process of where the vision comes in, where the language comes in to get to the action. Yeah, so maybe one way I could describe this is to start with visual language models.

39:22These are also sometimes called multimodal LLMs. So if you use like Gemini or ChatGPT and you upload an image and you ask some question, that's basically a VLM. And the way that these models work today is you start with a language model. A language model is very, very simple. It's a transformer that takes in text and predicts future text. And to get these things to process images, what we do is we train a vision encoder, basically another little piece of neural network that takes an image and puts it into the same space as the language tokens. That's sometimes called a vision encoder. So now you're going to feed images to this thing the same way that you feed language, because this little like virtual visual cortex kind of takes the images and turns them into, you know, semantic like looking thingies that the language model knows how to process.

40:11So now with VLAs, well, with first generation VLAs, as I mentioned, people basically just took exactly this model and just changed the output to literally like output numbers in text that represent actions. The second generation VLAs, which is what everyone's basically using today. They take inspiration from how VLMs add a visual cortex to, to the language model. And they also had a kind of a virtual motor cortex, a specialized little piece of circuitry whose job is to take the outputs from the language model backbone and decode them into continuous actions. And this is typically done with diffusion.

40:44Basically the same kind of technology that's used to generate images and videos is now used to generate trajectories of robot joints. That makes a lot of sense because trajectories of robot joints are a continuous spatial object the same way that images are continuous spatial objects. So it makes sense that the same technology would be applicable there. And I think this is really cool because almost like what we're, what we're doing is like building a brain piece by piece. Like there is this, the language model backbone, I guess kind of like a prefrontal cortex almost. There's the little visual cortex part that encodes images and now there's a little motor cortex part.

41:14Anyone familiar with biology would be very offended at this point because it's backwards, right? Evolutionarily, the motor cortex comes first, then vision, then prefrontal, but here it's the other way around. Yeah. And then, uh, to get the continuous action, it's outputting a stream of numbers that are telling the actuators how far to move or right. And control theory comes in at the end. Is that right? You're outputting. Well, and I should be very upfront about this. The amount of control theory in this is like pretty minimal. So currently the way that most of these systems work, including ours, is that the model outputs a short trajectory of future target joint angles.

42:06So roughly like half a second of joint angles to hit. And the actual actuation, the motor commands to reach those joint angles are computed with a very simple impedance controller. This is kind of like, you know, the most basic type of controller you can put on a robot car, basically. Yeah. You know, past robots were often built to do one thing really well, particularly, I mean, certainly before, you know, they were, AI was involved. How realistic is it that one robot with one of these models controlling it can do many different things? Yeah. And or and I had a conversation with a guy, interesting guy, maybe you run across him.

43:00Mike LeBlanc, I think his name is, he's got a startup called Foundation and he's doing humanoids for the military. and they're training them to do one thing, like put an explosive charge on a door, which is a very dangerous thing for a soldier to do. So they drop it from the Humvee, it walks, slaps this thing on the door, and comes back, hopefully in one piece. And that made a lot of sense. that rather than trying to build this generalized model, just train it to do one thing. But how much generalization is developing from yours and other robots in the field? So here's maybe how I can try to answer this question.

43:57I think every time that people try to take robots out of the factory and into open world environments, they very quickly realize that in the real world, there's a huge range of things that can happen. And it's very, very hard to get a system that is highly specialized and still robust enough to work in the real world. This lesson was probably learned very, like, I don't know if it's for the first time, but an early instance of this lesson was actually autonomous driving. Like in the very early days of autonomous driving, some people thought that, well, okay, driving is complicated. There's lots of stuff that can happen.

44:33But if we just like an instrument the road and like prevent other things from getting into like the autonomous car lane. Maybe then things will work. Like we'll install like magnetic sensors and all that stuff. And that really never took off because essentially the gap between a closed world and an open world is, is enormous. And you can't, it's like, you can't be just like a little open world. Like as soon as you're, you're out in the wild, immediately stuff can happen. Maybe it happens rarely, but that's not, that doesn't save you. Like even if it happens rarely, you have to deal with it. So here's an example from our work.

45:03Um, we had a project on using our systems to assemble boxes. So you take like a flattened cardboard box and you have to like build it up, fold it. And it's like a little bit like an origami problem. Basically you think that's pretty structured. Like it's one thing, just build a box. But sometimes you grab the boxes off the pile and you get two boxes instead of one. So you have to put one in the back and maybe there's someone, something is torn a little bit, so you have to discard it because it's torn and maybe someone left like their, uh, phone on the table. So you have to put the phone away. So there's just so many things that happen.

45:31that even like once you cross that boundary from the factory to the real world, that you can't actually have a narrow specialist. That even if what you want is to do one thing, you have to handle all these other things that can happen all around. And the generalist model, the one that could handle a wide variety of tasks actually becomes a better specialist because it can deal with all that weird stuff that arises. So I think that generality is really essential, even if you really want to do one thing, and I think that's a lesson that's been learned time and time again in robotics. And, and are you going to, uh, move into humanoids or have you?

46:03Um, yeah, that's a good question. So the main reason why we've stayed away from humanoids ourselves for the, uh, the data collection that we're doing here is actually, it's not ideological. It's just because humanoids are expensive, complicated and teleoperating humanoids is harder. So we've basically stuck to robotic platforms that can give us the most data, the most data diversity. But I think humanoids are really cool. Like we've worked with other companies that have humanoids, the model runs on humanoids. And I think that, you know, it's something that I'm sure we'll want to explore more in the future.

46:33My own take when it comes to robot morphology though, is that I think that, you know, this is maybe like a little bit, uh, idealistic, but I, I actually really hope that robots will kind of end up being a little bit like personal computers where there's like general software and the form factor of the device can be very different for different jobs. So when you get a computer, maybe you get a laptop, maybe your computer is your cell phone, basically, maybe it's a big desktop, maybe it's a big server machine. It's like whatever is the right for your job. And I think robots will be the same way that maybe like, if you live in a small apartment in New York, maybe you have like a little home robot that kind of attaches to the ceiling and pivots around and cleans up the thing.

47:09And maybe if you, you know, if you live on a farm or something, you have a big mobile robot, you know, tractor with a bunch of arms attached that can drive around and do stuff. And maybe there'll be one software stack that can accommodate these things with different applications and prompt engineering people do for particular domains. And there'll be this heterogeneity of platforms. like physical intelligence everywhere yeah well how does how do your models work on humanoids because your your models have not been trained on operating legs and walking yeah so typically right now when we work with folks that have humanoid robots uh they have their own software stack that takes care of balance and all that and all those things and then the model drives basically manipulation behaviors yeah uh and and what you were just saying do you think we'll end up with an iphone of robots that everyone uses and uh and maybe there are specialized robot bodies for for different uh tasks uh and would there be do you think there will be a dominant model Yeah.

48:18Yeah. I don't know, but, but I think that one, like one vision of this that's very appealing to me is that, you know, with robots, since we get to build them, we can build them however we want. And I, and I think that's one of the advantages of robots is that they can be designed not to replace people, but to do the, to do things differently from people. And so you could design form factors that are suitable for particular domains. They can be much smaller, much bigger. They can have, you know, five arms, seven arms, one arm, whatever is like appropriate for the cost point that you have, whatever is appropriate for your application.

48:52And people can be pretty creative about it. It's just the thing that's been preventing folks from experimenting with these things is that if you want a robot to do anything, you have to solve the intelligence problem. And if there isn't a ready-made off-the-shelf thing that at least gives you like a rough prototype, you just can't get started. So what I really hope is that good robotic foundation models will provide that kind of middle layer, you know, in a computer, that layer is basically taken up by the operating system. The operating system is the thing on top of which you put applications.

49:19So there's an intelligence layer that's pretty general on top of which somebody can essentially prompt engineer their tasks. And then they can experiment with all the aspects of that application, really nail it correctly, design the right form factor. Then we'll see a lot of experimentation, a lot of creativity. And then I think we'll actually see what robots should really look like instead of just thinking of them as like a metal version of a person. Yeah. Yeah. And on the humanoid question, I was asked to sort of write a brief for somebody late last year or mid last year about humanoids. And I wrote this very bearish thing about how, you know, all of the different problems from, you know, dexterity in the hand to models that can deal with unpredictable environments and things like that.

50:11But since then, there's been this explosion of humanoids. And everyone's talking about humanoids. I mean, did I get it wrong? do you think that humanoids are much closer to being in the world than, than I, than I thought? It's a question. Um, I mean, I think it's, it, it, it, I think there's a good reason to be excited about humanoids and like, in the sense of like, just emotionally, this really kind of captures the imagination. Uh, it also makes it easier for people to think about because like, we know what people can do. So if we kind of build a robot that does what people do, that like kind of makes sense.

50:46But I do think that it's a somewhat limited view to restrict ourselves just to that. Yeah. My sense is that humanoids will have a place. Like I think there will be humanoids that do something that people find useful. But I think there will also be a place for all sorts of other things. And, you know, as with any new technology, oftentimes what's most important to figure out the right way for it to exist in the world is the tools and machinery and structures to allow people to experiment, to prototype, to try out all sorts of different things. And I think the trouble with robots is that just like the barrier to entry for serious open world robotic systems is extremely high.

51:23Like you have to solve open research problems just to get your prototype out the door. And I think that's kind of what's actually limiting a lot of creativity. Yeah. And for example, with Boston Dynamics, but in the past they were pure control theory, but now they've they have an ai partner and they're they're using ai models to control the robots and as i said there's you know there's been a lot of talk of them being used in factories i think hyundai is is uh deploying them um and i understand the the argument that well it's the human form factor the world is built for that and if we can have robots operate in in that form factor with that form factor there's it can they can presumably eventually do everything that humans can do uh how much of this do you think is hype uh it's very hard for a layman when you when you watch videos or, as I said, the 60 minutes piece, to know how real it is and how much is our sort of fascinating demos that in two years people say, oh yeah, yeah, we had that pilot, but it didn't work out.

52:48So I have two things I can say about this. The first is, I think in regard to demos, I mean, I think you're right that for, for a demo it's like, there's a big difference between setting up a demo and setting up something that works. And you know, one thing that I've struggled with in my career, I think quite a bit is that, you know, I work on robotic learning and robotic learning that the purpose of it is to get systems that can generalize and work in open world environments. But it's very hard to illustrate. Like it's, it's, if you, if you show somebody a demo in a particular setting and the robot does something cool, like, yeah, it's obviously cool stuff going on.

53:21If you show doing something fairly simple, but in like a hundred different environments, well, then each of the videos of that is just a robot doing something simple. So the fact that it can do it in all these different settings is harder to convey. And that's kind of a science communication challenge that I think we have to be cognizant of when we look at the videos. But I think there's a deeper, more technical thing I want to say about humanoids and about robots in general, which is that conventionally when somebody thinks about building a really cool robotic system, they naturally start with building the physical robot.

53:55So if you start from the premise, I'm going to build one robot, it makes sense like if it's going to do something general, it should be a very general body. And you should really get that right because once you've committed to like that form factor, like you're kind of stuck. So I think there's actually a technical point here, which is that the old way of thinking about robotic software as something that drives one robot kind of naturally leads to that. Because if you want one very general robot, then you kind of have to like have it have everything. But if you accept the premise that we're going to have robotic foundation models that can drive lots of different robots to do lots of different things now, kind of on unchains your thinking.

54:31And now it's okay to experiment with different form factors, experiment with different applications, different ways of approaching things. It's just that you need this kind of general AI system to be able to do that. And without it, it actually kind of makes sense that you would go into this, you know, single ideal, perfect body plan and maybe a humanoid is a good choice for that. Yeah. And you've made tremendous progress since I first met you. What's on the horizon? What are you guys working? So for me, one of the things that I really want to figure out over the next year, or maybe the next few years, both in my academic work and in the work here at physical intelligence, is how to go from the foundation, which I think at this point, we have a pretty good understanding of, to something that is a true data flywheel, a true continual learning system.

55:17where the robot experiences more and more tasks and the more it experiences them, the better it gets. And that involves, I think a lot of autonomous learning, a lot of reinforcement learning, a lot of learning from other supervision signals, like these language feedback that I mentioned before. And turning that into a continuous cycle where the more the system is deployed, the more capable it gets. And I think there's some very deep technical challenges there that'll stress kind of all aspects of this. So that's what I'm most excited. Yeah. Okay. uh sergey i'm gonna ask one last question i asked feyfei lee this and uh david ha and some others what's your guilty pleasure do you have one oh uh hmm something you do to relax that that you you know is kind of silly or or a waste of time but you enjoy it nonetheless i remember you were you were a big video game guy.

56:13Yeah, I actually, back when I was in college, I actually thought that my career would be making video games. I think these days it's probably science fiction. I'm a big science fiction nerd. I made sure to put like a little epigraph with quotes from science fiction stories that I like into a lot of the papers that we've been putting out. I like that very much because I think it's just, it's very good, I think, especially for somebody working in science or engineering. And this is maybe a bit of advice that I'd have for folks listening to this too, is that it's good to not feel too constrained.

56:43And sometimes the rigor of doing serious engineering and scientific work forces them to forces them to very constrained thinking. So it's very important, I think, to try to get out of that mindset sometimes, not to get too carried away with it, but it's good to like, let yourself think things that are maybe not, not good to do in polite circles in science and engineering to think about things that are more out there that are more fanciful, more creative or more improbable, let's say. Yeah. Is there a favorite work of science fiction that you've read that you would advise people to read? When I was younger, I didn't read very much Robert Heinlein.

57:19So I think I only discovered Robert Heinlein's work in the last few years. I've been really getting into trouble. Yeah, that's wonderful. Classical American sci-fi. That's right. This very optimistic aspect of American culture that I think is very refreshing especially in today's day mage yeah okay sergey well i hope we have a chance to talk again and i'll see you at the conferences and this has been fascinating most ai is just speech to text plus a language model it's full for reading transcripts not understanding conversations velma from modulate an ai built on ensemble listening model architecture specializes in audio analysis.

58:00It orchestrates hundreds of smaller submodels purpose-built to understand the nuances of voice like tone, timing, stress, and intent. Perfect for fraud defense, deep fake detection, agent attrition prevention, or customer service moderation. Check out the live Velma preview at preview.modulate.ai. That's preview.modulate.ai to see how the model breaks down audio, providing timestamped, explainable signals. Stop transcribing. Start listening with modulate.ai.