#176 Sergey Levine: Decoding The Evolution of AI in Robotics

0:00Imagine someday building a home robot that can perform a variety of tasks in your kitchen. then it becomes a lot more than just a perception problem. Then you really need to be able to learn the individual manipulation skills. You need to be able to generalize very broadly. Hi, I'm Craig Smith, and this is Eye on AI. Today, I talked to Sergey Levine, an associate professor at the University of California, Berkeley, who does research at the Robotic Artificial Intelligence and Learning Lab at the university and is pushing the boundaries of what AI control of robots can do. Sergey talked about some of his recent work in reinforcement learning and the aggregation of data sets from robots around the world to help train a model that can generalize across different kinds of robots.

0:53It's exciting research into embodied AI, bringing the transformative technology out of the computer and into the real world. I hope you find the conversation as fascinating as I did. So, Sergei, could you start by introducing yourself? So I'm an associate professor at UC Berkeley. I previously did my PhD at Stanford. And I also spend one day a week at robotics at Google, where I also work on robotic learning. My research concerns, of course, robotics, but also many other related techniques in machine learning, reinforcement learning. Lately, my group has also been doing work on various things related to reinforcement learning for language models, computational design, things like this, other aspects of decision making.

1:42You know, everyone's talking about world models and they're combining world models and language models. Are you working on world models at all? What's your view of that? But yeah, I guess there are a few things I could say about it. So generally, if we want to control robotic systems, there are a number of ways that machine learning can enable that. One very simple way is imitation learning. Imitation learning amounts to taking demonstrations, typically provided by a person controlling the system, and then imitating those demonstrations to try to produce an agent. And that can work for robots.

2:23It can work for lots of other things. Arguably, language models are just giant imitation learning machines because they're imitating how humans generate text. There are lots of other ways to do it. So a world model is essentially a dynamics model that represents how the environment will evolve in response to the agent's actions. And we can learn that from data, too. Typically, in reinforcement, it's referred to as model-based RL. So model-based RL means you train a model of how your environment behaves, and then you use that model to figure out how to act in the world. And it's a very old discipline, actually.

2:55In fact, the first learning-based control methods before model-free RL became such a popular thing were actually model-based RL methods. Some of the earliest neural network control methods actually used dynamics modeling. And again, there are a million different ways to instantiate this. you could instantiate dynamics models or world models by, for example, taking image observations and doing video prediction. You could also instantiate them by learning non-reconstructive representations, representations that sort of, roughly speaking, capture the state of the system without necessarily grounding it back into pixels and then predict that.

3:31So there's a lot of different approaches for doing this. Recently, I've been talking to Wave about their Gaia model, and you know i've seen the the videos but they they have that model built into a controller connected to a controller to operate a autonomous vehicle what's different about that structure architecture from what you're working on with reinforcement learning uh i don't think there's that much i could say about it because i don't i don't know how their system works um i mean i mean I've seen the public material, same as everyone else, but I don't really have any insight about the details there.

4:12Maybe one thing I would say, though, that most methods for learning-based control don't necessarily need to predict the raw pixels that the robot's camera will observe in the future. That is one way to do it, and there's a lot that can be done with that. But I think that the more significant distinction is actually how well we're able to utilize data to then produce more optimal decisions. And going through prediction is one way to do it. And you could predict pixels, which is what video prediction models do. You could also predict outcomes or rewards, which is what value functions do. At the end of the day, they are actually not all that different.

4:55and maybe a bigger distinction as to whether you can get a system that really works in the real world is what data it's trained on. For example, if you want robotic manipulation systems that will actually work in broad open world environments, you need to train them on data for broad open world environments. So a lot of what I'm actually concerned with in my research is how do we develop learning-based control techniques that can use large amounts of data and how do we figure out what data sets we can acquire to get really generalizable, in my case, often robotic manipulation skills, but also robotic navigation skills and things like that.

5:28Systems for manipulation for things like warehousing, oftentimes those problems can be, to a large extent, reduced to perception problems. So if you structure your environment in the right way, then as long as you're able to detect where the objects are, then you can use hand-designed strategies for tackling that. That tends not to work so well if you want to take the robotic system into more open-world environments. Like if you imagine someday building a home robot that can perform a variety of tasks in your kitchen, then it becomes a lot more than just a perception problem. Then you really need to be able to learn the individual manipulation skills.

6:03You need to be able to generalize very broadly. So maybe one thing that I could discuss here that could be relevant is a project that we actually did recently. this was actually a collaboration between Google, Berkeley, and several other universities on trying to see how could we get robotic controllers that actually generalize across different robot morphologies. And that's actually really important because if a lot of this comes down to data, then it's very difficult to get the kind of breadth and diversity of data from a single robot that can allow the kind of broad generalization that you want from a home robot.

6:40But if you can pool data from lots of different robots, then maybe you can actually get that kind of coverage. And furthermore, if you can actually do that and you get a system that generalizes across robots, then you get something really cool, which is in principle, someone could put together some new robotic system and then plug this kind of robot brain into it and immediately get something that can control that robot. Now, the work that we did so far on this wasn't really that concerned with building better models so much as just getting this diverse data set and just applying kind of standard techniques that we'd already developed in previous works.

7:09And that actually worked decently well. So this project was called RTX. And the idea there was that we got data from, in the end, it was 34 different research labs. Google was one of them. Berkeley was, actually, there were two labs at Berkeley that contributed data to this. And then we trained a model on this to control, to perform basically language condition manipulation tasks. So think like you give the robot an instruction, pick up the tomato and put it in the bowl, and the robot is supposed to do that. And then we took this model and we handed it off to the different labs that contributed data and had them tested in comparison to whatever model they had for their research, basically trained on their own system.

7:44And the multi-robot model actually was, on average, about 50 % better in terms of success rate. And that's actually pretty interesting because this was competing with whatever each lab's individual system was. And presumably, if they're good researchers, they build something that works pretty well in their system. Now, this was actually an imitation learning approach, language condition imitation. I think that whether it's imitation or prediction or world modeling, I mean, I think many of those techniques could be made to work. I think the high order bit here that I want to get across is that by actually getting this data set, you could actually get a system that you could plug into all these different robots and actually get good results out of it.

8:16Hmm. That's fascinating. That model was being trained on data sets from all the various participating labs? Yeah. So in these experiments, we were not testing whether it could generalize to a new robot. And that's a very exciting frontier for this, but that's still in the future. This was just trying to answer the question, if you include data from the other labs, does that one lab's robot get better? Now, of course, if you're sort of in the minority, if you're one of the groups that contributed a relatively small amount of data, you would expect to seek apparently more benefit from everyone else.

8:48Interestingly, actually, even the majority contributors saw a lot of benefits. So probably the largest data set at about 100 ,000 trials was from Google's own robot, the mobile base that we used in a lot of the robotics research there. And with that system, we were able to actually test it on various, we have this kind of test suite of difficult queries. They're actually meant to be queries that require synthesizing, you know, pre-trained knowledge from the web, as well as good instruction following ability and so on. So these are, they require spatial reasoning, things like that. And on the hardest test, we actually saw 3x improvement in performance over just using the Google data set.

9:28Now, that's actually, in my mind, pretty profound because the Google data set was very carefully curated, collected by basically professionals that were collecting robot data. And the fact that including all these additional sources of data from a long list of academic labs actually led to that much improvement really suggests that there is something kind of magical that happens when you combine enough data from enough different sources. Yeah, so for these experiments, we were actually passing the model around. Okay. The data set is public now. So anybody could take that data set and download it and train their own models.

9:59And we actually have an ongoing effort at UC Berkeley that my students are, for that initial experiment, it was just the model weights. That's fascinating. Just the model weights. So the architecture of that model was being replicated in each lab. They weren't using their own model. Correct. Yeah. So it was exactly the same model with exactly the same weights that had to drive all of the robots in all the locations. And that is actually, if you think about it, it's a very non-trivial thing, right? Because all the model gets to see is what the robot receives through the camera and has to figure out that, okay, now I'm driving a UR-10 industrial robot versus now I'm driving a low cost WITOX robot, or now I'm driving a FRANCA or the Google robot and adjust the controls according.

10:43When I was at the lab, as I recall, that you had robots that were networked, So the learnings from one were updating a central brain, and that then was controlling the other robots of each robot. Have you done broader experiments like that, very much like this? Yeah, I'm glad you asked that. So this was a lot of what we were trying to do over the last five years, actually. And in some ways, this multi-robot training effort, it's kind of, you know, partly it came about kind of an acknowledgement of the limitations of this kind of arm farm approach. So getting lots of robots in a room is great if you want to prototype, let's say, reinforcement learning algorithms.

11:32But if you really want broad generalization out of it, like they can't be all in the same room. So you really need to get much better coverage of the world. And by aggregating data from robots in many different sites, Now you can get much better coverage. Now this is still a prototype for what might be a larger system, because these are still data sets collected by researchers essentially doing science experiments. So you could imagine that in the future, the aggregation wouldn't be across different research labs, it would be across different deployed robots. Now that of course is a much more complex undertaking that requires more than just science.

12:04It also requires some kind of organizational effort, consensus from companies and so on. But that I think is kind of the real thing once that comes about. and you could imagine a future where the data streams from a variety of different deployed robots in lots of locations are agglomerated and then used to train one centralized robotic brain which can then be handed back to these robots to improve their performance. And the key thing that we want to de-risk with this project is just if you do this at any scale, you know even at the scale of academic labs, can you even get a policy that can drive all the different robots?

12:36Because if that's not possible then aggregating heterogeneous data wouldn't work and we would need to somehow figure out standardization. Standardization is hard. So what we know now is that we don't have to worry as much about standardization. We should just worry about scale. Yeah, this model then, the weights are being passed around, and they're controlling different form function robots, right? I mean, or were they all just variations? So in these experiments, the robots were all arms with parallel jaw grippers. We are right now experimenting with generalization across single arm and bimanual systems.

13:12At some point in the future, we'll also look at multi-fingered systems, things like that. So so far, truth in advertising, it's one arm with a parallel draw gripper. They're just different brands of arms. Now, they do vary a lot. So the small-scale kind of hobbyist Widow X arm is like maybe 50 centimeters long, kind of smaller with a weak gripper. The big DR-10 robot is an industrial robot meant for manufacturing, quite a bit bigger, beefier, has a stronger motor, stronger grip, or that sort of thing. So there's variability. They're still the same sort. Right. And the model that you're training on this aggregated data, is that the reinforcement?

13:48Can you describe that model? We actually trained two models. One was based on the RT1 model, which was developed at Google last year. And the RT1 model is basically, it's a transformer that reads in language instruction commands, an image and then it outputs discretized tokenized actions. So it's kind of almost like the most obvious way to design a transformer-based policy. The second model was the RT2 model, which is a more recent development, which actually uses a backbone from a pre-trained vision language model. So vision language models are trained to look at images and output responses to textual questions.

14:27So you give it a picture and you say, like, is there a dog in the picture? And it will produce some text to answer that. And then we took this vision language pre-trained backbone and then further fine-tuned on robot data to output robot actions in response to robot observations. So you can sort of think of like the VLM has a number of tasks that it can do. It can answer questions. It can produce captions. Now there's one more task, which is given a robot instruction output the actions for the robot. And now that's a much more powerful model, because it has all that internet knowledge baked into it from the vision language model pre-training.

14:57And that's the one that we've used for the more complex queries with spatial relations and things like that. Yeah. Is most of your work on the data side or on the model side? Well, it's really both. And to some extent, they also go hand in hand because depending on what your algorithms are able to handle, that will inform the kind of data that you need to get. For example, a lot of the more algorithmic work that my lab does these days is concerned with techniques for offline reinforcement learning. Offline reinforcement learning is basically a a way to take data and produce more optimal policies.

15:30So imitation learning methods, they take in data and they produce policies that reproduce the behavior in the data. Offline RL methods take in data and attempt to produce behaviors that are better than the average behavior in the data. So intuitively, you can think of it as using data to figure out what options are available to you and then selecting the best among those options. In fact, methods that use world models, as we discussed before, can be seen as offline RL methods, because typically the way they work is they train the world model on existing data, and then they use it to extract better control strategies than the typical thing that was demonstrated in the data set.

15:59But there are also other ways to build offline RL techniques that don't rely on world models, that rely on value function, things like that. Where do you think the research is heading? Because everything's moving so fast. Is there for robotic control, do you think that the research will settle on one architecture and then there will be different flavors of that architecture, but everyone will agree, yes, that this is the best way, and then it's a matter of training, generalizing across robots and networking data. Or do you think that there will be a series of models that will be used for various functions?

16:42Yeah, good question. So I'll give you an answer. It's a slightly aspirational answer, so maybe this is more like where I wish things were headed. I don't know if this is necessarily where things will head, But I think it's very important for robotics to kind of adopt a paradigm where we are in the habit of having reusable models, where just like in computer vision and NLP, if a researcher produces a good model, other robotics researchers should be able to use it. Now, that might seem like a very obvious thing, but this is not actually how robotics works today. Most robotic learning research, the artifact that is produced is not actually the model.

17:20It's the code or the paper or the insights. The models themselves are almost never portable, never mind across labs, even across different locations in the same lab, different times of day in the same lab, that sort of thing. and I think we really need to move that towards a setting where we have models that are trained on data sets that enable generalization across different locations and systems different objects that sort of thing that we can then give to other researchers other practitioners that will also run on their systems and once we've got we've got a good flow for doing that maybe using things like this RTX data set that has multiple robots maybe using some other days but something where we can and just get in the habit of doing that, then we can actually make more progress as a community towards shared generalizable systems.

18:08Now, until that happens, there's absolutely no question about whether people will use the same architecture or the same model. If they can't even share anything across, then that won't work. But once we can share something, and probably the key to that is a data set that enables that, then the community can figure it out. Maybe at that point, perhaps it'll be realistic to have a single pre-trained backbone, that, you know, like the LLAMA model in natural language processing, an analog to that in robotics, and then people can build on top of that. Or maybe there will be several such things. Maybe there will be a few big ones that kind of the big, well-equipped labs produce that others will then build on.

18:43But before we get to any of that, we need to just get in the habit of actually building models that others can run. The other side to robotics is just the hardware. And, I was talking to a guy the other day who was talking about where robotic control systems are heading. And he's not a roboticist or an AI researcher. But he was waxing very optimistic about there being household robots within three to five years. And that sounded unlikely to me because just the hardware alone is not, at least the hardware that I've seen, is nowhere near being able to do, you know, releasing it into an unstructured environment full of randomness.

19:35Do you think that the hardware is moving along with the AI or is it lagging? That's a good question. I think that a very important part of that question is just what kind of hardware we need. I think to a large extent, learning methods should actually lower the bar for the hardware that's necessary. Basically, the exercise you can do is you can get one of these little trash picker devices and see what kind of tasks you can do around the house with it. I mean, obviously, it's very limiting, so there's some things you couldn't do. But there's also a lot of things you could do with it. Certainly you can tidy up the floor, put things in different locations in the kitchen.

20:18It's actually kind of surprising how much a relatively primitive robotic system can accomplish. So there's very nice work out of Professor Chelsea Finn's group that I also helped with a little bit by a student named Tony Zhao, who developed a bimanual robotic system out of two low-cost robots from Charleston Robotics. So these are not even the fancy industrial arms. These are basically very fancy hobbyist robots. So they cost about, I think,$5 ,000 each. And most of his kind of cleverness in his research was in devising a very convenient teleoperation system, a teleoperation rig that he could hold with his hands and control this fairly cheap bimanual system.

21:06And he could demonstrate all sorts of very complex behaviors. You get this thing to like put a shoe on a foot, use tape to tape down a box, things like that. And the learning methodology that could produce the autonomous policy was well-designed, but not particularly profound. It's sort of used state-of-the-art transformer-based techniques, but didn't really have any particularly surprising innovation. The key to it was really building a really good teleoperation rig that allowed him to produce those behaviors and then very high-quality engineering to then get that down to a policy. So this is called the Aloha system.

21:43For those who are listening, I encourage you to check it out. And it probably gives some idea of what even very primitive hardware is capable of if it's equipped with the right data, the right kind of teleoperation rig to provide that data, and kind of good, bread and butter, modern machine learning techniques. Now, that's still not going to do everything around the house. But I suspect that for folks that watch these Aloha videos, it'll kind of maybe slightly change their mind in terms of the kind of hardware we really need for everyday tasks. So probably there is still some innovation, but it might be actually less than you think.

22:14That's interesting. And then the controller side, the AI side, the model side, I mean, if that is adequate, that hardware, how much more improvement is needed on the control side? That's a complex question because that's probably very heavily dependent on the required bar for robustness and the degree of generalization. So in some ways, it sort of parallels the autonomous driving story, right? Like if you wanted to build an autonomous car that could succeed in like 90 % of cases, well, that's probably something that we've had for over a decade. But if you want an autonomous car that will succeed, that will avoid catastrophic failures with enough robustness that you could just deploy it on any road in any city, just dealing with all those tail cases, that's still an open problem.

23:05And I think with home robots, it's going to be the same way that if you want to lop off the bulk of the things and the bulk of the situations, maybe that's not quite there yet. But I think that it's reasonable to imagine that we get there soon. But how long it takes to get that long tail fully figured out, that's a much more complex question. I think that one thing that's pretty interesting is the degree to which vision language models have progressed over the last, really over just like the last 12 months. And that's especially relevant for robotics because while the way that vision language models are typically used is more for traditional perception tasks, question answering, that sort of thing, the ability to reason about visual observations, perform inferences about spatial arrangements of objects, that sort of thing, that is something that is likely to translate into better robotic capability.

23:58And because generalization is one of those big challenges, I mentioned this long tail issue, I think there is a lot of reason to be optimistic about the potential for those models to eventually improve the robustness of robotic controllers as well. People are talking about combining language and vision, or I should say language and world models into agents that can reason, plan, and take action. That sounded to me very much like robotic control. I guess what I'm asking is, is that research and the people who are in robotic control research on different tracks? The answer is a little bit complicated, but maybe the short version is that yes, it's closely related to a lot of robotics problems.

24:42In fact, there's plenty of work in robotics on using language models for essentially constructing plans and then connecting those plans up to some kind of control mechanism that can bring them about. Now, probably this stuff started maybe like roughly two years ago. Probably one of the more well-known works in this area is the SACAN paper from Google, which used a language model to plan long horizon behaviors for robots. initially in this field one of the big challenges that people were concerned about is how to connect up the language model to perception and action because standard language models have to operate on symbolic representations of the world so you have to take those symbolic representations and somehow weld them on to rich sensory perception and complex actuation now initially the way that this was done was kind of along the lines of what you described by trying to construct some sort of a joint planning procedure that would figure out both a probable sequence of, you know, symbolic steps, essentially language, and the corresponding behaviors that would bring that about.

25:55There's actually a paper from one of my colleagues, one's called Grounded Decoding, which proposes a Bayesian filtering approach to doing exactly that. That said, something that we've seen over the last, maybe like six to nine months is that increasingly with vision language models becoming more powerful a very appealing alternative instead of doing this is to actually train models end-to-end to solve the entire problem now those models can still be doing planning if you have a vision language model that outputs text and also outputs actions you can do essentially the analog of chain of thought prompting you can tell it okay here's some complex problem and produce steps for solving that problem and then once you produce those steps then produce the actions.