π0: A Foundation Model for Robotics with Sergey Levine - #719

24:39This was at least five years ago, probably more. And, you know, first of all, they had to set up the clothes, you know, just so. And then even still, the folding wasn't all that great. And it was a purpose-built laundry folding robot. You know, so a lot of care and, you know, curation of the scenario and poor results. Whereas, you know, in your example, the robot takes the laundry out of the dryer into a bin, pulls it out and starts folding it. You know, talk a little bit more about, you know, that specific example and, you know, to what degree you, you know, is it, you know, is it zero shot? in the sense that you didn't do anything specific to get the robots to perform well on that task rather than relative to other tasks beyond the post-training?

25:35Or was there care somehow that went into that scenario and other scenarios to make the robots perform well on them? Yeah, that's a good question. So there was a lot of care, but no care went into the code in the sense that the code that is running on the robot for laundry folding is exactly completely 100 % identical. to what's running when it's like building the cardboard box or cleaning the table. The care went into carefully working with the robot operators to come up with a good strategy that the robot could execute well. So again, this is coming back to that point about post-training, like, yeah, it really matters what kind of data you get for the post-training phase.

26:17Now, fortunately, you don't need an enormous amount of that. You need an enormous amount of pre-training data. And there you can be, you don't have to be nearly as careful. But for the post-training, like, yeah, the strategy of folding the shirt, there's like five different ways you could do it. And that particular way of doing it is better. It's more reliable. The robot can more easily get the shirt into that position. But at the same time, you know, one of the things that makes laundry folding a really great illustration of this pre-training, post-training principle is that while you could have a really smart strategy that you've designed and you say, okay, like this strategy, if you guys do this for like 20 hours, the robot will be pretty good at this.

26:49you can still get into lots of unpredictable situations. So you're still relying very heavily on that pre-training data to get the robot out of the weird situations and into the ones from which those nice strategies will work. And in the videos that we released, there's a few things that even kind of surprised us. Like we tried to, you know, obviously when you build a robotic system, the first thing you want to do is you want to mess with it and see how you can get it to fail. So Michael, who's one of the researchers working on this, he, in the videos, you can see him, he puts a shirt in the basket the robot takes it out starts falling then he takes another shirt and just drops it on the table and the robot just picks this thing up and just drops it back in right so it's like yeah get this out of here i'm working right it's even more interesting because at first it's it tries to just continue working yeah and then it realizes that hey i got to get rid of this thing and it's just whoop yeah exactly and uh like the post-training data doesn't have that obviously because it's a small amount of very carefully controlled data where everything is just right.

27:44But yeah, probably somewhere in the pre-training data, somebody messed up a little bit, took out two shirts by accident and put it back. Maybe they didn't put it back in the same way. So the robot has to generalize, but that diversity of lower quality data illustrates a lot more of these issues. And as long as you have a good generalizable model on top of it, that can extrapolate from those patterns, then you'll actually get those kinds of emergent behaviors. And that even surprised us. Like we kind of thought like, yeah, that'll generalize some cool way, but we didn't expect quite that kind of nice extrapolation.

28:11And along the lines of curating that training data, if you think about like folding a shirt in terms of stages, one stage is getting the shirt into the position, the next stage, then you execute your folding steps and then there is putting it onto the stack. Does the training data isolate those individual steps or is it all end-to-end run through the process? It's end-to-end. And so we basically, we instruct the operators to perform the entire task. We do, afterwards, for the language condition stuff, we do have human labelers segment the data and annotate it with text, but the collection is just fully intent.

28:57And I think this is important because if you, you can sort of imagine that for now we're doing this in the laboratory, but in the future you could have a robot that is actually like out there in the world really doing real work, maybe initially under teleoperation, but that teleoperation is creating the data that would later help it become more autonomous. Yeah. Yeah. I think the question came from the idea that, you know, once you've got the table laid out, your scenarios may be more well-defined, but when it's in the shirts in the basket, you've got a lot more, you know, think of it combinatorially, like a lot more positions that the shirts could be in that kind of thing.

29:32And so you might need more data of, you know, getting the shirt out of the basket and flat onto the table. But it sounds like you didn't do that. And maybe a follow-on question is, you know, are there areas that you worked with or introduced synthetic data into the process? Yeah, I think this question is getting at something really important that is important to think about when we're collecting robotic data and something that people sometimes don't put as much thought into, which is that very broadly speaking in machine learning, the one thing we know works is when training matches test so getting real authentic data is really really important um and we try actually pretty hard to make the data collection process for our systems as realistic as possible and as representative of what would actually happen uh for real tasks like even to the point where right now we're doing some experiments collecting data for like kitchen type tasks and the robot is like literally in our kitchen like our building has like a little office kitchen and the robot is like cleaning up like the actual office kitchen When we're doing the table cleaning task, we would like eat our lunch, put our lunch there, and the robot would go and like clean up the actual lunch.

30:42So the more realistic we can make it, the more training will match tests and the more the model will actually generalize to real world situations. Now, at the same time, to your question about synthetic data, something that I'm pretty excited to study in the future is the degree to which having this initial foundational understanding might actually make it easier to incorporate synthetic data. It's like if you're playing a video game, right? We understand how the real world works. So the somewhat abstract cartoony environment in a video game or in an animated film makes sense to us. But we're coming to that image with a lot of the physical priors we learn from the real world.

31:19And it may very well be that by having this really nice foundation of real world experience, it might actually be easier to incorporate less realistic data in the future because the robot will represent its experience in this way that abstracts away those difference. The Pi Zero model, I think the full model is 3.3 billion parameters. Can you talk a little bit about kind of the application of scaling laws and how you see that evolving? Yeah, that's a great question. So we wanted to start with a pretty lightweight model because, well, we didn't want to wait for a long time for things to train. We also have to run this thing in real time for inference.

31:54so we weren't particularly like deliberate in saying like this is the optimal size we just picked kind of the smallest model that seemed like it had you know enough of that internet scale knowledge baked into it but one of the technical challenges that I do think is really exciting to study in the future is how we can both have the benefit of large models for the more elaborate kind of semantic reasoning the you know like all the fancy chain of thought stuff solving complex problems and still connect it up to like a little motor cortex model that can run fast enough to control the robot. And there's been a little bit of work academically, including in my group and other research groups, that shows that if you have these vision language action models, you can actually do the same kind of reasoning stuff, the same kind of test time compute tricks that people have used for LLMs.

32:42So we had a paper, for example, called the Embodied Chain of thought from my lab at UC Berkeley, where we trained a vision language action model, a different one, predating Pi Zero. This is based on OpenVLA, which would actually reason through things in the task. So it would say like, okay, you're asking me to put the banana in the plate. Okay, well, to do that, I should first find the banana in the image, find the plate. I know the bananas are yellow, so let's find like the yellow thing. I need to find where my hand is. Okay, my hand is over the banana. That means that the right thing is to move down.

33:10Move down means coordinate, coordinate, coordinate, and then execute the task. And this actually helps. This helps a lot to get good results, especially in unfamiliar settings, because in those unfamiliar settings, while the robot might not kind of instinctually know what to do, if it reasons through the task, it can succeed more reliably. What was the performance like? It sounds very slow. It was slow, yeah. I mean, this was also on an academic compute budget, let's say. So yeah, it kind of like moves, pauses, moves, pauses, but it's 50 % better. So you do get a big improvement. Obviously, now the systems challenge needs to be overcome.

33:45I'm curious if one of the things that is clear, you know, both in our conversation and the paper is that the team has been really good at like taking tidbits of, you know, pieces, innovations from lots of different places and pulling them together into this work. Was there anything that came out of the recent DeepSeek, you know, R1 stuff that, you know, was inspiring or that you are looking forward to playing with in terms of improving this model? Yeah, that's a really good question. I mean, it's something that we have been thinking about a lot. I don't think there's anything concrete that I can really talk about now because, you know, when you have kind of vague ideas, you want to like at least test them out first before you risk saying something stupid.

34:25But a lot of us found that to be really inspiring. And I think that especially the way that they have this fairly simple recipe where it's just like pre-train and then use RL. Obviously, like the fancier recipe that's a little more conventional is like a little bit better, But just the fact that I pre-train and then use RL works so well, that's pretty cool. And I really wonder how some of those could be adapted to things like these VLA reasoning models where the robot could actually use RL to train itself how to think more carefully. But this is just speculation at this point. So you mentioned FAST a couple of times.

34:59Talk us through the motivation there and how it builds on what you're doing with PyZero. Yeah, yeah. Yeah. So we used a diffusion-based model for Pi Zero, which we had to kind of adapt with VLMs. The action expert in particular. Yeah, the action expert, exactly. But prior work on VLAs used discretization. And the reason that it used discretization is because VLMs naturally output discrete tokens. So if you want to adapt the VLM most directly to control robots, you take your actions and you basically turn them into text. In fact, in the first vision language action model ever developed in the RT2 model, they were literally like numbers.

35:37Like you represent the action as the number 132 and that is converted to floating point and then run on the robot. The trouble with this though is that we know from language models that it really matters how you represent those tokens. Like much as we'd like to imagine that language models are so brilliant that they can just handle like whatever you throw at them, the way that you represent your tokens makes a huge difference for performance, not necessarily because there's any particular knowledge baked into it, but because it makes the model train a lot more effectively. Basically, different letters in English occur with different frequency, and therefore they carry different amounts of information.

36:15And it's much easier for a neural network to learn when all the outputs are kind of equalized in terms of how much information they carry. So it really helps to represent that text in a way where every single token has about the same amount of information. And then the model learns very quickly and generalizes effectively. And of course, there's no reason to think that that wouldn't be true for actions either, except that that's not how actions were being tokenized before. If you're just representing them as numbers, the occurrence of those numbers on the internet has absolutely no bearing with how frequently those actions occur for robotic data.

36:49So the foundation of modern tokenizers is basically compression. Like it turns out that the way that you get every token to have about the same amount of information is to compress your text because the optimal compression will provide as much about as much information, about the equal amount of information for every single bit. So we can take inspiration from compression methods for continuous data to develop a tokenization for actions. And we thought about this and one place where you see compression of continuous data is images. So you use like JPEG compression. JPEG compression compresses images.

37:24JPEG compression basically represents an image with different frequencies. So different frequencies occur in different extents in different images. You put your image in a frequency domain, roughly speaking, and then you compress the weights on those frequencies. So we decided to try that. Essentially, it's like run JPEG compression on your action chunks. And that actually gets you a much better tokenization. It gets you the ability to represent the same actions with the same fidelity, but with a much smaller number of tokens. That by itself is not actually as important. What's important is that now those tokens contain about the same amount of information.

38:00and when you train on this your model trains about four times faster than it would if you used uh the the naive tokenization that's a that's a really big difference and it's not just that you spend less time waiting like if your model trains four times faster you can train for the same amount of time it'll train four times better right so that's a really big deal uh that allowed us to train much better models especially for uh tasks that required a lot of generalization to language following uh this part we're still trying to understand like what's the connection between this in language following.

38:29Like, you know, maybe it's something like if you fit the data better, you'll get to understanding language better. But one of the things this allows you to do is take, for example, the open source Droid data set, which is a big data set of Franca robotic arm manipulation that was collected across a number of different universities with language annotations, and actually get a policy that will generalize to a new Franca arm in a new location. We took the Pi Zero Fast model and we sent it to our friends at all sorts of different universities. There was a video that I posted on Twitter from some folks at UPenn that ran this.

39:02And it's like literally the first time they load this up on their Franco robot, they tell it's someone like pick up the pineapple and put it in the basket. And it actually goes and does it. And like, you know, the folks running this are not used to models working out of the box. Like usually robotic models don't work out of the box. So it's pretty cool that we could get that by just figuring out a better way to compress actions. And when you say that you train the model using this idea, what is the model in this case, the entire model, just the VLM, just the action expert? Yeah, so the model at this point, for the Pi Zero Fast experiment, is just the VLM component.

39:37And I don't know if that's necessarily a good choice, like we honestly didn't even really experiment with it very carefully. You could apply the same Fast Tokenizer to other VLMs, so we used it with the OpenVLA model, for instance, and the tokenizer is like available on Hugging Face, so anyone could grab it and run with their own VLA model. And so does the action expert change at all? Or is that the same with and without the fast tokenizer? Basically nothing else changed. Like the only thing that changes is the tokenizer. Okay. And so the mapping from the, you know, the fast tokens to the actions, that is still learned, but it's just learning a different thing, you know.

40:18Yeah, that's part of the tokenizer. Part of the loop, right. Okay. Interesting. Interesting. And so talk a little bit about the open sourcing of all this. That's kind of the most recent thing that you guys did. Yeah, yeah. So we figured that now that we actually have a model that can perform some pretty interesting tasks and also a model that other people could actually run. So we had like a little closed beta where we sent this to a few other folks, a few universities, a few companies. that this would be a really great thing to share with the community. Now, obviously, we have our own reason that we want to do that.

40:57Like, we want to see how people use robotic foundation models because that'll help us learn how to make them better in the future. But we also think that this is a great way to galvanize a lot more interest in this stuff because we saw how with language models, just getting the pre-trained checkpoints out there and allowing people to fine-tune their own models created this, like, wave of creativity where people come with all sorts of new things to do with them. So we're really looking forward to see what people will do with pre-trained robotic foundation models. So we really want to get that out there.

41:24We have a few demos where, you know, it'll run on some robots that will run on Droid. The truth is that it is like a very early prototype, I think, in the grand trajectory of robotic foundation models. So probably most things that people will try it for won't work. But just from seeing the kinds of experiments people do, I think we'll all learn a lot about it and we'll figure out a lot of new ideas for how to make robotic foundation models more applicable in the future. And so what specifically did you provide in open source? I know you provided the weights for the models. Are you providing any of the details around the training recipes and or data sets, those kinds of things?

41:58Does it allow you to fully replicate PyZero or does it allow you to use what's already been done? So the open source repo includes the code for fine tuning. It includes the base checkpoint. It includes another base checkpoint for the fast version of the model with the tokenizer. and then it has a few example fine-tuned models like it has a model fine-tuned for droid it has a model fine-tuned for aloha and these are really kind of intended as like demos if you just want to like try it out and see uh the main use case that we anticipate is for folks to take the base model and then fine-tune into their own robots because the robotics community is still very fragmented like everyone's setup is different so if you are lucky enough to somehow have a setup very similar to droid or very similar to a setup that we had then you might be able to try to run it in zero shot.

42:42But really, the intended use case is to collect some of your own data, fine tune it, and then try to use it to solve your task. And in order to do that, would you need a setup that offered the same kind of operator guided data collection methodology? That's a really good question. And this is actually one of the things that we hope to understand better. So, you know, we explored a few different ways to collect data. We kind of have our own intuition for what works and what doesn't work. But if somebody tries to fine-tune the Pi Zero model, they'll collect data. Maybe they'll do it the same way that we do.

43:16Maybe they'll do it differently. And I think it'll be really interesting to see what kind of recipes work and what kind don't. So we know what recipe worked for us. We described it in the paper. So hopefully if someone tries it, it'll work for them. But maybe we'll try other things. And then we'll find out that maybe you can get away with a lot less data if it's more consistent. Or maybe there's some other kind of thing that we didn't know was true that's actually true. So we kind of just really want to see how people experiment. And you, most of the robots that are depicted in the paper and in the demo videos are arm-based robots.

43:45You mentioned some work on a humanoid based robots. Are there other form factors that you see folks experimenting with? Yeah. So we ourselves have successfully been able to run the model. Well, either we ourselves or with our collaborators and partners on robots that include humanoids, single arm, dual arm, and mobile platforms. In principle, the model supports very flexible action representations, as long as your action is less than 32 dimensions. So, you know, we have to pick a maximum, so we pick 32. I've gotten videos of people that have successfully run this model with five-finger hands. I'm not entirely sure how they did it because no one has actually told me, They just send me videos like, look, it's working.

44:30In principle, it should be possible to run it for like navigation and things like that. We haven't done that ourselves, but that's very much within the kind of the constraints of the model. So as long as it fits to the dimensionality, somebody could try it. I'm really curious to see what works and what doesn't. I'm sure there'll be some limitations. Like if you train on an octopus arm, well, that's probably a little too different, but I'd be curious to see what happens. When you think about kind of the robot, the platform landscape, like are there accessible hobbyist types of or enthusiast types of arms that you could try it out on?

45:04Yeah, that's a really good question. So we actually were very lucky to get some help from Hugging Face, who helped us with doing a PyTorch port of the model. And they also have a very low cost arm. I can't vouch for its quality. I've never actually used it, but they seem to have been able to do some pretty cool things with it. So if anyone is interested in like a really low-cost robot, checking out LaRobot and their PyTorch port of PyZero, as well as some of the, you know, I think they've actually tried it out on their arms. That could be a good way to go. But honestly, like even the nicer arms that we've been using, the arms that we used for things like the Aloha experiments, these are not all that expensive.

45:46I think Aloha was like on the order of 20K. So the whole system is on the order of 20K. The arms are, I think, somewhere in the$6 ,000 range per arm. So if you just want the follower arms, like the minimum thing, I think you could probably get away with like 15K or so. But the cost is, you know, seems to be going down every year. So I wouldn't be surprised if like next time this year that these things are even cheaper. Okay. Awesome. Awesome. By the way, one of the things I'm really excited about is like, well, if we get these robotic foundation models into people's hands, if the cost of hardware keeps dropping, maybe it will be like very practical for anybody to just play around with their own robot.

46:20I mean, you know, 15K is a bit expensive, but if it drops like, you know, another factor of two or four, maybe that'll be actually pretty practical. Awesome. Well, what's next? Where do you see it all going? Yeah. So there are a number of next steps that I'm pretty excited about. One of the things that I really want us to be able to do better, and I think that that's something that we should be able to talk about more in a few weeks is do a much better job of following complex instructions. So one of the things that's really cool about ChatGPT is you can actually give it a prompt that describes in detail almost like a job you want to do.

46:52So you don't just tell like, oh, you know, please write me an email to my boss. Like you would actually describe like, write me an email to my boss that describes how I want to raise and blah, blah, blah, blah, blah, you know, whatever. It's kind of a complete description of a task. And it would be really cool if we can do that with robots too. Instead of just telling it like fold the shirt or clean the table. You can tell it like, hey, I'm throwing a party. I already put the plates on the table, but there's some trash. So put away the trash, but leave the plates where they are and make sure the fork and the knife is next to the plate in a nice, tidy way.

47:20So something that kind of actually like describes what you actually want. And maybe the robot would do the task. Maybe he might even ask you like, hey, I didn't get that part. Like, are you sure you want me to like leave that plate there that doesn't look like it belongs? Like you can have a much more intricate interaction. And the interesting thing there is not just the interaction part, it's the ability to instruct the robot to really do the job. I think that that can also be a really interesting mechanism, not just for getting robots to do sophisticated tasks, but also getting robots to repurpose their skills.

47:51So if the robot can actually get this more intricate task and think about, hey, I've learned to do these particular behaviors, how do I adapt them to solve this new problem that I've been presented with? That's something that you could do with a lot of that semantic knowledge inside of VLMs, but it requires a little bit more processing. Maybe it requires a little bit of that test time compute. And that's something that we've been working on that we hope to be able to tell people more about in the near future. Do you make a distinction between complex instructions to instruct the robot to do a complex multi-step task and instructions, complex instructions that instruct the robot how to do a complex task?

48:31Yeah, that's a really interesting question um i would like to not have to make that distinction like the distinction is this is actually important in the sense of like the capability but i think you can have a system that does both of those and in particular like people actually do it in a very flexible way where people bring to bear their own knowledge and their own skills uh but they also benefit from the information that they're offering so if you already uh you know if you are a professional laundry folder like it's enough for someone to just tell you like hey like go do your job and you already know what to do.

49:02But if you are not experienced at that task, then maybe you'll still be able to do it if only somebody provides you with more instruction. And if you have a model that is trained end-to-end that can perform this kind of reasoning, put together the steps that it knows, and flexibly decide what to do, then it can do either of those depending on the situation, depending on its prior knowledge. Got it. Was there anything else on that list? Yeah. So other things that we're doing that I'm pretty excited about, we are trying obviously to push the boundaries on the generalization for these systems. So I'm very happy with how we've been able to demonstrate pretty sophisticated tasks, but it's still a challenge if you want those tasks to work with any object and in any environment.

49:43Generalization means a lot in this context. It's generalization to task, generalization to environment, generalization to object, generalization to platform. And to instruction as well. Yeah. So it's a very big space. And because it's so big, it's also kind of hard to say anything particularly definitive about it. But it's something that we're studying quite a lot. We're trying to see how different ways of collecting data, different ways of training models, different ways of transferring knowledge from internet scale pre-training can facilitate generalization. And I think something that's really exciting there is like those moments, like I mentioned, when we had that droid model and somebody was able to run it at a different location, students were able to run it at another university and just see like that spark of life where they just give it some task and just goes and does it.

50:26Maybe it does it poorly, maybe it's slow. But I think that's really special. And I think that the more we can enable that, that kind of aha moment where you just load up the model on your robot, tell it to do something, and it actually kind of gets it. Like that, that I think is really, really important. I think that if we're careful with transferring knowledge from the web, curating data in the right way, and setting up our model in the right way, then I think we can get a lot more of that. When I think about, you know, this is an example like this, where, you know, someone is using this model, presumably the same type of robot, but just another environment.

51:00And you think, oh, well, that should work. It's a software program. You put it someplace else. And I think back to, I think it was one of, it was like a Peter Abiel demo of like the robot trying to do knot tying and like just different colors of rope and stripes and stuff like that totally confounded the robot. Yeah. My very first robotics project, which was actually with Peter Abiel, we have to use the same background in every trial because that was the background that worked for the robot. So I'm really glad that we've gotten past that at this point. That's awesome. Well, I'm looking forward to keeping in touch and keeping up with the updates coming out of the work.

51:38Very cool stuff.