Chelsea Finn: Building Robots That Can Do Anything

0:00Hi everyone. I'm really excited to talk about developing general purpose robots and how we might actually truly develop and bring intelligence into the physical world. So to start off, I'd like to talk about this problem, which is that if you want to truly solve a robotics application, you essentially need to build an entire company around that application. You need to build a different company for logistics, for wet lab automation, for robots and kitchens, for surgical robots, and so on. And this is really, really hard to do because that company needs to make new hardware, develop custom software, design unique movement primitives for that application, handle edge cases, and so on.

0:43You have to do all of that from scratch if you want to solve a robotics problem. And as a result, a lot of robotics companies haven't been very successful in actually bringing robots into the physical world successfully in our daily lives. I co-founded a company called Physical Intelligence that's trying to solve this problem. And in particular, we're trying to develop a general purpose model that can enable any robot to do any task in any environment. And we think that this sort of generalist model may work better and be easier to use than purpose-built models, just like we've seen in the development of foundation models for language and other applications.

1:24For example, if you want to build a coding assistant, you don't nowadays develop something specifically for coding, but you develop and you build on models that were trained on large amounts of data, not just on code. And essentially, this is the problem of trying to develop these sorts of foundation models and bring this sort of intelligence into the physical world rather than the digital world where they largely are today. So how do we do this? In this talk, I'd like to talk about how we go about doing this. And if we were to take a lesson from language models, we know that language models have taught us the importance of scale.

2:00And so one possible conclusion would be that perhaps scale is the most important ingredient for developing these models. And if you were to say this conclusion is true, then you might look to certain data sources for large-scale data. So for example, we might look at data from industrial automation. And you get tons and tons of data of robots doing tasks over and over again like this, but the sort of data isn't going to allow robots to go into disaster zones or to make a sandwich or to bag groceries. And so this massive scale doesn't have the diversity of behaviors that we need in order to solve this general problem.

2:42Alternatively, maybe we look at data from YouTube, which has also a massive data source and many videos of humans doing tasks that can be useful for training robots. But at the same time, we don't learn how to write by watching other people write, and we don't become expert tennis players by watching Wimbledon. And so even though there's a massive scale of data here, it's very challenging to use, and there's also a gap between the embodiment of robots and humans. And lastly, we might look at data from simulation. You can also get a massive scale of data here, but this data lacks realism and also has a gap from reality.

3:15And so I think the lesson here is that scale is necessary for developing these models that can generalize in open world conditions, but they're subordinate to actually solving the problem. So you need scale, but it's not sufficient for the entire problem. And so at Physical Intelligence, this is an example of a data episode that we've collected. This is in order of our first anniversary, which was a few months ago, where we, here you can see a teleoperator in person who's operating some leader arms to control the robot to light a match and light a candle with the match. And with this sort of data, we can train robots to do a variety of different tasks.

3:55And so what I'd like to talk about is some of our recent results at trying to develop sort of physical intelligence with large-scale real robot data. I should mention this is large-scale by today's robot standards and arguably a minuscule amount of data compared to the sorts of robot data that we should have in the years to come. And so in particular, we'll be looking at whether robots can do a variety of dexterous long-horizon tasks, whether robots can succeed in places they've never been, whether robots can respond to open-ended prompts and interjections. And even if you're not excited about robotics, I think that the lessons that we've learned from trying to address these problems are applicable outside of the physical world.

4:34So can we develop robots that can have complete dexterous long-horizon tasks? And in particular, in this first part, I'd like to talk about how we trained a Pi Zero Foundation model to do this task, which is to unload a dryer and fold laundry. And to date, I think this is the most impressive thing that I've seen a robot do in the physical world. It's really hard.

5:04It's an incredibly difficult problem. You can see that it's not perfect. Here it's making some miscarps, making some mistakes. But it's really, really hard because you have to deal with the variability in the clothes and the way in which they might be positioned and crumpled and be able to handle all those sorts of things. And as you're doing this task, which takes about 10 minutes for the robot, there's many opportunities to fail, to fail catastrophically. For example, dropping things on the ground, which is hard to recover from. And you have to be able to recover from even small mistakes. I was personally actually working quite a bit on this laundry folding robot along with Michael and Siraj, of course, supported and with contributions from the whole physical intelligence team.

5:45So how do you even approach this sort of problem? This is a really, really hard thing for a robot to do. And what we did is we started simple. We started with, can a robot fold a single size, single brand shirt? And can a robot dynamically flatten one shirt, again, single brand, single size? And if you start simple, this makes the problem quite a bit easier. We collected some data with teleoperation and trained a policy with imitation learning. And our model had around 100 million parameters mapping from images from robot's cameras to target joint positions on the robot arms. And we do this source of control at 50 hertz on the robot.

6:24And we founded the company in kind of mid-March of 2024. And a couple months later after we had set everything up, we were able to get a policy that could fairly reliably fold a single size single brand shirt. You can see that I'm testing the policy right here. And we also wanted to test some dynamic motions because you need to be able to match the control frequency accurately in order to do these sorts of dynamic motions. And so these were some of our very initial tests at addressing this sort of laundry folding problem. Then from there we wanted to make the problem incrementally harder. And so we instead of starting from the shirt flat on the table, we started in a crumpled position like these.

7:05And it turns out that this actually makes it a lot harder. And so here are some videos of some of our initial attempts at trying to train the robot to fold these shirts, and the robot struggles. The robot does some things that kind of look somewhat sensible, but generally isn't able to make progress on the task. With many tests, frequently we're getting 0 % success rate in our tests of this system, and really struggling to make progress. So really here is the, it introduces this challenge of handling the sorts of variability in the ways in which shirts might be crumpled on the table. We had some initial signs of life in late June, of last year.

7:43And so in this case the robot was able to kind of make progress on flattening the shirt. It's also then able to fold the shirt decently well from that initial state. Still not perfect. And as you can see it takes quite a while to do this. So this is a video that was sped up 8x. So not something that you might have the patience for a robot to do. So with some initial signs of life, also very low success rate, we started to transition to a slightly harder version of the task where the laundry starts in a laundry basket. We also introduced variable size shirts and shorts into the mix. And again, the robot really struggled.

8:22So in many of our tests, we're getting a 0 % success rate across the board. And we're really struggling to actually get the robots to learn how to do these tasks. At this point, we were trying to consider a lot of different things. We thought that maybe the robot needs memory, needs history in some way. Maybe we need to just train our models for longer. Maybe we should be doing control in end effector space rather than in joint space of the robot. Maybe our encoders, we knew that there were calibration issues and maybe we need that calibration to be more consistent. Maybe we need to condition the model on more information about the data.

8:53Maybe we need hierarchy because this is a pretty long horizon task and it needs to break it down into different subtasks. Maybe we need higher resolution images. Maybe we need to introduce interventions in data collection. A lot of these things we also tried. We had around two to three months of failure where nothing was really working at addressing this task. But then at some point, we actually had a bit of a breakthrough, which was that we found one thing that really seemed to make a difference in the robot's ability to do the task. And this was actually to take some inspiration from the world of language modeling to actually, instead of just training a policy on all of our data, we pre-train on all the data and then fine-tune on a curated, consistent, high-quality set of demonstration data.

9:35When we did this, we found that the robot was actually able to make progress and a lot more reliably fold articles of clothing. And so I think that this video was the first video where the robot was able to fold five items in a row and stack them. I went home very excited this day. This was in September of 2024, so multiple months after our initial tests. Now this is far from perfect. It takes 20 minutes to fold five items of clothes. And at the same time though, it kind of suggested that this sort of recipe was able to unlock the capability in the robot to actually fold these articles of clothing.

10:15So you can see these sorts of failures here. In this case, it attempted to fold the blue shirt around seven times before eventually actually figuring out how to do that. There's also other failure modes as well. So here's an example where the robot pushes the stack to the corner of the table and decides to kind of fiddle with it a bit, and then eventually slides it off the table. And then it proceeds as if nothing happened and it's going to continue to fold. We continued to iterate on this recipe. We selected and worked on our curation strategy for curating a higher quality set of demonstration data.

10:46We got it from 20 minutes down to 12 minutes for these five items. This is kind of how we were evaluating how good our robot system was. It still makes mistakes. It's still, the full quality still varies, but it's still significantly better than our previous curation recipe. Now, at this point, we were still training models largely, kind of we were pre-training and fine-tuning only on laundry data, and we weren't leveraging kind of pre-trained models in the community. And there were some folks working at physical intelligence that were working on developing a pre-trained model, trained on all of the robot data.

11:18and we then started to try to introduce these models into our recipe. And so we took an open source vision language model, a 3 billion parameter model called Polygema. Previously we were using, the previous videos were all with like 100 to 300 million parameters that we were iterating on. This model takes as input images from the robot, also a language command, and then has a head, a diffusion head, that's going to attend to all the internal values of the vision language model and with the joint angles predict a chunk of 50 actions into the future, so about one second of action steps. And we're using a flow matching, a variant of diffusion to actually output these actions and output continuous actions.

12:04So we took this pre-trained, this model, and instead of pre-training only on laundry, we pre-trained on all of the robot data that we had collected. And then we just fine-tuned it with the same exact post-training recipe that we had developed without using the vision language models. When we did this, we actually saw the robot continue to actually get better when we just plugged in that new pre-trained model. And so in the left video, it's able to do five items in nine minutes, which was faster than the 12 minutes we had before. In the right videos, we were testing with some novel clothing items and found that it was also quite efficient at folding multiple items in a row.

12:40And we also saw as a result, there's also more consistent bold quality by using this model that was about 10 times larger and had seen more robot data as input. To look at a few highlights of this, here's a pair of shorts that the robot hasn't seen before. And this is kind of a tricky scenario where to flatten it, it actually kind of needs to reach under the kind of the bottom of the shorts. And it's able to do that. It's able to kind of figure out that it should reach under the left part of the shorts in order to eventually flatten it and then once it actually successfully flattens it, it's able to fold it successfully.

13:16It also has to do something similar at times to fold shirts. So in this case, it needs to actually kind of fold the shirt over on itself, which actually puts it in a more crumpled state, arguably, but allows it to find the corners of the shirt and then go ahead and fold it. And then like I mentioned, it also is able to handle unseen clothing items. So here's an example of a shirt with a v-neck that is able to fold even though the post-training dataset didn't have, well, the shirt was completely held out and the post-training dataset didn't have any v-necks as input in the dataset. It's also able to fold shirts with buttons, so it has some degree of generalization to different clothing items.

13:55And then lastly, because this policy is a neural network and it's kind of taking as input the current image, it's able to handle interruptions. So here Michael is continuing to mess with the robot and the robot figures out that it should put the shirt away while it's trying to fold the other shirt. In this case, Michael is going to continue messing with the robot. So Michael unfolds one side and the robot reacts. Michael goes in again.

14:26And the robot makes some mistakes here, but is able to recover. Michael messes it up again. So those are some results of what the robot is able to do. Now, I talked about this pre-training and post-training recipe being really important. We can actually quantitatively measure that and actually make sure that this is actually what's leading to improvement. So we compared this pre-training and post-training recipe to not using any pre-training and only training on the curated data set versus no post-training where you're training on all of the data rather than fine-tuning on the curated data set.

14:59And we evaluated these models in terms of their progress on the task, where you make partial progress for getting it out of the bin, which is the easiest part, and then further progress for flattening, folding, and stacking the items. And we see that the pre-training and post-training recipe is able to get far higher performance than omitting pre-training and omitting post-training. And notably, omitting pre-training and post-training is basically able to get it out of the bin and make very little progress after that. Whereas when we combine pre-training and curated post-training, we get far higher performance where it's able to reliably flatten and fold objects.

15:33And then the last thing I'll mention on this note is that nothing in this recipe is specific to laundry. And so we took the same recipe and fine-tuned on other tasks. So here the task is to kind of clean up a table. And the robot was also able to successfully do this task, despite the fact that we primarily were iterating a lot on laundry, but is able to also apply this recipe to this task. It also is able to scoop coffee beans into a coffee grinder. This task is pretty hard. It has to construct the bottom part of a cardboard box, which requires quite a bit of dexterity. And then lastly, autonomously lighting a candle with a match, again with this kind of same pre-training and post-training recipe.

16:21And so this is pointing at this kind of the benefit of foundation models that I alluded to before, which is that to do these different tasks, you don't have to start completely from scratch. You can actually leverage pre-training across multiple robots and across multiple tasks. And then we're also able to apply that same recipe to robots at other companies. This is a robot that I've actually never seen in person before. They collected data, they sent the data to us, we fine-tuned our model on their data. We actually didn't even know exactly how the model is being controlled, exactly the representation of their actions.

16:56But by fine-tuning the model on this new robot, the model is able to control the robot in order to make a cup of coffee in this case. So, some takeaways for this part. We were able to independently develop post-training and pre-training and decouple the problem, and then eventually get the best of both. We found that training on all the data doesn't work for complex tasks, and this sort of pre-training and post-training on curated data leads to far better performance. And then we broke up this really hard problem of folding laundry by gradually starting with folding single shirts and going to more and more complex versions of the task.

17:33Now, there's a number of limitations here, And one limitation I'd like to point out is that these robots inevitably, in this case, were trained in the environments that they were tested. And so this means that, in principle, you can use these methods to collect a lot of data in one environment and then deploy them in one environment. But ultimately, there's going to be things that change about an environment and scenarios where we would want to actually apply these robots to environments that they've never seen in before. And so how can robots actually succeed in places that they've never been?

18:02The lesson we've learned from machine learning in other places is that we should collect diverse data. And so we started by collecting data of tidying bedrooms and kitchens in many different environments. And here's an example, kind of a sample of that data. And we collected robot data in homes across San Francisco here, and also collected data in diverse mock kitchens and mock bedrooms. And in total, we had more than 100 unique rooms represented in the data set. That ended up being part of a bigger pre-training mixture. So we trained on this diverse mobile manipulation data, including the low level action prediction, as well as predicting high level subtask commands for how to complete the task.

18:44But we also trained on previously collected static manipulation data that was also fairly diverse. Static manipulation data that we had collected in our office and in labs, as well as web data and high level instructional data. And I should point out here that the mobile manipulation data of tidying bedrooms and kitchens only accounted for 2.4 % of the overall pre-training mix. And so the lesson here is that you're basically able to spin up a new task and actually an entirely new robot. The rest of the mixture didn't have any mobile manipulation data with this particular mobile manipulator in it without redoing all of the data collection.

19:20We're able to build upon everything that had been done before. And it's kind of this kind of same story of foundation models being able to make it easier to spin up a new problem, a new application without starting from scratch. Now, this wasn't completely easy. We had a couple challenges. One of the challenges that we ran into is that naively, this model can ignore language instructions. So we actually, in this case, asked it to pick up the cutting board and it chose to pick up the plate instead. Now we're again asking it to pick up the cutting board. And instead, the robot had a mind of its own decided to pick up the plate, and then we tell it to put the plate in the sink, and eventually it decides that, well, after kind of moving away from the cutting board, it eventually decided that it would actually pick up the cutting board.

20:04And so in the early development of our model, we found that it often ignored language. And to solve this, we thought about how vision language models actually follow language well, and so maybe there's a way to preserve the inherent abilities of the pre-trained models when addressing this task. And so what we did is with this Pi Zero architecture, this action head that's using diffusion is randomly initialized. And this ends up actually deteriorating the pre-trained knowledge that's present in the vision language model. And we found that if we can prevent this deterioration, we might be able to get better language following.

20:43And so the recipe that we came up with was actually in some ways fairly similar, but instead we're going to be predicting tokenized actions. And when we have the diffusion head, we'll be stopping the gradient from the randomly initialized diffusion head to prevent it from deteriorating the language following abilities of the VLM backbone. And we found that this first led to faster training because the tokenized actions are a more direct supervision signal. And second, it also followed language far better, an 80 % follow rate rather than a 20 % follow rate, which suggests that we're able to preserve the kind of pre-training in the vision language model backbone.

21:19So we put those pieces together. We took that recipe and trained it, pre-trained it on all of our data, including the mobile manipulation data. We fine-tuned it on mobile manipulation data in a variety of environments. And then we tested the model in places it's never been in before. So we rented three Airbnbs that we had never been to before. We put the robot in those homes, in this case in the kitchen, and I asked it to close the cabinet. I asked it to put away the dishes. It's also never seen these dishes or these forks, these objects. And the robot's able to succeed even though it's never been the here before.

21:54There's different countertops, different furniture, different objects, and so forth. Lastly, I asked it to clean up the spill and the robot is able to oblige and wipe down the spill and eventually put the sponge into the sink.

22:16It's also able to do this for bedrooms, so Laura asked it, in this case, just clean the bedroom. And it puts articles of clothing in, it throws away the trash, and then is able to tidy the bed by putting the pillow at the top of the bed and tidying the blanket or the comforter of the bed.

22:41YC's next batch is now taking applications. Got a startup in you? Apply at ycombinator.com slash apply. It's never too early, and filling out the app will level up your idea. Okay, back to the video. So quantitatively, I talked about how there's only 2.7 % or something of the mixture. So how much does that other data actually help? Could we actually just train on that kind of 2.7 %? And we find that these kind of bars on the right, which are excluding data from static robots in labs and environments and so forth, reduces performance significantly. So the performance goes down to less than 60 % when you exclude that data when evaluated in novel homes, compared to if you use the full pre-training mixture, it has more than 20 % higher performance.

23:27Lastly, we also looked at is the diversity of data helpful? Is it important? And so we increased the amount of data from these environments to test this. since it's always good to like, you can kind of do vibe evals, but it's really helpful to actually measure how well these things work. And so this is what this is measuring. And we find that if we actually increase the amount of homes, the amount of locations that are represented in the data, the performance increases, which is great. And it actually gets to the same level of performance as if we train on data from that target environment. And so that means we're actually mostly closing the generalization gap and suggest that the bottlenecks at this point for this sort of task lie not in collecting more diverse data, but in actually getting higher reliability and higher performance.

24:14Now, I should also mention that there's failure modes. Like this success rate was around 80%. There's lots of room for improvement. Here are a couple examples of those failure modes. So here it's told to put the items in the drawer. It is able to put it in the drawer, but the item isn't fully in the drawer at the end. and it decides that it's done and kind of moves on to the next thing. Here the robot needs to put the clothes in the laundry basket. It drives over the shirt and then it gets stuck and is not able to lift it up. Here we asked it to put the dishes in the sink and it successfully is able to put a number of the dishes in the sink but it struggles to pick up the cutting board in this particular case because it's very thin and it's flush against the surface of the countertop.