Chelsea Finn: This is the State of the Art in Robotics

26:38You don't need pictures of avocado chairs in your data set in order to generate something like this. Or you don't need combinations of other things that you might ask the model to do when it's deployed. Now, even back in 2021, it wasn't perfect, but these signs of compositional generalization were really exciting for demonstrating these two attributes of the model. So we have these two goals that we'd like to do. We like an out-of-the-box model and compositional generalization. Now, the tried and tested recipe

27:08Chelsea Finn:for developing this kind of model is to first take a sufficiently large and diverse data set, and to second, train a model with sufficient capacity. And so what we're going to do is we're going to do that. We're going to try to use all of the data that we have available. This includes really diverse robot demonstration data, including really low quality demonstration data. It's also going to include policy rollout data, basically attempts from the robot of doing the task. Basically all of the training data that was used for reinforcement learning for the previous tasks will be included in the training recipe.

27:44We're also going to include videos of humans and we're also going to include data from the web. Basically all of the data that we have. And then to train a model with sufficient capacity, of course we'll train a model that's large enough. But to fit data that's so heterogeneous, we also find it particularly important to prompt the model with all of the context that it needs in order to predict actions. And we found that this idea was really the key unlock to using this kind of data and this data of this kind of degree of heterogeneity. And specifically what this looks like is we're going to train a foundation model that takes as input the memory like that I mentioned before, an instruction of what to do.

28:30But it's also going to take as input a subtask construction of kind of what the next immediate thing it should do is. It'll also take as input metadata that kind of indicates the quality of the data, the length of the episodes and so forth. And this metadata gives it a lot more information about how it should predict the next action. And then optionally we'll also train the model with a sub-goal image as a prompt to the model. It's essentially saying like a few seconds from now you should try to reach something that looks like this image. With this detailed prompting we find that the model can really make use of much more heterogeneous data and I'll show some comparisons later that really kind of show how important it is.

29:11And now to actually deploy this model we need to then provide things like this subtask instruction and sub-goal images and so with that we can train a high-level policy that predicts the subtask construction like what to do next, what is the next like subtask for the task of cleaning the kitchen, and we'll additionally train a world model to generate images for what the robot should do next as sub-goal image conditioning. And so with this, we'll train a single model with those attributes on all of the diverse data that we had available. And here are some examples of what that single model can do.

29:46So all of these videos are from a single model, specifically a model that we called the Pi07 model. And on left, you can see it doing things like folding a collared shirt. On the top right, it's doing a really precise assembly step where it needs to insert a screw and drill that screw into a robot arm. And on the bottom right the robot is replacing a trash bag in a trash can. And so we had two goals at the start of this. The first was to move towards an out-of-the-box model and even those videos showed that out of the box the model is able to do quite a bit. But really the key question here is how does this pre-trained model compared to the specialists that were trained specifically for coffee making, specifically for box building that I talked about previously.

30:37And if we measure the throughput and the success rate of this single Pi-07 model versus the fine-tuned Pi-06 model, we see that the across the board, the single pre-trained Pi-07 model matches or outperforms the fine-tuned specialists that were developed with reinforcement learning post training for those downstream tasks. And so we see that it's able to match the performance of specialists. It also holds for SFT specialists, not just RL post trained models as well, suggesting that we do indeed have a single model that can do a lot of different tasks with a really high degree of performance out of the box.

31:16Chelsea Finn:Okay, so that was the first goal of out of the box models. And the second goal is compositional generalization. And there's a few different ways to measure this. There's many different ways you might try to combine concepts in robotics. The first test that we wanted to do was to see if a robot could interact with an appliance that's quite rare, like an air fryer. And this is kind of an example. We basically wanted to see if it could open an air fryer, put a sweet potato in the air fryer, and close the air fryer. And we picked this because we thought that the data set didn't have any air fryers in it.

31:50We didn't intentionally collect any training data with air After we did some analysis on the dataset, we actually found that our dataset was so diverse that it did actually have three episodes with air fryers in it. We expect that they likely weren't having an impact and that even if we didn't include those exact three episodes, it likely would still work. But what we found generally is that the robot was able to interact with the appliance that was basically hardly represented at all in the training dataset and combine the skill of interacting with it, like opening it, closing it, and so forth, with this object that it hasn't seen before.

32:28And then after instructing it, like Lucy did, we can train a high-level policy to do this task fully autonomously, and you can see the robot doing that in this video. So that's the first form of compositional generalization. The second compositional generalization test that we wanted to look at is we wanted to see if we could compositionally generalize between tasks and robot platforms. And so we wanted to take a robot platform called a Biarm UR5E robot. It's actually a very large kind of industrial robot platform. And we wanted to see if it could fold clothes, despite the fact that we didn't collect any folding data on this robot platform.

33:05So specifically, we had data of folding clothes, like folding a shirt on the robot platform that's pictured here on the left and then we wanted to see like out of the box without collecting any folding data on this very different robot platform on the right could the robot successfully do the task and what we see in this video is the we indeed did see that it kind of compositionally generalized in this manner the first time we saw robot do this we were like floored because there was no training data for this task. And the robot here is quite different from the other robot, not just in size, but also in the lengths of the linkages of the robot, in the configuration of the joints of the robot, and so forth.

33:52Okay, we can, and this is a 1x speed video so it's not the fastest thing and obviously if you haven't seen any training data on something you might not, if it's literally the robot's first time folding a shirt. It might take a few attempts, but eventually it will get to the folded shirt. You can also see the generated sub-goal images on the top left. So those are basically the models that are trying to generate images that will make progress on the folding task. And then those are passes input to the model and we see the folded shirt here. I think it's going to make a couple small corrections at the end to try to make it a little bit smoother.

34:29Cool. So the takeaway here is that both in terms of like language object interactions and in terms of task robot interactions we see strong signs of compositional generalization in this model. Okay, and then quantitatively we also see that as we kind of get to these more advanced models like the Pi-07 model, the performance of folding towels and folding shirts on this platform that hasn't seen before increases dramatically and it even approaches the performance of human teleop despite the fact that we didn't have any robot specific training data for folding clothes.

35:08Chelsea Finn:And then for the last experiment that we did here, I think this is maybe perhaps the most interesting experiment is we wanted to test how important are the two ingredients that I mentioned. How important is diverse data and how important is this sort of like capacity or detailed prompting for the kinds of results that I showed. And so if we remove the most diverse data from the model training shown in like the grayish color, we find that the performance on held out tasks decreases dramatically. Whereas if we just take out a random 20 % of the data that's less diverse than the most diverse subset, the performance only decreases a little bit.

35:48So this suggests that actually having really diverse data plays an important role in enabling it to generalize to new tasks. And then we tried to also ablate the fact that we are prompting the model with metadata. And for this experiment, we looked at with and without prompting with metadata. So with prompting is shown in yellow and without prompting is shown in the gray color. And with prompting, it helps kind of significantly. But the most interesting thing is if you look at when you add, so this is showing as you add more and more data and physically as you add more and more low quality data, what is the performance?

36:23And without metadata prompting, when you add lower quality data from 80 % data to 100 % data, the performance actually decreases, which is perhaps not too surprising because you're adding low quality data to your data mixture. Whereas with the metadata prompting, the performance actually increases when you add that low quality data. So Justin is actually able to make, get a lot more juice out of even low quality data when you include this kind of prompting. Cool. So the takeaways here are that we found that we're able to train a single model to control the robots that matches or exceeds the performance of specialized post-trained models, kind of like going from a BERT-like pre-trained model to a model that really works out of the box like GPT.

37:07We also saw strong signs of compositional generalization in a Dolly-like way. For example, in compositionally generalizing skills applied to appliances and skills applied to new robots in ways that weren't seen in the training data. Cool. And then all the videos and experiments that I showed were just evaluating the model out of the box without any post training. And the paper and the technical report online have a lot more experiments and a lot more details. Okay. So we talked about long-term autonomy. We then showed how we can kind of develop that in a single general purpose model. Where are we at now?

37:48The first thing that I'll mention is if we go back to the timeline of generalist AI, I think that we now kind of firmly have physical intelligence in the right side of this timeline. We're kind of firmly more in like a GBT and DALI-like era for robotics and physical intelligence, which is really exciting. And I think that we kind of went there in just a few years. And lastly, we also have these models that are actually deployed in real world circumstances. So the two videos on the top are actually two YC companies, Ultra and Weave, that have taken Pi models and post-trained them in deployment to do tasks like folding laundry and packaging in a warehouse.

38:31The kind of video on the bottom left is the video that I showed previously. And this kind of model works for a really diverse set of robot embodiments. The ones on kind of the top and the left are kind of a more standard by manual platform, but it also can be adapted to things like drones, quadcopters, surgical robots, and on the bottom right for things like tractors. And so this is really kind of truly showing how physical intelligence can make an impact, not just in demos and research and so forth, but actually in real world deployment. And I think that we'll, as we go, we'll start to see robots that are more and more robots actually deployed for real in the physical world with all the advances that we've been seeing over the past few years.

39:17Awesome. So the last thing that I'll mention shamelessly is that we are hiring at Physical Intelligence. So if you're excited about some of the stuff that I talked about, we encourage you to take a look at some of the open roles and apply. And yeah, definitely have time for questions and happy to get all your thoughts.

39:34Chelsea Finn:Thanks.

39:44Okay, so the first question is how far away are we from a chatGPT moment for robotics and what will that look like? So I'll start with the second part actually, which is that I'm not sure it will really look like the chatGPT moment that we saw in language models, which is that with something like ChatGPT, we saw like what, like it passed like what a million users in five days. I think that the distribution channel for physical models is going to be slower, unfortunately, because you actually need a physical robot there. I mean, I think that we've seen for something like Waymo, it's like the rollout, it's actually been like incredible to see the rollout that they've had, but it still takes time to actually deploy things on physical devices.

40:26So I don't know if we'll have single moment that has the distribution that ChatGPT had. At the same time, in terms of the capabilities of these models, I think that we are really starting to get to the point where these models are actually useful in the real world. And I think that getting to the capabilities of ChatGPT, I think is very much on the horizon in the next few years. Cool. The second question is, when should a small team switch from scaling per-site models to a generalist policy and what does that transition actually look like? What signals tell you it's time? So this is a good question.

41:03I think that the, I mean at the very least I actually think that just starting with a generalist policy and then fine-tuning it, even like right off the bat, can be really effective. And we've, and like fortunately a lot of generalist policies, like they're actually really powerful generalist policies that are open source. And so the PI0 and Py05 models are open source, for example, and we've seen a lot of people get a lot of use out of those models already. And we're also working with a lot of partners like the tractor company, like Ultra, like Weave, to actually develop, to take our most recent models and get even more juice out of even more powerful models for their own applications.

41:44So even right off the bat, I think that you can use them. The only scenario in which I actually wouldn't use them is if you're really in a really constrained environment. I've talked to some folks working on surgical robots that are like in an operating room in the basement with no internet connection and a really bad GPU. And so sometimes like it's really hard to use a larger model, but you still can do local inference with on a workstation with these models. And so I think that, yeah, right away, I just taking like Pio5 or your favorite model and fine tuning it is the way to go. And I think that we'll see lots these small companies and I think there's like yeah so much work to do in terms of actually getting these robots to work with this technology in the real world.

42:24Great, the next question is given how fast robotics is moving in industry, what are the real advantages and drawbacks of doing a PhD today, especially for someone who wants to go into industry afterwards? So I was not planning to do a PhD. I was always planning to go straight to industry. My parents are engineers and worked in industry. And I kind of thought that the way to have impact was to go to a company and so forth. And my dad even told me that he wouldn't hire someone with a PhD. And so I was like, maybe I shouldn't get a PhD if I wouldn't be able to get a job. But he's in a different field as well in civil engineering.

43:07And so So the, but at the same time, I think that like a PhD is an incredible opportunity and I, I love my PhD and obviously it depends a lot on the advisor, depends a lot on like what you would be doing and so forth. But I think a PhD is an incredible opportunity to first learn a lot about how to handle uncertainty, how to pick good problems to work on. And I think that basically like in research, you don't, no one kind of even gives you the problem to work on. you have to pick the problem. You don't know for the problem that you picked if it is achievable to make progress on that problem in a six-month time span, two-year time span, 10-year time span.

43:44And so you learn about how to deal with that uncertainty. And then the, so that's really useful. And then it also, it also is an opportunity to like do amazing research and do whatever, in many cases do like have a lot of freedom to work on the research that you find most exciting. And so I think that today it's still an amazing opportunity to do work, to learn about uncertainty. I think that learning about uncertainty is really useful in the startup environments, in being at the frontier of AI, because we don't know now. No one knows what the best route is to make these models more and more powerful.

44:20At the same time, there's also a lot of incredible opportunities in industry. I think that the first in terms of what goes into developing like everything that I showed, it's not just the research. There is a whole like software stack that needs to run on the robot, needs to run reliably. There's obviously everything on the hardware side. And then there's also even the machine learning infrastructure, the data infrastructure and all that. And so even with like you don't need a PhD necessarily to do a lot of that engineering work. And then on the research side, there's often opportunities to get involved as well.

44:54And a lot of research is engineering these days as well. And so I think it kind of depends on, yeah, it's a very personal decision and what you want. Even today, I think I probably like kind of retrospectively would want to do a PhD just to learn about how to handle uncertainty to do research. Because I really love being at the frontier and thinking about challenging problems. but there's also a lot of really amazing opportunities in both paths.

45:25Chelsea Finn:Okay, the next question is, large language models learn from the internet, but robots don't really have an internet scale data set, a physical experience. What's the robotics equivalent and how do we get it? So in robotics, well, so maybe in language models to start off, the data on the web is language data and not all of it's high quality, but some of it is like really informative and useful and it is data that is in kind of the that that kind of reflects a lot of what the like what you want a model to do like you want it to be able to predict text and be able to complete text answer questions and so forth and there's a lot of questions being answered on the internet and a lot of text that's being completed on the internet and so in general with machine learning you want trained to match test test and so you want the thing that you're gonna be training your model on to be reflective of the thing that you you're going to be asking it to do later on.

46:17And so I think the equivalent in robotics is data of robots operating in real world circumstances. And the way that we approach it at physical intelligence is to collect data, like to collect robot experience of robots doing all sorts of tasks. You can collect this with teleoperation to get initial data of robots doing useful things. But in the long run, I think it will also contain a lot of autonomous experience of robots deployed attempting things. And just like how we see in language models, how now a lot of time is spent actually generating data, generating synthetic data by actually running the model and having it think through things.

46:56I think a lot of the data in the future in robotics is going to be the robot attempting to do lots of tasks in lots of real world circumstances. And so, yeah, I think that that's kind of what it looks like. I also think that there are other possible sources of information that's really useful for model training, like videos of people doing things like YouTube, like web data and captioned images that tell you like this is a kitchen that has a fridge on the right of the sink and so forth. And all of that data I think can be really useful for developing a kind of frontier multimodal model that can control robots to do things, reason through how to do a long task, and also control the robot to do those tasks.

47:37I think that there's no substitute for the robot experience itself. You can't just like, if you watch a human do something, like if I watch Roger Federer play tennis, it doesn't mean I can play tennis as well as him, unfortunately. And likewise, robots can't like watch a person doing something and then figure out how to do it themselves directly. They really need their experience on their own platform to learn effectively. And I think that we will need large data sets. I think that doesn't mean the human video isn't useful. It's useful to watch Roger Federer play tennis, but the actual experience on robot platforms will be a critical component of developing an analogous data set for robotics.

48:16The next question is, is it possible that general purpose robotics models get democratized via open source the way that large language models did? Or will the cost of embodied data and hardware keep the best models concentrated in a few well-resourced labs? So I think this is a good question. So I do think the cost of embodied data and hardware will like could very much make this look different because I think that it's harder to get data even to like distill a model for example just like readily on the internet. I also think that we've seen kind of pretty large data sets get open source as well and pretty powerful models get open sourced.

48:59I think it's really hard to say exactly what will happen. And so, yeah, I don't know. The one thing that I will say is that with language models, even aside from like democratization like Kimmy and like really getting models that like perform at the state of the art, even then companies that are focusing a lot on closed source models are also doing a lot of open sourcing. And so there are like Gemma, for example, and the GPT open source and so forth. I think these companies like to support open source because it actually helps build the ecosystem around the things that they're building. And so I imagine they're being, I guess I'm optimistic that there will be a strong open source community regardless, but I don't know if it will exactly play out exactly the way that Language Balls played out.

49:48Okay, the next question is, does the model output raw motor commands directly or does it output a target hand position and let a controller solve for the joint angles? and what makes that the right level to learn at. So all the models that I showed were outputting target joint positions. So like, what is the angle of this joint? What is the angle of this joint and so forth that you want to hit? And then there's a controller, like a PD controller that is trying to then hit that target position for those joints. The model actually is also trained to predict target gripper, like, yeah, gripper positions, like where in 3D space should my gripper be?

50:24the um and you could also use that as well and back out the joint positions um you could also like another thing you could do is you could go directly to like motor torques or to uh like voltages or efforts uh there are pros and cons of different options uh we have found controlling joints uh and controlling um in like the the 3d space of the gripper uh to be to both work well and so yeah there's pros and cons I think the one thing that would be nice about going directly to the like the voltages is that you could also get a more stiff kind of output or a less stiff output whereas is the controller if you have a fixed controller then you you're not letting your model control that that aspect and so yeah there's different pros and cons the yeah, I think that the, it's like what we're working seems to work.

51:20It doesn't seem to be a bottleneck. And I often like to focus on the things that seem to be bottlenecks versus things that don't seem to be bottlenecks. Okay, next question is, do robots need something like imagination? The ability to picture what should happen next before they can become truly useful. So the PIO7 model that I showed has something like this, where it can kind of imagine what a future image should look like and then try to accomplish that. We found that that leads to improvement. And we saw in the short folding example, we saw like a quantitative bump from using that sort of imagination compared to not using it.

52:00At the same time, I think that the model actually performed surprisingly well without that as well. and we were actually hoping, we were thinking about writing an entire paper, an entire technical report just about that capability and that model. But the model without that was so good that we felt like we needed to actually have that play a bigger part of the story because it seemed like that was really delivering in terms of actually getting really strong results. And so, I mean, it seems like one design choice. I think that the, I think it's hard to say if it's going to be a critical component or not.

52:33But I guess the good news with these kinds of models is that if you develop a good data set, you can kind of run experiments and you can continue to test things with the data set that you have quite effectively. I also think that being able to predict the future seems like a very relevant objective compared to predicting future actions. So that should, I would imagine, kind of help in terms of learning from all the data that you have available to you. So yeah, hard to say if this should be a critical component or not. It seems like empirically so far it seems to help, although perhaps not as much as you might expect.

53:09And even without that imagination, the robot can do pretty incredible things. Okay, next is right now it seems that robots are doing amazing tasks, but in a very slow manner. What is needed to improve the speed? I'm really excited about improving the speed. And we did see speed improvements from reinforcement learning. We also have another release called the RL token that where we showed actually even faster speed and actually faster speed than human teleop. I think one of the bottlenecks is that when you teleoperate robots to do things, which is the easiest way to teach a robot to do something, people are kind of slow at teleoperating the robot.

53:47We have a couple of projects in the pipeline that I think have really promising results in terms of getting fast policies. and so I think more to come there. And yeah, I think it's either you need to figure out how to make the data faster or you need to figure out how to be faster than the data. We see the evidence of being able to be a little bit faster than the data and in terms of the next steps, it's either to kind of go even further than that or make the data faster. Cool, what's the most surprising task you've seen a robot complete recently? What do you want to see it do next? So the most surprising thing was not really a task, but when we were working on Pi07, I personally trained one of the policies for some of the initial tests for assembling this pinwheel.

54:35And when I was working on training it to construct the pinwheel, one thing that really surprised me was in all the data, we kind of carefully controlled the strategy for how to assemble the pinwheel, where you basically will take the kind of pre-cut piece of paper and take a little pin and insert the pin into a hole in the paper. And in all of the data, we picked up the pin with the right hand and picked up the paper with the left hand and inserted it. And the robot kind of started doing that. And then it actually made a mistake. And the paper ended up on the right side and the pin ended up on the left side.

55:09And what the robot did is it picked up the paper and it picked up the pin with its left gripper and it put the pin with its left gripper and inserted it into the paper with its right. And it had never seen data of inserting the pin with its left gripper. And it kind of showed that even that wasn't in the post-training data at all, like it was only like, and it wasn't even in pre-training either. The robot essentially had learned this sort of equivariance between its left hand and its right hand so that it could actually transfer behaviors from one hand to another, despite the fact that that was never in the data.

55:40And so that was like a really cool moment. I feel like, I don't know if other people were as excited about it as I was, when I shared it with some people. But it kind of shows this kind of emergent capability in these models that I hadn't seen before. And then, yeah, in terms of what I'd love to see, I don't know. I love seeing robots do anything. And I guess I think that there's still a long way to push in terms of reliability for robots being able to do tasks for really long periods of time. I don't necessarily think that much about individual tasks, but more so about capabilities and how to get the next capability from these models.

56:22Yeah, and so, yeah, anything. I guess the, yeah, a robot doing anything always gets me excited, even if it's something that hasn't been done before. I guess one thing that we've been doing recently is having robots use knives to slice vegetables. I think that there's a lot that you can do there once you actually can use knives safely, which is one thing that we've done recently. Okay, and then the last question is, how can someone break into robotics from a software engineering background? Great. So I think that the, I guess first there's a lot of software engineering in robotics. And so there's, I think you could try joining a robotics company as a software engineer.

57:05And then another thing that I would mention, I've actually seen someone take this path. uh so someone um who actually now works at physical intelligence her name is jenny she uh worked in trading algorithmic trading for a while um then she worked at harvey and was doing legal stuff and um she was really excited about robots and so she bought a cheap robot uh and basically in her bedroom um like played around with it uh and tried like fine-tuning an open source model and trying to get it to do something and then she shared what she had done and um she She sent me a cold email and was like, hey, I'm interested in working in your lab.

57:39And it seemed like her profile was promising and that she actually got out there and tried it and done it. And she was really excited about that. And so now she works at Physical Intelligence. But yeah, I think just getting your feet wet, trying stuff out and learning from that experience and then using that experience to kind of share with people, have it on your resume and so forth, I think is a great way to do stuff. And fortunately, there's a lot of open source stuff out there that can allow you to get started on those kinds of things. Great. That was the last question. Thanks, everyone, for listening.