Training General Robots for Any Task: Physical Intelligence’s Karol Hausman and Tobi Springenberg

29:13And the reason for that is that it's actually really, really hard to model. Again, we can get back to the long tail of failures that you see when you do deployment. I can give you a lot of examples from the tasks that we've actually looked at for this release where there were failure modes that we saw that if you had just done a simulation of it, you might not have seen it. So to give you an example, we have this one task, which is you have to build a box. So this is an actual deployment task where the goal is we build these little cardboard boxes to put chocolate into such that they can then be packaged up and sent out, basically.

29:51So that's building a chocolate box, basically. And building this box initially worked great, And then there is new shipments of boxes coming in and they come in as like a flattened sheet of cardboard. And then these cardboards that came in in this new shipment were kind of not perfectly perforated. So they were sticking together. Right. And then the robot starts like grabbing them, puts them on a table to try to build this box. And it has two boxes suddenly on the table. Right. And this is something that wouldn't happen in sim if you had written like a nice simulator where you would just get individual cardboards and like fold them.

30:25And so now you have to deal with this problem. And if you just learn everything in SIEM and then try to deploy it, you wouldn't encounter it. So we encounter it. And then our kind of method can kind of figure out that, oh, actually, what I need to do is I need to separate this. And I need to move that second piece back and build the box. And we see a lot of successes for RL being applied in SIEM and transferred to the real world, especially in locomotion. and we we haven't really seen that kind of success in manipulation for for these kind of methods and i think maybe one reason for that is that with with locomotion with trying to move around it seems that the the biggest part of the problem is modeling your own body so if you can figure out how to model you yourself as a robot you're basically like almost there so uh you can do this modeling simulation exercise once because you only can do it you only have to do it for you yourself for this one robot and then you're basically done if you do it really really well it should transfer with manipulation however the problem is not how you move your own body it's how the world reacts to it you're actually changing the world around you it's not difficult to figure out how to move your hand from a to b it's difficult to figure out how this affects the objects you're interacting with and now the problem is no longer just modeling your own robot you have to model the entire world right like every single object that you might be interacting with every single task you can think of and that's where we see scaling problems and that's i think why we haven't seen those kind of methods be as effective in in manipulation what was the headline of the results from pi star 0.6 and you know where where did you see the model get after rl on the on the test that you cared about and what do you think that means about your overall training recipe going forward yeah so i think for me the most impressive thing honestly for me personally to see was just have these models run for hours at a time, recover from lots of different failures and basically just keep going.

32:28And at the same time, do that at a rate that is actually much better than the initial model that we started with. So the headline figures were we increased throughput of the policies by over 2x on these three tasks. So there's one task was this box building task I already talked about. One was making coffee with an actual kind of industrial scale espresso machine. And the other one was kind of like folding laundry. And so for each of them, we managed to like make the base policy that was trained just from demonstrations much, much faster and also make it be able to recover from theaters much, much better.

33:05And so seeing that actually in action when you just you sit there, right, we have if you go to our website, you can look at the videos. we have the robot serve coffee for 13 hours in a row or fold laundry for four hours, things like that. Actually seeing that live changes the way you think about these models. You know, changes the way at least I think about it actually being realistic that we can deploy it and that we can do it in a way where it's not just a toy demo which is shown once but is actually kind of doing the real thing fully. and that's been really a challenge in robotics that i don't think many people are aware of yeah like you know you see so many videos of robots doing cool things and you know we post these videos too they're like there's basically like anything you want robot to do there's probably already a video of a robot doing that um but you know you can take as many takes as you will as you want you can you can keep on recording until you get the perfect shot and the problem that i think everybody encounters is the reliability of these models how performant they are, how fast they can go about the task, for how long you can actually deploy them without failure.

34:17And I think this is the biggest bottleneck in terms of deploying these models in the real world. Because if they break every other trial, they're not really deployable. Right. And this is, I think, the most important breakthrough for us with this PyStar06 release, that we can actually start getting to a place where they are deployable, where we use these robots in our office to serve us coffee or we can give them to people at Pi to fold laundry in their home or we can deploy them and have them fold boxes for real and that is really, really exciting. Should we think about what you guys are doing with reinforcement learning as primarily a customer deployment reliability point then?

35:01Like you can now make sure that you can go reliably deploy the coffee making model on a customer site and it's going to be fast enough, It's not going to fail over long time horizons. So it's more of a customer deployment innovations versus like a fundamental kind of capability innovation. Or is it both? I think it's both. I think, I mean, Carol, you said this a little bit earlier. I think to some extent, the robots that we really, really want, right? The robot that you want at home, which can do your laundry, do your dishes, cook for you, drive around. and also the robot that people want in these smaller businesses, maybe solving a real problem that they have that they don't want to automate in a classical way because it's too expensive, like building a chocolate box.

35:44Those are things where the robot has to be reliable, it has to be good, and it has to have the capability to do a new task that it hasn't seen in initial training stages. I think it's unrealistic for us to assume that, you know, we can just go with like more and more human data collection, go bigger, bigger, bigger. We will do that. But there is always going to be a limit to how good and how much data you can get and how good the initial policy is going to be. So I think it is that what you said in terms of we want deployment, we need this. But also, I think increasingly over the next years, I expect we will see that we will do this deployment and that data will actually become really valuable as a source for pre-training, for making our models better themselves.

36:29And we rely more and more on autonomous data collection, is my prediction at least, over the next coming years to kind of build that host of data, that convex hull of all the tasks that we want robots eventually to do such that the models ingest this and becomes good at doing them and interpolating between them. And I think of it as a new capability. We haven't so far figured out how to learn from your own experience. There's been many attempts, but I don't think we've seen it done at scale. to the extent that actually shows a convincing result that allows you to deploy something. And this is why this result was really, really important to us.

37:10We wanted to get to the point where they can learn from their own experience. Because similarly to how we learn, you can learn a little bit from watching videos and maybe learning from others, but at some point you need to learn on the job. You need to try the thing yourself. You need to see how your actions impact what you actually want to achieve and make your own conclusions and try to learn that way. Yeah. And I think this is the first step towards that. You're reminding me of the, do you guys read the Rich Sutton Age of Experience paper this year? I love, I thought it was very profound. Do you think that this unlocks kind of continual learning in robotics real?

37:48Will this be part of that? It kind of depends what people mean by continual learning i think it's um it's definitely more continual than what we've done in the past where you know you have like a big pre-training mixture and maybe like a post-training mixture and you like you know you you sit down you work really really hard and then you come up with an artifact and like that's it yeah right like the artifact is done and there's not much you can do to change it now this is a much more of a living thing right like we we start with a process similar to this but then you deploy it and then it keeps on learning right so it's much more continual in that sense that it tries new things it tries to learn from its own experience and it keeps on getting better yeah now i think there is still room to for it to be much more continual where it can acquire new skills that way or it can be even much faster in doing this yeah um it can probably reason throughout this process so i i think there's a spectrum of like how much you can learn on the job.

38:51And this is really promising because it shows that you can do it, but I think we can make it much, much better. Yeah, I would agree. I would say we're at the very beginning of this, right? And it's definitely not continued learning in the classical sense that people would have thought about it of like data streams and then the whole thing churns and it just ultimately leads all the way to, I don't know, AGI or something like this yet. But it's a first step, I would say. We're moving in the right direction there, and there's lots more to be done. And I think I will say, even from this release, I was personally impressed and to some extent shocked how good these models actually are at picking up little things that you put back into the data.

39:34I was surprised that even with just human corrections, there was one example for tamping when we do. So tamping is a specific part of making an espresso, right? You put the beans and you have to tamp down the best part. Yeah, the best part. You have to tamp down the coffee before you put it in. I don't get it right myself. There you go. See, I'm not a coffee expert. It's a skill issue. I'm going to get it just right. That's right. And so our robot in the beginning tamped way too hard because it just happened to be the case that the initial human demonstrations were just making sure that, you know, let's make sure the coffee grounds are flat so we can put it in.

40:11and then the robot was like tamping really hard and like almost lifting itself off the table. When we looked at it, we're like, that's a bit much. And so with just, I don't know, it was I think 34 to 50 episodes. There's a really small range of corrections that humans did and we feed that data back and the model actually starts like being much more gentle and doing the correct thing. And I was really surprised by that because you think, you know, this model has been pre-trained on these millions and millions of episodes and now you're just doing a little correction and that actually works. So seeing that happen was a thing that I think is pointing towards this continual learning part, which I find impressive.

40:46Can I ask though, and the thing I'm still hung up on is generalization. So as I learn how to tamp better, does that make me better at folding boxes or not? In this specific case, no. But the mechanism is the same that you can also employ to fix the, oh, I have two boxes in front of me that are sort of stuck together and I need to pull them apart, right? Because you can get 30 corrections for the stamping part. You get 30 corrections for the pulling boxes apart. Yeah. But you get 30 corrections for, oh, you know, this box wasn't like neatly folded together. And all of this accumulates together to then give you this more generalized improvement, I would say.

41:28Okay. So it's a repeatable recipe, but they don't necessarily cross-pollinate. Yeah. I mean, I would expect that as we scale this up, we might see also things actually kind of transfer from A to B if there is motions that are kind of similar across tasks. But at this point, yeah, I would say it's more like a repeatable recipe. And we see a lot of generalization from pre-training where you train on more and more tasks, more and more data. You see that it's much easier to onboard a new task or you see tasks that appear zero shot that you didn't expect before. And this keeps on improving. doing we uh we kick off a pre-training run at certain cadence and every single time we start seeing that the model keeps on getting better because there's more data being fed in there's more improvements that we're making to the pre-training process and so on and I also suspect that as we have more and more of these models deployed doing all kinds of different tasks they also bring data back in and I think one way where we where I'm quite certain where will see more generalization from that process.

42:29That as you deploy these models, the data comes back, the models get better, you can deploy them more, then the models get better, you can deploy them more and so on. And I think maybe it's worthwhile for this point that you brought up, we haven't really talked about one crucial detail aspect of this 5 star 06 recipe, which is that the model has kind of two parts. One is the policy that is trying to improve via corrections and RL feedback. And the other part is how do you actually get this RL feedback, right? So we've talked a little bit. I've mentioned like, you know, humans might correct and that's the human correction part.

43:04And the RL feedback part is a little different and it's kind of, and already has some of these aspects of generalization that I think you're like trying to search for, which is that the way we do this is we first basically get humans to tell us basically whether a specific attempt of making the coffee or doing the box was, successful or not. So there will be human labels provided with these episodes. And then we train something which is called a value function to try and predict basically from my given point of where I am in the task, will I likely be succeeding or failing basically. And this value function is then used as kind of a baseline to decide whether for this data point should I bump that up or should I bump that down depending on whether I expect that I will be moving towards success or I'm more likely to move towards failure.

44:00And one thing that we saw when we trained these value functions, so those are trained basically from the same kind of backbone, the same kind of model, but they're pre-trained before the actual policy is trained that actually runs the task. When we train these value functions, we see that adding more data from different tasks actually helps there. And the model starts being actually really quite good at at least for for certain tasks and knowing when it will fail beforehand and before it is obvious for me for example when i look at a video of it trying to insert uh the um uh the portafilter thank you see i'm i'm not good at making and it's trying to insert a portafilter into the the coffee machine it kind of knows that it doesn't quite have the right angle before that happens so like 30, 40 steps before that actually happens, the value function kind of, if you look at the prediction, drops and saying, oh, this is not good in this specific episode.

44:57So I shouldn't include this data. Interesting. And this gets better with more data and more tasks. So this is an interesting counterpoint to the Karpathy slurping bits from a straw thing, right? Because you're not waiting for that final bits at the end. You're actually getting a lot of signal along the way. I think a rel is just like such a vast field and there's so many different approaches to it. And people often associate RL with something like a policy gradient method or, you know, very specific on policy learning approaches. And to me, RL is more of a problem definition. And there is many, many approaches that get around the problem that you're referring to, which is that, you know, you only get the reward at the very, very end.

45:40And it's not really scalable for very long horizon tasks. There are things like value functions. There are things like temporal difference learning that try to get around this problem, where you constantly make predictions and you do it in a sequential way. And this is maybe another one of these things where I think robotics can really help the broader AI community. Because we don't have the advantage of having a perfect language simulator where you can run as many simulations as you would like. Instead, you need to do it in the real world. So you need to make more efficient methods. And therefore, you need to learn value functions.

46:15and things like this. And I think this will, these will be really valuable everywhere. Yeah. Can I push a little bit on, I'd love to understand, you know, internet video seems like it's part of the recipe, but not a huge focus right now. As I see it, like, do you think that there's gold left to be mined in internet video? And then if you look at what's happening in video models right now, world models, to what extent do you think that's going to be a, you know, discontinuous jump in model capabilities and an important part of your model pipeline. Yeah. I think maybe there are two questions there.

46:51One is about the data. How do you bootstrap yourself to the point where you can start deploying? And the other question is, what about video models and kind of the world model aspects of it? So on the data point, I think we are now in this bootstrap phase where basically anything goes. like whatever you you can figure out how to like add to the model and to to to its benefit i think it's good whether you can add sim whether you can add human videos some kind of handheld devices human teleoperations i think it kind of doesn't matter you just need to figure out some way to bootstrap yourself to the point where you can deploy these models because i think in the long term there's going to be this bootstrap phase but then there's going to going to be the deployment phase and i think deployment phase will be will provide much much more data than anything you could do in the bootstrap phase so we're in this kind of like weird spot right now where we tried many different things straight to see what what sticks to just get us to the deployment threshold i see and once you can deploy i think that will vastly be be much greater than than anything you can do uh before that um so so that's also what we are sprinting towards that's why we want to start deploying these models that's why we want to do this you know with many different tasks in in many different environments so that we can just have this very powerful data engine now on the on the world modeling side of things i think the world models and aurel approaches are kind of targeting the same problem the problem of counterfactuals of how do you or a credit assignment problem right like how do you figure out which actions were the ones that actually matter for your success and how would the world have evolved had you had had you taken a different action and one way you can do this is by predicting what would have happened right like rolling out a full video of you know if i if i put this portafilter a little bit differently you know where would i end up and would this be a failure or a success or you can do this through reinforcement learning and it does it through a slightly different mechanism a little bit more implicitly but it fundamentally targets a very similar problem uh we are exploring all of those approaches and try to see you know how to how to really solve the counterfactual problem um i don't think there was a an answer yet uh but we see we see a lot of progress with with reinforcement learning that we that we've just shown with with pi star with pi star of six but i think there is probably room for for many many other approaches too awesome can we talk about once you guys get past that bootstrap phase let's talk about customer deployments a little bit what do you bring to a customer what do you sell them and then um how do you imagine that's going to evolve over time like are you selling them a fully vertically integrated robotic solution are you selling them a model that they have to figure out how to integrate into their operations like how does this all work uh the the the real answer is we don't know yet yeah um we are we are still figuring that out yeah we are still quite early in in the technology as you can as you can tell we are just starting to to even get to the threshold where we can start deploying these things so we believe we should focus on the on the technology first to figure out how to get it to the point where it's actually easy to deploy and expand this aperture that we're talking about initially and robotics the history of robotic startups is as very often gets to this point where you develop a technology for for some period of time you started with this grand vision of what it should be able to enable how general purpose it will be and as soon as you pick an application that you want to apply it to you're kind of stuck you start cutting corners you start uh figuring out very special purpose solutions just for this application and very quickly you become you know an application company that just focuses on let's say warehouse pick and place robots and that's it and we really want to avoid that future we think we have a chance to really solve physical intelligence and the the benefits of doing this will far outweigh any single applications that we can focus on now so we want to make sure that the technology is as general as possible as easily deployable as possible.

51:07This aperture is as wide as possible. And then we'll start figuring out how to commercialize it. And as you said, there could be many different ways of doing this. There's probably ways that we can't think of just yet because they will depend on how the technology goes, whether you can be a model provider, a fully vertical solution, or you sell robots or whatever else. But I think it's a little too premature to answer this question. It will give you a lot of comfort, you know, just to like pick one of those. It will give Alfred a lot of comfort. Alfred will be happy with us, but I think it's just too early.

51:41No, you guys have a grand, grand vision. So thank you for working on physical intelligence. It's a wonderful, wonderful improvement. Just for pi star 06 is just a huge sort of breakthrough. And so congratulations on all the success you've had. Thank you. Can I follow up with a spicy question? Sure. So as you said, this vision is so grand, so broad. You're doing all these different things. I'm sure you've studied all previous robotics efforts, and they've largely, as you said, applied to an application, and they get narrower and narrower. And one of the most successful cases of a large application is self-driving.

52:24And Waymo or Tesla have done enormously well. But if I had to go back in history, I learned about self-driving when Sebastian Theron was on the stage of TED in, I think, 2009, 2010. And he talked about the thing where they won the DARPA Challenge. That was 2007. And we're in 2025, and the thing barely goes from San Francisco down here. They kind of can do it now, but they take local roads. They can't even get on the freeway. If you do such a generalized job, how long is the runway or the timeline that you're thinking about to build for generalization and performance? Yeah. So there are some aspects of the problem that make it easier than self-driving and some that make it harder.

53:15Yeah. One thing that makes it easier is that we don't need to deploy it only when it's 100 % reliable, right? There's many, many tasks out there that even if you're at 95 % reliability, you're totally fine. If you have a robot in your home folding your laundry and every 100 bite them, you know, it doesn't fold it perfectly, you'll be totally fine. You just call your child to go fold the 100 bite. That's right. It's an additional benefit. We still need chores. We still have that, yeah. Exactly. And with self-driving, that's not the case, right? Like if you fail every 100th time catastrophically, that's a big problem.

53:51Yeah. So I think in terms of deploying this technology, it might be easier. Now, we also benefit from the fact that this is a different era of technology. We are at the era of vision language models or foundation models that have some common sense. And we learn a lot of lessons between, what was it, 2009 and 2025. And we can benefit from all of those. So I think that also really, really helps. and these are much more general purpose solutions than what we had in the past. At the same time, there are some things that will be very challenging, right? Like there isn't just a single application. This is a very general purpose solution that can be applied to driving, but also to manipulation and locomotion and flying and all kinds of other things.

54:37And I think it's to be seen how much harder this is. So far, based on what we've experienced, it doesn't seem to be that much harder, to be honest. It seems that if you tackle this with a very general purpose kind of mindset from the get-go, it turns out that it can generalize fairly well. And there is something about physical intelligence that we don't fully understand that allows these models to generalize between driving and making coffee and flying a drone and operating a surgical robot. even though they seem so far apart from each other and it seems that these should be all different models and different applications, these models somehow can make sense out of all of that data.

55:18And that gives me a lot of hope that maybe the problem is not that much harder and it might be actually easier. So I think it's a fair question, but I also don't want to draw the wrong conclusions from what we've seen from self-driving. That's beautiful. Congratulations. What result has impressed you the most outside of results that you created? That's a great question. yeah that's a good question actually i can start i've been i've been really impressed by the video models what you mentioned earlier i saw them a few years ago i worked on on aspects of them a few years ago and i didn't expect this trajectory to be that the improvement to be so steep like they're basically indistinguishable right now from reality and they can do incredible things um so that's been really, really impressive and really surprising to me.

56:06Yeah. I would say I'm still in awe to some extent that we've gotten to this place where we do seem to get models that do seem generally intelligent to a level that I really didn't foresee coming out of just next token prediction. I'm still amazed with this. And like every little advance that I see, winning IMO math challenges or applying it to finding new stuff in science. To me, yeah, there are so many things this year where I thought like, wow, there's still a lot of progress to be made, even though it felt like at the beginning of the year, maybe this whole pre-training business of LLMs is kind of maybe petering out a bit.

56:48Yeah, realizing that there's like this whole almost second breath of fresh air basically coming in. Yeah, I would maybe add to this, just like the fact that this whole thing works. it's kind of mind-blowing yeah i don't think we like fully realize how ridiculous this is right like you you build this like loosely brain inspired thing that has very general purpose learning algorithm you feed it data and it somehow gets it and gets it way better than anything we've ever had before and this applies to robots and it applies to vision and language and sound and all kinds of other things and like i think if you stop for a second and just think about it how it works and and that it works it's just like absolutely mind-blowing like the fact that we can have robots you can put it in a home and it kind of knows what to do in a home that it's never been to before or it can make coffee for 13 hours straight or you know things like that and this is from this very general purpose thing that that trains fully end-to-end that we don't fully understand but it seems to start to get it that to me is just mind-blowing we're in a simulation it's what that's what sonya believes that we're living in a simulation but it is interesting right like in science they teach you to take a big problem and break it up into smaller and smaller problems and then basically somebody realizes that that's maybe not the best way to train machines or robots of any kind and to be honest the whole machine learning like ai field made that same mistake actually to some extent right we were working for a long time people were working on solving individual problems very deeply basically right and then over time there is this like notion of oh if we can put it all together like do multitask learning if we could do that really really well we'd do much better and then but then the fact that that all happened just because we switched to this you know general pre-training objective and then it just all falls out that's the part that is the surprising bit, right?

58:48Do you think it's like an accordion where we go from one framework to the other framework? We take big problems, break them up into small and small ones that work for a period of time, then it stopped working. And then we're like, all right, let's go back to the big problem and try to solve it more generally and go back and forth. I don't see us going back. Yeah, I don't see us going back. I think there is a lot of approaches or a lot of people saying that, you know, you need the best of both worlds and you need some kind of way of incorporating the rules that you already know about like you know newtonian physics you don't need to learn that we already know how it works so can you just like put it somehow into the weights but uh from what we've seen so far it doesn't work if you try to do this you kind of limit the ability to to learn new things and i don't think there's the best of both worlds i think we just go all the way learning and it's kind of interesting to you know how similarly to how we learn you would think that if there was a way to pre-bake all of the intelligence, the evolution would have figured this out.

59:44You would have just been born, you know, knowing everything there is to know. And we see this with some other species, right? Like I think deer, when they get born, they're basically like as smart as they will ever be. Like they don't really learn much throughout their lifetime. But for intelligent species like humans, but also I think crows, for instance, they have these childhood periods, adolescence period, where they're not very smart to begin with, but they have to learn from their own experience. And it doesn't come pre-baked. You kind of have to earn it on your own. And I think there is something to that.

1:00:21You need to just experience the world and learn from that. And I think that's the lesson we're learning in machine learning as well, in AI, that we think we know how we think, but we actually don't. and we just need to let the algorithm learn it from data. Same thing with raising a child. I think I know how my son is thinking, but I don't. Yeah, I have a small daughter and yeah, it's just so surprising. They learn so fast. They learn so fast and you don't know where they get it from. Hopefully from the parents. Hopefully. She definitely knows some things that I didn't teach her. Thank you guys so much.

1:01:00It's a really beautiful mission you're building after. Thank you for coming to share. Thank you. Thanks for having us. Thank you. Thanks for having us.

1:01:32Thank you.