Intelligent Robots in 2026: Are We There Yet? with Nikita Rudin - #760

29:50It's like you have a digital twin of your humanoid robot in a high-fidelity simulation environment and you're training to a very fine level of detail, which sounds very computationally expensive. No, so we're much closer to what you described first. So we have a very generic simulation environment, but there are some very specific things that are important. One clear example is what are the torque and velocity limits of a motor? You cannot expect it to do something that is not possible on the real robot. So you need to add those limits. And there are a few more things like that, like what kind of delay can you expect between a command.

30:29So you need to identify a few of those parameters. And we are actually doing this usually in what we call a real to sim process. So we take the real robot, we literally hang it in the air, we let it shake a little bit, collect data of all the different motors, and then we know which are those important effects that we need to identify. We identify them and add them to the simulator. But simulation speed is the most important thing. So you cannot afford to simulate all those different effects, currents, magnetic fields, etc. You need to abstract all of it away. That sounded very hard and expensive.

31:07My personal take is that you still need to understand them even though you're not simulating them. Got it. And so is the, it sounds then that it sounds then like the result of that process is not a general model that you could deploy to any humanoid robot, but one that is specific to the humanoid robot for which you took the reel to send, you know, those key parameters. but that because you're able to abstract it out to these some handful of or several handfuls of key parameters, it's relatively easy to do new robots. Yeah, and this was also the surprising part of one of the key learnings of this year is switching robots is fairly easy as long as the hardware performs reasonably well.

32:04So now as a company, we work with a few different suppliers of robots and a few different partners as well with whom we're working closely. We've deployed controllers on, let's say, between five and ten different robots. And we see now that making a new robot walk is a few days of work. And it should be less. It should be less than one day of work if we optimize some of our processes. Now, bringing them to a new task, this is a bit more challenging. this requires more engineering today and this is what we're focusing on one of our key metrics is how much human effort is involved in bringing a new robot to a new task a new robot is very easy today, a new task is something we're working on and in this context like how we've kind of talked a little bit about this but how specific is a task meaning like is a task pick-in-place or is a task robot in this warehouse picking off of this line and placing into these bins?

33:10That's a great question. More like pick-in-place. But there is an interesting concept there, which is we are trying to leverage the information contained in large VLMs to orchestrate and break down complex tasks into clear subtasks. Even though that's not what we're focusing on today, cooking is a great example, a great metaphor. If you wanted to, cooking, yes, if you wanted to train the robot to cook every single meal on the planet and you would say each meal is its own task, you would never finish that, right? Like the set of tasks is huge. But what you could do is you give the recipe to a VLM.

33:58you also give it images of what robot sees if the recipe says cut a cucumber the VLM would say grab the knife grab the cucumber and do this sort of motion to cut it and then you can break it down into much simpler primitives like cutting things, holding a pan putting it down somewhere, filling a glass of water, pouring it things like that and suddenly the set of these primitives is not infinite anymore the challenge is that now you need a higher level intelligence that will orchestrate all these primitives. But what's interesting is that that part is basically solved with a VLM. It's not 100 % there, but it's moving much faster than the actual physical interaction of doing all these motions.

34:46So can you elaborate on that? The orchestration is solved, and if so, how is it solved? and what's the relationship between that orchestration and what we talk about is like the reasoning capabilities of these large models. Is it the same thing or related? It's similar. So maybe one way to describe this is on our website, we have two videos. We have a video of a robot walking in a forest and picking up trash. This is mostly there to showcase what's possible and also play a little bit on our Swiss angle, using our nature. We have another video where the robot is doing the same thing in our office.

35:29And in that second video, it's 100 % autonomous. So you give it a text prompt. I think we're saying something like, pick up the toys in front of you and drop them in the basket at the end. And we're using an off-the-shelf VLM for that. The way this works is we're giving the images to the VLM and we're allowing it to do tool use to call specific skills of the robot. So the VLM would say, oh, I see a toy there on the ground. Let's walk to the toy. And this let's walk to the toy is a skill that is actually triggered and executed by the robot. Once we're there, it will trigger, pick up the toy, then go to the basket and drop it off.

36:10And so by having a few of those primitives, which are walk to things, I mean, the walking is locomotion, as we discussed, is itself very complicated. So you can walk on stairs, can walk on a bunch of different complex terrain. But by having like walking, picking things up from the ground and then dropping them somewhere else, we can recombine it in many, many different ways without any retraining, just by prompting an off-the-shelf VLM. When I hear tool use, I hear like separate process or module or model. is that the is that the case and like how do you think about you know then I think of like if you know if we got a bunch of tools you've got a bunch of these you know separate models or modules like are they actually does this architecture imply that they are in fact separate and you know trained separately or are they more universal somehow?

37:13Another great question so that in our case specifically today, they are separate, but we are actively working on merging them together into one single model, a more general model. And the hope with that, again, is that you see some generalization. So you see some interpolation between those different models.

37:37And the way we would do that is actually by still training those different modalities primitive separately, and then using them as data generators to collect a massive amount of data in simulation to then train one of those larger VLAs across the whole data set such that it can perform everything. We're seeing early results of that in our company. Things are going in that direction, but we still need to prove that this actually leads to the generalization we were talking about before. And hearing you describe that, You know, often when I hear like these kind of student teacher types of approaches or, you know, I think of like distillation and trying to get to smaller models, you're not necessarily trying to do that.

38:23But talk a little bit about the hardware capabilities from a model inference perspective and like where we are in terms of, you know, model size, that kind of thing. In our plan, if we go through that whole process, we train, let's say, 50 of those primitives. We distill everything. we are still developing a hierarchical pipeline where you have three models interacting with each other. It would start with a relatively large, let's say, VLM that would be allowed to use reasoning. And the output of that one would be that VLM would go from a very abstract task to clear subtasks such that if you have a robot here and I tell it, go pick something up in the fridge and would say, turn around, go through the door, open the fridge, grab the thing, close the fridge, et cetera, et cetera, clear instructions.

39:16Then those clear instructions would go to that VLA that we were describing before, where if it receives as instruction, open the fridge, it will plan a kinematic motion for the arm to grab the fridge handle and open it just a few seconds into the future. And finally, we have what we call a whole body tracker, which will receive this plan of how the hand should move and how the whole body should move. And then we'll control the motors of that specific robot to execute that motion. Okay. Now, the size and frequencies of these models are very different, and that's why we think it makes sense to have three of them.

39:53The final one, the whole body tracker, is a very simple, very small model. Typically, it's a very small transformer. To be completely honest, it doesn't even need to be a transformer. But today, everything should be a transformer. and that can run very easily. We typically run them at 50 hertz, 50 times per second, even on the CPU of the onboard computer of the robot, not even the GPU, because it takes more time to send it to GPU and get it back.

40:23Then I'll skip the VLA for now, going to the VLM. That one is typically fairly hard to run onboard on the robot. For now, it's running off-board either in our office, in a server rack or even in the cloud, which creates some challenges. Once you want to deploy 100 robots in a warehouse, either you have an amazingly good internet connection or you have to install server racks in that warehouse. So we are hoping that compute keeps progressing such as we can finally feed those things on the robots themselves, typically on a Jetson. And then the VLA, this is where compute is the most limiting today because we cannot really put it off board it still needs to run fairly fast let's say 10 times per second with with minimal delay so it needs to be on board and they also typically use diffusion which means that you don't infer it just once at this part of the network is inferred is inferred multiple steps so this is where compute is the most critical the on-board compute of drawbot is the most critical yeah i hadn't thought about you know in the case of the um the one except we were talking about these other robots in the home that like they're essentially like their brains are in the cloud i assume that you know somehow we were able to get these models small enough to run locally which seemed like a lot but um just from a latency perspective uh that i don't know it's it's hard to imagine that being particularly tenable and consistent as well.

42:03So some of these models, just to clarify, some of these models can fit on the robot. But what we're seeing today is if you want to do some of this more abstract reasoning, so you give it a very abstract task and then it has to orchestrate something for multiple minutes, there you would really benefit from larger models. Yeah, and I would imagine that you would want even more abstract models in the home, you know, for consumer tasks than you would require in an industrial setting. Is that true? Yes, I would say that's true because in an industrial setting, if the task is repetitive, you can more or less pre-compute those very abstract instructions or go from very abstract instructions to clear instructions.

42:54In a home, if you have a human just telling you, something. For sure, you need the scale of large models. Are you using off-the-shelf RL environments or is part of what you're creating the simulation environment for creating these models? It is a big step of what we're creating. So we are not building simulators ourselves. We're using existing simulators, including ones from NVIDIA. We also test and experiment with many others. But one of our key know-how is how to properly build those simulation environments, and we have our own custom RL algorithms on top to benefit from that as much as possible.

43:35Got it. So the simulator and the simulation environment are distinct. Is the simulation environment, when you say that, is that the configuration of the environment in the simulator? Like the simulator is like the platform and the simulation environment is like the thing that you create about your scenario? Or is there... Are there just those two levels of abstraction or are there three levels of abstraction I guess is... I guess you would add the RL algorithm as a third component that interacts with both. But that's it. The simulator itself is basically a physics engine and a renderer. And then you have to put a robot in there.

44:20If you wanted to walk on stairs, you have to create stairs. But you cannot just ask the robot to randomly figure out how to walk on very complex stairs. So you have to create a whole what we call a curriculum of difficulties. So you would start with very small stairs and progressively make them harder. And the same is true for all sorts of tasks. When we're training a robot to open a door, we have to create a simulated version of the door. Then we have to help the robots. We have to figure out exactly all these training processes on top of just the scenario itself. And so as we've talked about this, you've kind of

44:59positioned RL and imitation as these two alternatives. But is it also possible to use imitation in conjunction with RL to kind of bootstrap learning and like, you know, help the robot get over these, you know, figure out stairs more quickly? Like, is that still a research problem or is that something that, you know, we're able to do in practice now? That's a good question. It's still a research problem, but we are seeing good signs of life of it, I would say. I can talk about two different ways to combine them. One way is use a few demonstrations to help the RL process. And this is something we're doing very actively.

45:46So if you have a human showing, like doing the task, you can extract just a little bit of information from that to help the exploration process of the reinforcement learning, such that the robot is not, you know, just randomly shaking and trying to figure out everything from scratch, but you're guiding it a little bit towards the right solution. And a completely different way to approach imitation learning plus RL is what we're seeing a bit more in other companies and in academia, which is doing more imitation learning for pre-training and then adding some flavor of RL on top to try to improve the behavior after the fact.

46:26A big question there is if the imitation learning pre-training was done without any simulator, do you suddenly need to add a simulation again to do the fine tuning or not? And I think this is still a very, very open question. You know, when I think back to, you know, some of the earliest conversations I had in robotics, like, you know, these are folks like Peter Rabiel. And I remember, I think, you know, this was like pre-VLM stuff and maybe even pre-Transformer. I don't remember. But like, you know, some of that earliest work was like they'd have hundreds of robots, you know real robots at google like and they were starting an experiment with rl i believe you know but like it was the robots were rling and it was very expensive it used a lot of you know you needed to have a hundred you know time on a hundred robots um you know but they didn't have to deal with this rl to sim gap um you know you're more focused on you know simulation and other ways to close that gap, but there are still proponents of RL in real life.

47:38How do you think about comparing and contrasting those approaches? There are definitely people who are big proponents of RL in real life, or another way to put it, who are against simulation at all. In some cases, there are good reasons for that, because some things are fundamentally hard to simulate. I can talk about a specific task we're focusing on, which involves the robot manipulating cardboard boxes. The robot has to walk, pick up the box, bring it somewhere, put it on a table, open it, take what is inside out of the box, and put it on a shelf, for example.

48:22In that case, most of the... If you break it down into subtasks, most of them are very well simulatable, except one very specific piece of it, which is opening the box. Because you can imagine there is maybe tape on the box. You have to take a knife, cut through the tape to be able to open it. And it's possible, but it's still a lot of effort to simulate the interaction of the tape with the cardboard and exactly how the knife cuts through it. So what we are trying to do there is identify those specific cases where simulation is still limited and use real data, but only for those very specific cases.

48:59and then mix it with simulated data of everything else. And we think this is how we basically get the best of both worlds. We get as much simulation as possible, and as simulators develop, they'll take more and more of the whole set of tasks. But while there is a gap, we'll reuse real data for those specific cases. You know, for the folks that say that, you know, RL in real is better, you know, in what ways would it be better for that particular scenario? It sounds like you would just go through a lot of boxes, but you still, like, unless you're giving your robot a utility knife, like, it seems like it's the same problem in real, right?

49:44I agree with you. I think it is much harder to do it in real life compared to simulation, especially with reinforcement learning. What you could do is do a little bit of imitation learning for that specific case. And that's much easier than letting the robot learn everything from scratch. I would guess the only argument for real life, for pure real life RL, would be that you don't need to deal with simulation, which can be very, very hard, especially if you don't have expertise in-house in terms of how to create those simulated environments and how to tune the simulators to behave nicely. When everything is in real life, well, you already have the perfect simulator in a way.

50:28But on the other hand, you have very expensive hardware and then any failure and then you reset is way more expensive than in simulation. Yeah, I think it's clear why it's compelling and also why it is aspirational. Like the idea that as humans, we don't simulate the world to learn things. We explore in the world and we learn that way. And so I'd want my robot to be able to do that. But we're nowhere near the sample efficiency in robots as we are in humans. So you would end up breaking a lot of robots and or boxes to get there. And then you've only solved one task. I think another interesting point is that the human reward signal is extremely complicated.

51:23if you think like if you're doing some tasks with your hands, the amount of information you're getting from all the nerves and all your skin, also from your muscles that are tired, etc. is extremely complicated. And we don't have that information at all with a robot. Typically, tactile sensing is very primitive. So if the robot is slowly damaging itself, you wouldn't know until a motor breaks. So if you're doing reinforcement learning in real life, getting rid of behaviors that would damage the motors simply won't work because you'll get maybe one event per week the reward is not there. And this is where simulation helps once again because in simulation we have perfect information about everything.

52:07We can design reward functions that will avoid breaking motors damaging the mechanics of the robot for example. Based on your earlier point about incorporating vision reducing performance or making it more difficult to convert on a model, you know, one approach to that is, oh, let's just add skin or let's add additional sensors. But, you know, any additional sensors or each additional sensor rather increases the burden, the computational burden on these models. Absolutely. Plus, mechanically, it makes everything more brittle. So with cameras, I would say we are there today. We can add cameras to our robots.

52:52They're very cheap. They're very reliable. Tactile sensing is just not there today. And thinking about crafting that reward function, talk a little bit about that process. That is key in any type of RL, is figuring out what that objective function is, a reward function.

53:16And how standardized are they for a given task? Or do they vary very widely and require a lot of hand tuning? And maybe as a secondary question on the language side or in coding agents, there's a lot of talk now about trying to incorporate value functions that allow that provides signal for positive behavior before the end objective. And I'm curious if value functions are a practical thing in robotics today as well, or a conversation. Is it research or is that something that we're using today? Absolutely. Value functions are part of the RL algorithm itself. So we are training, for example, we're mostly using some variant of PPO, which is an actor-critic algorithm, which means that you're training both an actor and a critic.

54:20The critic is basically a value function. It's not used at deployment. It's only used to help the training of the actor during the training process. Now we're seeing some research into how it could actually be used even at deployment. I think this is more on the research side. It's not proven yet. And then about reward tuning, it is a big topic. We have 35 people in a company and quite a few of these people still spend hours tuning rewards. And typically this is obviously referred to in a negative way. You don't want people tuning rewards. But I think you have to make a distinction between two types of tuning.

55:08There are general rewards that just simply come from the task itself. So if we think again about how a robot is, just locomotion, how robots are walking, you would say the task is just, you know, go from point A to point B. But in reality, it's a bit more complicated. You want the robot to go from point A to point B. You don't want it to use too much energy to do that. You don't want it to hit the ground too hard. You probably don't want it to slip everywhere. You don't want the arms doing completely crazy motions around it. So by the time you describe even in text what you actually want, you already have, I don't know, maybe 15 lines.

55:45And so that translates to 15 different reward functions that you have to come up with in Tune. And I think my personal take is that that part of Tune is fine. There is another kind of Tune that tends to happen a lot in RL, which is related to exploration. Once you've described the perfect task that you want, how do you guide the policy, the training process towards that? For example, if you're on a robot that opens a door, you might need to tell it, put your hand close to the handle, then close your fingers, then pull on the door. And these are really things that don't scale across tasks. And this is something we're trying to avoid as much as possible.

56:27And this is why we're working on other techniques where we can use one or two demonstrations from a human to help the learning process instead of all these manually tuned reward functions. You know, we're at the end of the year and this is kind of a natural time for folks to make predictions. Do you have any predictions for the upcoming year or, you know, several years, whatever horizon you'd like to offer in terms of robotics? Like, how do you think about the future? You know, I said before that I think there isn't a single humanoid robot providing value in the world today. My prediction is that will change around the end of next year, maybe beginning of 2027.

57:11So we won't, again, prediction, hard to say exactly what's going to happen, but I don't think we'll get this, you know, chat GPT moment where suddenly you have a billion robots everywhere just because you need to build the hardware. It doesn't scale like getting access to chat GPT, right? But I would predict that around the end of next year, we'll start seeing robots doing actual work. It will be just a few here and there. And then in 2027, 2028, we'll scale both the numbers of robots per task, but also the set of different tasks that these robots can do. Which means that in the coming years, we'll go from, I don't know, hundreds of robots to thousands and then very quickly to millions, tens of millions, et cetera.

57:55And presumably you see that happening first in industrial settings and then consumer? This is my prediction, yes. We'll see that first in industrial, then consumer, so at home. And I hope that after that, we can go to more, to crazier applications. Like we should really send humanoid robots to Mars to build colonies before humans land there. When you think about the currently available robots bots that you have, you know, seen and or worked with, like what are they all kind of the same? Like are all the, you know, the dogs the same, all the human noise are roughly the same, or do you see, you know, big differences between them from a hardware perspective?

58:46And if so, like, are there, you know, ones that are particularly exciting for you now? That's a great question. There, there probably three or four different strategies you can take in terms of how you're designing your human and robot, specifically what kind of actuators you use, what kind of gearboxes. And since there are many companies exploring that space, everything is happening in parallel. No matter which of those strategies you take, there are a few companies in the U.S., maybe one in Europe, and probably 50 in China building that exact thing. So the competition is really fierce. One part where hardware is not there yet today is on the end effectors, on the hands.

59:35It's still debatable if you actually need very dexterous hands. I think one of the big reasons why many companies develop hands with high dexterity, so like more than 20 degrees of freedom in a hand, is because they're using imitation learning. They're imitating humans, which means that you need to be able to imitate everything a human does. Once you go the RL route, you can learn other kinds of behaviors with much simpler grippers. But the debate is still open on that. Again, we've been talking about dogs and humanoids. But there's this broader question, which is, is humanoid the best form factor for a robot?

1:00:15Like, should we be making robots with two arms, two legs? Do you think that that's the way to go? or are we kind of anchored on this because it's our form but there are better forms that you've seen or think about? That's another great question. So I worked on more than 25 robots I think by now. So any number of legs and arms you can imagine from zero to probably four, five, six legs. And there is room for all sorts of robots in the world. Honestly, I mostly use, and as a company we use the word humanoid for the lack of a better word. What we mean by that is not the human form factor, but human capabilities.

1:01:01So very basically, we want robots that can go where humans go and can manipulate their environments in a similar way to how humans do that. So probably you would need at least two arms with some sort of ender factor to interact with the environment. and then whether you have legs or wheels both are fine both have their own applications so you can go a long way especially in industry with a wheeled platform and we're working with robots like that as well having said that it's surprising how quickly wheeled platforms get stuck a very common thing that is easy to imagine is if the floor is not perfectly flat If you have something, you have cables or, of course, stairs, your wheel platform is stuck.

1:01:48But another very important part is also the footprint. With those wheel platforms, you have two choices. Either you make them very large, and then they're stable by default because they have a very large footprint, but they don't fit through tight spaces anymore. And very quickly, especially in slightly older industrial settings, you have tight spaces. And the alternative is maybe some gyroscopic Segway-like thing? That could be one. Typically what we see is that you just have a small platform, which means that you have to be very, very careful how you move the torso on top, because if you lean too far, it just falls over.

1:02:27I haven't seen the gyroscopic platform yet. Okay. I still think a three-legged robot is probably the coolest robot I've seen so far, but it's maybe not the most applicable for industrial tasks on Earth. And then maybe one more question. There are, you know, quite a few now like robotics kits or, you know, robots are getting more accessible for folks that are interested in the space and want to play, but, you know, don't have access to a humanoid robot like or don't have any robots. It's like, you know, what are some cool things that someone can, you know, order now, you know, maybe get by the holidays or soon thereafter, meaning not like pre-order for 2027 and start playing around?

1:03:12Like what, you know, if you were, you know, advising someone who is like excited about getting their hands dirty, like, you know, what would you tell them to start doing? there's an amazing community around Hugging Face and their little robot project where they have very cheap robot arms and they can help you learn about the whole teleoperation data collection training and deployment pipeline with those arms so that's a really good way to learn about that for the locomotion and reinforcement learning aspect it's a little bit harder because you probably want a robot with legs which also means that the robot should be able to fall and stand up without completely breaking.

1:03:55I think the best bet there is the Chinese quadrupeds that are really getting fairly cheap. It's still multiple thousands of dollars, but let's say it's affordable for a university or a school or if you really want to go much deeper into that at home as well. and you get your quadruped and you unbox it, what can you do with it? Or where do you start with trying to do some experiments with it? So when you unbox it, typically it can already do quite a lot. So it will be able to walk. In some cases, they even have things like slam pipelines. So it can do some navigation, can avoid obstacles, things like that.

1:04:46But then the challenge is that you want to get rid of all that software and basically recreate it from scratch. And again, there are many communities online. There are many GitHub reposts that help you get started. The unit we go to is probably the most standard platform, so we started there. And there are people who open source already everything from training to deploying these policies on those robots. Okay. Cool. Awesome. Well, Nikita, thanks so much for jumping on and sharing a bit about what you're up to. Very cool stuff. Thank you so much. I really enjoyed this. Thank you.

1:06:05you