Robot-Use Agents: Why General-Purpose Models May Win in Robotics

0:00Waddle Labs Founder:One of the big surprises the last few years has been the generalizability of coding agents across different domains. And now frontier researchers are showing that this includes controlling robots. This led MIT professor Philip Isola to suggest in a recent viral essay that we may be entering the era of robot use agents, where general purpose models could make different robots more capable. So today, Francois and I invited the founders of Waddle Labs and RoboCurve, two groups of startups that are working at the very frontier of making robots more capable with LLMs.

0:35Waddle Labs Founder:Maybe you guys want to briefly introduce yourselves and just say a little bit about what each of your companies focuses on. I'm Hanmei. I'm from Water Labs together with Vincent.

0:43Philip Isola:We work on building LLMs that control robots. And we do this by doing two things. Building a harness that allows the LLMs to do this very effectively, collecting data and using that data to train better LLMs. Hi, I'm Jay, co-founder of RoboCurve. We are an evals company for physical AI. We measure everything, any robot, any model, including LLMs and also classical approaches such as VLAs, Vision Language Action Models, and World Action Models. And we evaluate all kinds of embodiments like hands, grippers, arms, humanoids, quadruplets, all kinds of stuff.

1:20Waddle Labs Founder:So in the last few weeks, videos from both of you guys went quite viral on Twitter and I think inspired. I think both of them are actually quoted in that Philip Isola essay talking about robot use agents. The videos showed things like LOMs being able to unscrew caps, being able to communicate between multiple robots, being able to do tasks like uncapping a pen, for example. And so I thought maybe what would be fun is for Francois and I to dig in with you guys on some of the research that led to this moment to begin. And then maybe we can also show some demonstrations of what this can actually do.

1:49Waddle Labs Founder:To start, why don't we talk about some of the early research around transfer between language models and robot control. So I want to talk about some of the early work on pre-training language models for decision making, and then also the RT2 paper, which established the VLA. So maybe, Jay, do you want to tell us a little bit about some of this early work involving being able to use language models to do any kind of robot control?

2:12Philip Isola:One of the earliest successful approaches of using AI on robots is the RT2 paper, where they use a pre-trained language model on web text and images and use that to control robots. And the interesting thing about that is that it's a fine-tuned version of a language model. And instead of outputting English, for example, they just output what we call an end-effector pose, which is the coordinates that you can then translate into join commands that can control the robots. In some cases, in some sense, it's very similar to what we see now with large language models, just that instead of fine-tuning, the models are good enough to just do it out of the box.

2:59RoboCurve Founder:And this is actually quite similar to the mapping that I have in my head is kind of like the COT moment. And so when we did GSM8K and you said, Susie had$8, she spent$5, how much does she have now? It had to output four hash marks and then the answer and then EOS. there was no ability for it to chain of thought. And so then we allowed it to actually have like, okay, let me go eight minus five is three. And so like, hash, hash, hash, hash, three. And so you allowed it to do this kind of thinking before it actually gave an action or an answer. And similarly now, like VLA is where basically like, in RT2 is basically like, it has to output an action.

3:39RoboCurve Founder:There's no, like, I can't allocate more compute for a more complex task. And then now I have this code chain of thought thing that I can do. And I can say, okay, even if you were using the example where you're not actually outputting the code, you're just giving it to Astra and let it think, think, think, think, think and then output an action. Similarly now, and then with code, it's a more Kolmogorov complexity or lower Kolmogorov complexity code length to just use code and really compact code to just like, here's the code, just go.

4:07Philip Isola:I totally agree with you on that. I feel like at the end of the day, it's a lot about a bit of less, right? If you give the agents So you give the AI model more autonomy and you, if you unshackle it a bit more and give it more resources, it can actually do a lot of the things that we fine tune it to do. I'm just going to hop in here as well. I think the really interesting thing about the bitter lesson here is that, um, like VLA is by design architecturally. They're built on top of language models as well, right? They can potentially reason they can write code. so perhaps the bitter lesson here isn't necessarily what architecture you build necessarily but it's what data is most useful right like people have been sort of nagging at this uh vla data bottleneck for years now and we're seeing very very slow progress and so the lesson here is maybe we we have a modality of data that we know works very well and this idea of transferring across different modalities which ham and i are super excited about like how do you take something that's traditionally out of distribution like robotics and make it something that's in distribution like maybe the way of harnessing a bit of lesson is saying, let's pick a data for which we know this modality is pretty blessed.

5:13Philip Isola:There's a lot of data, these models understand it well, and then use this as a way of unlocking a lot of other domains as well. Yeah. And to add to that, the reason why RT2 is such a good model is because it is using a language model that taps into the modality of all the data that a language model is trained on. So all those web images and web text actually improves the VLA compared to just training a specific robotics foundation model without any pre-training.

5:45Waddle Labs Founder:So you're benefiting in the case of RT2, these early VLA approaches from pre-training. I guess, what's the limitation in the, I guess, action-taking, fine-tuning approach that you can now get around if you can directly write code? You're referring to a difference in data there. What exactly is that difference in data you're referring to?

6:05Philip Isola:There's a few things that models have got much better at since RD2. Like one, I mean, they're much better tool use, for example. So, you know, one kind of data is now these models can, well, I mean, they write code much better. So now they can write complex policies as code. They're also much better using tools to explore the kinds of environments they're in, like what arms they have access to. And a lot of this comes from in-context learning. like the key difference between like these VLAs and like GPT-6 or LLM isn't necessarily the architecture, but kind of the approach you take towards training.

6:37Philip Isola:We want to be bidder less than pill, right? We want to benefit from all kinds of data. We want to pour in computer use data into our robot models. We want to pour encoding data into our robot models. But when you do that, you just end up getting what we think of as like these general purpose LLM agents. And then so to me, it's like, okay, why not just build a really good foundational LLM and then use that to control robots rather than training like a model that's specifically for robotics and that's more dependent on just robot data.

7:04Waddle Labs Founder:On that note, maybe one of the things that I think if we could rewind the clock a couple of years may have been a good sign that we were trending in a good direction to this is research on code as policies, right? Do you guys want to talk a little bit about when the research community refers to code as policies, what exactly that means? And I think it's worth putting this in the context of when this paper came out, which was coding agents just starting to work. I think this was still in the era in which people were like putting comments and auto-filling, you know, Python code blocks, which, you know, now we think of as ancient history, but this was like two years ago.

7:35Waddle Labs Founder:So, you know, how does code as policies work and how did that inspire some of what you guys are now seeing as possible?

7:40RoboCurve Founder:I probably rewind back to Voyager, where like Voyager was like the one of the first, what is, what does a coding agent mean? Coding agent requires good tool use and on the fly tool creation that's called code okay you have python built-ins let's just say i have only python built-ins those are tools and i have sort and i have like if and i have four and i have while and i have all these things and i need to like use those tools to assemble a new tool called a new francois.py right and like now that's a new tool and i get to use that that was voyager and like yeah i don't i'm not gonna say voyager was the first one to do this voyager was the most popular one to do this for minecraft and then they created tools that they could invoke on the fly uh to help them play the game better and compressed thinking and experience into a new tool that i can later invoke and then i think because everyone in 2024 had this insight was like okay if we just pour if you know daria and sam both all realized if i pour all of my heart and soul and resources and compute and intelligence into getting better at coding, we will have liftoff.

8:45RoboCurve Founder:Then I can automate the ML engineer. Then I can have liftoff and I'll have all the things. I'll solve all the things. And so then that's what everyone did. And I don't think that they had the insight. I don't think they were so insightful to know that, oh, if I did this, then we will have LLMs that would be good enough to write policies for robotics. And I can displace all of robotics with actual coding agents. And I could displace artists with like, you know, coding agents that are writing JavaScript to make beautiful, like, art. Like, I'm not sure that they had that insight, to be honest.

9:18Waddle Labs Founder:Can we talk a little bit about what some of these early code as policy methods were even able to do? So even, you know, this is well ahead of us having coding agents that are widely available and people intuitively understanding how they work. It's well ahead of people using RL methods to actually train those coding agents to be really good at tool calling. What did some of those early ones, maybe even before Voyager, like code as policies, which I think the paper came out in 2022 at the end, right around when ChatGPT came out. What was the capability of those compared to what you can see now?

9:43Philip Isola:Yeah, those CODIS policy papers, especially ones for Google DeepMind team, were incredible because what they did is they created these kind of functions, like pick up an object, lift up, or like move to a certain pose. Like they provided this list of functions to a coding agent in the form of like literary Python functions. And then the coding agent will be able to write code involving these functions to then control the robot to do very complex hacks. And what was very surprising or what just excelled was that coding agents can do this very one shot. They did not need additional robot data to, in order to work with this code, because they're already trained on so much coding data.

10:22Philip Isola:They already have a sense of like how to, what to do first, what to do second in order to like move a block into a bowl, for example. And I think it's this one shot ability, this in context exploration ability that really motivated a lot of later work to continue exploring, including us to continue and exploring how we can apply LLMs to robotics.

10:41Waddle Labs Founder:I know, Francois, you have this framework we've talked about of the various ways that learning can happen. There's going to be learning that's embedded into the weights versus being in context learning and so on. Do you want to quickly talk about that? And then maybe we can think about where all the research over the last few years has fit into that framework, especially the direction it seems to be going.

10:57RoboCurve Founder:I mean, I've been working on this experiment, and actually I haven't really crystallized it into a paper yet. But basically it's like, what is the most efficient way from an intelligence sample perspective to input a learning, let's say in this context, a state action result or state action reward tuple back into the policy. There's ICL. There is... So that's in-context learning. Yeah, it's in-context learning where I just append it. This is how most people are using LLMs. They're just like, oh, no, don't do it like this. Do it like this. And I was like, okay. And then it's still in the context and then I can remember, I can iterate.

11:35RoboCurve Founder:And that really only works if you train the LLM or at least post train the LLM to be able to learn and improve. And that self-refines a reflection like literature from way back that like actually allowed multi-turn into the training set. If you don't have that, it doesn't learn, it doesn't learn. But even then there's a limit to how good ICL can go. So I've done this experiment where like you take an LLM that we've trained on and I've held out a task. let's just say gsmak for for uh simplicity and and then i icl it and i want to measure on the val set how much it improves on a per sample basis and it improves greatly very cheaply it doesn't cost much i don't have there's no sgd so i don't have any flops and so very quickly i can adapt and improve on my val set example example example one is non-monotonic improvement which is wild so it gets worse it gets better it gets worse it gets better like pretty grat aggressively number two is that it caps out very quickly and so after like 20 30 maybe 40 examples it is basically saturated and more examples back into the context does not improve you're like you're constrained by the

12:43Waddle Labs Founder:model's ability to intelligently use all of its context exactly from the post training how many

12:47RoboCurve Founder:multi-turns did it actually get to be able to improve and do self-reflection and then the most important one it for surely can't do is after it hits the context window context length of the model that it was trained on if it was trained on 100 000 really that means you have a context window of like 50 ,000. Once you exceed 50 ,000, you don't improve anymore. You actually just get worse because the model can't attend over everything. There's a really good example you said, which I didn't really think about is like, well, if I can rag or I can load in my act into active memory, and this is very prime agent, continual harness kind of like thinking where I can pull stuff in on the fly, similar examples, then it's like me, like, hmm, I have an exam or I have an exam with a textbook.

13:29RoboCurve Founder:I'm going to do better if I have the textbook to look up as reference. And so that is definitely going to work. And it also scales, I would imagine, much better. And then the last two paradigms would be LoRa with rank one or rank two or whatever, LoRa with rank 10 or rank 100, and then all the way to full SFT RL. And if you're Tesla and you have all the data, infinite data, and you're learning, doing self-driving car by ICL, what are you doing? Like, are you kidding? It's like, clearly we wouldn't do that, right? And if you're figure or something like that. But it's amazing how good in low data regimes you can do with ICL.

14:11RoboCurve Founder:And then even cooler, we can compress the ICL into tool use, into tools, or like compress it into learnings. And that's the hierarchy that we haven't really figured out. that you guys are probably on the forefront of.

14:22Waddle Labs Founder:Yeah, do you guys want to elaborate on that a little bit? Because I imagine that's a central part of how you guys think about building Waddle.

14:27Philip Isola:There's a few things that are quite interesting here, I think, to unpack. Yeah, the first thing is we've been sort of thinking about this idea of a harness almost as a form of domain specificity. Like when you, for example, deploy a robot in a new environment, maybe it's in a wet lab and it just needs to do a lot of tests you're picking up, for example. You can learn a lot of this in context, but then one way of consolidating this context for future agents is sort of packaging each of these skills that you've learned into specific programs for example like writing out these skills and write out these memories is a form of consolidating it's it's like a form of distillation from past experience for your future agents to use so this is quite interesting the other thing that's quite interesting is it almost seems like there's some relationship to the broader meta-learning literature throughout in machine learning history.

15:18Philip Isola:Like you have this sort of bigger model that then programs a smaller model to do things. And one of the very interesting things is that, yes, these smaller models are going to learn in context. They're sort of wrapped inside the harness and they're doing a task. You can make them maybe smaller. You can make them run faster as long as the bigger model can sort of do this domain specialization of your harness quite well. So this is something that we're pretty interested in right now. And last point as well, I think the really interesting theoretical question is how far in-context learning can get you.

15:47Philip Isola:And I think it's really anyone's guess as to that. There's papers that show that these models sort of can approximate gradient descent during in-context learning as well. So this line between weight space and symbolic space, I think, is a little bit blurry.

16:00RoboCurve Founder:YC's next batch is now taking applications. Got a startup in you? Apply at ycombinator.com slash apply. It's never too early, and filling out the app will level up your idea.

16:12Waddle Labs Founder:Okay, back to the video. So what exactly are we watching here?

16:16Philip Isola:We're watching Astra controlling the arms to pick a block off the table and put it into the bone.

16:22Waddle Labs Founder:So here Astra is using just these camera inputs and presumably it's aware of the robot that it's controlling. And it's going to write code that controls the robot at its individual joints.

16:31Philip Isola:Yes, so it does look at all the cameras. It has all the camera feeds. Instead of code, it's more like a tool call. So it sends commands to the robot to control and put the block into the ball.

16:46Waddle Labs Founder:So in this situation, we're using Astra directly. But if we were using, say, Waddle's harness, what's the kind of difference between directly using Astra to control this versus doing this through Waddle's API?

16:56Philip Isola:Sometimes having Astra directly command the post a robot should go to is not the optimal tool to use. If it's a repetitive task, you don't want Astra in the loop. maybe you want to write code that can run repeatedly very, very fast. Or if it's a task, you've done a similar task before, you should be able to call a skill that has compiled and use that skill to do the task faster and handle edge cases better. So here we can see it seems to be approaching in on this guy. Oh, very dramatic. Yeah, you can notice that the latency is a bit slow because we are bottlenecked by the latency of Azure. But if you look at the trends, the latency of these models are improving very rapidly.

17:34Philip Isola:Right. So one thing that we saw is that for Babel class LLMs, their latency is improving by around 2x per month, which is very, very fast. Very fast. Yeah. If the trends continue, we could get real-time control by end of the year.

17:50Waddle Labs Founder:So now that we saw this task go from end to end, why don't we walk through what that actually entailed? So there's a coding agent in the loop here running this. Yes. What are the steps that the coding agent would have taken? And let's kind of contrast the direct coding agent version versus the waddle harness in the loop version. Yes.