Redefining Robotics with Carolina Parada

0:03The next two years are going to be pretty defining for the field of robotics. there's just a lot of things that are coming together understanding dexterity whole body control you can see how this could actually merge into a very strong solution

0:21welcome back to google deep mind the podcast i'm professor hannah fry sometimes maybe even often the terms ai and robot are used interchangeably in casual conversation you know people talking about chatting to a robot on an app. But robots have a physical body. And here at Google Deep Mind, they care about robots with AI embedded in the real world. And while AI has made huge strides, embodied intelligence has lagged behind. But perhaps all of that is about to change. Carolina Parada leads robotics research here at Google Deep Mind, the international team responsible for some extraordinary advances in robotics.

1:04Most recently, Gemini Robotics, which brings Gemini's multimodal understanding to the physical world. Welcome to the podcast, Carolina. Now, I know you've been working with these robots for quite a long time. How have you seen them evolve? Thanks for having me. Yeah, it's been super exciting. I have been excited about robotics since I was 10 years old. Super excited because of what I've seen in cartoons. Like you see robots like Rosie the Robot helping do all the chores. And as a kid, you're like, of course, that's what I want to build when I grow up. And really, I've been at the Google Demi Robotics team for about seven years.

1:39And really, things have changed dramatically. In the last three years in particular, we've always believed from the very beginning that AI was going to be completely transformative to robotics. I mean, there's a lot of robots out there that are really helpful today. There's robots in manufacturing lines. There is robots that are navigating the moon. There is robots that are in our oceans. But these robots have been programmed to do specifically those tasks. They make a lot of assumptions about those environments or the objects they might encounter, or they might be remotely operated by humans.

2:10But we have believed from the beginning that AI is the way to transform robotics so that we can build robots that are truly intelligent so that they can interact with you, that they can reason about their environment, and they can take action in a way that feels very general. So that has been our mission from the start. And so I think three years ago, you had robotics in your podcast. And back then, we were doing reinforcement learning for robotics. And so essentially, we were teaching robots to stack blocks by giving them a simple reward, like plus one, if you tower got taller. We made some progress there.

2:44But a lot since we've been at the forefront of AI, we've been bringing more and more of AI into the entire world of robotics. So about 2022, we introduced, for example, LLMs to robots. And that was the first time that you could actually talk to a robot and say something like, I'm thirsty. And it would know what you meant. And then later on, we brought VLMs so the robot could understand natural language, but it could also understand the visual input that it was getting and then make decisions based on that. And then in 2023, we introduced robotics transformers. And this is the first time that the transformer architecture was actually included in robotics.

3:21And it basically showed us that robot performance scales with data. And that essentially started a new foundation or a new era of large-scale data-driven robot learning. And then more recently, we introduced just now Gemini Robotics, which is essentially our most advanced model for actions. and it essentially takes the multimodal world understanding of Gemini and brings it to the physical world by adding actions as a new modality in Gemini. And that really enables models to be very general because it's understanding the world through Gemini's understanding and enables it to be interactive. In fact, it can understand any language that Gemini supports and enable it to be dexterous.

4:03So it can still do very complex manipulation while talking to you and also understanding a completely new situation, which today is actually very hard for robots to do. In terms of your big goal, your big ambition, how will we know when we get there? I think it's definitely going to be gradual, where robots are able to understand a new situation and reason about something they need to do that they haven't seen before. And that's exactly what we're seeing right now. But it's still going to be difficult for them to learn more and more complex tasks. In fact, that's what we see. The robot can feel sort of like a two-year-old toddler that can understand its world around it.

4:46It can start to play with objects. It understands concepts. But if you teach it to do something more complex, like we have an example where we're teaching the robot to do an origami fold, it actually needs time to practice that. And once it has more practice in that case, it can actually do it. So that's roughly where we are today, but that's far from where we need to be if we want robots to be in everyday spaces doing all kinds of tasks for us. I thought that what we could do is take a little look at some of what these robots can do, because there's a video that you guys have recently released.

5:19What we have here then is we have a humanoid robot who is packing a lunch for its human, also playing noughts and crosses. Is it any good, the noughts and crosses? I think we still beat it because it's very simple understanding. Tell you what it's doing, though. It's picking up the pieces and moving them around quite easily. There's also a bit here where it can make its own anagram based on tiles that appear. What were you particularly impressed by? I think that's what's most exciting about these models is that in many occasions, our own researchers were excited and impressed by what it was doing.

5:54And it was primarily because the way we were testing it was by putting the robot in front of situations that it's never seen before. So even us didn't know whether the robot was going to be able to get it right. And in many occasions it did. So many of the examples that we show in this video, as well as the other videos where you have two arms moving around, is that it's actually understanding a complex concept. So a really cool example where we were all like gasp was when we showed the video where the robot is actually doing a slam dunk. And what was cool about that case is that that day we were just having the creative team come and film the robots.

6:31And we asked them to bring toys. We didn't say anything else. They're like, just bring toys to play with the robot. All things the robot hadn't seen before. Yeah, they had no idea what the robot was trained on, right? So they actually brought this little basketball hoop that was a little cute toy with a little ball. And they put it in front of the robot. Again, the robot had never seen anything related to basketball. It certainly has never seen this toy. And they asked it to do a slam dunk of the ball. And we were all like, I have no idea if it will work. And actually, it took not even a quarter of a second.

7:02And it actually decided to put the ball inside the basketball hoop. And we were all like, that's amazing. And it was just essentially drawing from Gemini's understanding of what basketball is and what a slam dunk is, right? which is a concept we couldn't have thought of teaching it to do. And it essentially did the right motion. So that was a really cool example. Talk to me about the packing lunch one. It kind of has a conceptual understanding of what a banana is, for example. Does it know how to grip a banana in the sense that you can't grip a banana in quite the same way as you could a clay pot or something even more fragile than a banana?

7:36Actually, one of the things that is super impressive is that these robots are extremely simple. They actually don't have touch sensing. They don't have depth sensing. They don't have force sensing. So they're literally doing eye-hand coordination and using an understanding of how you grasp a banana. So it actually is looking at the object and grasping it. And once he sees that he has it in hand, that's how he knows that he has detected it. There's other robots out there that are much more complex, but this forces the model to really reason about what he's seeing and making a decision about how to pick that up.

8:08And that's the thing that's really original here. Yeah, that is one of the many things, is the fact that he's doing it not just because we taught it a thousand times how to pick up a banana, it's because he's pulling this out of his understanding of how to pick up objects from Gemini and then adapting it to the world of actions. Because I can imagine, I mean, there've been lots of videos doing the rounds on the internet for a number of years of extremely impressive looking robots doing backflips and, I don't know, being kicked over and sort of running up and down mountains and things. In comparison to those videos, picking up and putting down a banana, you know, into a lunchbox.

8:44Seems like quite a simple task, but we're talking about a different type of robot here, aren't we? Yeah, I mean, this is a completely different problem you're trying to solve. Many of those videos are basically rehearsed sequences that the robot has learned and memorized, and we're actually very impressed by them. But it's a different problem that you're trying to solve. What you're trying to solve here is for the robot to reason about what it means to pack a lunch, given the objects in front, what it needs to do in order to put a piece of bread inside of a bag and then what it means to close it.

9:15And it's never going to go as you expect because these are very flexible things that move around. So it needs to react and respond to what's happening and then actually complete the task. It's that idea of generality. That's right. Yeah. So how do you compare one robot against another? How do you decide whether this robot is doing generality better than another? That was actually one of the things that was hard for us to express. When we were even recording for the demos in this release, a demo is by definition like scripted. So we were like, this doesn't quite capture what we want to share. That's why we asked the team to bring a bunch of toys and actually start playing with the robots and see what emerges.

9:56And the best way to capture it is that we're able to change the behavior of the robot by talking to it. And you can see that in the videos, we're actually able to put raw objects that it's never seen before, and we move objects around to make sure that people understand that this is actually not a prescripted behavior. In fact, in our benchmarks, we evaluate our models in all kinds of ways in terms of generalization. So we will change the visual background. We will change the background. The objects were new. We will add objects to distract the robot. We would also ask it to do completely new things, or even you can talk to it in a different language.

10:33So I could just give it the instruction in Spanish and it would just actually work. I want to talk about interactivity too, because in a few of your videos, there's one where a human is sat at a desk and the robot is kind of clearing up after him as he goes. In another, you've got a human moving a cup around and the robot sort of chasing it, trying to put an object inside. How much more difficult are those interactive scenarios than just a static task? Yeah, I mean, the significantly more advanced behavior. And a lot of the interactivity sort of just fell out of the model. Like we were not thinking, for example, how fast can we move these objects before the robot would react.

11:11We certainly knew that we wanted a model that could react quickly. But a lot of these examples that we posted on videos just fell out of people playing with a model and seeing how it would behave. Same with organizing the desk, that was actually someone playing with the robot, deciding to see how much it could game it until it actually was able to complete the full task. So yeah, it actually is amazing to see how a lot of these other capabilities that are already there in Gemini are actually extremely valuable when you bring them into a robot, which is now able to adapt based on what you're saying.

11:44So you could actually have a full conversation and change the behavior of the robot as it's moving. So you can say, I want you to do this. Oh, no, actually, never mind. I want you to do this other thing. And it would actually just follow you. It's actually kind of comical. And then you could also change the objects around and it will just do it. I think it's kind of a good job sometimes that these robots don't have feelings because they feel very sort of forlorn, like just being chased around on a table by researchers. Yeah, it's super fun. That's the large language model sitting underneath it that's helping it do that, right?

12:18That's giving it that conceptual understanding of the objects that it's manipulating. That's right. So we're leveraging Gemini's multimodal understanding to take the visual input that the robot is seeing through its cameras and the natural language that is hearing from the human and then translate that into how to act. And it actually also speaks back. So you can ask it a question about whether it's done. You can ask it a question about how far it is in the process of folding an origami figure. It actually understands that I can respond. I remember when Gemini was first being launched and there was sort of people went to great lengths to talk about how it was multimodal.

12:54This is sort of the payoff for putting in all of that extra groundwork and making sure that it can understand videos and photos and so on. I mean, one of many. I think us humans capture the world through many different senses. Right. So I think it's super important if you want to build an intelligence as powerful as our brains to be able to undertake input in a multimodal way. And definitely robotics is a perfect example where you can see that it absolutely requires to have understanding of natural language and visual input, and presumably in the future also touch sensing in order to make decisions about how to act the same way humans do.

13:31Why does it matter, though, that robots should have a conceptual understanding of what they're doing? I mean, OK, maybe you wouldn't call them intelligent, but there are robots like, I don't know, dishwashers or like lawnmowers, right? They don't have a conceptual understanding of what a plate is or what grass is. I mean, is it actually necessary? I'm sure that there's applications where you can have a robot that can just repeat the actions and it will be just fine. But we're interested in actually building robots that can really reason and act in a very general way. Just because the world is really messy.

14:05Things will never go exactly according to plan. And there's a lot of tasks where things are constantly changing. and it actually just opens up the opportunity of applications for these robots. They could literally be anywhere that a human could be doing a task. So that enables them to be helpful in home environments, but also in manufacturing environments. There are some things that are important in robotics that I think now with the sort of standard Gemini, actually quite easy. Things like pointing or drawing bounding boxes. Just explain to us what those are. Well, basically, this is one of the areas that we actually had to improve Gemini in order to help with robotics.

14:48So if you have, for example, an object in front of you, what we mean by pointing is that I can literally identify any point in that object. So I can say, imagine that you have a T-shirt in front of you. If I point to the color, it should say, this is the color. Or if I say color, it should identify where the color is. And you might imagine that this is not that important. But actually, if you're trying to fold that T-shirt, you need to know where the color is, where the bottom of the T-shirt is, and all of the different components. Bounding boxes, what it means is that you can identify all the edges of that object so that you know where the object ends and the rest of the environment begins.

15:28So these kinds of examples are, I think, trivial for us humans. We don't even think about it. But if robots are actually able to have access to that kind of information, then they can be smarter about the way that they take action in the physical world. This is what we call embodied reasoning, essentially. How is it different from the kind of reasoning that you get in the standard Gemini model? Yeah, we refer to embodied reasoning as reasoning about the physical world in a lot more detail, the way humans do. If you're going to take action, say that you're trying to pack a lunch for your kid. You're actually, in order to do that, you have to understand where all the objects are in 3D space.

16:04Then you need to understand how to grasp each object in order to pack it into that box. And then you need to figure out how to organize all those pieces so that they fit. All of this is what we mean by embodied reasoning. So is this things like, I don't know, let's say you've got two camera view. You're there and I'm here, for instance. I can see your microphone and so can you, but we've got a completely different view of it. Is it that kind of stuff? Yeah, I mean, it can understand, for example, how far the microphone is from our face. But also if I move around, it can do object correspondence, meaning it understands that microphone is the same one that I'm seeing from the other point of view, which you can imagine is super important if a robot is moving and reasoning about its environment.

16:44How hard is it to switch from a 2D image, like a single camera view, to a 3D understanding of the space? So actually today what robots are doing is that they're taking camera views from different places. So actually the robot has cameras in this wrist and has a camera on top. And it is actually taking all of the inputs from the three images and doing this on its own. It's actually reasoning, oh, I'm closer to the object because now this camera looks closer. This camera, I can see my hand. And it's doing all of that association on its own. We're not explicitly adding depth as an additional input.

17:22We're just giving it multiple camera views and is realizing how to use them in order to understand depth. And how much of that was you deliberately setting that as a task for the robots? Or how much of it sort of emerged from the conceptual understanding that you get from the Gemini models? It simply emerged, actually. So we were able to give it multiple cameras and just see if we actually could reason between them. I mean, that's got to be quite shocking. I mean, I imagine you spent many, many people must have spent many, many years thinking very hard about that problem of how do you how do you align different camera views to make it so that you know, you're tracking an object across different angles.

18:04And then all of a sudden you get these large language models, you know, like Gemini, and it can just do it automatically. Yeah, I mean, it's actually wonderful to be able to leverage these models to bring simplicity to the system. You really don't need to have all of these different stages that you extract depth. Then only then you extract where the objects are and only then you plan how to move. And then only then you're able to do the task. And that's because the foundational model is effectively like a Swiss army knife, like you can do all of the things. Yes, exactly. And you can reason between them, right?

18:36OK, so you enhance the physical reasoning almost. Exactly. You enhance physical reasoning and special understanding. And then motion understanding would be the next thing. It's understanding what would happen if I put a glass at the edge of the table, what's actually likely to happen. All of these areas is the areas that we enhance. But that's not enough. You actually have to take it another step and essentially start to teach Gemini the language of actions. And actions for us means understanding how you are actually moving each joint in a robot. So if this is my robot arm, then I'm teaching Gemini how to move the robot, how to move my arm like this.

19:15And these are all essentially numbers, right? And it's learning to translate what it means to pick up a glass versus move my arm in order to pick up a glass. And so you're essentially teaching it a new language. You're connecting those different ideas. Exactly. Can we think of this as two systems working in tandem then? I mean, I'm thinking of the analogy here of system one and system two, the Danny Kahneman thinking fast and slow thing. Yes, exactly. So essentially the model that we built actually has two models. It has a system that is slow, but very powerful at reasoning and thinking, and a system that is faster, but very good at reactivity.

19:51This is like how human brains work, right? Like that you have the kind of the part of your brain that's very good at calculation and analysis, and then you also have your very instinctive reactive side too. Yes, that's right. In fact, what we do today is that one of these models is much bigger than the other, as you can imagine, and actually lives on the server. and the fast model lives on device and it can respond very quickly. Talk me through how this works then. In terms of the system one and system two and that example of a slam dunk, something it's never seen before, how does it work? So what happens is when you ask the robot to take the basketball and do a slam dunk, the system two has to understand what that means.

20:31You know, what is basketball? It has to understand where the objects that are in front of it are, like where the basketball is, understand there is a hoop. And then that slam dunk actually means picking up that ball and putting it there. So it understands all of that and predicts a rough trajectory of what the robot should do in terms of how it should move. And then hands that over to the system one, which is on device and is able to take that trajectory. But it also takes the visual input and is able to adjust that trajectory. So if I were to, for example, get in the way, put my hand in the middle or move the object around, it would still be able to respond because it already understood the concept of what a slam dunk was and respond very quickly.

21:13Why do you need two systems at all though? Why couldn't you just use the slow, clever one? Yeah, I mean, we could actually just use the slow, clever one, but then it would actually be significantly visually slower and it won't adapt as quickly to changes in its environment. And that's important, especially if you're doing something where the objects will move around. So if you have, for example, imagine when you're folding a T-shirt in the air, right, which as humans do pretty clearly, you're actually moving this T-shirt and things are moving for you in ways that you don't predict. So you need to be able to respond quickly in order to actually complete the task.

21:47So you need definitely a fast system. And the slow system simply enables us to do much more complex reasoning. So you could also just live with a small system if you could do tasks that don't require advanced reasoning. Was it a direct copy of how things work in the human brain? I mean, you know, the Daniel Kahneman work comes back to the 1970s or so, right, that we've understood that that's how the human brain works. Was it a direct? No, not at all. I think we started definitely with a slow system, as you said. Why don't we just solve this with one model? And we found that actually if you want to do highly dexterous behaviours with complex or any kind of complex manipulation, you need to respond quickly.

22:24And that was the best combination that we could find. Wow. It's almost like evolution is a really good optimisation. Finds really good strategies for like quick but clever things. Yes, definitely. I do think that there's sometimes where the human body knows stuff before your brain does, as it were. You know, like you can catch a falling glass without thinking or you can commit things to muscle memory, like, you know, playing a piano where you can actually just be thinking about completely different things. Are you seeing similar things with the robots, that they almost have a physical intelligence that's separate from the slow, clever system?

23:01So we definitely see that if you take the model that can reason and you give it a lot of examples of a particular task, it will get really, really good at that task. But at the moment, if you do too much of that, it will start forgetting some of the generalization. So this is an active area of research, is how do we enable the robot to get really, really good at a task? like a really extremely difficult one, and then not lose any of the generalization. So it actually right now is a balancing act. I mean, in some ways that does happen with humans too. Like I know some people who are really, really, really, really, really good at maths and terrible at tying their own shoelaces.

23:39Okay then, so if this is what's going on behind the scenes, right? So if we've got system one and system two, as you described it, it's also definitely true that these robots have these very impressive new abilities and capabilities which are very different from where we were before. Last time I visited DeepMind's robotics lab, I think it's fair to say that the robot's movements were a bit clumsy. I think that's probably the kindest way to say it. Let me just play you a little clip. So there's only one way round that it can hold this red object and successfully pick it up. And it hasn't worked out which way.