Unveiling Google's RT-2: A Robot's Journey Through Internet Learning

0:00Today we have some breaking news out of Google's DeepMind on AI mixed with robotics. So let's jump in. Essentially, Google's DeepMind team is really pushing the boundaries of AI and robotics, and they're bringing a new, you know, vision language action model. So this is the Robotics Transformer 2, which is the RT2. And this cutting edge AI model was trained on a massive amount of text and images collected from the internet. And this enables it to generate robotic actions, which is a massive step forward from traditional AI based chatbots like ChatGPT or Google Bard, which essentially just generate text snippets for you right so this is really bridging the gap between web knowledge and robotics control so robots unlike chatbots need a very firm grounding in the real world to you know truly be able to be helpful to humans and i think this has always been a very large task that they have to you know comprehend and carry out really complex tasks in a lot of very diverse and often very unpredictable environments.

1:03To date, we have a lot of factories that have robots that are operating doing very simple tasks. But as soon as you want them to be able to not just perform a repetitive task over and over again, but kind of do new things and learn and adapt, it becomes a much more challenging task to undertake. And because of this, I think the training of models like RT2 represent a much more intricate undertaking compared to, you know, training just a large language model or an LLM. So I think it's not just about, you know, a robot recognizing an apple, for example. It has to understand the context. It has to differentiate it from a red ball.

1:43And it has to be able to perform various tasks related to that object. So I think historically practical robot training demanded billions of data points about, you know, the physical world. But RT2 has ushered in a more streamlined approach. So building on RT1's ability to essentially just generalize information across systems, RT2 can actually create a single model capable of complex reasoning with just a fraction of the amount of, you know, robot training data that is typically required. So let's talk a little bit about RT2's kind of novel abilities. So what makes RT2 unique, I think, is its ability to transfer knowledge from a vast corpus of essentially web data and then be able to navigate complex situations and human made requests.

2:34So it can comprehend and execute tasks like, you know, depositing a piece of garbage, even without explicit programming for that specific action, right? So normally when you have a robot, you literally just train it on how to do a task. A lot of times it's very automated. This is not very automatic. and so now you know you're telling it go pick up that piece of trash and throw it in the garbage and it based off of everything it's learned and pictures it's seen will go and try to perform that task which is absolutely a massive step right it's the difference between you know perhaps you could train a robot to drive a car from you know point a to point b because you just you know have it automatically programmed that every inch it's going to turn right or left it's the difference between that which is what previously has been done to something like tesla's self-driving where you know that it's actually looking at the environment around it and making adaptations to how it drives you know it could avoid a pedestrian or it could stop at a stop sign so it's very very this is a massive step and so of course we have something like tesla self-driving which is follows roads which have you know rules and signs and a lot of information like that which it's specifically designed to do this is a whole new step where this is a robot that essentially has to be able to do tasks in the world where there's an infinite amount of actions it can accomplish and there's an infinite amount of ways to do that so i think this is really really impressive um and i think that really kind of demonstrates the model's capability for learning beyond its initial training which i think is key so google engineers put rt2 through the paces with over 6 000 robotics trials and in tasks based on the training data rt2 performed comparable to its predecessor which is rt1 however when faced with novel unfamiliar scenarios rt2 doubled its success rate from rt1's 32 percent to a you know i think it's around a 62 percent success rate so i think this improved adaptability is a monumental advancement in the ai models capabilities And, you know, I can already hear a lot of doubters or haters or whatever you want to call them, right?

4:46Skeptics, perhaps, is a better word, saying, oh, look, this thing's only 62 % efficient. It only has a 62 % success rate. but like look rt1 was not put out that long ago and between then and now we've already doubled our success rate from 32 to 62 for completely unfamiliar scenarios meaning you plop this thing down in a park you say go pick up that piece of trash go throw in that garbage can it's never been to the park before it doesn't know necessarily what that garbage can is or what type of trash that is and it's able to have a 62 success rate executing that task and imagine you know they they ran this through 6 ,000 robotic trials.

5:22So imagine how many of those 6 ,000 tasks it was able to do. I'm absolutely blown away. And for the critics or skeptics out there, I think this is incredibly impressive. And especially the speed of adoption and improvement we're seeing here, I think is incredibly, incredibly impressive. So the RT2's model is an advancement on its predecessor, obviously, right, the RT1, which was trained on multitask demonstrations, which was essentially in an office kitchen environment and that happened over 17 months so the data collected by 13 robots informed rt2's development which allowed it to demonstrate improved generalization capabilities and semantic and visual understanding beyond just the robotic data it was exposed to right so it's actually able to see the world around it actually able to learn and it's be it's able to improve based off of right it's it's experience with everything around it not just the data that was originally given, which is very, very impressive.

6:20The sort of backbone or, you know, the most important part of the RT2 is the adaptation of high capacity vision language models, which are called VLMs. But essentially, they have been successfully trained on web scale data. So these models are brilliant at kind of recognizing visual or language patterns and kind of operating across different languages. But to control a robot, these models need to be trained to output actions. So this challenge is addressed by essentially representing actions as tokens in the model's output, similar to language tokens, right? When we have an LLM like ChatGPT, it's not that it knows, you know, what word necessarily goes next.

6:59It just knows what token goes next in a sentence, which is a fraction of a word. And by making it fractions of words, it's actually a lot more accurate. And that's essentially how these LLMs are able to generate sentences. So they've done the same thing, except now they're calling actions or parts of an action tokens, right? So you might need 100 tokens to complete a specific task. And it doesn't have to know exactly how to do the whole task. It just knows how to do micro movements, putting it towards that task, which is absolutely fascinating, in my opinion, and well done to the DeepMind team on doing that.

7:32So DeepMind actually performed a series of qualitative and quantitative experiments on RT2 models. And in this process, they categorized skills into three areas. So they had symbol understanding, reasoning, and then human recognition. So tasks requiring knowledge transfer from web pre-training essentially demonstrated its, you know, pretty remarkable potential, but with significantly improved generalization performance compared to previous models. I think that that's absolutely important. So on tasks seen in robot data, RT2 retained performance while improving on unseen scenarios by almost double, which essentially was showing a really massive benefit of large scale pre-training.

8:17and it outperformed multiple baselines on unseen tasks and achieved a success rate of 90 % on the open source language table suite of robotics tasks in simulation. Furthermore, RT2 actually proved capable of carrying out more involved commands that required reasoning about intermediate steps. So this actually showed that RT2 can plan from both image and text commands and essentially is, you know, enabling visually grounded planning, which is really, really impressive. I think RT2 showcases how vision language models can evolve into really powerful vision language action models, and that can really directly control a robot.

9:01So I think this is promising, and this whole development really signifies the dawn of a new era in robotic control. And I think it really raises hopes for the creation of general purpose robots capable of reasoning, problem solving, and performing a diverse range of real world tasks. So while Google admits there's still work to be done, the success of RT2 is a testament to the potential advancements in generative AI and LLM technology. And I think the leap from RT1 to RT2 really just kind of underscores the immense strides being made in the field. Now, I will say, of course, this is incredibly exciting, incredibly interesting, but I will say that there are some concerns here, If you really think about it, as these robots essentially are sitting here and learning how to take in the environment around them and do certain tasks, it brings up the question of robotics ethics and a whole Terminator-style event essentially happening where the robots are able to learn how to do something.

9:59right? Maybe they learn how to drive a car and maybe they learn how to rob a bank or maybe they learn how to fly a fighter jet or maybe they learn how to commandeer a freighter ship, right? Like there's all sorts of things that these can essentially learn how to do. And I know a lot of people think like, oh, this is so far fetched. But in my opinion, based off of, you know, things I've seen from chat GPT or inflection AI, I think there's all sorts of dangers that definitely we need to be aware of. I think the number one most important thing is making sure that the preservation of human life is the top priority for these robots and then of course that begins at the ai model which is kind of the brains of a lot of this um so it's going to be really interesting to see how this evolves and i definitely think that is a major area of focus we need to continue to be very vocal about in whatever respective fields we have