Jim Fan on Nvidia’s Embodied AI Lab and Jensen Huang’s Prediction that All Robots will be Autonomous

0:00So from the chip level, which is the Justin Soar family, to the foundation model, project group, and also to the simulation and the utilities that we build along the way, it will become a platform, a computing platform for human robots, and then also for intelligent robots in general. So I want to quote Jensen here. One of my favorite quotes from him is that everything that moves will eventually be autonomous. and I believe in that as well. It's not right now but let's say 10 years or more from now if we believe that there will be as many intelligent robots as iPhones then we'd better start building that today.

0:57Hi and welcome to Training Data. We have with us today Jim Fan, Senior Research Scientist at NVIDIA. Jim leads NVIDIA's Embodied AI Agent Research with a dual mandate spanning robotics in the physical world and gameplay agents in the virtual world. Jim's group is responsible for Project Groot NVIDIA's humanoid robots that you may have seen on stage with Jensen at this year's GTC. We're excited to ask Jim about all things robotics, why now, why humanoids, and what's required to unlock a GPT -3 moment for robotics. Welcome to training data. Thank you for having me. We're so excited to dig in today and learn about everything you have to share with us around robotics and embodied AI.

1:39Before we get there, you have a fascinating personal story. Thank you for the first, the first intern at OpenAI. Maybe walk us through some of your personal story and how you got to where you are. Absolutely. I would love to share the stories with the audience. So back in the summer of 2016, some of my friends said there is a new startup in town, and you should check it out. And I'm like, huh, I don't have anything else to do because I got accepted to PhD. And that summer I was idle, so I decided to join the startup and that turned out to be OpenAI. And during my time at OpenAI, we were already talking about HGI, back in 2016.

2:19And back then, my intermentor was on Dr. Kapparthy and Ilya Soskever. And we talked about, and we discussed a project together, it's called World of Bits. So the idea is very simple. We want to build an AI agent that can read computer screens, read the pixels from the screens, and then control the keyboard and mouse. If you think about it, this interface is as general as it can get. Right, like all the things that we do on computer, like you know, Replying to emails or playing games or browsing the web. It can all be done in this interface, mapping pixels to keyboard mouse control. So that was actually my first kind of attempt at AGI, at OpenAI, and also my first journey, the first chapter of my journey in AI agents.

3:06I remember World of Bets actually. I didn't know that you're a part of that. That's really interesting. Yeah, yeah, it was a very fun project and was part of a bigger initiative called Open a Universe Which was like a bigger Platform on like integrating all the applications and games into into this framework What do you think were some of the unlocks then and then also what do you think were some of the challenges that you had with agents back then? Yes, so back then the main method that we use was reinforcement learning There was no LOM no transformer back in 2016 And the thing is, reinforcement learning, it works on specific tasks, but it doesn't generalize.

3:42Like we can't give the Asian arbitrary language and instructed to do arbitrary things that we can do with a keyboard mouse. So back then, it kind of worked on tasks that we designed, but it doesn't really generalize. So that started my next chapter, which is I went to Stanford, and I started my PhD with Professor Faith Ely, and we started working on computer vision and also embodied AI. And during my time at Stanford, which was from 2016 to 2021, I kind of witnessed the transition of the Stanford Vision Lab led by Feifei from, you know, static computer vision, like recognizing images and videos, to more embodied computer vision, when an agent learns perception and takes actions in an interactive environment.

4:31And this environment can be virtual as in simulation, or it can be the physical world. So that was my PhD, like transitioning to embodied AI. And then after I graduated from PhD, I joined Nvidia and have stayed there ever since. So I carried over my work from my PhD thesis to Nvidia and still working on embodied AI till this day. So you oversee the embodied AI initiative at Nvidia. Maybe say a word on what that means and what you all are hoping to accomplish. Yes, so the team that I am co -leading right now is called gear which is a GER and that stands for Journalist Embodied Agent Research. And to summarize what our team works on in three words is that we generate actions because we build in body AI agents and those agents take actions in different worlds.

5:21And if the actions are taken in the virtual world, that will be gaming AI and simulation. And if the actions are taken in the physical world, that will be robotics. Actually, earlier this year, in March, GTC, at Jensen's keynote, he unveiled something called Project Group, which is a video's moonshot effort at building foundation models for human -erobotics. And that's basically what the gear team is focusing on right now. We want to build the AI brain for human -erobots and even beyond. What do you think is Nvidia's competitive advantage in building that? Yeah, that's a great question. So one is for sure like compute resources.

6:02All of these foundation models require a lot of compute to scale up. And we do believe in scaling law. There were scaling laws for like LOMs. But the scaling law for embodied AI and robotics are yet to be studied. So we're working on that. And the second strength of Nvidia is actually simulation. So, Nvidia, before it was an AI company, it was a graphics company. So, Nvidia has many years of expertise on building simulation, like physics simulation and rendering and also real -time acceleration on GPUs. So, we are using simulation heavily in our approach to build robotics. The simulation strategy is super interesting.

6:41Why do you think most of the industry is still very focused on real -world data, the opposite strategy? Yeah, I think we need all kinds of data and simulation and real world data by themselves are not enough. So, at GIR we divide this data strategy into roughly three buckets. One is the Internet scale data, like all the tags and videos online. And the second is simulation data, where we use Nvidia simulation tools to generate lots of synthetic data. And the third is the real robot data, where we collect the data by tele -operating in the robot, and then just collecting and recording those data on the robot platforms.

7:22And I believe a successful robotic strategy will involve the effective use of all three kinds of data and mixing them and also delivering a unified solution. Can you say more about, well, you're talking earlier about, you know, have data as fundamentally the key bottleneck in making a robotics foundation model actually work? Can you say more about your kind of conviction in that idea? And then what exactly does it take to make great data to kind of break through this problem? Yes. So I think like the three different kinds of data that I just mentioned have different strengths and weaknesses. So for internet data, they are the most diverse.

8:00They encode a lot of common sense prior. Like for example, most of the videos online are human centered. because humans, we love to take selfies, we love to record each other doing all kinds of activities, and there are also a lot of instructional videos online. So we can use that to kind of learn how humans interact with objects and how objects behave under different situations. So that kind of provides a common sense prior for the robot foundation model. But the Internet skill data, they don't come with actions. We cannot download the motor control signals of the robots from the internet. And that goes to the second part of the data strategy, which is using simulation.

8:43So in simulation, you can have all the actions and you can also observe the consequences of the actions in that particular environment. And the strength of simulation is that it's basically infinite data. And the data scales with compute. the more GPUs you put into the simulation pipeline, the more data that you will get. And also the data is super real time. So if you collect data only on the real robot, then you are limited by 24 hours per day. But in simulation, like the GPU accelerated simulators, we can actually accelerate real time by 10 ,000x. So we can collect the data at much higher throughput, given the same work of time.

9:24So that's a strength. But the weakness is that for simulation, no matter how good the graphics pipeline is, there will always be this simulation to reality gap. The physics will be different from the real world, and the visuals will still be different. They will not look exactly as realistic as real world, and also there is a diversity issue. The contents in the simulation will not be as diverse as all the scenarios that we encounter in the real world. So these are the weaknesses. And then going to the real robot data. And those data, they don't have the same to real gap, because they're collected on the real robot.

10:05But it's much more expensive to collect, because you need to hire people to operate the robots. And again, they're limited by the speed of the world of atoms. You only have 24 hours per day, and you need humans to collect those data, which is also very expensive. So we see these three types of data as having complementary strengths. And I think a successful strategy is to combine their strengths and then, you know, to remove their weaknesses. So the cute groups robots that were on stage with Jensen, those are such a cool moment. If you had to help us dream in one five, ten years, like what do you think your group will have accomplished?

10:45Yeah, so this is pure speculation, but I hope that we can see research breakthrough in robot foundation model maybe in the next two to three years So that's what we call like a GPT -3 moment for robotics and then After that, it's a bit uncertain because to have the robots Enter like daily lives of people. There are a lot more things than just a technical side The robots need to be affordable and mass -produced and we also need like safety for the hardware and also privacy regulations and those will take longer for the robots to be able to hit a mass market. So that's a bit harder to predict, but I do hope that the research breaks through a coming in the next two to three years.

11:29What do you think will define what a GPT -3 moment in AI robotics looks like? Yeah, that's a great question. So I would like to think about robotics as consisting of two systems, system one and system two. So that comes from the book thinking fast and slow where a system one means this low -level motor control that's unconscious and fast like for example when I'm grasping this cup of water I don't really think about how I move the fingertip at every millisecond So that would be system one and then system two is slow and deliberate and it's more like reasoning and planning That actually uses the the conscious brain power that we have So, I think the GPU 3 moment will be on the system one side.

12:17And my favorite example is the verb open. So just think about the complexity of the word open, right? Opening the door is different from opening window. It's also different from opening a bottle or opening a phone. But for humans, we have no trouble understanding that open means different things when you're interacting. It means different motions when you're interacting with different objects. but so far we have not seen a robotics model that can generalize on a low -level motor control on these verbs. So I hope to see a model that can understand these verbs in their abstract sense and can generalize to all kinds of scenarios that make sense to humans.

12:56And we haven't seen that yet, but I'm hopeful that this moment could come in the next two to three years. What about system two thinking? Like how do you think we get there? Do you think that some of, you know, the reasoning efforts in the LLM world will be relevant as well in the robotics world? Yeah, absolutely. I think for System 2, we have already seen very strong models that can do reasoning and planning and also coding as well. So these are the LOMs and frontier models that we have already seen these days. But to integrate the System 2 models with System 1 is another research challenge in itself.

13:33So the question is, for Robo Foundation model, do we have a single monolithic model or do we have some kind of cascaded approach where the system two and the system one models are separate and can communicate with each other in some ways? I think that's an open question and again they have pros and cons, right? Like for the first idea, the monolithic model is cleaner, there's just one model, one API to maintain, but also it's a bit harder to kind of control because you have different control frequencies. Like the system two models will operate on slower control frequency, let's say one hertz, like one decision per second, while the system one, like the motor control of me grasping this cup of water, that will likely be a thousand hertz, where I need to make these minor, like these tiny muscle decisions at a thousand times per second.

14:25It's really hard to encode them both in a single model, so maybe a cascaded approach will be better. But again, how do we communicate between system one and two? Do they communicate through text or through some latent variables? It's unclear, and I think it's a very exciting new research direction. Is your instinct that will get there in that breakthrough on system one thinking? Like through scale and transformers, like, is it going to work? Or is that, you know, cross your fingers and hope and see? I certainly hope that the data strategy I described will kind of get us there. because I feel that we have not pushed the limit of transformers yet.

15:03On the essential level, like transformers take tokens in and outputs tokens. And ultimately, the quality of the tokens determines the quality of the model, the quality of those large transformers. And for robotics, as I mentioned, like the data strategy is very complex. We have all the internet data and also we need simulation data and the real robot data. And once we're able to scale up on the data pipeline with all those high quality actions, then we can tokenize them and we can send them to a transformer to compress. So I feel we have not pushed transform to limit yet. And once we figure out the data strategy, we may be able to see some emergent property as we scale up the data and scale up the model size.

15:47And for that, I'm calling it the scaling law for embodied AI and it's just getting started. I'm very optimistic that we will get there. I'm curious to hear what are you most excited about personally? When we do get there, what's the industry or application or use case that you're really excited to see this completely transformed the world of robotics today? Yes. So there are actually a few reasons that we chose humanoid robots as kind of the main research thesis to tackle. One reason is the world is built around human embodiment, the human form factor, all restaurants, factories, hospitals, and all equipment and tools, they're designed for the human form and also the human hands.

16:28So, in principle, a sufficiently good humanoid hardware should be able to support any task that a reasonable human can do in principle. And the humanoid hardware is not there yet today, but I feel in the next two to three years, the humanoid hardware ecosystem will mature and will have affordable humanoid hardware to work on. And then it will be a problem about AI brain, about how we kind of drive those humanoid hardware. And once we have that, once we're able to have the group foundation model that can take any instruction in language and then perform any tasks that a reasonable human can do, then we unlock a lot of economic value.

17:08Like we can have robots in our households helping us with daily chores, like laundry, dishwasher and cooking or like elderly care, and we also have them in restaurants, in hospitals, in factories, helping with all the tasks that humans do. And I hope that will come in the next decade, but again, as I mentioned in the beginning, This is not just a technical problem, but also there are many things beyond the technology. So I'm looking forward to that. Any other reasons you've chosen to go after humanoid robots specifically? Yeah, so there are also a bit more practical reasons in terms of the training pipeline.

17:49So there are lots of data online about humans, right? It's all like human centered, all the videos, all like humans doing daily tasks or having fun. And the humanoid robot form factor is closest to the human form factor, which means that the model that we train using all of those data will be able to have an easier time to transfer to the humanoid form factor other than other form factors. So let's say for robot arms, right? Like how many videos do we see online about robot arms and grippers? Very few, but there are many videos of people using their five finger hands to work with objects. So it might be easier to train for human robots.

18:33And then once we have that, we'll be able to specialize them to like the robot arms and more kind of specific robot forms. So that's why we're aiming for the full generality first. I didn't realize it. So are you exclusively trading on humanoids today versus robot arms and robot dogs as well? Yeah. So for project group. In simulation, yeah. Yes, so for project group we are aiming more towards humanoid right now But the pipeline that we're building including the simulation tools Right the real robot tools like those are general purpose enough that we can also adapt to other platforms in the future So yeah, we're building these tools to be generally applicable You've used term general quite a few times now I think there are some folks from especially from the robotics world who think that you know a general approach won't work can you have to be domain environment specific?

19:26Why have you chosen to go after a generalist approach? And the Richard Sutton bit or less and stuff has been a recurring theme on our podcast. I'm curious if you think it holds in robotics as well. Absolutely. So I would like to first talk about the success story in NLP that we have all seen. So before the chat GPT and the GBD3 in the world of NLP, there are a lot of different models and pipelines for different applications, like translation and coding and doing math and doing creative writing. They all use very different models and completely different training pipelines. But then, Chargibit came and unified everything into a single model.

20:13So before Chargibit will call those specialists. And then the GB3 and Chargibit is, we call them the journalists. And once we have the general list, we can prompt them, distill them, and fine tune them back to the specialized tasks. And we call those the specialized general list. And according to the historical trend, it's almost always the case that the specialized general list are just far stronger than the original specialist. And they're also much easier to maintain because you have a single API that takes text in and then spits text out. So I think we can follow the same success story from the world of NLP and it will be the same for robotics.

20:54So right now in 2024 most of the robotics and applications we have seen are still in the specialist stage. They have a specific robot hardware for specific tasks collecting specific data using specific pipelines. But Project Group aims to build this general purpose foundation model that works on humanoid first, but later will generalize to all kinds of different robot forms or embodiments. And that will be the journalist moment that we are pursuing. And then once we have that journalist, we'll be able to prompt it, fine tune it, distill it down to specific robotics tasks. And those are the specialized journalists.

21:36But that will only happen after we have the journalist. So it would be easier in a short run to pursue the specialist. It's just easier to show results because you can just focus on a very narrow set of tasks. But we at Nvidia believe that the future belongs to journalists. Even though it will take longer to develop, it will have more difficult research problems to solve. But that's what we're aiming for first. The usually thing about Nvidia building grew to me is also what you mentioned earlier, which is that Nvidia owns both the chip and the model itself. What do you think are some of the interesting things that Nvidia could do to optimize Groot on its own chip?

22:15Yes, so at the March GTC Jensen also unveiled the next generation of the edge computing chips. It's called the Jensen store chip And it was actually co -announced with project Groot So the idea is we will have kind of the full stack as a unified solution to the customers So from the chip level, which is the Justin Soar family, to the foundation model, Project Groot, and also to the simulation and the utilities that we build along the way, it will become a platform, a computing platform for human robots, and then also for intelligent robots in general. So I want to quote Jensen here. One of my favorite quotes from him is that everything that moves will eventually be autonomous.

23:02And I believe in that as well. It's not right now, but let's say 10 years or more from now. If we believe that there will be as many intelligent robots as iPhones, then we'd better start building that today. That's awesome. Are there any particular results from your research so far that you want to highlight anything that gives you kind of optimism or conviction and the approach that you're taking? Yes, we can talk about some prior works that we have done. So one work that I was really happy about was called URACA. And for this work, we did a demo where we trained a five -finger robot hand to do pen spinning.

23:42So very useful. And it's super humorous with respect to myself because I have given up pen spinning, in launches chart. I'm not able to do it. I'm doing live demo. I will film, miserably, at this live demo. So yeah, I'm not able to do this, but the robot hand is able to. And the idea that we use to train this is that we prompt an ROM to write code in the simulator API that Nvidia has built. So it's called the i6 -SIM API. And the ROM outputs the code for reward function. So a reward function is basically a specification of the desirable behavior that we want a robot to do. So the robot will be rewarded if it's on the right track or penalized if it's doing something wrong.

24:33So that's a reward function. And typically the reward function is engineered by a human expert, a typical robot assist, who really knows about the API. It takes a lot of specialized knowledge. And the reward function engineering is by itself a very tedious and manual task. So what URICA did was we designed this algorithm that uses OOM to automate this reward function design so that the reward function can instruct the robot to do very complex things like pen spinning. So it is a general purpose technique that we developed and we do plan to scale this up to beyond just pen spinning. It should be able to design reward functions for all kinds of tasks, or it can even generate new tasks using the Nvidia simulation API.

25:20That gives us a lot of space to grow. Why do you think? I remember five years ago, there were people that were research labs working on solving Rubik's cubes, the robot hand and things like that. It felt like robotics went through maybe a trough of disillusionment. And in the last year or so, it feels like the space has really heated up again. Do you think there is a why now around robotics this time around? And what's different and where, you know, we're reading that OpenAI is getting back into robotics. Everybody is now spinning up their efforts. Like, what do you think is different now? Yeah, I think there are quite a few key factors that are different now.

25:59One is on the robot hardware. Actually, since the end of last year, we have seen a surge of new robot hardware in the ecosystem. There are companies like Tesla, working on Optimus, Boston Dynamics, and so on, and a lot of startups as well. So we are seeing better and better hardware. So that's number one. And those hardware are becoming more and more capable with, you know, better dexterous hands, better whole body reliability. And the second factor is the pricing. So we also see a significant drop in the price and the cost, the manufacturing cost for the human robots. So back in 2001, NASA had a humanoid developed and it's called a robot knot.

26:42I remember if I recall correctly, it cost north of $1 .5 million per robot. And then most recently, there are companies that are able to put a price tag of about $30 ,000 full -fledged humanoid, and that's roughly comparable to the price of a car. And also, there's always this trend in manufacturing where a mature product, the price of it, will tend towards the price of the raw material cost. And for the humanoid, it typically takes only 4 % of the raw material of a car. So it's possible that we can see the cost trending downwards even more, and there could be an exponential decrease in the price in the next couple of years.

27:24And that makes these still very hard to be more and more affordable. That's the second factor of why I think humanoid is gathering momentum. And the third one is on the foundation model side. We are able to see the system too problem. The reasoning, the planning part being addressed very well by the frontier models, like the GPs and the clouds and the llamas of the world. And these LOMs, they are able to generalize to new scenarios. They're able to write code. and actually the URICA project I just mentioned leverages these coding abilities of the LOMs to help develop new robot solutions. And there are also a search in multi -modal models improving the computer vision, the perception of it.

28:06So I think these successes also encourage us to pursue robot foundation models because we think we can write on the generalizability of these frontier models and then add actions on top of them. So we can generate action tokens that will ultimately drive these humanoid robots. I completely agree with all that. I also think so much of what we've been trying to tackle to date in the field has been how to unlock the scale of data that you need to build this model. And all the research advancements that we've made, many of which you've contributed to yourself around Centauril and other things. And the tools that Nvidia's built with Isaac Sim and others have really accelerated the field alongside tell operation and cheaper tell operation devices and things like that.