NVIDIA's Plan To Build AI That Understands The Real World — With Rev Lebaredian

0:00Let's talk about NVIDIA's push to generate AI that understands the real world with technology that can influence the future of robotics, labor, cars, and more. We're joined by the company's VP of Omniverse and Simulation Technologies right after this.

0:19You're used to hearing my voice on the world bringing you interviews from around the globe. And you hear me reporting environment and climate news. I'm Carolyn Buehler. And I'm Marco Werman. We're now with you hosting The World Together. More global journalism with a fresh new sound. Listen to The World on your local public radio station and wherever you find your podcast.

0:47Welcome to Big Technology Podcast, a show for cool-headed, nuanced conversation of the tech world and beyond. Today, we're joined by Rev Leberetian. He's the vice president of Omniverse and Simulation Technology at NVIDIA for a fascinating conversation about what may well be the next stage of AI progress, the pursuit of world models that provide common sense to AIs. Rev, I'm so happy to see you here. We actually spent some time at your headquarters a couple months back, and I'm really glad that you're here today and to introduce you to the big technology audience. Welcome to the show. Thank you for having me.

1:22All right, before we jump into world models, obviously, we're having this conversation in the wake of the DeepSea revolution. I don't know what you want to call it. And everyone is talking about NVIDIA now. You're in quiet period, so we're not going to go into financials. But I can and do want to ask you about the technology side of this, specifically about Jevons Paradox. I keep hearing NVIDIA, Jevons Paradox, Jevons Paradox, NVIDIA. What is Jevons Paradox and what do you think about it? My understanding of what Jevons' paradox is, is essentially an economic kind of principle that as you reduce the cost of something, of running it, you create more demand for it because it unlocks essentially more uses of that technology when it becomes more economically feasible to use that.

2:13But I think that really does apply in this case in the same way that it applies to almost every other important computing innovation over the last 40, 50 years, or at least as long as I've been alive. You know, at the inception of NVIDIA in 1993, NVIDIA selected, very carefully selected the very first computing problem to address in order to create the conditions by which we could continue innovating and keep growing that market. And this was the problem of computer graphics and particularly rendering within computer graphics, generating these images. The reason we selected it is because it's an endless problem.

3:07No matter how much compute you throw at it, no matter how much innovation we throw at it, you always want more. And throughout the time I've been at NVIDIA, which is now 23 years, many times I've heard, well, graphics are good enough. Rendering is good enough. And so soon, soon, NVIDIA has big GPUs and more computing power. It's not going to be necessary. We'll just get consumed by SOCs or integrated into another chip as integrated graphics and it'll disappear. But that never happened because the fundamental problem of simulating the physics of light and matter was endless. We see this in almost every important computing domain.

3:59AI is one of these things. I mean, can we really say that we have now reached the point where our computers are intelligent enough or the intelligence we create is good enough? And so it's just going to shrink. We're not going to have any more use for more compute power there. I don't think so. I think intelligence is something that is probably the most endless of all computing problems. If we can throw more compute at the problem, we can make more intelligence and do it better and better. So making AI more efficient will just increase its economic value in many of the applications we want to apply it to and increase demand.

4:46And can we talk about the progression of AI models becoming more efficient? I know it's like a hot topic right now, but it does seem to me that over the past couple of years, we've definitely seen models become more and more efficient. So what can you tell us about this? We'll just talk about large language models on this front. The efficiency gains that we've seen over time with them. I mean, this isn't new. This has been happening for the past 10, 12 years or so, essentially since we first discovered deep learning on our GPUs with AlexNet. If you look at the computational curve, what our GPUs can do in terms of tensor operations, the AI kind of math that we need to do, over the last 10 years, we've had essentially a million X performance increase.

5:43And that increase isn't just from the raw hardware. It's also through many layers of the software, algorithms. So we're getting these benefits, these speedups continuously at a very rapid rate, exponentially, by compounding many layers, all the different layers at which this computing happens. From the fundamental hardware, the chips themselves, systems level, networking, system software, algorithms, frameworks, and so on. And so what we've seen here with DeepSeek is a great advancement that's on the same curve that we've been on for a decade now. Okay. And 23 years at NVIDIA. I'm going to save a question to ask you about that as we get later on or towards the end of the interview, because I'm very curious what your experience has been being at NVIDIA for so long, especially given that the company's technology, at least from the outside world, was viewed as in favor.

6:48and then people question it and back in favor of people question. Obviously, we see what's going on now. I mean, we're leaving through a mini cycle at this point. So very curious about your experience. But I want to talk about the technology first. And let me just bring you into a conversation that we had here on the show with Jan LeCun, who's met as a chief AI scientist, really right after ChatGPT came out. And one of the things that Jan did was he said, go ask ChatGPT what happens if you let go of a piece of paper with your left hand. And I typed it in. It gave a very convincing answer. It was completely wrong because with text, you don't have the common sense about physics.

7:27And try as you might to teach a model physics with text, you can't. There's just not enough literature that describes what happens when you drop a paper with a hand, and therefore the models are limited. And Jan's point here was basically like, if you want to get to truly intelligent machines, you need to build something into the AI that teaches common sense, that teaches physics, and you need to look beyond words to do that. And so now I turn it over to you, Rev, because I do think that right now within NVIDIA, a big initiative is to build a picture of the world to teach AI models that common sense that Jan had mentioned was lacking.

8:09And I have some follow-ups about it, but I want to hear first a little bit about what you're doing and whether your efforts are geared towards solving the problem that Jan brought up. Well, what Jan said is absolutely true, and it makes intuitive sense, right? If an AI has only been trained on words, on text that we've digitized, How can it possibly know about concepts from our physical world, like what the color red really is? What it means to hear sound? What it means to feel felt? You know, it can't know those things because it never experienced them. When we train a model, essentially what we're doing is we're providing life experience to that model, and it's pulling apart patterns or it's discerning patterns from all of the experience that we give it.

9:11And what was really, really amazing about GPT, the advancements with LLMs, starting with the Transformer, is that we could take this really, really complex set of rules that humans had no way of actually defining directly in a clear and robust manner, the rules of language. and we were able to pull that out of a corpus of data. We took all of this text, all these books and whatever information you could scrape from the internet about that. And somehow this model figured out what all the patterns of language are in many different languages and could then, because it understands the fundamental rules of language, do some amazing things.

10:08It could generate new text. It could style some text that you give it in a different way. It can translate text from one form to another, from one language to another. It can do all of this awesome stuff. But it lacks any information about our world other than what's been described in those words. And so the next step in AI is for us to take the same fundamental technology we have, this machine we have, where we can feed it life experience and it figures out what the patterns and the rules are and feed it with actual data about our physical world and about how our world works so that it could apply that same learning to the rules of physics instead of the rules of grammar.

10:58the rules of language. It's going to understand how the physical world around us works. And our thesis is that from all the AIs we're going to create into the future, the most valuable ones are going to be the ones that can interact with our physical world, the world that we experience around us, the world created out of atoms. Today, the AIs that we're creating are largely about our world of knowledge, our world of information, ones and zeros, things that you could easily represent inside a computer in the digital world. But if we can apply the same AI technology to the physical world around us, then essentially we unlock robotics.

11:45We can have these agents with this intelligence and even super intelligence in specific tasks, do amazing things in the world around us, which is, if you look at global markets, if you look at all of the commerce happening in the world and GDP, the world of knowledge, information technology, is somewhere between$2 to$5 trillion a year. But everything else, transportation, manufacturing, supply chain, warehouse and logistics, creating drugs, all the stuff in the physical world, that's about$100 trillion. So the application of this kind of AI to the physical world is going to bring more value to us.

12:41So it's interesting. It's not just basically inputting that real world knowledge into LLMs. Right. So they can get the question about dropping the paper with a hand. Correct. It is also something that you're working on is building the foundation for robots to go out into our world and operate within it. So, yes, it's not it's not inputting it in the same way that we do for these text models. We're not just going to describe with words what happens when you drop a piece of paper. We're going to give these models other senses during the learning process. So they'll watch videos of paper dropping.

13:33We can also give it more accurate, specific information in the 3D realm. Because we can simulate these physical worlds inside a computer today, we have physics simulations of worlds, we can pull ground truth data about the position and orientation and state of things inside that 3D world and use that as another mode of input into these models. And so what we'll end up with is a World Foundation model that was trained on many different modes of data, essentially different senses. It can see, it can hear, it can touch and feel and do many of the things we can do or many things other animals or even things no creature can do because we can provide it with sensors that don't exist inside the natural world.

14:28And it can, from that, kind of decipher what are the actual combined rules of the world. And this encoding of the knowledge of how the physical world works can then be the basis for us to build agents inside the real world, to build the brains of these agents, otherwise known as physical robots. Right. And so this is your recently announced Cosmos project. So talk a little bit about what Cosmos is. I mean, obviously, it's a world foundational model. But how long have you been building it? And what type of companies and developers might use it and what they might use it for?

15:17We've been working towards Cosmos for probably about 10 years. We envisioned that eventually this new technology that had formed with deep learning, that that was going to be the critical technology necessary for us to create robot brains. And that is ultimately what's going to unlock this incredible amount of value for us. So we started working towards this a long time ago. We realized early on that the big problem we were going to have is in order to train such a model, to train a robot brain to understand the physical world and to work within it, we're going to have to give it experience. We're going to have to give it the data that represents the physical world.

16:15And capturing this data from the real world is not really an easy thing to do. It's very expensive and in some cases very dangerous. For example, for self-driving cars, which is a type of robot, it's a robot that can autonomously, you know, on its own figure out how to get from point A to point B by controlling this physical being, a car, by braking and accelerating and steering. How are we going to ensure that a self-driving car really understands when a child runs into the street as it's barreling down the street that it should stop? And how can we be sure that it's actually going to do that without actually doing that in the real world?

17:02We don't want to go capture data of a child running across the street. Well, we can do that by simulating it inside a computer. And so we realized this early on. So we set about applying all of the technologies we'd been working on up until that point with computer graphics and for video games and video game engines and physics inside these worlds to create a system to do world simulation that was physically accurate so that we could then train these AIs. And so we call that operating system, if you will, Omniverse. It's a system to create these physics simulations, which we then use to train AIs that we could test in that same simulation before we put them out in the real world.

17:57So we use it for self-driving cars and other robots out there. So building Cosmos actually starts first with simulating the world. And so we've been building that stack and those computers for quite a while.

18:15Once the Transformer model was introduced and we started seeing the amazing things large language models can do and the chat GPT moment came, we understood that this essentially unlocked the one thing that we needed to really push forward in robotics, which is the ability to have this kind of general intelligence about a really complex set of things, complex set of rules. And so we set about building what is Cosmos today, essentially a few years ago using all of the technology we had built before with simulation and AI training. And what Cosmos is, it's actually a few things. It's a collection of some open weight models that we've made freely available.

19:15Along with it, we also provide essentially all of the tooling and pipelines you need to create a new world foundation model. So we give you the world foundation models that we've started training, which are world-class, especially for the purposes of building physical AI. And we also have what's called a tokenizer, which are AIs themselves that are world-class. It's a critical element of building World Foundation models. And then we have curation pipelines. The data that you select and curate to feed into the training of your World Foundation model is critical. And just selecting the right data requires a lot of AI in and of itself.

20:06So we released all of this stuff. And we put it out there in the open so that the whole community can join us in building physical AI. And so who's going to use it? Is it going to be robotics developers? Is it going to be somebody that's building, let's say, LLM-based application, but just wants them to be a little smarter? Both? It will be all of them. Yes. We feel that we're, as the industry, the world, is right at the beginnings of this physical AI revolution. And no one company, no one organization is going to be able to build everything that we need. So we're building it out there in the open to encourage others to come build on top of what we've built and come build it with us.

21:01And this is going to be essentially anybody that has an application that involves the physical world. And so that's definitely robotics companies are part of this and robotics in the very general sense. That includes self-driving car companies, robo-taxi companies, as well as companies building robots that are in our factories and warehouses. Anybody that wants to make intelligent robots that have perception and operate autonomously inside the real world, they want this. But it's not only about robots in the way we think about them as these agents that move around. we have sensors that we're placing in our spaces, in our cities, in urban environments, inside buildings.

21:54These sensors need to understand what's happening in that world, maybe for security reasons, for coordinating other robots, changing the climate and energy efficiency of our buildings and data centers. So there's many applications of physical AI that are broader than what we generally think of as what you imagine when you say a robotic application. There's going to be thousands and thousands of companies that build these physical AIs. And this is just the beginning. Now, you mentioned that the transformer model was an important development on this path. and that obviously was the thing that underpinned a lot of the real innovation we've seen with large language models can the real world ai learn from the knowledge base that has been sort of turned into these ai models with text like if you're if you have a model that's trying to understand the world with common sense do they take text as an input they take all of it as input how does it work then with with text i mean it's very interesting because Because it seems like that's like when we talk about the progression towards general intelligence, that is a very, you know, kind of amazing application of being able to read something and then sort of intuit what it means in a physical space.

23:22Don't you think? Yeah, I think the way I think about it, and I think this is right, is these AIs learn the same way we do. when you're brought into this world you don't know who is mommy who is daddy you don't even know how to see yet you don't have depth perception you can't see color or understand what it is you don't know language you don't know these things but you learn by being bombarded with all of this information simultaneously through the many different senses so when your mommy looks at you and says, I'm mommy pointing, you're getting multiple modes of information coming at you, including essentially that text that's coming through an audio form there.

24:15And then eventually when you learn how to read, you learn how to read because a teacher points at letters and then words and sounds them out. So you have this association that you build between the information that you understand, like mommy, and the letters that mean that thing. AIs learn in the same way. When we train them, if you give them all of these modes of information associated with each other at the same time, it'll associate them together. That's how image generators work today. When you go generate an image using a text prompt, The reason why it can generate an image of a red ball in a grass field in an overcast day is because when it was trained, there was an association of some text along with the images that were fed into it.

25:20It knew that during the training process that these words were related to that image. And so we can gather that understanding from that association. What we're trying to do with World Foundation models is take it to the next level by giving it more modes of information and richer information. But part of that will still include text. We'll feed in the text along with the video and other ground truth information from the physical state of the world. Yeah, so this is going to be a multi-part question, and I apologize, but I don't really know another way to ask it. So what are the other modes of information that you're feeding in there?

26:06And do you really need to go through this simulation process? And I'll tell you, it all sounds like a worthwhile endeavor to me, and I'm sure it is. But I also see video models today. And that is something that's really surprised me when we've seen the video generation models, is that they really have an understanding of physics. Like they know just as an image, like an image generation is not moving, right? So you know that let's say the guy sits on the chair. But video, you could see people walking through a field and you watch the grass move. And that means that those models inherently have a concept of how physics works, I think.

26:47And I'm going to run it by you because you're the expert here. But like, again, and Jan's going to come on the show in a couple of weeks. So maybe this is just in my mind because I'm gearing up and thinking about our last conversation, but I'm going to put this to you also. Maybe I'll ask what your answers, I'll ask him to weigh in on your answers on this. But the thing that he always talked about is a human mind is able to sort of see infinite possibilities and accept that. It doesn't break us. So if you have a pencil and you hold it up, you know it's going to fall, but you know it could fall in infinite ways, but it's still going to fall.

27:20For an AI that's been trained on different scenarios, it's very difficult. for them to understand that that pencil might fall in infinite ways when asked to generate it. However, they've been doing a very good job with the video generators of like showing that they understand that. So just to sort of reiterate, what different modes of information are you using and why do we need this broader simulation environment or this Cosmos tool if we are getting such good results from video generation already? All very, very good questions. So first First off, we use many, many modes. The primary one though for training Cosmos is video, just like the video generation models.

28:00But along with that, there's text. We also feed it extra information and labels that we can gather from data, particularly when we do, when we train, when we generate the data synthetically. If you use a simulator to generate the videos, you have perfect information about everything that's going on in every pixel in that video. We know how far each object is in each pixel. We know the depth. We know what the object is in each pixel. You can segment out all of that stuff. Traditionally, what we've done for perception training for autonomous vehicles, so we've used humans to go through and label all that information from hours and hours of video that's been collected, and it's inaccurate and not complete.

29:07So from simulation, we can get perfect information about the actual videos themselves. Now, that being said, your question about, you know, these video models seem to really know physics and know it well. I think it is pretty amazing, you know, how much physics they do know. And it's kind of surprising we're here at this point. Like, had you asked me five years ago, would we be able to generate videos with this much physics plausibility at this stage? I wasn't sure, actually, because I continually had been wrong for years prior to that. I didn't expect to see image classification in my lifetime until we saw it with AlexNet.

29:56But I would have bet against it. And so we're pretty far along. That being said, there's a lot of flaws in the physics we see. So you see this in the video. One of the basic things is object permanence. if you direct the video to move the camera, point away and come back, objects that were there at the beginning of the video are no longer there or they're different, right? And so that is such a fundamental violation of the laws of physics. It's kind of hard to say, well, these models currently understand physics well. And there's a whole bunch of other things in there. You know, my life's work has been primarily computer graphics and specifically rendering, which is a 3D rendering is essentially a physics simulation.

30:51It's the simulation of how light interacts with matter and eventually reaches a sensor of some sort. We simulate what a camera would do in a 3D world and what image it would gather from the world. When I look at a lot of these videos that are generated, I see tons and tons of flaws because when we do those simulations and rendering, we're attuned to seeing when shadows are wrong and reflections are wrong and these sorts of things. to the untrained eye, it looks plausible. It looks correct. But I think people can still kind of feel something is wrong, you know, when it's AI generated, when it's not.

31:39In the same way that for decades now, since we introduced computer graphics to visual effects in the movies, you know, when some, you don't know what it is, but if the rendering's not great in there, it just feels CG, it feels wrong. We still have that kind of uncanny valley thing going on. That all being said, I think we're going to rapidly get better and better. So the models today have an amazing amount of knowledge about the physical world, but they're maybe at like 5%, 10 % of what they should understand. We need to get them to 90%, 95%. Right. Yeah, I just saw a video of a tidal wave hitting some island.

32:22and I looked at it, it was like super realistic. It was, of course it was on Instagram because that's all Instagram is right now is 3D generate, I mean, AI generated video. And it took me a second and it's more frequently taking me a minute to be like, oh, that's AI generated. And sometimes I have to look in the comments and just sort of trust the wisdom of the crowds on that front. But you might not be the best judge of it as well. Humans, I mean, we're not particularly good at knowing whether the physics will really be accurate or not. This is why movies, directors can take such license with the physics when they do explosions and all kinds of other fun stuff like tidal waves in there.