NVIDIA's Jensen Huang on AI Chip Design, Scaling Data Centers, and his 10-Year Bets

0:05Hi, listeners, and welcome to KnowPriors. Today, we're here again, one year since our last discussion with the one and only Jensen Huang, founder and CEO of NVIDIA. Today, NVIDIA's market cap is over$3 trillion, and it's the one literally holding all the chips in the AI revolution. We're excited to hang out in NVIDIA's headquarters and talk all things frontier models and data center scale computing, and the bets NVIDIA is taking on a 10-year basis. Welcome back, Jensen. 30 years in to NVIDIA and looking 10 years out, what are the big bets you think are still to make? Is it all about scale up from here?

0:39Are Are we running into limitations in terms of how we can squeeze more compute memory out of the architectures we have? What are you focused on? Well, if we take a step back and think about what we've done, we went from coding to machine learning, from writing software tools to creating AIs, And all of that running on CPUs that was designed for human coding to now running on GPUs designed for AI coding, basically, machine learning. And so the world has changed. The way we do computing, the whole stack has changed. And as a result, the scale of the problems we could address has changed a lot. because if you could paralyze your software on one GPU, you've set the foundations to paralyze across a whole cluster or maybe across multiple clusters or multiple data centers.

1:34And so I think we've set ourselves up to be able to scale computing at a level and develop software at a level that nobody's ever imagined before. And so we're at the beginning of that. over the next 10 years our hope is that we could double or triple performance every year at at scale not at chip at scale and to be able to therefore drive the cost down by a factor of two or three drive the energy down by a factor of two three every single year when you do that every single year when you double or triple every year in just a few years it adds up and so it compounds really, really aggressively.

2:13And so I wouldn't be surprised if the way people think about Moore's Law, which is 2x every couple of years, we're going to be on some kind of a hyper Moore's Law curve. And I fully hope that we continue to do that. What do you think is the driver of making that happen even faster than Moore's Law? Because I know Moore's Law was sort of self-reflexive, right? It was something that he said, and then people kind of implemented it to Yeah. The two fundamental technical pillars, one of them was Denard scaling and the other one was Carver-Mead's VLSI scaling. And both of those techniques were rigorous techniques, but those techniques have really run out of steam.

2:56And so now we need a new way of doing scaling. You know, obviously the new way of doing scaling are all kinds of things associated with co-design. Unless you can modify or change the algorithm to reflect the architecture of the system and then change the system to reflect the architecture of the new software and go back and forth. Unless you can control both sides of it, you have no hope. But if you can control both sides of it, you can do things like move from FP64 to FP32 to BF16 to FP8 to, you know, FP4 to who knows what, right? And so I think that co-design is a very big part of that. The second part of it, we call it full stack.

3:41The second part of it is data center scale. You know, unless you could treat the network as a compute fabric and push a lot of the work into the network, push a lot of the work into the fabric. And as a result, you're compressing, you know, doing compressing at very large scales. And so that's the reason why we bought Mellanox and started fusing InfiniBand and MVLink in such an aggressive way. And now look where MVLink is going to go. You know, the compute fabric is going to scale out what appears to be one incredible processor called a GPU. Now we've got hundreds of GPUs that are going to be working together.

4:29You know, most of these computing challenges that we're dealing with now, one of the most exciting ones, of course, is inference time scaling. It has to do with essentially generating tokens at incredibly low latency because you're self-reflecting, as you just mentioned. I mean, you're going to be doing tree surge. You're going to be doing chain of thought. You're going to be doing probably some amount of simulation in your head. You're going to be reflecting on your own answers. Well, you're going to be prompting yourself and generating text silently and still respond, hopefully, in a second.

5:09Well, the only way to do that is if your latency is extremely low. Meanwhile, the data center is still about producing high throughput tokens because you still want to keep the cost down. You want to keep the throughput high. You want to generate a return. And so these two fundamental things about a factory, low latency and high throughput, they're at odds with each other. And so in order for us to create something that is really great in both, we have to go invent something new. And EnvyLink is really our way of doing that. Now you have a virtual GPU that has incredible amount of flops because you need it for context.

5:51You need a huge amount of memory, working memory, and still have incredible bandwidth for token generation. all at the same time. I guess in parallel, you also have all the people building the models actually also optimizing things pretty dramatically. Like David and my team pulled data where over the last 18 months or so, the cost of a million tokens going into a GPT-4 equivalent model has basically dropped 240X. Yeah. And so there's just massive optimization and compression happening on that side as well. Just in our layer, just in the layer that we work on. You know, one of the things that we care a lot about, of course, is the ecosystem of our stack and the productivity of our software.

6:31You know, people forget that because you have CUDA foundation, and that's a solid foundation, everything above it can change. If everything, if the foundation is changing underneath you, it's hard to build a building on top. It's hard to create anything interesting on top. And so CUDA made it possible for us to iterate so quickly. Just in the last year, I think we just went back and benchmarked when Lama first came out, we've improved the performance of Hopper by a factor of five. without the layer on top ever changing. Now, well, a factor of five in one year is impossible using traditional computing approaches, but accelerated computing and using this way of code design, we're able to invent all kinds of new things.

7:16How much are your biggest customers thinking about the interchangeability of their infrastructure between large-scale training and inference? Well, you know, infrastructure is disaggregated these days. Sam was just telling me that he had decommissioned Volta just recently. They have Pascals, they have Amperes, all different configurations of Blackwell coming. Some of it is optimized for air cool, some of it's optimized for liquid cool. Your services are going to have to take advantage of all of this. The advantage that NVIDIA has, of course, is that the infrastructure that you built today for training will just be wonderful for inference tomorrow.

7:59And most of ChatGPT, I believe, are inferenced on the same type of systems that we're trained on just recently. And so if you can train on it, you can inference on it. And so you're leaving a trail of infrastructure that you know is going to be incredibly good at inference. and you have complete confidence that you can then take that return on the investment that you've had and put it into a new infrastructure to go scale with. You know you're going to leave behind something of use. And you know that NVIDIA and the rest of the ecosystem are going to be working on improving the algorithm so that the rest of your infrastructure improves by a factor of five in just a year.

8:36And so that motion will never change. And so the way that people think about the infrastructures. Yeah, even though I built it for training today, it's got to be great for training. We know it's going to be great for inference. Inference is going to be multi-scale. I mean, you're going to take, first of all, in order to distill smaller models, it's good to have a larger model to distill from. And so you're still going to create these incredible frontier models. They're going to be used for, of course, the groundbreaking work. You're going to use it for synthetic data generation. You're going to use the models, the big models to teach smaller models and distill down to smaller models.

9:13And so there's a whole bunch of different things you can do. But in the end, you're going to have giant models all the way down to little tiny models. The little tiny models are going to be quite effective, not as generalizable, but quite effective. And so they're going to perform very specific stunts incredibly well, that one task. And we're going to see superhuman tasks in one little tiny domain from a little tiny model. Maybe, you know, it's not a small language model, but, you know, tiny language model, TLMs or, you know, whatever. Yeah. So I think we're going to see all kinds of sizes and we hope.

9:49Is that right? Just kind of like softwares today. I think in a lot of ways, artificial intelligence allows us to break new ground in how easy it is to create new applications. But everything about computing has largely remained the same. For example, the cost of maintaining software is extremely expensive. And once you build it, you would like it to run on a large of an installed base as possible. You would like not to write the same software twice. I mean, you know, a lot of people still feel the same way. You like to take your engineering and move them forward. And so to the extent that the architecture allows you to, on one hand, create software today that runs even better tomorrow with new hardware, that's great.

10:31Or software that you create tomorrow, AI that you create tomorrow, runs on a large installed base, you think that that's great. That way of thinking about software is not going to change. NVIDIA has moved into larger and larger, let's say, like unit of support for customers. I think about it going from single chip to, you know, server to rack and VL72. How do you think about that progression? Like, what's next? Like, should NVIDIA do full data center? In fact, we build full data centers. The way that we build everything, unless you're building, if you're developing software, you need the computer in its full manifestation.

11:06So we don't build PowerPoint slides and ship the chips. And we build a whole data center. And until we get the whole data center built up, how do you know the software works? Until you get the whole data center built up, how do you know your fabric works? And all the things that you expect it, the efficiencies to be, how do you know it's going to really work at the scale? And that's the reason why it's not unusual to see somebody's actual performance be dramatically lower than their peak performance as shown in PowerPoint slides.

11:46And computing is just not what it used to be. You know, I say that the new unit of computing is the data center. That's to us. So that's what you have to deliver. That's what we build. Now, we build a whole thing like that. And then we, for every single thing, every combination, air-cooled, x86, liquid-cooled, grace, Ethernet, InfiniBand, NVLink, no NVLink, you know what I'm saying? We build every single configuration. We have five supercomputers in our company today. Next year, we're going to build easily five more. So if you're serious about software, you build your own computers. If you're serious about software, then you're going to build your whole computer.

12:21And we build it all at scale. This is the part that is really interesting. We build it at scale, and we build it vertically integrated. We optimize it full stack and then, and then we disaggregate everything and we sell it in parts. That's the part that is completely, utterly remarkable about what we do. The complexity of that is just insane. And the reason for that is we want to be able to graft our infrastructure into GCP, AWS, Azure, OCI. All of their control planes, security planes are all different. and all of the way they think about their cluster sizing, all different. But yet we make it possible for them to all accommodate NVIDIA's architecture so that CUDA could be everywhere.

13:05That's really, in the end, the singular thought, that we would like to have a computing platform that developers could use that's largely consistent, modulo 10 % here and there because people's infrastructure are slightly optimized differently, and modulo 10 % here and there. but everything they build will run everywhere. This is kind of one of the principles of software that should never be given up. And we protect it quite dearly. It makes it possible for our software engineers to build once, run everywhere. And that's because we recognize that the investment of software is the most expensive investment.

13:44And it's easy to test. Look at the size of the whole hardware industry. And then look at the size of the world's industries. It's$100 trillion on top of this$1 trillion industry. And that tells you something. The software that you build, you have to basically maintain for as long as you shall live. We've never given up on a piece of software. The reason why CUDA is used is because I told everybody, we will maintain this for as long as we shall live. And we're serious. And we still maintain. I just saw a review the other day. NVIDIA Shield, our Android TV. It's the best Android TV in the world.

14:17We shipped it seven years ago. It is still the number one Android TV that people, you know, anybody who enjoys TV. And we just updated the software just this last week. And people wrote a new story about it. GeForce, we have 300 million gamers around the world. We've never stranded a single one of them. And so the fact that our architecture is compatible across all of these different areas makes it possible for us to do it. Otherwise, we would have software teams that are 100 times the size of our company as today. if not for this architectural compatibility. So we're very serious about that. And that translates to benefits to developers.

14:56One impressive substantiation of that recently was how quickly you brought up a cluster for X.AI. Yeah. And if you want to talk about that, because that was striking in terms of both the scale and the speed with which you did that. You know, a lot of that credit you got to give to Elon. I think the, first of all, to decide to do something, select the site, bring cooling to it, power, and then decide to build this 100 ,000 GPU supercluster, which is the largest of its kind in one unit. And then working backwards, we started planning together the date that he was going to stand everything up, and the date that he was going to stand everything up was determined quite a few months ago.

15:48And so all of the components, all the OEMs, all the systems, all the software integration we did with their team, all the network simulation, we simulate all the network configurations. I mean, it's like we pre-staged everything as a digital twin. We pre-staged all of the supply chain. We pre-staged all of the wiring of the networking. We even set up a small version of it, kind of just a first instance of it, ground truth, reference zero, system zero, before everything else showed up. So by the time that everything showed up, everything was staged, all the practicing was done, all the simulations were done, and then the massive integration.

16:34Even then, the massive integration was a monument of gargantuan teams of humanity crawling over each other, wiring everything up 24-7. And within a few weeks, the clusters were up. I mean, it's really a testament to his willpower and how he's able to think through mechanical things, electrical things, and overcome what is apparently extraordinary obstacles. I mean, what was done there is the first time that a computer of that large scale has ever been done at that speed. Unless our two teams are working from a networking team to compute team to software team to training team, you know, and the infrastructure team, the people that the electrical engineers to the, you know, to the software engineers all working together.

17:25Yeah, it's really quite a feat to watch. Was there a challenge that felt most likely to be blocking from an engineering perspective? Just a tonnage of electronics that had to come together. I mean, it'd probably be worth just to measure it. I mean, it's, you know, tons and tons of equipment. It's just abnormal. You know, usually a supercomputer system like that, you plan it for a couple of years from the moment that the first systems come delivered to the time that you probably submitted everything for some serious work. Don't be surprised if it's a year, you know. I mean, that happens all the time.

18:04It's not abnormal. Now, we couldn't afford to do that. So we created, you know, a few years ago, there was an initiative in our company that's called Data Center as a Product. We don't sell it as a product, but we have to treat it like it's a product. Everything about planning for it and then standing it up, optimizing it, tuning it, keep it operational. The goal is that it should be kind of like opening up your beautiful new iPhone and you open it up and everything just kind of works. Now, of course, it's a miracle of technology making it like that, but we now have the skills to do that. And so if you're interested in a data center and just have to give me a space and some power, some cooling, you know, and we'll help you set it up within, call it 30 days.

18:51I mean, it's pretty extraordinary. That's wild. If you think, if you look ahead to 200 ,000, 500 ,000, a million in a super cluster or whatever you call it at that point, what do you think is the biggest blocker? Capital, energy, supply in one area? Everything. Nothing about the scales that you talked about. Nothing is normal. But nothing is impossible. Nothing is, yeah. No laws of physics limits. But everything is going to be hard. And of course, is it worth it? Like you can't believe. to get to something that we would recognize as a computer that so easily and so able to do what we ask it to do, otherwise general intelligence of some kind.

19:44And even if we could argue about, is it really general intelligence? Just getting close to it is going to be a miracle. We know that. And so I think there are five or six endeavors to try to get there, right? I think, of course, OpenAI and Anthropic and X and, of course, Google and Meta and Microsoft. And, you know, this frontier, the next couple of clicks up that mountain are just so vital. Who doesn't want to be the first on that mountain? I think that the prize for reinventing intelligence altogether, it's just too consequential not to attempt it. And so I think there are no laws of physics.

20:33Everything is going to be hard. A year ago, when we spoke together, you talked about, we asked like what applications you got most excited about that NVIDIA would serve next in AI and otherwise. And you talked about how you let your most extreme customers lead you there and about some of the scientific applications. I think that's become like much more mainstream of you over the last year. Is it still like science and AI's application of science that most excites you? I love the fact that we have AI chip designers. Here at NVIDIA. Yeah. I love that we have AI software engineers. How effective are AI chip designers today?

21:15Super good. We couldn't have built Hopper without it. And the reason for that is because they could explore a much larger space than we can. And because they have infinite time, they're running on a supercomputer. We have so little time using human engineers that we don't explore as much of the space as we should. And we also can't explore combinatorially. I can't explore my space while including your exploration and your exploration. And so, you know, our chips are so large, it's not like it's designed as one chip. It's designed almost like a thousand chips. And we have to optimize each one of them kind of in isolation.

21:52You really want to optimize a lot of them together. And, you know, cross-module co-design and optimize across a much larger space. Obviously, we're going to be able to find local maximums that are hidden behind local minimums somewhere. And so clearly, we can find better answers. You can't do that without AI engineers. Just simply can't do it. We just don't have enough time. One other thing that's changed since we last spoke collectively, and I looked it up, at the time, NVIDIA's market cap was about 500 billion. It's now over 3 trillion. So, the last 18 months, you've added$2.5 trillion plus of market cap, which effectively is$100 billion plus a month, or two and a half snowflakes, or a stripe plus a little bit, or however you want to think about it.

22:42A country or two. A country or two. Obviously, a lot of things have stayed consistent in terms of focus on what you're building, et cetera. Walking through here earlier today, I felt the buzz when I was at Google 15 years ago. You felt the energy of the company and the vibe of excitement. What has changed during that period, if anything? Or what is different in terms of either how NVIDIA functions or how you think about the world or the size of bets you can take? Well, our company can't change as fast as the stock price. Let's just be clear about that. So in a lot of ways, we haven't changed that much.

23:16I think the thing to do is to take a step back and ask ourselves, what are we doing? I think that that's really the big observation, realization, awakening for companies and countries is what's actually happening. I think what we were talking about earlier, from our industry perspective, we reinvented computing. Now, it hasn't been reinvented for 60 years. That's how big of a deal it is. that we've driven down the marginal cost of computing down probably by a million X in the last 10 years to the point that we just, hey, let's just let the computer go exhaustively write the software. That's the big realization.

24:00And that in a lot of ways, we were kind of saying the same thing about chip design. We would love for the computer to go discover something about our chips that we otherwise couldn't have done ourselves. Explore our chips and optimize it in a way that we couldn't do ourselves. In the way that we would love for digital biology or, you know, any other field of science. And so, I think people are starting to realize, one, we reinvented computing. But what does that mean, even? And all of a sudden, we created this thing called intelligence. And what happened to computing? Well, we went from data centers.

24:37Data centers our multi-tenant, stores our files. These new data centers we're creating are not data centers. They don't, they're not multi-tenant. They tend to be single tenant. They're not storing any of our files. They're just, they're producing something and they're producing tokens. And these tokens are reconstituted into what appears to be intelligence. Isn't that right? And intelligence of all different kinds. You know, it could be articulation of robotic motion. It could be sequences of amino acids. It could be, you know, chemical chains. It could be all kinds of interesting thing, right?

25:08So what are we really doing? We've created a new instrument, a new machinery that in a lot of ways is the noun of the adjective generative AI. You know, instead of generative AI, it's an AI factory. It's a factory that generates AI. And we're doing that at extremely large scale. And what people are starting to realize is, you know, maybe this is a new industry. It generates tokens, it generates numbers, but these numbers constitute in a way that is fairly valuable and and what industry would benefit from it then you take a step back and you ask yourself again you know what's going on nvidia on the one hand we reinvented computing as we know it and so there's a trillion dollars of infrastructure that needs to be modernized that's just one layer of it the big layer of it is that there's this instrument that we're building is not just for data centers, which we're modernizing, but you're using it for producing some new commodity.

26:12And how big can this new commodity industry be? Hard to say, but it's probably worth trillions. And so that I think is kind of the, if you were to take a step back, you know, we don't build computers anymore. We build factories and every country is going to need it. Every company is going to need it. You You know, give me an example of a company or industry that says, you know what? We don't need to produce intelligence. We got plenty of it. And so that's the big idea, I think, you know, and that's kind of an abstracted industrial view. And, you know, someday people will realize that in a lot of ways the semiconductor industry wasn't about building chips.

26:51It was about building the foundational fabric for society. And then all of a sudden everybody goes, oh, I get it. You know, this is a big deal. It's not just about chips. How do you think about embodiment now? Well, the thing I'm super excited about is in a lot of ways, we're close to artificial general intelligence, but we're also close to artificial general robotics. Tokens are tokens. I mean, the question is, can you tokenize it? You know, of course, tokenizing things is not easy, as you guys know. But if you're able to tokenize things, align it with large language models and other modalities, if I can generate a video that has Jensen reaching out to pick up the coffee cup, why can't I prompt a robot to generate the tokens to pick up the – you know?

27:43And so intuitively, you would think that the problem statement is rather similar for a computer. And so I think that we're that close. That's incredibly exciting. Now, the two brownfield robotic systems, brownfield meaning that you don't have to change the environment for, is self-driving cars and with digital chauffeurs and embodied robots, right? Between the cars and the human robot, we could literally bring robotics to the world without changing the world because we built a world for those two things. It's probably not a coincidence that Elon's focused on those two forms of robotics because it is likely to have the larger potential scale.