NVIDIA's AI Moat & Origins — With Bryan Catanzaro

0:00The NVIDIA executive who started its AI push joins us to talk about what makes the company so indispensable, unpacking the secrets to its success. That's coming up right after this. Welcome to Big Technology Podcast, a show for cool-headed, nuanced conversation of the tech world and beyond. We have a great conversation for you today. We're going deep inside NVIDIA with Brian Catanzaro. He's the vice president of Applied Deep Learning Research at NVIDIA, and he's going to be speaking at the company's forthcoming GTC conference, March 18th to 21, in San Jose. and his conversation is going to be specifically on practical AI agents that reason and code at scale.

0:40So if you're in the area or you're thinking about heading out, you can mark that down. But let's get to the conversation. Brian, welcome to the show. Thanks. I'm glad to be here. Great to have you. You're kind of the guy that kicked off this whole AI push within NVIDIA. Well, you know, it took a whole company to transform NVIDIA into what it is today. So So there's tens of thousands of people that deserve a lot of credit. But I was honored to be one of the people that helped NVIDIA get started a long time ago. What is exactly happening within your company? What do you offer that allows anyone that wants to build AI and then run AI models to do it effectively?

1:20NVIDIA is an accelerated computing company. Wait, wait. So what does that mean, accelerated computing? I hear you guys talk about it all the time. We've been talking about it for a long time and the world still doesn't quite understand it. So I'm glad to try to explain it. The idea is that the world faces a lot of computational challenges that can't be solved without faster computers. But building a computer is not enough. In order to actually deliver acceleration, all of the pieces have to line up and plug together and be fully optimized across the entire stack. so that people have the chance to do things that they just couldn't do otherwise computationally.

2:00And AI is a great example of that. Training and deploying the awesome generative models that are changing the world right now is extraordinarily computationally intensive. It's the biggest computational challenge the world has ever faced. And the reason that NVIDIA is providing something useful here is because for decades, we've taken on this mission of optimizing the entire stack to build software, algorithms, libraries, frameworks, compilers, systems, networking, chips, the whole thing, and optimize them together for the most important workloads. And that's AI. So it's interesting because if you think about like a traditional chip company, and by the way, I might be totally off base on this and correct me if I am.

2:47The manufacturer will buy the chip, put it inside, let's say their computer, right? And then build all the software around it. But you guys also build, you not only manufacture the chip, but you build the algorithms and the software that surrounds it. So that enables companies to get the most out of it. So is that a right characterization of what you're doing? Yeah, I think so. I mean, the core thesis that powers NVIDIA is that a chip could never be enough. Just the same way that a chip couldn't be enough for my Apple phone, for example. Apple makes awesome chips, but the experience of using my phone is a lot more than the chip.

3:27And the way that Apple's able to vertically integrate and optimize their entire system in order to create an amazing consumer experience, I think is pretty incredible and super valuable. What NVIDIA is doing is not the same, but it's related in the sense that we understand that the value of the technology we create is only understood in composition, in context. It's really about, are we delivering acceleration, transformative acceleration to the most important computational workloads of our time? So why couldn't other companies then just go and build their own software to train using NVIDIA chips or other chips?

4:09Because it seems to me like, correct me if I'm wrong here also, but if I'm reliant on NVIDIA's software, it's closed source, right? So I'm going to train my model with it. It's kind of difficult to switch to another chip. So why particularly rely on the NVIDIA software? Yeah, I mean, we have a lot of open source software as well as some closed source software. You know, we make the decision about what to open source based on what we think is going to help the market most. But I mean, the reason why people work with us is because we deliver transformational acceleration, we enable people to do things computationally that they just couldn't do.

4:48And we know that it would never be enough to just provide a chip that said it was really fast and had like a lot of operations per second inside the chip. Because the gap between, you know, what a particular chip can do and what the experience of a scientist or engineer trying to invent the future, that gap is quite enormous. And if any one of the links in that chain, whether we're talking about systems design or networking or data center design or the compilers, frameworks, libraries, applications, algorithms, you know, if any of those links are to fail, the acceleration is lost and the value, therefore, is lost.

5:31And so NVIDIA has a unique way of approaching this problem, co-optimizing the entire stack in order to deliver that acceleration to the end scientists and researchers that are trying to invent the future. And that's what differentiates us from other companies. Now, could other companies do that? I mean, absolutely. It's not a secret. In fact, we've been shouting it from the rooftops for decades that this is what we do and that it's different from being a chip company. But, you know, we're continuing to test that thesis. You know, is there value in accelerated computing that's above and beyond what you get just from making and selling awesome chips?

6:11And I think the answer to that is yes. And, you know, I think that's the reason why we've been so successful. Right. And to me, I think this is just for anyone listening. Like, I spent the whole week making calls on NVIDIA trying to figure out because I was wrong. I thought that, like, everything would kind of slow down this year. and I was like speaking to customers and analysts tell me exactly what I missed. And this was the thing that I underestimated, which is that it's not just the chip, but it's the chip, the software and everything that goes along with it. And that's why the company has been so successful.

6:43So let's talk a little bit about like what it actually goes into training an AI model. So let's say I'm an organization. I come to NVIDIA, say I have a bunch of data or maybe I even don't have data and I'm looking to build a large language model, what do I do now? Deep breath. That's a great question. You know, the first thing that's on my mind is like, you know, what data center are you going to use to train this model in? And, you know, that's a really important question because it turns out that the AI market is growing pretty fast because there's so many institutions that are training these huge models and you actually have to have a building to put these machines in and they're not small.

7:23and you need to hook it up to power. So that would be one of my first questions is like, okay, are you ready to stand this up? Or are you going to be working with a CSP, like for example, AWS? And we love to support our customers through cloud providers as well. Okay. And so then what happens next? Let's say I'm set up. You're set up. Okay. So we will definitely point you to our reference implementations of the various LLMs and their training setups on these clouds. We'll show you how to scale it to many thousand GPUs efficiently. We'll tell you what kind of speed you should expect to get while training the model.

8:12And we'll also discuss things about reliability. how do we make sure that the job is actually progressing properly and yielding intelligence? So we definitely help our customers with things like that. I think also then when the model's trained, there's a question about how do we deploy it? And we'd love to help people deploy AI as well. I think Jensen said in the earnings call this week that somewhere around 40 % of our data center GPUs were going for inference, which I think is pretty amazing and definitely a shift from where things have been a few years ago. And so we're spending a lot more time helping our customers accelerate the deployment of these models as well, making sure that they get the best speed so that they can get as much out of the systems they're deploying these models on as possible.

9:11And what are they using the models for? Language models are starting to be used in a lot of different parts of a lot of different companies. Things like question answering, I think, are really important to help people understand answers to their specific questions, especially relating to private data stores that they need to answer questions with. you know I think we're seeing a lot of people use AI and office type settings I don't know if you've interacted with Microsoft co-pilot at all but it can be really helpful at least to me when I'm looking at a summary of a meeting and what the the action items are for everybody at the meeting you know and other sort of office automation tasks you know we're also pushing forward with the use of these models for our own internal work at NVIDIA.

10:08We have a project called Chip Nemo that is using language models to help our chip designers and verification efforts be more efficient as we build our own products. It's called Chip Nemo? Yeah, Nemo is kind of our most user-friendly open source software for training and fine-tuning language models and other kinds of conversational AI, it also has a lot of speech capabilities as well. And, you know, we've been building it for quite a few years because we believed that conversational AI was really going to transform industry. We wanted to make a platform for companies to build and deploy their own conversational AI.

10:52So that's what Nemo is. And so when we talk about chip Nemo, we're talking about using that for our own chip book. Wait, so how do you use it for your own chip work? At the moment, a lot of it has to do with improving communication between chip designers. So you have like a thousand people working on this project and there's a lot of interfaces that need to be described and people have questions. They don't know exactly who to talk to. So basically we're making knowledge bases about our own work that then people can use to answer questions. and we found that it's kind of like having a more senior engineer that you can talk to all the time that helps you find the things you need to find in a huge code base.

11:37And so that's the primary thing that we're doing right now is augmenting the engineers on the team with kind of superpowers to understand our own code better and interact with it better. Over time, I expect that Chip Nemo is going to do other things as well, improving the quality of our designs. Our Hopper GPUs, for example, have a lot of circuits in them that were designed by AI that we built ourselves that have better speed and power and cost characteristics than we knew how to build with any other tool. Wait, generative AI programs designed some of the chips? Yes, Hopper is designed with generative AI.

12:21That's insane. It's wild. So let's dream a little bit. Obviously, we know that like knowledge repositories inside companies is something that this stuff is going to be really good for. Maybe a little bit of like consumer agents or consumer chatbots like ChatGPT. Is this where it ends? Like, where do you see it going? I don't think this is where it ends. You know, I've been thinking recently about past revolutions in the media space. You know, we got books, which transformed society, you know, when, because we could distribute ideas, and we could reference the same ideas in a new way, because everybody could read the same book, you know, audio, you know, as soon as we got audio recordings that created an entirely new industry, you know, the recorded music industry, which continues to be totally vibrant and important to our culture.

13:14Movies, TV, and video games. Every time that we come up with a new technology, we find a way to explore ideas as humans and explore our culture together in a way that helps us solve problems better and also creates a new form of culture that we interact with. And I think that the most exciting applications for AI are ones we haven't really even dreamed up yet. In the same way that it would have been hard to imagine how books were going to change the world back when Gutenberg first made the press, I think AI is going to create a new form of media that is much more interesting, much more engaging, much more useful.

14:02and ultimately we're going to use that to refine our ideas and explore them together in the way that we have with other media. It's just going to be much more interesting and useful. When I hear you say media, it leads me to believe that you think that this is going to be more of like an agent or a digital friend that people will start interacting with, right? Because that's media. Unless is there something else or something else I could be thinking about, I'm not thinking about that could take the form of media? Yeah, something along those lines. I mean, I think, you know, I'm expecting AI is going to change the lives of all of us here on planet Earth.

14:37And when I think about how 8 billion people on this planet live, you know, most of us aren't reading and writing that much, you know. But we do love virtual worlds. People love interacting in video games. We love interacting with each other. And I think that the primary way that people are going to interact with AI is going to be in virtual worlds, because I think that's going to be the most natural way of interaction and the most useful way. And I think we're going to perceive that as a new form of media that really touches all aspects of our work and our play. It's going to be something new.

15:12So you're a real believer in this metaverse vision that you'll just kind of end up in a digital world and the people and the scenery will all be AI generated or maybe mostly AI generated and you go. people we we have a culture it's very important to us you know the the the ideas that we share together and the sort of shared humanity that we have is more important to us than uh the content of uh the the things that we're interacting with so for example ai is probably going to be really awesome at playing soccer but do i think that people are going to go to watch robots play soccer, even if the robots are kicking around the ball better than humans, I don't think it's as interesting because I don't think that it is related to us.

16:01You know, I think the primary thing that we're interested in is ourselves. We're trying to understand ourselves and how we relate to other people. I think AI is going to give us new ways of doing that. We are going to be interacting in virtual worlds. You know, NVIDIA has been a big believer in virtual worlds for, you know, the past 30 years. Right. You've been on gaming before you were on AI. That's right. And we've had this initiative called the Omniverse long before Meta renamed itself because we believe that simulating the world and providing virtual agents a place to interact with people is hugely important to the future of technology.

16:37I see these things coming together. I think there's a lot of opportunities to use virtual worlds to make AI stronger, to teach AI how to understand the real world and act better in the real world. And then, of course, giving humans the opportunity to interact with AIs in much more natural and useful ways. I think a lot of that's going to happen in a virtual world. Okay, I want to talk about reasoning and a few other ways about how companies are going to work with NVIDIA and what might be coming down the pike. So we're going to do that right after the break, coming up right after this. Did you know your credit card points and miles can lose value to inflation?

17:13Credit card companies often reduce the redemption value of your points and miles. Now, imagine a credit card with rewards that can grow in value. With the Gemini credit card, you can earn Bitcoin or one of over 50 other cryptos instantly with no annual fee. Every swipe at the store or gas pump earns you instant rewards deposited straight to your account. Plus, sign up now for a$200 Bitcoin bonus to kickstart your rewards. Visit Gemini.com slash card today. Check out the link in the description for more information on rates. Again, if you're looking to invest in Bitcoin but don't know where to start, the Gemini credit card makes it easy.

17:51The Gemini credit card is issued by WebBank. In order to qualify for the$200 crypto intro bonus, you must spend$3 ,000 in your first 90 days. Some exclusions apply to instant rewards in which rewards are deposited when the transaction posts. This content is not investment advice and trading crypto involves risk. The Gemini credit card cannot be used to make gambling-related purchases. What the hell is going on right now? And why is it happening like this? At Wired, we're obsessed with getting to the bottom of those questions on a daily basis. And maybe you are too. I'm Katie Drummond, the Global Editorial Director of Wired.

18:28And I'm hosting our new podcast series, The Big Interview. Each week, I'll sit down with some of the most interesting, provocative, and influential people who are shaping our right now. Big interview conversations are fun. I want a shark that... That eats the internet. That turns it all off. Unfiltered and unafraid. So in a lot of ways, I try to be an antidote to the unimaginable faucet of reactionary content that you see online, to the best of my ability. Every week, we're going to offer you the ultimate luxury of our times. Meaning and context. True or false, you, Brian Johnson, the man sitting across from me, one day, at some point, as of yet undefined in the future, you will die.

19:12False. Tell me more. Listen to the big interview right now in the same place you find Wired's Uncanny Valley podcast. Subscribe or follow wherever you get your podcasts. We're back here on Big Technology Podcast with Brian Catanzaro. He's the Vice President of Applied Deep Learning Research at NVIDIA. What made you think, okay, AI is going to be big enough that I should get to Jensen, CEO of NVIDIA, and say we need to really work hard to make this part of our core offering? I had been spending my research career at Berkeley as a PhD student on the future of computing, and we knew that computing was going to have to change.

19:55Back in 2005 or so, it was obvious that computers would have to be different. The standard way of making computers wasn't working anymore. We would have to be more specific. We'd have to be more parallel. And so I had been spending my time as a grad student thinking about what kinds of applications could take advantage of the computers that will be possible to build, but then are going to provide enormous amounts of value to humanity. And at the time, AI was not a very big field, and it wasn't actually super popular to work in it. But when I was thinking about it, I felt like from first principles, it made sense to me that this was something that had the potential to really change the world.

20:38And, you know, NVIDIA's approach to solving this, I think, was also, you know, fairly careful and iterative, you know. So, you know, I published my first paper in 2008 on machine learning on the GPU. And NVIDIA really jumped in full steam ahead for the whole company to become an AI company in 2013. So it took about five years of sort of testing that thesis, like, is AI actually going to be something that could really change the world? And we started getting some early indicators of success. You know, one of those was, of course, the ImageNet competition in 2012, which really shocked the world with the quality of results and wouldn't have been possible without accelerated computing.

21:29You know, the results that they got were so incredible because they built a very fast system for training neural nets. And that wasn't generative, right? That was just identifying those in photos. That's correct. Yeah, it wasn't generative at the time. But, you know, the idea of generative AI is fairly old. I mean, when I was a grad student, generative AI was a thing that we talked about all the time. It's just that we weren't using neural nets for it. We were using other models like graphical models. These are other mathematical approaches that are a little bit more clever, but don't scale as well.

22:05And so this is another part of the thesis that I had is that, you know, the thing that's really going to help AI succeed is scale. You know, if we can apply huge data sets and huge amounts of compute to AI, then the results are going to get much better. And this is also controversial back then. And even today, some people really don't like this idea because they would like AI progress to be mostly held back by our smarts, like our mathematical skills in coming up with more clever models to describe our data and the world. But it does seem these days that there's a lot of evidence that the most important thing is having really good data sets to learn from and then enormous computational scale.

22:50And so that was my thesis. And I was advocating for that at NVIDIA. I wrote this little prototype of a library for training neural nets on the GPU, which then became QDNN, which was our very first library for AI on the GPU. And the process of getting the company to rally around that and build that as a product and ship it, it took some time. Um, but because there were these, um, you know, early indicators of success that there was a lot of, uh, demand picking up, um, even back then, uh, it, it made sense for the company to, to really pay attention. And then, you know, Jensen himself is such a visionary.

23:36I remember when he first started interacting with me about this back in 2012, I, um, I felt like, uh, he was just so hungry to learn you know so i felt like i i gave him all the things that i learned from my phd in the course of like an hour about like um how ai could change nvidia's business and what uh nvidia could potentially build and my ambitions for what that meant were like a thousand times smaller than jensen's were you know um he he took it immediately and then elaborated on it and thought about where is this going? You know, one of the things he first said back in 2012 was this is an entirely new way of writing software rather than having humans enumerate all the different cases that software needs to understand.

24:20We're going to have models that learn from our data how to solve problems. And these days that sounds like the truth, right? Like we see that happening every day when we interact with these models. But, you know, 12 years ago, that was a pretty bold thing to say and i was a little bit nervous about it because um you know the history of ai uh over the past you know 70 years had been one of um over promising and under delivering in a lot of ways which then caused a lot of booms and busts right yeah a lot of ai winters it can do something and then just totally dry it up until it's that's right prove itself again and so when when jensen like immediately glommed onto this and started like um thinking about what it what it could mean I wanted to slow him down a little bit and be like, Jensen, this is a big, huge idea, but I'm not sure if it's going to happen now.

25:13It might be 30 years from now. But it turns out that Jensen was right about this. This was the right time to apply enormous data and enormous compute to AI and get these results. Right. But 2012, wasn't it? I mean, it took another 10 years, 11 years, really, for the boom to come. So what did it feel like? I think NVIDIA, yeah. Go ahead. Oh, I was going to say, I think NVIDIA is really good at decade-long technology development. I've seen that happen at NVIDIA many times. You know, ray tracing, I was in meetings in 2008 with Jensen on ray tracing, and we launched our first ray tracing GPU in 2018.

25:52You know, it took 10 years of continuous development and research in order to make ray traced virtual worlds a reality. and uh CUDA itself you know uh the the projects that led to CUDA uh they started in the early 2000s CUDA was released as a beta in 2006 right and I think it wasn't where that all AI programming is done with on with the H100s pretty much and a100 right yeah CUDA is our our framework for programming the GPU and making it do stuff that's interesting and um and uh you You know, that project was crazy for a long time. You know, Wall Street hated it because it subtracted value from our earnings reports.

26:36Like they looked at the costs of our products and they're like, these products are too expensive. Your margins are too low. You know, back then the margins were quite low. And that's because there wasn't the applications and the ecosystem that were using CUDA yet in order for us to build a strong business around it. But NVIDIA continued investing in CUDA in the libraries, the software, the compilers, the frameworks, and of course also the chips for 10 years. Actually, maybe more than 10 years before all of a sudden CUDA became an overnight success. You know, it's like 10 years of hard work that everyone ignored and Wall Street criticized NVIDIA for mercilessly.

27:19Why are you wasting your time on this? You know, everybody knows the GPU is just for gamers. Why are you trying to make the GPU do something else? And, you know, we did it anyway. And, you know, that's one of the things I love about this company. I think it's one of the reasons why we're successful at the accelerated computing mission is that when we decide to do something, we do it out of our convictions about how technology will unfold. And we base those convictions on a speed of light analysis about what's actually possible to try to keep ourselves honest. And then, you know, once we have that conviction, we're able to follow through.

27:52What did you see in those years that everybody gave up on this? I mean, obviously, there were big advances that were made in things like machine learning, right? Like computer vision and natural language processing. And that's where we had Facebook really take the lead as the public spokesperson for this stuff, talking about image recognition. And they even built this fake generative chat bot called M that I had access to that basically would be like it's supposed to be a large language model. We didn't even know it was going to be a large language model as a pre-transformer. Right. But like you would talk to this bot and it would talk back and they were trying to figure out like what people were interested in, if they're going to build a bot.