Bill Dally: The Evolution and Revolution of AI and Computing

0:05Hey everyone and welcome to Generative Now. I am Michael Magnano. I am a partner at Lightspeed And this week, I'm speaking to one of the most forward-thinking minds when it comes to computer hardware and architecture, Bill Daly. Bill Daly has been NVIDIA's chief scientist and senior vice president for research at NVIDIA since 2009. His decades-long career started in academia at Caltech, MIT, and later becoming the chair of the computer science department at Stanford. We talk about his early experience in the 1980s, playing around with neural networks at Caltech, the pace of AI evolution, and why he believes that AI is the technology that will revolutionize all human endeavors.

0:43So check out my conversation with Bill Daly, NVIDIA's chief scientist and senior vice president for research. Hey, Bill, how's it going? It's going well, Michael. Thank you so much for doing this. Really, really appreciate the time. Very excited to talk to you. You've been looking forward to this one for a while. You have obviously incredibly impressive background and role at NVIDIA. And there's so much we could get into about NVIDIA and the state of AI and GPUs and all of the research that you and your team of, I believe, hundreds of researchers are working on. But maybe before we get there, like I said, you have such an impressive career.

1:24I think the audience would love to hear a little bit about what you've done over the past several decades across academia, entrepreneurship, this role as chief scientist of NVIDIA. Give us a little bit of the background of your story. Okay. What's relevant to AI and the like probably started when I was a graduate student at Caltech. This was 40 years ago in the 1980s. I took a course on neural networks. And I thought that was just a really cool technology. We built little multi-layer perceptrons and conf nets and these things called hot field nets that were little associative memories. But it also impressed me that it was a toy, that it was a great technology, but the compute wasn't there at the time.

2:09But that was a formative thing. And then later on, I was a professor at MIT and I was building parallel computers. and it kind of struck me that parallelism was the technology, it was a way to scale performance in a way that you couldn't do with serial processors. But at the same time, existing software was a huge inertia that with Moore's Law in effect then and the Moore's Law about serial processors, not about transistors, people could just wait. And every 18 months or so, their performance of their computers would double. And so why rewrite all your software? where if you went with parallel computing, your performance would go up by a factor of four.

2:50If you just wait, it goes up by a factor of two, and that's just too easy a path to compete with. So it wasn't really until that ended that the parallel computing took up. Until Moore's Law ended. Yeah. And so then in the early 2000s, when I was on the faculty at Stanford, we developed this technology called stream processing, which is a way of really making parallel processing more accessible by managing the data movement in a very effective way. And we partnered with NVIDIA, the development of the NV50, which came to market as the G80, to make that technology broadly available in the form of CUDA.

3:29Now, another thing that was going on about the same time when I was on the faculty at Stanford, I was chair of the computer science department, when Sebastian Thrun won the grand challenge for an autonomous car to drive itself across the desert from, what is it, Barstow to Las Vegas or something like that. And the technology that made that work, I remember going to one of Sebastian's meetings and they were talking about how they're having trouble having their car tell the difference between the road and the desert. And it's actually harder than it seems because those dirt roads, they're dirt and the desert is dirt.

4:00And how do you tell one dirt from the other dirt? And they had the smartest graduate students trying to code up manual feature detectors to do that. And it wasn't working. And so they just acquired a lot of data and they used statistical methods to do it. It wasn't neural networks at the time. Again, the compute wasn't quite there for that. I'm trying to remember what it was, but it was a way of automatically discovering features by mining lots of data. And it just struck me that that was a very powerful technology. And a few years after that, after I'd left Stanford and joined NVIDIA about 2010, I had a breakfast with Andrew Ng.

4:36And at the time he was working at Google Brain, you know, finding cats on the internet using 16 ,000 CPUs. And it struck me, okay, you know, that's a lot of compute power. You know, I should say it's a lot of expense for that compute power, but we've gotten there. These neural networks that I played with, you know, back in the 1980s, we finally have the technology to make them real. And it also struck me that CPUs aren't the way to do this, right? What we should do is get the stuff running on GPUs. So I got somebody in NVIDIA research to port his cat-finding code to GPUs. And that code ultimately became QDNN.

5:13That's kind of the path I took, starting in the academic world at Caltech and MIT and Stanford, sort of seeing all the pieces come together, the original neural network technology, the parallel computing, evolving that into stream processing, into GPU computing, and ultimately converging on where we are today, building the engines that are basically powering this revolution in AI. And these engines, obviously, there's enormous demand, unlike anything I could have ever imagined, even just 18 months ago. One of the questions I like to often ask people sitting in your seat on this podcast is, did you expect what has happened over the past 12 to 18 months?

5:59Obviously, you've been thinking and working on this stuff for the past couple of decades, But but did you even know what we were about to experience, you know, through the explosion of generative AI? And I didn't expect it to happen this quickly. OK. So I was convinced that this was the technology that's going to revolutionize all of human endeavor. Right. You know, how we play, how we work, how we, you know, educate, how we get medical care, everything about your life would be profoundly affected by AI. I knew that was going to happen, but I thought the change was going to be more gradual and not quite as frenetic as it's turned out to be.

6:33And it's interesting because it was a slow start. You know, things were moving along. You were seeing lots of applications of AI starting with the Confinets back, you know, 10 years ago, maybe 12 years ago. So people were starting companies to, you know, in agriculture, you know, tell what is plant, what is weed and squirt the herbicide on the bad one. And, you know, it was it was happening. There was growth. But then, you know, when chat GPT came out, it was like somebody turned the rate knob way up and things just became a lot faster. I was not expecting that. Could data be a limitation at some point in the near future?

7:15You know, it certainly is one of the key ingredients that you need to make this work. But there's so much that can be done, you know, to both mine private, you know, repositories of data that many companies have that has not been addressed yet. And also to create synthetic data, which we found very effective in many applications. And I don't see that as an immediate concern. I think that there's going to be plenty of data, both on the private side and on the synthetic side. that, you know, the usual thing people have done is kind of scraping the web and getting data. They may be nearing the limits of what can be done with that, but there's a whole lot more data out there.

7:55Yeah, makes sense. So talk us through what does it mean to be the chief scientist at NVIDIA? You know, give us a day in the life of Bill Daly at NVIDIA and your team. It's got to be the world's most fun job, I think. So I get to do a lot of interesting things. You know, my role is as chief scientist and senior vice president of research. And it's actually two distinct jobs. So as chief scientist, my job is really to kind of poke my nose into everything going on in the company and try to make the technology better. And so whether it's, you know, I'll attend the meetings I'm planning for the next generation of GPUs, you know, for autonomous vehicle projects, for robotics projects, and try to stay up to date and connect people to, oh, there's somebody at this university doing something really exciting that could make this better.

8:40Let's take a look at that. Or, you know, maybe we should be pushing harder on a new packaging technology for the next generation GPUs and just trying to push people out of their comfort zones a little bit, get them to try, try things that could make stuff better. And then the flip side is I run the research organization, which is like a giant playground. We get smart people from all over doing exciting things, you know, ranging from circuit design, you know, on what I call the supply side of the research lab, supplying technology to make the GPUs better. And then we have people doing, you know, all sorts of AI, autonomous vehicles, graphics, robotics on the demand side.

9:19And it's just, you know, it's fun to meet with these people. They're smart people, talk to them about, you know, what ideas they're doing. And I try to my job is to get obstacles out of their way. I try to enable them by finding out what's blocking them and remove the blockages so they can do do amazing things. Yeah, so it almost sounds like the chief scientist part of your job is really about thinking about the future and planning for the future. And then the research org is about, hey, what is the research we can be doing now to make the technology and the product offering better for our customers?

9:49Is that a good one? Yeah, it's really right. So the two fit together. And very often, you know, for the chief scientist job, thinking about the future, we try to identify gaps. We try to have this vision of, you know, where we want things to go with, you know, both the GPU hardware, the software, the applications. And we say, why can't we do that today? And on the research side, we try to fill those gaps. We try to, you know, what technology can we develop that will make that possible? Got it. So maybe starting with the latter on the research side, the research org, like what are some of the what are some of the things that are that are most exciting to you and the team right now?

10:22Either areas of focus or specific papers or or bits of research that you're working on right now? Yeah, well, generative AI has to be kind of the most exciting thing going on. And so we're trying to develop new technologies for that and trying to develop some fundamental understanding of it as well. Our research group in Finland wrote a paper a little while back that basically really kind of laid out how diffusion models really work. And actually, the process made how they are applied much more efficient. And so that's the kind of thing we do is try to look at the technology that everybody has jumped on, sometimes without really understanding how it works, and try to dig down and understand what makes it tick and how we can make it better.

11:09We're doing really exciting things across the generative space, both with language models, with vision and video models. And probably most exciting is multimodal models that combine all of that stuff together. And it's fun to watch it happen, and there's a lot of energy there. People were pretty excited about it. Yeah. I mean, on the topic of multimodal models, you know, we're recording this just a few days after the recent announcement from OpenAI with GBD 4.0. I mean, I think that's an example of what you're talking about. Super impressive. Super impressive to see these things come together, right?

11:42You know, again, I'm not an expert on the technology, but my understanding is, you know, this is the same approach that we've seen to these large language models and other types of models over the past couple of years. But now when you put these things together into one, just enables a whole other type of interaction and experience. That's pretty mind-blowing. Yeah. And it also opens up the data to orders of magnitude more. You asked this question about data. Yeah, where do you get that? When you're just dealing with language, there's so much data. But then all of a sudden when you say, okay, let's add in videos and images and audio, now all of a sudden there's enormous amount more data.

12:18And you think about how how people learn and experience the universe, a little bit is by reading books. But a lot of our experiences is visual and is really experiencing it through seeing things. And now our models can do that as well. Yeah. How does how does the research to understand what's happening with these models, whether it's a diffusion based model or a transformer, like the sheer size of these things? How do you go in and understand what's actually happening under the hood? Yeah, I mean, it's a case-by-case thing, and you have to do pilot studies. In fact, very often when we build our big models internally, it actually stops being researched.

13:00It becomes a production task, right, because a lot of resources are being applied. A lot of people are being applied. We're curating lots of data. But before we set up for that, we'll try to do little pilot studies. We'll say, you know, you do ablation studies. Let's take this away and see what happens. to take this way and see what happens. And then we also try to just develop some math behind what's going on. So we can sort of predict that if we do something, what will happen. And then from that, we start developing an understanding of what's going on, what's really represented by this embedding, by this latent space, and being able to anticipate and predict what would happen if we make a change to the model or to the process or something like that.

13:41Right. So like you said, it's a challenge of production. And so even just this research to understand what's happening, just like building any of these models must require enormous scale in terms of compute, in terms of data. I mean, it's like you're building these models on your own from scratch. Yeah, well, we are actually in some cases. But we try – when you put that many resources into something, you have to be pretty sure it's going to work or you have a difficult conversation with Jensen at the end of the day. And so you try to do the little pilot experiments in advance of that so that when you run off a big training run, you have a very high probability of success.

14:19Right. What else on the research side is getting you excited right now? I've heard you talk a little bit about autonomous vehicles. Is that an area where your team is spending a lot of time on research? Yeah, we have one group doing autonomous vehicle research, and they're very closely collaborating with our autonomous vehicle product team. team. There's some exciting things going on there, actually applying foundation models to autonomous vehicles, but both as a way of creating a training environment, creating, being able to write a prompt and as a result, get a scenario that you can then simulate and run your car through, but also having a model that you can use for the perception and planning and prediction of what the other actors in the scene are going to be doing.

15:04And so there's a lot of very exciting things coming together at that nexus of generative AI and autonomous vehicles. Now, there's also a bunch of exciting things going on on the supply side. I mean, we're constantly pushed to say, you know, how can we stay ahead? We, you know, in my opinion, I think we have the best platform for AI today. And, you know, but people, as soon as we announce a GPU, other people can sort of copy what we've done, right? And, you know, in four years' time, they'll have a platform that's probably as good as ours today. So how can we continue to stay ahead? And there's some pretty exciting things we're doing on that front as well in terms of, you know, new number of representations for AI, new ways of handling sparsity of both the weights and activations in these models and just ways of making the platform more efficient.

15:56So, you know, for a given amount of silicon area, given amount of power, how can we get more out of it? Yeah, I've seen, you know, I've seen other companies recently announce, you know, really large chips or, you know, lots of different takes on the architecture. And, you know, it often makes me wonder about, well, I wonder what NVIDIA is going to do next. It sounds like you're thinking multiple steps ahead and maybe already have the roadmap planned out on the supply side for multiple iterations of chips and designs. Yeah, we have to stay ahead. I mean, the things we're going to do for the next couple of generations are already pretty much in the bag.

16:30And I can't, I can't talk very much about that. Of course. Yeah, of course. But in research, we're trying to look then beyond that and say, what, what are the things, you know, you know, three, four or five generations out. Right. And, and it's fun. It's, you know, with, in some sense, it makes computer design even more fun than when Moore's law was in place. Cause back then you made small tweaks and run stuff in the new process and you get a faster CPU. Now, you know, we're getting maybe 10 % out of a new generation of technology. Um, so, you know, most of what makes, makes it better is better computer architecture, better, you know, better design, um, you know, things from the creative process, not from the, the semiconductor process.

17:09So maybe, uh, applying what we talked about earlier with, you know, the explosion of generative AI over the past couple of years back to, um, autonomous vehicles, what, what has impressed you about, you know, maybe some of the advanced advancements in autonomous vehicles, uh, over the past year or two, it seems like on the consumer side, there's been some pretty big breakthroughs. Waymo is now, you know, doing, I think, tens of thousands of trips. Obviously, Tesla full self-driving seems to be getting more and more reliable. Like, are we getting closer to this? And how does the work that NVIDIA does contribute?

17:39Yeah, I mean, it's hard to say. I mean, it's one of these things that's a very difficult problem. In fact, you know, I'm on the record of a decade ago saying that we were almost there. And, you know, a decade later, we're not. And the reason is it's one of these things where you've got to, you know, get that long tail. And it's really a problem of chasing the rare cases and making sure they're handled well. You know, I think, you know, the leaders in this field, the Waymos of the world are doing a great job of that and involves an enormous amount of data. And it involves, you know, having a great discipline and a real safety culture to make sure that you really made sure that under situations that you have not anticipated that the vehicle is going to respond correctly and everybody's going to remain safe.

18:31So it's, you know, we've done some really exciting things that I find interesting from a technology point of view with these generative models. And, you know, we've played around with this, you know, various approaches to the architecture of the vehicle, you know, where we have the classic stages of perception and planning and control and the like. And then we also have versions that are end to end. And sometimes we try to combine those where we have the usual stages so we can reach in and both observe and control. But we also have trained them together. So we've actually trained our perception with a loss function that is conditioned by what the planning is.

19:13So it's a perception tuned to what it's going to be used for. um and so they're exciting things in the technology there but ultimately it's it's a tough game of chasing down the um the the rare cases and making sure you can handle them well um and it would be a much easier task if it was only autonomous vehicles on the road right you have to deal with these pesky humans that are out there that do difficult to predict things right so you're saying the autonomous vehicles they'll do predictable things what humans will do is completely unpredictable and if we see something or You can predict them, but every human is different and they may act in a different way.

19:50So you're predicting what the most likely human would be. And you even try to develop technology which characterize by observing the actors, both the cars and the pedestrians and the like, it tries to characterize them. That one is distracted. This one is aggressive. That one is about to fall asleep. And then predict what they will do based on that characterization. But even then, it can be hard. That's interesting. So the models will characterize drivers down to the individual car and say, this driver is this type of driver, this driver is that type of driver. Yeah, you need to characterize what a particular car will do because they're not all going to do the same thing.

20:25And you can observe, just like you would out on the road and observe and see what you think they're going to do. That's fascinating. That's fascinating. You mentioned something earlier about autonomous vehicles, how you're even using large language models to invent scenarios and test cases, I think I remember you saying. I think I read somewhere that that NVIDIA has basically been applying generative AI to many stages of chip design. Even talk to me a little bit about that. Like, how is NVIDIA actually leveraging this technology to help make chip design more efficient? Yeah, that's a great question.

20:59So we have a bunch of projects to apply AI to chip design to make, you know, to basically eat our own dog food in some sense. Probably the most exciting one is one where we've taken large language models and then specialized them with what's called domain-specific pre-training. So we basically take a lot of data that we have, our entire repository of previous GPU designs, tests, design documentation, and train up the model on this. And what we found is that then we can get a model which is much better than just a general model, even a huge model like GPT-4. We can take a LAMA-70B or something, train it up on our own data, and it's better than a larger model at a number of tasks.

21:46And the most important ones are tasks that assist a designer to make them more productive. One thing we found is that junior designers tend to use a lot of senior designers' time asking questions. and it's part of the process of becoming part of the team and learning how GPUs work and all that. But now we can have them ask the model a question and it gives them pretty good answers, which not only makes them more productive, but makes the people whose time they were using to answer the question more productive. These models have also been very good at summarizing bugs. So you get a bug report that may be many pages long and it's a bunch of logs out of some test case with where the test went awry.

22:25And it can summarize that bug now And in many cases, also, you arm somebody to say that there's an action required by a particular designer to now fix the bugs. That makes the process go better as well. And in some cases, we have the models writing code, but we more often will have them writing test code or code that configures a particular design tool to do something than writing the code for the GPU itself. And then there are also applications where we take this technology and we use it as part of the design process. One that I particularly like is we've developed a graph neural net that can take a circuit design and predict what the parasitics are going to be.

23:06And this is a huge productivity gain because, you know, it used to be the circuit designer would draw the circuit, hand it off to a layout designer. And a couple of days later, the layout designer would finish the layout and you'd extract the parasitics and the circuit designer would find it. It doesn't work because the parasitics are worse than I thought they would be. And they would have to try something different. And so the design cycle was, you know, a couple of days around that loop. But with this tool, which, you know, it doesn't get it exactly right, but it's very good at predicting what the parasitics will be.

23:37It's now seconds around that loop. So the designer, you know, draws a schematic, predicts a parasitic, runs a simulation. Now they can iterate quickly while they still have everything in their head about what they were working on. You know, another really cool one is we apply reinforcement learning to designing the adders in our GPUs. This is a critical circuit. And it's also something that people have been thinking about really hard since the 1950s. And so there are textbooks written about, you know, how to design, you know, good adders. And it boils down to this problem of structuring a tree that does what's called a parallel prefix calculation.

24:10calculation. It's doing a running sum across the carries of bits to decide whether you have a carry into a particular bit of the adder. And in this problem that people have been beating on since the 1950s, it turns out we applied reinforcement learning. We treated it like an Atari game of where you put the next carry look ahead node in the tree. And it wound up beating the best known techniques by a substantial amount as well. And so this is something that's applied now to the design of the arithmetic circuits in our GPUs. You know, another neat one is the productivity increases. Every time we move to a new technology, we go from five nanometers to three nanometers to two, we have to redo the entire standard cell library.

24:54Or even within a particular node, if we are targeting, you know, a different foundry, we have to redo the standard cell library for that foundry. That used to take a team of about 10 people about nine months. Think of 90-person months. Now we have a reinforcement learning program that basically designs the standard cells, and it achieves a higher quality. So the average cell is smaller than the ones designed by the humans and better in a few other metrics as well. But it does it in an overnight run on one GPU. And so that's a great example of applying this AI technology to making the GPUs better.

25:34It's really cool. I mean, you're obviously hearing stories like this from all different types of companies about how they're leveraging AI. And so it makes sense that NVIDIA, the company that has many ways inventing and creating AI, is leveraging AI to make AI more efficient. It's really cool to hear some of those examples. Maybe let's get into the team a little bit. You mentioned, I think you have like several hundred researchers with PhDs at NVIDIA. Is that right? I read that somewhere. Yeah, it's about 400. How do you recruit and build a team like that? I mean, these are some of the smartest and brightest people in the world.