Turing Award Special: A Conversation with David Patterson

26:19So you could have given a pool of deep use and maybe accelerators, database accelerators make more sense is that you only need to use memory from the pool rather than you have to justify the cost of having DRAM that is only can be used in these narrow situations. So I'd say the bottom line of that paper is pooling of DRAM is realistic. We should be thinking of being more memory focused, thinking of the problem of how do we get access to the data rather than thinking of it as computing is the focus of what's being done. yeah well and i wonder and you said you're not as familiar on the software side so redirect me if i'm going off into an area that isn't in your domain but does that require the software explicitly managing the memory because i feel like a lot of right now at least higher level software can optimistically ignore memory hierarchy and then it you have to start being aware of it when your performance falls down you're saying oh i've broken cache locality or something like that but But when you have this...

27:20Yeah, this is a little different. So how do computers work, right? It's how could you have gigahertz processors and this DRAM that takes 100 nanoseconds? How do you make that work? Well, caches were invented and we just keep sticking in levels of caches and trying to hide, you know, give the illusion that you have this incredibly large memory and it's incredibly fast. And it's up to the hardware memory hierarchy to hide that from the programmers. For some tasks that works, illusion works pretty well, but for others it doesn't. And so programmers do need to be aware. But I think this one isn't so much the memory hierarchy as it is just the memory capacity, which I think programmers have had to worry about for a long time.

28:04But this idea that, well, I think the argument is if you re-architect your software to be aware of the pooling of memory across different servers, you have this ability to be able to get a lot better cost performance for data intensive applications. Yeah, that would make sense just in terms of being able to trade off more memory versus CPU and not having to stack them. I was also wondering if you make memory ownership visible to the software, could you hand off between different processes without having to do a mem copy where you basically say like yeah here's your address go yes that's kind of this distributed shared address space i think there's i believe in the c there's a whole bunch of cxl protocols but i think the latest cxl protocols will allow that type of access to the shared memory but it's different is you know it's shared between different servers and that used to be impossible thing to do and the way that servers are constructed there's just a limit to the amount of DRAM you can stick under one-on-one server.

29:08But using CXL, you can get this illusion of having a much more memory for individual pieces. So it's kind of a practical way to get a tremendously bigger memory footprint without a gigantic cost of some kind of supercomputer. Yeah, I remember earlier in my career, I was involved with a lot of high-performance computing and stuff and shared memory models. If you wanted to do it across lots of CPUs, you had to go to like SGI had these like mega things or something like that. Anything else you were passing messages essentially over a network bus in some way. So it sounds like in some ways this could enable shared compute, shared memory style programming models, but across some number of servers.

29:48Yeah. So that was John Hennessey and I. So John Hennessey did this project called Dash, which was a big shared memory processor. And this was this kind of switch to parallel computing or that was kind of in the air. And we had a contrasting project at Berkeley called the network of workstations. And this is in, I guess this is in the 90s, I guess. I think that's right. And John's thought, well, the hardest problems with parallelism is programming. And if we keep it a coherent shared address space, that's going to make it easier. And so we at Berkeley, what we said, well, that's one way to go. But we think independent, it's so cost effective rather than have these big servers with this, you know, interconnect to be able to provide that.

30:34It's tremendously more cost effective if we could use, you know, we said workstations, but PCs, put a bunch of them together, use local area networks to connect them. And that's going to be the cost performance of that's going to be amazing. And so how did that settle? Well, what happened is, you know, the Internet came along and Internet services wanted to, you know, the parallelism was based on number of people, not on a single program that you had to parallelize across under processors. So it was a huge throughput demand. So what ended up happening is the, you know, the network of workstations.

31:07What is that's how all Internet services standardized around that? And, you know, John's model was more efficient in DRAM, but the price of DRAM at that time was so low and you could scale up. And it was also very reliable that a single server could fail and the software could keep working around it. So it was much more reliable, much more scalable and much cheaper than the coherent address space model that those SGI machines did. But it was it's an example. How do we settle debates in computer architecture? Well, we get companies to spend hundreds of millions of dollars or billions of dollars to put it in the marketplace.

31:43And then we fight it out. Oh, that's how I won. That's how we do it. Absolutely. Well, and as you highlight, a huge amount of the demand and compute growth for many, many years was internet driven, essentially embarrassingly parallel. You just split it out across things, stateless servers that maybe access some shared state. And then you've got databases or things like that that have to actually have that coherent view of state. Yep. Now we're at a scale where that's not super cost-effective, needing this pooled memory for databases. Are there other domains? We've talked about machine learning, particularly at inference time being very memory intensive.

32:18Databases are another one. What other domains do you think this type of memory centricity is likely to make a lot of sense? Yeah, that's kind of the question when people talk about domains. So my examples have always been, besides me, data analytics, databases. interfaces I don't think I have any other areas that are obviously memory intensive my guess is that'll be increasingly the problem going forward for lots of applications but it's hard to I'm not sure I don't think I have any great example no worries so looking at this and looking at you know so much is being driven right now by machine learning and as you highlighted like folks doing machine learning, they'll take as much as you can throw at them.

33:03I saw something recently where like one of these machine learning coding tools, Cursor was saying, hey, Anthropics throttling us because they are literally out of GPUs. They can't run enough inferences, right? Like everybody is struggling on that domain. But I'm curious, what do you see as like the big unsolved problems for the next five or 10 years? Well, one of the questions I had is when, you this machine learning organization. It's like, well, we'll see how significant this is. And it's very hard to see when you're in the middle of it, whether this is a paradigm shift or not. Retroactively, it's much easier.

33:42Like a decade later, you look back, wow. And in my career, the microprocessor, the internet, maybe mobile phones, smartphones. And when you look back, you say, wow, that was a giant change in our technology base. So now that I've been there eight years, I think this is one of those things. So agree, this is a big paradigm shift. I think that's becoming more and more clear. What are the still big unsolved problems here? Yeah. Well, because we're at this, well, I'd say a couple of things. We're not at an upper bound on intelligence. We're not at, you know what, that's good enough, right? I think with intelligence, I don't know if there is going to be an upper bound, but we're certainly not there yet.

34:29It's still there's things where it screws up. There are things where we'd like it to be a lot smarter. So I think on the machine learning side, can we figure out how to deliver something useful that people can depend upon? You can ask it questions and trust its answers. And can we deliver that economically? And can we reduce the carbon footprint of our solutions as well? And which is that's a topic I ended up being involved in quite a bit, the carbon footprint of this. So I think I think if we only focus on the part of the industry that's machine learning, there's giant challenges there. What's the best architecture?

35:12Can we improve the algorithms to reduce the cost of training and serving? can we come up with architecture ideas that can reduce and hardware ideas that can reduce hard service there's just a huge set of problems there so it's this very exciting time like i said where my career have been kind of boring times this is not one of them and what's great is if you're a researcher then if you've got a good idea people are anxious to hear it there are times when everybody's making a lot of money and it's kind of boring is it's hard to get people's attention because they're making a lot of money and they don't need to change anything that's not where we are today.

35:46So it's a very exciting time if people are interested in hardware to get into this space or algorithmic advances. I'd be curious to dig a little bit deeper on the carbon footprint side. I saw something recently that Microsoft is projecting they're spending $80 billion on data centers this year. I haven't seen similar numbers from Google and Amazon, but I know that they're also deeply investing in capacity for this. And there's a lot of worry out in the world around like, what's the environmental impact? What's the energy impact? So can you talk a little bit about the research you did in that domain?

36:20Yeah, I know I've worked a lot on this topic. So I got started in it because I guess in 2021, there were papers coming out. They were making alarming claims. And I would ask my friends in machine learning at Google, like, is this true? So there was a paper that came out in IEEE Spectrum, which was the flagship of IEEE, one of the big organizations in the world. And it said, it was 2021. And he said, I think it's by 2024, training a model would cost$100 billion and produce as much emissions as the city of New York in one month. Like, oh, my God, is that true? So we started investigating it and we found that there was a particular paper that inspired these concerns.

37:08And it was actually a paper by a group at the University of Massachusetts who were trying to guess what it costs for one of the Google projects. So in machine learning, there's a thing that you do to try and find better models, more efficient models. And that's called neural architecture search. So you're using kind of machine learning to find better models, more efficient models. And so it was actually a more efficient version of the transformer model called Evolve Transformer. So they tried to estimate what was the carbon emissions of that search. And because they didn't have internal Google information, there's a thing that we call in that paper, we call the 4Ms, which affects the machine costs, which is the model itself.

37:55there's the machine it's running on and then the third m is mechanization which is an m word that means how efficient your data center is and the fourth one which was the big surprise was max because the the cleanliness of the energy is highly dependent on geographies where you are if you're near a hydroelectric dam and you're solar or wind you know the energy is going to be much cleaner. So those 4Ms. So they didn't have access to the 4Ms. And so they did an estimate based on averages. So that was fine. So they were a little about a factor of five higher than Google was because we optimized a bunch of those things.

38:35But unfortunately, they misunderstood how we did the neuroarchitecture search. So they were off by another factor of 18. So we did a small proxy model to search the space and they assumed we did four more. So the paper itself that everybody based their work on was off by a factor of about 90, too high. But then they misunderstood what the paper was about. They thought it wasn't searching for a new model, which they do that occasionally. And then they put the model out, they publish it and put it on GitHub and people download it and use it thousands of times. The people who read that paper thought that was the training of the model.

39:09And so not surprisingly, it takes more than a thousand times as much energy energy to find a model than to train one. So we multiplied together that the conclusion was too high by 120 ,000. And so you end up with the claims like it's going to cost$100 billion in New York. So now the problem was, now that we knew that, how do we get the word out? So there's no real good mechanism because this paper appeared in the conference to say, hey, by the way, remember that paper everybody's citing? It's off by 100 ,000. So I tried to go around and give talks in life, but it's basically an unsolved problem.

39:44So that's got me into the space. I think what happens today is people just have a hard time because you hear these, you talk about tons of emissions. It's hard to put it in perspective. So there's something called the International Energy Agency, which is like an organization for 50 countries. And what they said was recently, how much energy is going into data centers? Well, it's about 1%. 1 % of all the electricity is going into data centers. That doesn't include the internet, doesn't include crypto. This is kind of Amazon and Google data centers. And then AI is only a piece of that. Our measurements were that it was about 15%.

40:23So it's less than a quarter of 1 % is AI. That's where we are today. And so this IEA agency, they said, looking into the future, even with strong growth, it'll get bigger, but it's compared to the other things that are going on, it's not that big a deal. So like air conditioning, air conditioning is going to grow a lot in the next five or 10 years, and it'll be much bigger energy driver. And also they talked about high electrical things like aluminum plants, just plain old economic growth was going to be another thing. So when you put things in perspective, it's going to grow, but it's small relative to other things that are going on.

41:06But it's very hard to communicate that kind of message today because people will notice things. Because what happens is that's the worldwide average. But in particular regions, if somebody builds a lot of data centers in the same region, then that utility company could be taxed. And that's new. So you'll see some cities where, wow, they want to build a lot of data centers here. We have a limited energy supply there and that's going to tax it. So it's kind of a local problem. But if you have the big picture, it's probably not a global problem. But nevertheless, the growth is so much and companies want to be able to build these data centers.

41:43They're investing or they've made plans for nuclear energy. So these so-called small modular reactors or even fusion, which is hard to believe. I saw that. Yeah. I'll believe it when I see it, but that would be incredible. Well, yeah. The other thing that's in the press all the time is quantum computing. And one of my friends said, you know, there's a chance that fusion is going to work before quantum computing. So, fusion's been one of those that we're going to solve it in the next 10 years for about the last 50, right? Right. But the people are more optimistic in the last five years that we're five years away than for decades we've been 10 years away.

42:24And they are using ML to help figure this out some. But it does seem like I can't believe it's going to be real. Obviously, if fusion happens, it'll be amazing. But the small – I have one of my nieces is a nuclear engineer. The small modular reactor thing, you know, we've had reactors in submarines for decades. They're training 18-year-olds to operate, you know, nuclear reactors. So there's another part of the space that technically could be very helpful. But, you know, the kind of the sociologically putting reactors into neighborhoods may be a bridge too far. Yeah. No, absolutely. And I think, you know, in that power domain, so actually, first off, let me restate in case anybody missed it, right?

43:10If you're worried about the environmental impact, the paper that you're probably tracing your worries back to is off by 100 ,000. And the papers, you know, in academia, we count citations. Its citation rate's gone way up. Our paper says, by the way, there's a little flaw there. We're way behind. We're not catching up. So that's kind of, I was talking to a guy who does a data center and stuff. And he says, that's what it works. If somebody makes a mistake in a calculation and people read about, wow, I didn't know it was that bad. And then that's news. And once it's news, it is very hard to fix.

43:44Even today, if you were to search Google about what's the cost of training, you might get this so-called same as five car lifetimes. That's that original paper that's off by 120 ,000. But Google will help you find that erroneous result. So another topic that I like to talk about with folks who you mentioned, right, you have five decades of experience here, almost as long as the industry. And you just said something. You've seen boring times. You've seen exciting times. And we're in one of those exciting times. I'm kind of curious with that perspective, what do you see going on in the evolution of the tech industry right now?

44:22Like what is changing the most? What's staying the same? What are you excited about? Well, right now it'd be like asking about right after the microprocessor was invented, which I asked some of my grad student friends. I was a grad student when it was invented. They told me I said I thought it was a big deal, which I'm very happy that I said that. Got that one, right? Yeah. Because it was kind of a toy. Actually, in the early years, there was real computer conferences. And also they had these kind of pretend computer conferences where these toy computers were at. They weren't real computers. But yeah, so because it's right then, it's kind of hard to see around the AI stuff.

45:03And what's got nothing to do with AI that's very exciting. I think the quantum computing stuff isn't really about AI. You know, quantum computing is like zero degrees Kelvin almost. And, you know, AI success is machine learning, lots of data. So quantum computing and lots of data don't really fit together. but you know i think what jensen said was pretty plausible what did he say it was 15 years is too early 30 years is too late 20 years and and i think so it may have a big impact on pieces of computing in decade from now or say but we're not going to have quantum cell phones right this is big science and there'll be some problems that are that it can solve that we thought were unsolvable.

45:50But most things, as far as we know right now, we'll still need regular general purpose computers for many other things, including, as far as we know, AI and machine learning. I'm not a person who thinks that's going to be this gigantic paradigm shift. It'll be an amazing accomplishment to do it, amazing scientific accomplishment, but I don't see it as changing the industry. We just don't know how far AI is going to go, right? It's how much of our own technology will we throw away and replace with AI? So for the people closer to AI, like in the vision community, have friends in the vision community, once there was that tipping point with AlexNet and ImageNet competition where it won, and within two years, everybody had to abandon what they were doing yet.

46:36They completely changed all their courses. So it just, everything went away and it became machine fun. So it was absolutely revolutionary. And that, how much of computing technology will that happen to? Or where are the walls and where it can happen? Because when it does happen, it's hard to see the limits or it's hard to see how far it can go. So we don't know where that wave of AI, how far it's going to spread into the computing field, but it's definitely affecting lots of pieces. People are talking about using large language models to make it easier to do what's called register transfer language of hardware design, which is, you know, that's not something I would have seriously thought about.

47:17So it's hard to ignore how pervasive AI is going to be, but we don't know how wide. But if we knew the answer to that, I think we'd have a better understanding. But there's this chance it's just going to infect all of our underlying technologies doing things very differently that we've done from the past. And so it's hard to predict given how extensive this change we might be undergoing in the next few years. Yeah, absolutely. I feel like it's already dramatically changing the way people do software development. You can write software with this thing and it does a pretty good job and can speed you up dramatically, but it also shifts the types of architectures you want to write, which then changes the education path of what type of software is good software and how does at work?

48:06Yeah, I was actually involved in another paper, kind of out of my depth, that some friends wanted to write. And it's about shaping AI for the public good, is the theme. And I think the title is Shaping AI for Billions and Archive Paper. And there was actually, I did an article in Economist about that topic. But I think the way we think a great use of technology is to act as kind of assistant with the human involved. So there's a human expert and it's working in conjunction with the human expert, which makes the human more valuable economically. So rather than getting rid of hundreds of programmers, let's make hundreds of programmers tremendously more productive.

48:47And in that paper, it talks about, you know, people worry about job loss. Well, in economics, there's this question of whether it's the product you're doing is elastic or inelastic. And what the question, what that means is, if you make it more efficient and cheaper, what's going to happen? Well, if it's agriculture, you're making food, there's almost so much food that people can eat, probably the number of jobs will go down. But in topics like software, where in the past, it's made the number of jobs go up, even though you bring the price down, there's tremendously more demand for it. So we think there's a, you know, if research focuses on elastic fields and improving human productivity, that could have this very positive effect that's going on.

49:30But I certainly see it right now that if you're an expert and you can use AI to make yourself much more productive and you can see when it screws up, it's a very powerful technology. On the other hand, if you're a novice, you know, how do you avoid the hallucinations or the mistakes it makes from having you do something that's embarrassing. So it looks like that's where it is right now. Who knows where it's going to be in a few years. But I think it's this reinventing how we write software and how we design hardware is an example of this extensiveness of where AI might go. Yeah. And I think you're spot on, right?

50:06We're entering a period of software abundance, but there does not seem to be... Software is like dreams made life. We have no shortage of dreams. There's always more software to write. To me, it's this, I talked about this, no, like that's smart enough. We don't need to go any further. I think same thing for software quality. I mean, the things over my career, the embarrassing stuff is how insecure our technology is, you know, enabling Purple Hill herd people in Lithuania to steal money from your grandmother, right? We helped make that happen. So if we could make a serious dent on the security problem, that would be amazing if that could happen.

50:42And maybe AI can help us. I mean, that's an example of a quality improvement that would be wonderful for our field. So we're coming to the end of our time together. Is there anything we haven't talked about that you would like to share with folks? Let's see. Upcoming, I think one of the in the embodied carbon space I'm very proud of, I think within a couple of weeks, Google's going to publish a paper that talks about the two parts of carbon footprint is when it's operating and what it costs to manufacture. and people have speculated a lot about how expensive it is to manufacture chips. We're going to release a paper or a blog, I think.

51:20We're going to have done what's called a life cycle analysis. And the title of the paper is Cradle to Grave. So the whole lifetime, we'll have the data out there to show how expensive it is to build these AI accelerators like GPUs. What are the carbon emissions associated with it? How does the operational piece compare to the manufacturing piece. And that'll be the first time that data's out. So it was nice that Google has both environmental experts and computer experts, and we collaborated together. This, I hope, will come out over the next month. And then reading about thinking, we tried to make a thoughtful paper under the assumption that there's people who are open-minded and would like to hear about AI.

52:00It's less clear to me now based on the kind of Twitter-like instant reactions based on one sentence in the paper, how many open-minded, thoughtful people are out there. But I think, hopefully, if you were to read about that, it'd give you something to think about. We try not to talk about the upsides and the downsides, not just the upsides of AI or the downsides, only the downsides of AI. Yeah. There are a lot of outrage merchants out there. Yeah. Yeah. Andy Konwinski is a former PhD student. We did this paper a long time ago on cloud computing where it was not as controversial, but it was controversial when we wrote the paper.

52:41Everything's cloud computing. There's nothing there. And so we kind of explained it. What are the upsides? What are the downsides? How researchers could make it better? He wrote that paper. He said, we need to do that again by AI, like you said, because of the outrage merchants. So I don't know if we're making a dent in the conversation, but we gave it a shot. I hope it gets better uptake than the carbon footprint corruption. Yeah, that would be a goal. Can we get as many citations as the paper that's off by 100 ,000? If we got there, that would be success. Awesome.