Turing Award Winner: TPU vs GPU vs CPU, Computer Architecture, RISC vs CISC | David Patterson

28:21When Google came along, they decided it was a clean slate. They didn't care about graphics. So at the heart of neural networking is a matrix multiply. That's the thing. So they designed a processor for at the time, microprocessor at the time, had a giant matrix multiplying unit. That was the main thing. And then they threw a bunch of stuff out that they didn't need. So a lot of general purpose computing is what's on the chip is maybe three levels of caches to try and have. So you don't spend all your time going to the relatively slow memory. Well, for machine learning, they knew when their memory accesses were so they could schedule that.

28:59So a hardware cache didn't make any sense. They would just have a memory that the software understood and would transfer in time. So those are some of the innovations. Also, they innovated on the floating point format. You didn't need hot scientific computing, cares a lot about precision. Most of it's done in 64-bit floating point where the exponent is less than 10 bits and most of it is the precision. you know, the fraction can be 50 some bits. Well, what they realized for machine learning and AI, they don't need all that precision. They needed the range. So, Google did the first floating point format where the exponent was bigger than the fraction.

29:40That was a radical idea, that so-called brain float 16, the part of Google that was doing this was the brain research group. So, they brought out this architecture. They had this narrow floating point format. They had big matrix multiply unit. It only had one processor in it, unlike the other ones. It was just dedicated for machine learning and it just kind of blew the doors off of everybody in the field. It was like 30 times better at inference than the contemporary GPU and 80 times better than CPU by having this dedicated one. So when Google made this announcement at their annual retreat a year or so after they had deployed it inside, it just shook everybody up.

30:26Intel started buying companies. NVIDIA started modifying the design to be much more enhanced machine learning. And then a bunch of other competitors started, hyperscalers started doing their own efforts. So I think that was the watershed. I think the TPU announcement was the watershed moment there.

30:47The Peterman Pod Host:So it's almost like levels of specialization, like the CPU is the most general. Yeah, if we could still, you know, if we still had Denard scaling, you know, Moore's Law and Denard scaling, where would we be today with general purpose processors? We should have 100 terahertz, you know, microprocessors. If we could build 100 terahertz microprocessors, that's what we'd do. GPUs would still be a niche, you know, if we could do that, you know, it raises all boats, that would be fantastic if we could do that. But that's long in the history. We haven't been able to do that for 20 years. And so there were probably two or three gigahertz microprocessors in 2005, and that's kind of where they barely improve today.

31:33So we can't do that. So there's still areas where it's important to have CPUs. They're the right, you know, you need operating systems, compilers and things. They're the right solution. But things have been specialized. And then now in terms of industry terms, huge emphasis is being put into the AI accelerators or just AI in general is where the money is being invested. And so the question for any processor designers was, how does my processor design fit into this AI universe? And what should I do? How should I change it for this important application?

32:12The Peterman Pod Host:So you mentioned that kind of Moore's Law is slowing down. And I watched this other talk from Jim Keller saying that Moore's Law isn't dead, but is it controversial to say that it's slowing down? No, I mean, not to real engineers. Moore's Law is very simple. It says in a chip, the number of transistors will double, originally said every year, and then he minuteed it to every two years. Just look, do the chips, are the number of transistors doubled? And no. Now, I think what people assume that means if we are no longer on Moore's Law, that technology is not improving. That's not the same thing.

32:54It's not improving at the rate that Moore projected. And what was amazing about Moore's Law, which lasted 50 years, is that it guided the investment of semiconductor manufacturing. We need to deliver on doubling transitions every year or two. How are we going to do that? How are we going to build the equipment to do that? So it was a guideline for the whole industry, which was kind of remarkable. So what we are doing now is there's pieces of the technology to get better, and there's pieces that don't improve at all. So one of the big pieces, important pieces on a chip is the static RAM, SRAM. So that's hardly improving at all.

33:35logic gates still continue to improve that they are the gates are getting better so if you're doing like uh adders or multipliers those are getting there so pieces it's not uniform improvement anymore and we're also going to more exotic packaging to be able to deliver it uh so people not uh it you know forever it was a single ship was the best way to package everything fits on one chip and that's what we're going to build. And now there's these ideas of chiplets or packaging multiple chips together, the latest GPUs. And I think the latest TPUs has actually two, what are called full reticle design dies package put into a package.

34:18And, uh, so a full reticle design is, uh, the maximum you can build on a, well, on a semiconductor, it, they have a step and repeat motor and there's, you know, in the old days, you'd have many chips inside that. Now there's just one. So two maximum reticle designs form a node. So they're using packaging. So if you look from outside, you can say, well, look how many transistors. It's, quote, Moore's law is continuing. But if you see what's inside the chip, that has tapered off. And I think part of it is for a fair amount of the industry. If you were in a semiconductor manufacturer and people ask you what you do is I make Moore's law.

34:58I sustain Moore's Law what I do and if you've been doing that for decades and somebody says Moore's Law is over it's like my career is over so I think there's an emotional side of it but you know just look at the data the data doesn't back up what Keller says but I also didn't say that the technology is not improving the technology is continuing to improve and specifically domain specific architectures but if you look at the general purpose architectures or even other memory technologies. You know, it used to be the DRAMs would improve by a factor of four in density every three years like clockwork.

35:34And now it's maybe 10 years between factors. So you can see plenty of evidence that Moore's law no longer applies.

35:41The Peterman Pod Host:Is there some new version or some analog to Moore's law that is kind of guiding, you know, microprocessor design and architecture these days? I guess a question is both NVIDIA and Google are continuing to deliver much faster processors for machine learning, tremendously better. What are they doing? Part of it is what I said about packaging, you know, to be able to get more transistors and you keep them closer together because distance matters. Part of it is innovating on the floating point formats. So, you know, unlike the supercomputers of 64-bit, it's not even 64-bit, not even 32-bit, not even 16-bit, but 8-bit and 4-bit floating point is going on.

36:33So narrowing of the data types. The, you know, having kind of what's called the matrix multiply instructions. So these powerful units that are put in there that can do special purpose applications that are very important. making those bigger and faster. So those are the things that are going on, but we don't have this simplifying guideline kind of underlying all this. You have to be aware of each piece of the technology, gauge how fast it's improving, whether they can deliver on what's going on, and then you assemble that together and make your bets. There's a paper that we've just got approved that I think we're going to put on archives soon, which is talking about the Google TPU line.

37:24And pretty remarkably, the Google TPU line, particularly for training, has stayed pretty constant in the basic architecture design. Things have gotten bigger and faster. But if you look at the design of it going back to the first training TPU, that block diagram still works all these, you know, a decade later. So the people who designed that in 2015 or two did a really great job. But the main components of a big matrix multiply unit, the so-called high bandwidth memory, a vector unit to go with the matrix unit, the basic building blocks are still there.

38:06The Peterman Pod Host:I think in your career with the CPUs, benchmarks played a huge role in measuring which computer architectures were performant and not. In this new space of floating point operations for AI, is there a benchmark that people use for GPUs? And can you also apply it for TPUs? I and some friends were helped involved in what called the MLPerf effort. So it was inspired by the spec CPU effort with the spec benchmarks that there were these competing companies and they'd all make claims about theirs was better than the others. And they realized that wasn't good for the industry. So they agreed on a set of benchmarks.

38:48So the ML perf effort that's run now, but what's called the ML comms, is an attempt to do that. If we're going to do this comparisons, let's not argue about what the benchmarks are. So that's a serious effort in that. Kind of interestingly, what's happened, I'd say, is because there's two pieces to the design. There's the machine learning libraries that are to implement a lot of features that you need for these applications. And the libraries will even be rewritten for specific applications rather than just having general libraries. This has turned out to be a big advantage for NVIDIA because they have a large corporation with lots of people that are available, lots of engineers who could build these libraries for them.

39:39So it's turned out not quite a, it's not a neutral evaluation, right? It's the architecture plus the libraries that go with it and the compiler too, but specifically the libraries that you tailor each time. So NVIDIA brings out, when they announce a new architecture, they create a new set of, they modify the libraries to run that really well or to run applications really well. So that's a powerful combination. Kind of in business terms, people refer to NVIDIA as having this CUDA moat. And part of it is CUDA, the programming language, but a big part of it is the libraries that NVIDIA makes. So this has made it difficult for startups to be able to compete with NVIDIA, partly because, you know, they didn't put enough emphasis in the software and partly because, you know, NVIDIA just has many more engineers than they do to be able to tailor the libraries.

40:42So it's the libraries plus the architecture that's this powerful advantage why most people do things on GPUs. Now, Google has been able to develop their own libraries. They don't have as many engineers. They use compilers more than I think NVIDIA does. So they have a set of libraries that they can do. But up until recently, Google has, it's all been internal. It's just for Google to use, or you could use it being the cloud. But these startups have to have a difficulty. As a result, that's why the MLPerf hasn't been as popular. Not everybody runs them because NVIDIA runs them really well, really better than everybody else.

41:29And the startups have a hard time showing off what they can do, given they don't have the engineering effort going into libraries that you need to do well in MLPerf.

41:39The Peterman Pod Host:You gave this popular talk about how to have a bad career, and it's kind of the negation of advice. to kind of have a good career. And I was just wondering if you could kind of summarize maybe the top three things that you kind of think people should take away. Yeah. So, yeah, well, there's been a few things. So the story is early in my career, my friend, a good friend and I said, how can we teach grad students how to give a good talk? And we thought it'd be funny to explain how to give a bad talk. And then if you didn't want to give a bad talk, this is, so this is how to do it badly. And if you don't, here's the things you do not to do it badly.

42:17And so then I later did a how to have a bad career. That's pretty much focused towards academia, you know, so for researchers to be able to do that. I later did a how to have a bad, how to build a bad research center, how to build a bad research lab. And then recently I've been giving how to give AI a bad carbon footprint. That's my latest one. But I think the thing that might be more relevant is at the end of my talks, I would kind of reflect on my career and talk about lessons learned. So I've written a paper called, I think it's Life Lessons from the First Half Century of My Career. I wrote that recently.

42:56And so those are divided up into kind of career advice and personal advice there. On the personal side, I'd say if you have a family, you know, make sure your family's first, keep your family first. The technology we've invented makes it really easy for your, you know, to take your work home with you and not pay attention to your family. Uh, I had a, when I was first here at Berkeley, uh, I gave a senior faculty member a ride home, dropped him off late at night coming up to Silicon Valley. And he said, well, Dave, if I had to do it all over, all over again, I wish I'd spent more time with the family.

43:32And I never wanted to say that. And nobody in their deathbed says, you know, I wish I'd spent more time in the office. So, so you gotta, you know, whenever it's easy to kind of, you know, let the, let your career overcome the needs of your family, but you just, you gotta just remember that. I would say in my life, kind of growing up in the fifties, you know, the idea was to be happy, you had to be wealthy, but those are actually two different goals, wealthy and happiness. So I always made decisions towards happiness versus wealth. And I felt very good about that. And even now, why would I pick something that made me wealthy and unhappy?

44:15Why would you do that? And in the field that we're in, it turned out there was a lot of wealth to go with it. So optimizing happiness didn't mean you had to suffer beyond that. I think it's important to have fun personally. I think when you're a kid, you don't have to tell kids to play. But as a adult, you get so busy, you don't think you have time to have fun. But, you know, you only get to do this once. So I play soccer. I run my bicycle to the interview. I lift weights. I body surf. I, you know, do things with my wife and family and my sons. So it's important to have fun. I think on the career side is one of the pieces of advice is by Akanthe.

45:05He wrote this book, The Habits of Very Effective People, I think. And he has a little quadrant and he divides it of urgent and not urgent and important and unimportant. And kind of there's so many things in our technology like email and texting for you to focus on the urgent things. but you really shouldn't be spending a lot of time on the unimportant and urgent things. And it takes self-discipline to set aside time for the important non-urgent things. But if you don't block that out, you can just not have time to do anything. I get to see other people's calendars at Google, and there's managers that every half hour from eight to six, five days a week are scheduled.

45:48And I don't see how you have time to think and reflect on things like that. Other career advice things is I remember I kind of woke up one morning and it was like God spoke to me. I was like thunderstruck. And it says it's not how many things you start, it's how many things you finish. It seems relatively obvious, but, you know, that's not the way I was acting. I had they had many things going on. But after that, I was like, there's one main thing I'm doing at a time. So when Hennessey and I wrote a textbook, that was the main thing. when I was department head here, that was the main thing I did.

46:26I would do some other little things there. And John Hennessey, he wrote a book to a kind of a career advice book based on his presidency. And one of the things he said in there, you know, you're only going to be remembered for the five or six things you've done in your life, not for the hundreds of little things. So to give yourself a chance to have some things you're really proud of, it's better to concentrate on a few of them, hoping that some of them will turn out to be a big deal rather than scatter yourself to many things. But if you're interested, you can, yeah, if you look for life lessons, David Patterson, first half century, you can see the whole list of, I think there's 16 lessons altogether.

47:11The Peterman Pod Host:You mentioned that you didn't want to reflect back on your life and feel like you didn't have enough family time. And yeah, I've, I don't think I've ever heard anyone say the opposite. Why do you think that is? Why is it that everyone looks back on their life and they never regret, you know, a lot of family time? I imagine there's got to be at least one person that says, I spent too much time with my family. My career suffered. Well, you know, what's, you know what's it's it's pretty philosophical i mean what's what's what's life all about what's success right you have to you have to figure out what that means for you i mean if you if you have i don't know if you have financial goals of being a you know a millionaire or billionaire or something like that and that's how you judge your life okay good luck you know it's pretty hard to make it, it's pretty hard to do.

48:09But I think it's the, you know, when I was finishing my PhD, I read this book by Studs Terkel called Working, where he interviewed all these people in his careers and they'd look back and what they liked and what they didn't like, what they felt about their careers. And what I got out of it, the people who worked with people like ministers or teachers or doctors felt really good about what they did with their careers. And the people who did more ephemeral stuff, you know, like, you know, technology things or airplanes or something like that are long gone, didn't feel as good about it. It was the people that they worked with that they really cared about.

48:43And I went out with a retired engineering dean here and to a meeting who I knew. And he said, you know, Dave, as I flink my hair, it wasn't the projects, it was the people that I worked at that mattered. And I thought, I knew that from a long time ago. So that's, I mean, this is kind of senior persons offering advice. I think you're going to care more about the people you've worked with and people you've helped will be a bigger deal. That's part of, you know, one of the other things about personal happiness is they studied happiness. Psychologists used to just study crazy people, but they started like, why are people happy?

49:17And they know what the reasons are, you know. Have a job that you like, you know, have friends and family. Helping other people. Helping other people makes you happy. They know this. Having something kind of either a religious side of it, it doesn't have to be a formal religion, but like contact with nature, the grandeur of nature. But the kind of the list of things you need to do to be happy is well understood. And, you know, there are these hats, and our world is filled with unhappy billionaires, right? If money was the thing that made you happy, why are these people, very wealthy people, so mad about things?

49:56So yeah, this is me passing on advice.

49:59The Peterman Pod Host:In one of your talks, you had, it said, what worked well for me? And it was kind of some reflections. And one of the things in there I thought was unique and interesting, you mentioned that courage was a big part of your career. And I don't hear that too often. I was curious why you say courage is so important in a career. Yeah, I think that's, I mean, that might be partially my personal makeup. But, you know, I was kind of the youngest kid in my class and kind of small. It took me a while to grow despite age, so I was always small. But my parents encouraged me to go out for wrestling. And wrestling gives you physical self-confidence because you, you know, spend years doing that.

50:46I did it in high school and college. And so I think partly, you know, technically, you know, having courage to do things, it kind of goes along with the advice is fortune favors the bold. That's this that goes that's advice is 2000 years old. I mean, it's hard to figure this out. Helen Keller wrote, you know, even even trying to play it safe, you still get caught. And so it turns out you might as well you might fail no matter what. And if you take a big chance, you can succeed. If you don't take the chance, you probably won't succeed if you play it safe. So fortune favors the gold, and it takes courage to do that.

51:28I think also just for me intellectually, I feel like if there's something not right, I need to stand up and confront it. And I think that kind of ironically comes from the wrestling side of my personality, where if I see somebody getting picked on or something like that, I'm going to stand and try and stop it. And I feel that same responsibility intellectually if people are making bad arguments or doing something that we need to stand up and do it. And I feel good about that. The cautionary start about that is because I guess one of my senior faculty members saw this nature of me. He said one of the sayings is friends come and go, but enemies accumulate.

52:11This is an old saying. So if you think about it, you kind of, people you went to high school with a while, friends, you kind of forget them. But somebody who you really does dislike, you never forget that you dislike that person. So standing up when it's important, but be careful when you make enemies because, you know, they're going to stick around for a long time.

52:29The Peterman Pod Host:When you say something's gone bad technically, do you mean someone was incorrect? Yeah, when they're weak, either politically or technically, when it's a weak argument. I think I really like there to be a marketplace of ideas and we hone the ideas by arguing them. And so if it's a specious argument, even if a person is a leader of a company and it just doesn't make sense, I feel it's kind of the responsibility. It's better for the company if somebody stands up and points that out than to just let them get away with it. And then, you know, it's a little bit confrontational. But, you know, as long as people all agree that, you know, this is for the greater good, we need to get the right ideas out there.

53:22And so let's argue about the ideas to see, you know, polish them to make them stronger. I think that's important in science and engineering and kind of in life too. There's a lot of stuff going on right now in the country that is worrisome. And I've certainly stood up and wrote op-eds about things that I think are wrong and need to be corrected. And if people are afraid to do that, it's hard to be optimistic about the future. of people are afraid to stand up when there's wrongs and try and stop them.

54:02The Peterman Pod Host:You also mentioned optimism in the talk, and you had this story I wonder if you want to share. So I would say in engineering, it's hard to know, right? But I think you need to be kind of optimistic or have a positive outlook because so many things could go wrong. And then some of my story, personal story that illustrates it's going back to high school when I'm 16, I'm dating this very attractive girl. And I screw up my courage and ask her if we would be exclusive. At the time, the phrase we used was going steady. And she looked at me and said, and she was 16. She had dated other guys and thought we were pretty young.

54:42And she said, well, Dave, you're such a nice guy. I don't know how to say no. For me, as a logical person, not a no sounded like a yes. And so I hugged her and said, great. And so she, in her mind, she thought, well, I'll let him down gently later. But we've been married 59 years now, and she hasn't let me down yet. So that was the case where optimism paid off.

55:06The Peterman Pod Host:I think everyone that hears a healthy relationship for that long, they might wonder how you did it. I used to tell people, you know, if you go to weddings, the marriage vows are really great, right? But nobody can remember their wedding vows. I used to say remember your wedding vows, but nobody remember that. So we boiled it down to nine magic words. And it's just three sentences. And they start I, you, I. And you got to say all three. And it's I was wrong. You were right. I love you. Okay. Those are the, those nine words. And this applies to both, both partners in a relationship, not just one partner.

55:43But yeah, if you can say them all and no substitutions, I was wrong. You're right. You're a jerk. You know, you can't do that. If you can remember those nine words, that can help you have a long relationship like my wife and I have.

55:56The Peterman Pod Host:And then last question for you, like knowing everything you know now from your career, if you could go back to yourself when you had just entered the industry and give yourself advice, what would you say? I mean, I think when I got here, because, you know, the imposter syndrome, I was a UCLA graduate student and suddenly I'm a Berkeley professor. So that just doesn't seem like that was very intimidating. But after a while, I just thought, well, I'm probably not going to get tenure, so I should just have a good time. So I think I already had a right attitude about it. I think that first year, I think it was very stressful while I was trying to handle the imposter syndrome and be a Berkeley professor.

56:40But I think after that, I handled it pretty well. I did all the things with the kids and stuff. So there's a version of that question is like, is there anything I would do over again? There's one thing I would have. I was chair of the architecture community, the SIGarch, as it's called. And they have an annual conference this year. And I was, this was in the 1990s, I think, I was the chair. And what I wasn't aware is at these conferences, there were men who were harassing young women at this conference. I just didn't think, you know, people like, you know, young people, people like me, nobody would do that.

57:22Only an idiot would do that. That can't possibly be happening. But it was happening. And I wish somebody had said something to me about it. And because I would have straightened out any man doing that. I would have threatened his life if he were to do that today. The only comforting thing is Sarita Abwe, who is a famous computer architect. And she said she also was not aware that that was going on. It became clear later, you know, five or 10 years later, it became more clear that this was going on and there were mechanisms. But that's the one thing I wish, you know, if I could go back in time, I would have figured that out and I would have straightened men out who were doing that.

58:05And they wouldn't, that would have stopped. I believe that would have stopped them.

58:08The Peterman Pod Host:Yeah. Thank you so much for your time today. I really appreciate it. All right. And thanks for the interview.

58:43The Peterman Pod Host:split keyboard so there's two sides this is in the case but yeah we launched on kickstarter and we hit our goal within eight hours of launching i really appreciate it if you were one of the people who grabbed one of the early units we're now working on the long journey of building the tooling now and so if you still want to pick one up i've left the late pledges open on kickstarter so you can grab one there i'll put a link in the description thank you again for watching the podcast and i'll see you in the next episode