Turing Award Special: A Conversation with John Hennessy

0:00John Hennessey is a computer scientist, entrepreneur, and academic known for his significant contributions to computer architecture. He co-developed the RISC architecture, which revolutionized modern computing by enabling faster and more efficient processors. Hennessy served as the president of Stanford University from 2000 to 2016, and later co-founded MIPS Computer Systems and Atheros Communications. Currently, he serves on the board of the Gordon and Betty Moore Foundation and is the chair of the board of Alphabet. John received the 2017 Turing Award for, quote, pioneering a systematic, quantitative approach to the design and evaluation of computer architectures with enduring impact on the microprocessor industry.

0:45In this episode, he joins Kevin Ball to talk about his life and career. Kevin Ball, or KBall, is the Vice President of Engineering at Mento and an independent coach for engineers and engineering leaders. He co-founded and served as CTO for two companies, founded the San Diego JavaScript Meetup, and organizes the AI in Action discussion group through latent space. Check out the show notes to follow KBall on Twitter or LinkedIn, or visit his website, kball.llc.

1:25John, welcome to the show. Thanks. Delighted to be here. Yeah, I'm excited to dig in. So, I mean, somebody with your background, you get introduced all the time. It kind of speaks for itself and you've got so many things you could say. I'm actually curious, if you were designing your own introduction, how would you introduce yourself to an audience? That's a good question. I'd say I have been extremely fortunate to have entered the computer field in its early days and to be able to do incredible things because of the remarkable advances that have been made in the field. And that's been just incredibly exciting.

2:06And I'm so glad I decided to be a computer person. It has definitely been a wild time in the computer world. Though, interestingly, you started early, but risk is still running. I mean, with RISC-V, that's kind of the hot topic now. What are your thoughts on what is our continued bandwidth in the risk space. Yeah. I think what happened is interesting. I think in the end, what really made the risk ideas really take off was the demand for more efficiency. And that comes in a number of different ways. I mean, because now a lot of the devices we use are battery powered, not plugged into the wall. So energy efficiency is really important and risk is much better at that.

2:50But also because we've gone ubiquitous and there are computers everywhere, right? Figure, look how many computers are inside a brand new car. I mean, there are 50, 100 microprocessors inside there. So the price does matter all of a sudden. We're not just building chips that cost several hundred dollars each. We're building chips that cost$10 each or$20 each. So the whole efficiency thing won out in risk. And now even in the large data centers, You see these companies that are the hyperscalers are building out of risk chips because the energy consumption is a big part of the bill that they pay in their data center.

3:27So they worry a lot about this energy efficiency issue. And that in the end, that was the key inside of risk that we knew how to build processors which were much more efficient in their use of silicon area and their use of power. And that's been a winning combination now for probably the last 15 or 20 years as we switch to a new computing world from the old world of having desktops and things plugged into the wall. Yeah. Well, and as you highlight, our constraints may have shifted, but efficiency is still super important. Yeah. So we've been on this long run for a really long time. Moore's Law carried for so long in scaling.

4:08And each generation of chips getting smaller, like even if it's slower, it's still mind-boggling how far we're going. But I feel like we're kind of seeing the end when that comes and we're having to embrace something different. Yeah, yeah, we're plateauing. I mean, first of all, Moore's law isn't a law. It's a kind of objective for the industry to scale against. But we see it slowing down. Now, let me point out that it's slowing down. If you look over the last 50 years, 50 plus since Gordon made his prediction, we've scaled by a factor of about 10 million. And we're off from Moore's projection by about a factor of 25.

4:46But the gap is getting bigger. And it's really been the last few years it's opened. And it's opening more and more and more. And so that's going to demand that we rethink computation. We think about efficiency. We think about different ways of doing things. Well, and I think one of the things that it's pushing people towards is more heterogeneous computing, right? Less. Absolutely. I mean, you look at the Apple chips, there are multiple processors, but there's a high performance processor, there's a low power processor, there's a AI processor, you know, there's a signal processor. So we're moving more and more of that.

5:23And that's, again, this drive for efficiency and using the silicon and power efficiently. Both matter. Yeah. So I'm interested in your thoughts on what that ends up looking like for a software development team, right? This is software engineering daily. So we're writing software, not just the hardware piece of it. So how does that heterogeneity play out into the tools we use to write software? Yeah, I think it basically requires more work on behalf of the programmers to really get a good fit between the processor and a processor, whatever processor they're using in a heterogeneous world. and the application.

6:01And that, for better or worse, that problem has gotten pushed off to the software, actually beginning when we went to multi-core. The reason we went to multi-core is that we didn't know how to build faster single-thread processors. We didn't have any idea how to do it. We were at a dead end. We'd used up, over a period of 15 or 20 years, we used up all the good ideas, and mostly instruction-level parallelism, and they ran out of steam. So then we had to go to multicore. Now, of course, when we go to multicore, then the programmers have to find the parallelism and decide what threads to run where.

6:35And now as we've gone to heterogeneous, as you alluded to, things get even more tricky because you've got to figure out, okay, not only what are the threads that I can run in parallel, but which thread should run on which processor. So that's going to require, I think for better or worse, programmers are going to be responsible for more efficiency going forward and getting efficiency out of the thing. It's funny that many years ago, I was talking to Maurice Wilkes, who was the last living pioneer from the golden age, from the ENIAC age, you know, in the post-World War II era. And I said to him, Maurice, what's going to happen if, you know, we can't continue to build hardware that's faster and faster and faster?

7:18We've been going, you know, one and a half times every year. And what's going to happen when this slows down? He goes, programmers are going to have to get a lot smarter and a lot more careful about the code they write. And I think he's right. And that's what we're seeing now. In some ways, it kind of actually reminds me a little bit of like what happened when RISC came in, where you were saying, this used to be in the hardware. You have these complex instructions that are doing all this stuff and you're saying, well, let's make software do it. Yeah, I think that's right. I think there is a parallel.

7:45And, you know, a part of what drove risk from, at least from my research group, was the notion that you should never do anything at runtime if you can do it at compile time. And a lot of what was going on were things we could do, we could do at compile time. So rather than reinterpret complex instructions, compile down, get rid of a layer of microcode and compile right down to the hardware primitives. And I think nowadays that's changed in that the processors have gotten a lot more complicated. Memory hierarchies are getting more complicated. If you look at GPUs or TPUs or anything, there's a lot more focus on controlling the memory system by the software rather than by the hardware.

8:29Now, today that happens with a combination of smart compiler tools and people who understand how to write their algorithms so that they compile well for those kinds of machines. And it's that combination. So it requires, I think, a level of understanding of the underlying hardware mechanisms to really become a good programmer that can program something efficiently. Yeah. Well, and it's like when single-threaded performance just kept getting better and better and better, we didn't have to worry about it. And you could almost completely disconnect those. Hardware teams working on their side, software teams working on their side.

9:04So long as you end up generating the bytecode, it's going to work and it's going to keep getting faster. I don't think we're in that world anymore. We're definitely not in that world anymore. I think it's just, you can't just rely on the hardware guys to make things faster because it's not going to happen, unfortunately, if we could. I mean, there were a lot of incentives not to rewrite software because, you know, a year later, it was going to run 50 % faster. Well, no more. Now, a year later, it runs 5 % faster if you're lucky, you know, so you're going to have to find ways to rethink that interface.

9:35And I think it's interesting because it's really about, you know, how do you think about the interface between the hardware and the software system? How do they come together? How much does the programmer have to know? What's the compiler responsible for? How does that all fit to deliver performance? I saw in one of the talks that you gave that when the first risk revolution was happening, one of the challenges was that the tooling didn't exist. And in fact, the tooling was being generated inside of academia because companies weren't doing it. What do you feel like the missing layer of tooling is for this generation of, okay, now we're moving into the heterogeneous world?

10:12You know, I think we still have this gap. And when you move to these domain-specific architectures, things that are tailored for particular classes of algorithms, right? Today, lots of machine learning things, obviously, but a wider range of things. Graphics clearly has this. Lots of signal processing has this special purpose aspect to it that can be captured. The key thing is to figure out, can you build an architecture that does really well in these kinds of applications, but it's sufficiently flexible to allow a wide range of applications? And then, of course, figuring out how to get that match between what the hardware can do well and what algorithm the programmer really wants is still an open issue.

10:58And we've got it for some things. But if you look at lots of the things we run, whether they're on graphics units or they're on something doing machine learning, they're doing linear algebra problems. And they're comparatively well-structured, even with sparse linear algebra, it's comparatively well-structured compared to a random piece of code you want to run, right? So figuring out how to align these things and how general can these architectures be, how wide a range of things can they run is still a critical open problem. And the tools will determine that to a large extent, how to get that interface to work between the hardware and software.

11:37Now, certainly if you're, you know, when you're shipping graphics units and you're manufacturing tons and tons and tons of these, you need that level of generality. But I think another thing that's kind of interesting is like with cloud FPGAs, you can sort of create your own architecture for your problem space. Is the efficiency good enough there? or is that still, when you go to FPGA, is that still leaving too much on the table? Yeah, I mean, you can do this. I think, you know, there's a efficiency loss that's pretty significant, but if there's a lot of gain from the flexibility that's achieved and you can really change that flexibility, change the structure of the FPGA to do some other problem, you can imagine situations where it makes sense, particularly when the algorithms are changing quickly.

12:24Rather than build an architecture that's adapted to a particular class of algorithms, it might be smarter to go to an FPGA structure that would allow the algorithms to continually evolve and still be able to match pretty well to the new algorithm. So there have been some people at Microsoft that have done some experiments with this kind of approach. Probably they've moved the furthest. But lots of hardware developers use FPGAs as starting points now anyway to get something that works, that's reasonable in terms of getting the hardware before they go to something that's a more customized design and is going to cost not only a lot more to design, but a lot more to fabricate as well.

13:04Yeah. Yeah. Another area here that I think is interesting, I'm going off of something I saw you talking about, I think in a talk in 2023 was essentially treating machine learning as a way of programming where you're programming software per se, but you're programming it with data rather than programming it with code. I'm curious how you think about that with relationship to efficiency, right? Like on the one hand, it's almost as flexible as you can get, right? Give it some text, you'll get text out. One of these LLMs, you'll get amazing, or you can train it on some other data domain. But it's also massively expensive.

13:42So how are you thinking about the role of programming with data in this ecosystem? Yeah, I mean, you're right. That programming with data is the right way to think about it. You've shifted to the use of data for programming. But then, of course, the cost is the training, particularly if it's a large data set that you need to get trained on is what's really costly, right? And the model, depending on how big the model is. I think one of the interesting things we've seen is that some of these smaller models that are trained more carefully and that are inspired by a large model have achieved enormously incredible results.

14:17Okay, so the giant models do these incredible things, but a model that's a lot smaller, let's say a billion parameters versus 500 billion parameters, is able to do pretty well for lots of applications. So one of the things I think we're going to see is the models for endpoints. For example, what's on my phone? I want a machine learning model on my phone that'll help me with text and search and some other things, but I'm not going to put a model on that has 500 billion parameters in it. So I'm going to have a small model on the phone that's going to do a lot of things. Probably one of the outputs of that model is, I'm not sure, call the big model in the cloud and go do that.

14:58And we're going to have to figure out how to make that work in a way that's appropriate and seems smooth and works well for people. But I think we're going to see more and more of that and particularly smaller LLMs adapted to particular domains, whether it's inside a camera, inside a phone, inside some kind of other device that may be on a lot of the time. So, yeah, I think that's a fascinating domain. And the more you can constrain the problem, the more you can fine tune the model to particularly do that. I love that. So curious, in that same interview that I'm thinking of, you predicted that LLM enabled technology would be truly useful in a year or two.

15:37And I think this was end of 2023. I feel like one of the things I've seen is this was the year that LLMs broke through for software development, right? Like coding assistance with LLMs have gone from a niche that a few people were exploring to just exploding. What other domains are you seeing that type of breakthrough in? Well, I think coding is certainly, you're right, and coding is amazing because it's, you wouldn't code anymore without an LLM assistant of some sort, right? I mean, you wouldn't do it because the leverage you're going to get from it for lots of code is just so high, right? So it's probably delivered, you know, it's obviously things like abstract data types and various forms of polymorphism delivered lots of programming productivity.

16:19Well, this is delivering another big hit in terms of improvement in productivity. I think we're seeing it in writing. We're seeing it around things that help you digest complex and large documents. I'm thinking of something like Notebook LM, where you can ask it, tell me what the key things I need to understand in this 100-page manuscript are. What are the key insights? And you get reasonably good answers out of these things, amazingly good. And for college instructors, you can say to it, design me five test questions based on this material here, and you could get great things out of it. So I think we'll see a lot of help on that.

16:57You know, one of the things that instructors generally hate to do is grading. All teachers hate to do the grading part. They like to see their students succeed, but the grudge work of... But I think now we've seen some systems based on LLMs that could do grading as well as people. And I think that'll be a big improvement. I'm very big on this idea of using machine learning and AI to eliminate human drudgery. We're not going to completely replace jobs, but we're going to replace some of the stuff that people really don't like doing in their jobs that is more rote, more straightforward that we could do with an LLM.

17:34And I think we're going to see more and more of this occur. Yeah, no, I completely agree. And I think it's a really interesting problem domain because you have to, sometimes you have to completely reshape how you're thinking about it, right? Using the coding with LLMs example, right? You have to shift how you're attacking your software problems. But what you get out of it is the elimination of a lot of drudgery. Yeah, I think one of the key problems, and I'm curious where you're seeing this is almost what you talked about there with regards to when does the small model call out to the big model?

18:06I think similarly, we need a question of when does the big model call out to the person and say, you know what, like, I can't do this. I need you to get involved. Yeah. So I think one thing we're going to have to do in all these LLM-based systems is tune the system so that they say, I don't know. Not my best answer is X, but X might be my best answer, but I don't have a high degree of certainty in that. And we've got to get there. I mean, there are these examples you hear about periodically of people using LLMs for writing and then making up citations to things that don't exist. It should never do something like that, right?

18:45Just as it shouldn't write a piece of code that it really doesn't have high confidence is the right way to write the code, right? And I think as my colleague Dan Bonet pointed out, one of the problems with these coding tools is that they'll sometimes write a piece of code that has a big flaw in it, and it won't know that it's got a flaw in it. And of course, that's tricky because as a programmer, reading somebody else's code, whether it's another person's or a machine's, and figuring out, is this right, is a hard task. But I think that's the sort of thing that we're going to have to navigate through and try to make the systems better at being more cautious when they don't have high confidence in what they're predicting.

19:29Absolutely. So text and LLMs and coding have been getting a lot of buzz and image and video gets a lot of buzz. But in some ways, I'm more excited about things like AlphaFold or other things like this that are not in the text domain. I saw that one of the DeepMind founders won the Nobel Prize for chemistry this year. And actually, I think the Nobel Prize for physics was also in a machine learning related domain. And so just I feel like those are the dimensions that are going to completely change the world. And I'm curious. Yeah. So science, I think, is going to change dramatically. I think these machine learning tools are going to be the new tool of science.

20:06as important as microscopes have been, as important as various tools for looking at the structure of molecules and DNA have been. And this is already happening. The chemistry example was a great example. I mean, AlphaFold has discovered more protein structures than 50 years of protein structure work discovered. And that's an amazing result. So I think we're going to see more and more of this. And people are doing all kinds of problems that are computationally not tractable if you do them from basic scientific principles. But where the LLM, where a machine learning system can be used to reduce the search space so dramatically that you can get the answer.

20:52You're still doing a little kind of physics simulation things that we traditionally do in much of science, but you're using it over a much smaller domain than you would have before. You figure out the basic structure of the protein by knowing what other proteins with similar molecules, similar atoms in them have. And then you use that to guide the process of getting the detailed structure. And it results in significant improvements in performance, ability to do much, much more. And I think we're going to, we're seeing this in lots of science. We're seeing it in astrophysics, where people look at the structure of galactic systems and understand how they're evolving.

21:32We're seeing it in one of the things I thought was amazing. There are people working on this to understand turbulent flow, one of the hardest computational problems we do. And solving that problem is extremely difficult from basic principles. On the other hand, you might be able to use these tools to kind of get the basic structure and then use simulation to get the accuracy that you really need in these systems. Look at weather prediction. I mean, amazing result. The deep mind people have beat the best weather prediction system out there, which was developed over a period of 20 years in terms of computational ability.

22:07And they're able to outperform it. So I think I'm really excited about what this is going to do for science. Yeah. Yeah. So there's something you talked about there that I think is a really interesting big picture theme, which is like these generative systems don't have to get to the right answer. They just have to narrow the search space. We have all sorts of domains in which we have formal validation that works when you have an answer or a small number of answers, but we're exploring the entire search space is totally intractable. And if you can use this system to narrow you in, now you dump it into a formal validation.

22:42I think mathematics is another interesting area here where we have formal validation checkers, but proof generation is hard. So use an LLM of some sort to generate viable truths, narrow your search space, and then now you can dump it into a formal validator. With that model, I'm curious, actually, you probably know more than I do about what are the different domains. We talked about a few. We talked about weather. We talked about chemistry and protein folding. What are some other domains that this can open us or us in terms of narrowing the search space down, and then we can dump it into either a formal validation or a human validation.

23:16So, well, I mean, lots of classic problems, which are NP complete, right? So that we don't know how to do them efficiently. If you can narrow the search space, you can come up with an answer. And it may not be the optimal answer, but it may be very close to the optimal answer. And that could certainly be appropriate. I mean, there are lots of interesting problems that reduced down to these very fundamental computational problems. For example, generating test patterns for software or hardware, generating sets of tests that will test everything. That's a really hard problem if you have to do it completely.

23:55But if you were guided by a system, you might be able to narrow the range of it so you could get a reasonable number of tests that would adequately test the system. And I think we'll see other examples like that, where, as you said, narrow the search space. You're doing a complex optimization problem, but if you can narrow the search space, then you can get to something that is close, if not perfectly optimal, close to the optimal solution very quickly. Yeah. I love that as a kind of idea generator for domains to attack, right? Anything where you have an NP-hard problem, but you can validate any particular solution, this might be a useful technology to try applying.

24:36Yeah, agreed. All right. So bringing this back around to software engineering and the tech industry, we're obviously in a very tumultuous time, lots of things changing here. How do you see all of these breakthroughs impacting the tech industry and the world of software development over the next few years? So I think one of the things we're seeing is we're seeing a kind of almost a back to the future evolution in the tech industry in the following sense. If you look at the tech industry prior to about 1985, 1990, there was a lot of vertical integration. I mean, IBM did everything. They designed their own chips.

25:16They designed their own disks. They did everything. It was a vertically integrated, all the way up through the entire software stack, right? They did everything. And then the industry moved to, particularly with the PC and the emergence of shrink-wrap software, it moved to very horizontally. So you had Intel down here at one layer with the disk guys over here as another. And then on top of that, you had Microsoft. And then on top of that, you had the application layers. Now, all of a sudden, because of the need to vertically integrate much more to get the applications closer to in touch with the hardware, we're seeing a reintegration in the vertical direction.