Turing Award Special: A Conversation with Jack Dongarra

0:00Jack Dongara is an American computer scientist who is celebrated for his pioneering contributions to numerical algorithms and high-performance computing. He developed essential software libraries like Linpack and Lopac, which are widely used for solving linear algebra problems on advanced computing systems. Dangara is also a co-creator of the Top 500 list, which ranks the world's most powerful supercomputers. His work has profoundly impacted computational science, enabling advancements across numerous research domains. Jack received the 2021 Turing Award for quote, pioneering contributions to numerical algorithms and libraries that enabled high performance computational software to keep pace with exponential hardware improvements for over four decades.

0:44He joins the podcast with Sean Falconer to talk about his life and career. This episode is hosted by Sean Falconer. Check the show notes for more information on Sean's work and where to find him.

1:07Jack, welcome to the show. Yeah, thanks very much. It's a pleasure to be here with you. Yeah, thanks so much for being here. So you've spent a lot of your career working on high-performance computing. So first of all, and maybe it seems like a basic question, but I think it's probably a good spot to start, is what defines high-performance computing? And is that a moving target as mainstream computers that we use every day become more powerful over time? Well, that's exactly right. So high-performance computing, or I would call supercomputers, are usually specified as the fastest computers at any time.

1:40So there's the marker as time. As time goes on, these computers change, of course, and they get faster. And things that were supercomputers, let's say three or five years ago, are no longer considered supercomputers. And they're replaced by the next generation of machines. And supercomputers are fast in terms of floating point operations, adds and multiplies. And they're characterized by being quite expensive as well. So the fastest computer that we have today is a machine that's located at Lawrence Livermore National Laboratory. So it's the fastest by some metric. And the cost of that computer is about$600 million.

2:24dollars. And that computer is a supercomputer today, but I would say five years from now, it's going to fall off and not even be considered one of the fastest computers. So there's an investment that has to be made if you're interested in having a supercomputer or need a supercomputer that has to be replaced with that frequency. Yeah. And how did you first get interested in this research area? Well, let's see, I guess I wanted to be a high school science teacher. And I went to college to do that. And in my last year as an undergraduate, I was encouraged by my physics professor to apply for an internship at Argonne National Laboratory.

3:10So Argonne National Laboratory is a Department of Energy laboratory located just outside of Chicago. And that's where I was going to school. So I applied for this position and I received a word that I got the appointment. And the appointment was to spend one semester with a scientist. So I worked at Argonne National Lab as my last semester, and that was transformational. It changed everything in terms of my outlook. I no longer wanted to be a science teacher. I felt I had an ambition to go into research and to take on this challenge, let's call it, of working at a national laboratory. So I worked there for the last semester, and then I decided to switch and become a computer scientist.

3:57So I applied and was accepted at Illinois Institute of Technology in Chicago in the computer science program as a master's student. So I worked for my master's degree, And Argonne National Laboratory offered me a position of one day a week working at Argonne. So I lived in Chicago, went to school at IIT, and then worked at Argonne National Lab one day a week. After receiving my master's degree in computer science, Argonne made an offer to me. I decided I didn't want to go on and get a PhD. So Argonne offered me a position at the lab, a full-time position, working alongside of the researchers there.

4:39at Argonne in what was called the Applied Math Division. And that was, again, a wonderful experience working with experts, designing software for solving certain mathematical problems. And that was something which drove me. And we had frequent visitors to Argonne National Lab from outside. Visitors from various universities would come and spend a day or a week or a month working at Argonne alongside of the other researchers that I was part of. And they encouraged me to go back to school and get a PhD. So I went back to school with one of the people who gave me encouragement as his student, and that was at the University of New Mexico.

5:21So I went there and worked on my PhD. And while I was there, I had the opportunity of working at Los Alamos National Lab. So Los Alamos is another Department of Energy laboratory located maybe a couple of hours north of Albuquerque, where the University of New Mexico is located. And I worked there and that was, you know, again, a wonderful experience. So both Argonne and Los Alamos had supercomputers or machines that were at the top rank at that time. Los Alamos had just acquired a computer called the Cray, Cray 1 computer. and that computer was different in terms of its architecture. It had vector instructions.

6:04So I had an opportunity to experiment and to basically to play with this computer that was there that was going to be used for scientific computations, but I had an opportunity. I was in a position to use it and to develop ideas and methods that would work well on that computer. And that was, again, a wonderful experience. And then I received my PhD in 1980 and went back to Argonne National Lab as a researcher and worked there for a number of years until transitioning to Tennessee, where I am today. So I've had just a few jobs in my life. I'd like to say I had three jobs. One job was at Argonne National Lab that was followed by the job at the University of Tennessee and Oak Ridge National Lab, where I am today.

6:51But before the job at Argonne, I made pizzas. So those are the three positions I claim to hold. Pizza maker and then researcher at Argonne National Lab and then finally professor at the University of Tennessee and working as a researcher at Oak Ridge National Laboratory. Well, it's wonderful that you were able to early on in your life find something that you fell in love with and were able to do a long career in without having to move around a lot. But in terms of motivation, is the fact that what defines a high-performance computer or supercomputer such a moving target where state of art today is yesterday's news three years from now?

7:32Is that something that's helped keep you motivated to focus on this field throughout your career? Oh, yeah. It's exciting to see new architectures and then try to understand how they can effectively be used to solve problems. So the way I look at it is I've helped design numerical libraries. So these are software components which are used by other applications. And these software components are sort of basic in terms of the operations that they do. And those libraries basically have to be reorganized or rewritten, refactored every 10 years. And that refactoring is caused by the architecture changes.

8:11So, you know, we have computers. If I go back in terms of where we came from, we had scalar computers, that is computers that had the ability to execute a single stream of instructions one at a time. And those machines were replaced with machines that were vector computers. So instead of just operating on one number, adding one number to another number, they operated by taking two vectors and adding them together, let's say, to produce a product. So you issue one instruction, and that has an effect across this array of data, and that allows things to run much, much faster in terms of the flow of the data through the system.

8:51So vector computers caused this revolutionary idea to change the way the software was written to adapt to it. After vector computers, those computers were a special purpose for scientific computations. So they were very expensive, and only a limited number would be manufactured. And what happened essentially to the computing area was that we had this incredible improvement in performance of microprocessors. So we had this thing that I'll refer to as the attack of the killer microbes. So those microprocessors became faster, more powerful, and were able to basically do the same function as those special purpose scientific computers that were characterized by vector computing.

9:40So microprocessors became the basic commodity component, which was used in our supercomputers. computers. So microprocessors then took on characteristics of being put together in a parallel context. So scientific computers evolved to using this microprocessor that were aggregated together in a parallel computer to help solve problems. So we have computers which had maybe 10 or 20 or 100 of these microprocessors together, communicating, passing messages back and forth, over a high-speed network to allow them to effectively solve problems. And that had a major change in terms of the software that we dealt with.

10:27Using that large number of processors together causes us to reorganize the algorithm that can effectively do that. And those microprocessors then were aggregated together, and we reached the point where we had thousands of processors being used to help solve our problem. Multicore came in and that, of course, added to the complexity, I'll say, but also to the layering of how these computers were effectively used. So in the end, the supercomputers that we have today are based on, I'll say, commodity processors in terms of the basic organization is around the x86 instruction set. So Intel and AMD have licensing rights for that instruction set.

11:15Our commodity processors use that. So our supercomputers basically have that as the core of their processing. And today they're augmented by GPUs. So graphical processing units have been added to the mix to help boost the ability to do floating point operations. So we have machines today which are hybrid, having commodity components plus GPUs, which are effectively used. And again, with each of these changes, it causes us to rethink the algorithms, rethink the software, rethink the numerical libraries, so they can effectively be used on this architecture. In terms of like solving like these kind of hard science problems that motivate a lot of the work around supercomputers, like is there particularly a advantage that you get trying to solve that with something like a supercomputer versus being able to use some sort of like distributed network on the cloud over like multiple computers and sort of split that work in parallel?

12:18Or is there certain things that you just can't split in that fashion? Right. So in general, the computations that are done on these supercomputers are very data intensive. That is to say, they do a lot of arithmetic and then they do a lot of communication, transferring information from one part of the machine to another part of the machine. So there's that kind of thing going on continuously in these computations. And if we think about a cloud-based system, the cloud-based system incurs a certain overhead associated with the latency of moving data over large distances and the bandwidth associated with that.

12:59So if we think about a truly distributed machine, let's say a machine that puts together components in various locations to create this virtual computer, that would not work very well in terms of a architecture for a machine that would be used for large-scale scientific computing. But if we think about a cloud-based thing, so if we think about, take Amazon as an example, using Amazon to do the computation on a single Amazon site, which had high-performance processors, plus perhaps graphical processing units that could be used to do these computations. So the question then is, we have to move the data from where it's located over to the cloud-based service, do the computation there, and then drag the results back to the home base.

13:48So moving data is a very expensive operation, and you don't want to do that very often. So you're basically using a cloud-based system. You may get locked into that cloud-based system to do the computations over some number of years if you're going to be doing it rather than transporting the data back and forth, because that's a very expensive operation itself. So the cost, if we think about buying a computer and think about using a cloud-based system, so a computer that would be on-premises as opposed to using a cloud-based system, there's been many studies that look at the cost of doing that.

14:27And those studies usually come out with on-premises computing is a better financial arrangement. And by better, I mean, perhaps by a factor of two over using a cloud-based system. So there could be a factor of two in terms of the cost that you would pay if you add everything up in terms of doing the computation. So again, the cloud providers are providing the service, they set the pricing. So that's based on them setting a price that's not competitive in some sense with the on-premises computations that might go on for large-scale, large-scale computations. So you mentioned like data movement there in communication.

15:09And even at the hardware level, I think that's like probably one of the biggest challenges to essentially scaling up the amount of, you know, flops that these machines are capable of. Like, what are some of the approaches to reduce the amount of communication that's happening at the hardware level? Right. Right. So you've hit on really the bottleneck or the point of contention on these computers. So the thing which is most expensive on our computers is data movement. So it's not the floating point operations. So it's not the computation that we do. It's the movement of data to the place where the computation is going to take place.

15:45So that turns out to be the biggest bottleneck. And our machines today are really over-provisioned for floating point operations. So they have too much capacity and we can't get the data to them. So we need in future designs, I'll say we need ways of trying to overcome that memory bottleneck. So that really is the biggest challenge that we have. So there's a lot of ideas that are floating around and how we could overcome that. Ideas like processor and memory. So we take processors and embed them in the memory itself so that the processors are very close to the point at which the data is located and data then can flow into those processors.

16:29We also have to come up with other techniques to avoid the latency of moving data. So some of the techniques that are used are organizing the computations around a directed acyclic graph. So a direct acyclic graph has the ability to uncover the maximum amount of parallelism in a computation so we can exploit that parallelism and hopefully get a much more optimal solution. But this issue of moving data is really critical in terms of the efficiencies that we see on our computers. And I'll let you in on a dirty little secret that we have. And that is, if we take a look at the peak performance of our supercomputers, and we look at the applications that are running on those supercomputers, and look at the performance that we actually see from those applications, we're getting roughly 10 % of the peak performance on these supercomputers.

17:26And that's a result of moving data. We can't move the data to the processors effectively. And that's causing this low efficiency in terms of our applications. So why do we end up with this case where things are over-provisioned for the floating point operations? Right. So it comes about from a number of reasons. You know, we have certain ways of measuring performance of machines. So I created a benchmark a number of years ago, and that benchmark is called the LINPAC benchmark. LINPAC benchmark solves a problem. It's a system of linear equations, and that system of linear equations is used in many applications.

18:06This benchmark was created back in the late 70s when floating point operations were very expensive. And this benchmark, as a result of floating point operations being very expensive, mimicked the way real applications would perform. But as the hardware changed over time, the benchmark itself has really less reflected how real applications are evolving. And architectures are trying to do very well on this benchmark. So they put instructions and design things to effectively do this benchmark very well. So the benchmark has at its core a matrix multiply, two matrices being multiplied together. So that's an operation which can be highly optimized.

18:57And the hardware on our machines today, hardware on our machines and on our GPUs, can do that operation very efficiently. So that operation has been optimized to a level which gets very close to the peak performance. So if our applications did that operation, we would match the peak performance. Unfortunately, matrix multiply is not the way in which we tackle problems today. So we use other techniques which don't have that ability. So matrix multiply has the ability of moving n squared pieces of data and doing n cubed operations on it. So that's a surface of data and a cube of floating point operations.

19:42So that gives rise to a situation where you could move little data and do a lot of operations on the data that you move. N squared data movement, n cubed operations on the data. So that is something which would be ideal for an algorithm if it just did that. Today's algorithms, unfortunately, don't just do that operation. So what would we do on our supercomputers? Our supercomputers are used to solve scientific problems. And those problems span the range of looking at weather forecasting, climate modeling, looking at applications that are trying to optimize combustion engines, looking at nuclear reactors.

20:23Some are used for nuclear weapon design. So all of those things usually center on solving the three-dimensional partial differential equation. So that three-dimensional PDE that's being solved is going to solve a system of linear equations. And that system of linear equations is not dense, it's sparse. And by sparse, I mean it has very few elements in this matrix. and we're going to try to solve a system of equations where we have a lot of zeros in the matrix itself. In order to solve that system, we use what's called an iterative method. And that iterative method has the property of doing really n operations on n pieces of data.

21:07So we have to move n pieces of data and then do n floating point operations on it. That's the basic pattern at which these algorithms operate. And that's very unlike what was done for a dense matrix, where we had n squared pieces of data and n cubed operations on it. So we move a little bit of data into a lot of operations. With the PDEs, unfortunately, we're in a situation where we just move n pieces of data and do n operations on it. And that causes this only less than 10 % of peak performance because of the movement of data, which is very poor compared to the rate at which we can do the floating point operations.

21:51So we're basically starving the floating point potential of these machines. The floating point units of these machines are being starved for data waiting for the information to flow to it. Yeah. And back to WinPak and where that started, There's a saying in business of you optimize the things that you measure, so be careful about what you measure. Yes. Because you end up creating these biases, essentially. It sounds like that's kind of what's happened there. So is this solved for that, that essentially there should be more than one measurement or KPI that is used to benchmark supercomputers? Right.

22:28So, of course, the best benchmark is the application that you have. So if you have an application that you're intending to run, that would be the thing to really benchmark. So benchmarks, the ideal benchmark would be multiple benchmarks that sort of span a space of applications. So today we have this, in the past we developed LINPACK and we thought that was a good measure. Today it's not a very good measure of how our computers really operate. So we've developed other benchmarks. I mentioned solving PDEs and this iterative process. So we've developed another benchmark called the HPCG benchmark, high performance conjugate gradients benchmark, and it uses an iterative algorithm.

23:11So it matches or tries to imitate what applications do in solving that three-dimensional partial differential equation. So it's trying to get a handle on what the performances and what the bottlenecks are on machines today on problems that are important to solve. So that's another component or another benchmark that we have that can be used to effectively deal with our situation today or our understanding of how these machines can be used and where the bottlenecks are in the machines themselves. So we need more of those things. That would be the ideal situation. Right. And do you think that that will change essentially how or lead to an overhaul of like the top 500 lists that people use to measure supercomputers today?

23:59Right. So we have this top 500 list, which measures the 500 fastest computers. In retrospect, 500 may be too many, but okay, that's what we have. We have all that data. I view the top 500 is giving us a handle on trends, and it sort of plots things in a nice way so we can see where we are today, at least in terms of what we think of as the theoretical peak performance for these machines. So if you don't do well on the top 500, you're probably not going to do well on other applications. That's one way to look at it. So we have all this data, and it presents a good point to keep. So I don't want to lose that top 500 data.

24:40I'm going to augment it. I'm going to augment it with other benchmarks. So again, we have this thing called HPCG, which measures another aspect of the computers. It really is trying to get a handle on this data movement to see how well we do with that. And that sort of augments the performance. So there's another list. It doesn't have quite 500 machines on it. It has only a few hundred machines on it. And that sort of shows the difference between the peak performance or the LINPAC benchmark and this thing which is more realistic in terms of what our applications can do today. And we need to develop others.

25:20So I shouldn't say we just have two. We should develop other benchmarks indeed. Yeah. It relates to that. I read your paper, Reinventing High Performance Computing Challenges and Opportunities, which I encourage anybody listening to check that out as well. But one of the things that you talk about in there is how there's this shift essentially to AI-driven workloads, a lot of that happening by these cloud vendors. And there, typically with AI-driven workloads, you're talking about like 16-bit arithmetic versus 64-bit floating point arithmetic. How does that sort of change the way that you need to think about measuring the performance of high-performance computing?

25:57So AI is a tremendous force in terms of computing and in terms of our understanding of how things are happening. So it's a great tool, and we use it all the time, and it's being used by all of the applications to really strengthen their ability. So going back to the situation that we have, I'll say in the old days, we had computers which had 32-bit and 64-bit floating point operations. Those were sort of the basic thing that we had to work with. With GPUs, we had another level coming in. So we had 16-bit floating point operations. So NVIDIA provides us with 16-bit. IEEE has a standard for doing 16-bit floating point operations.

26:43and that is implemented in the hardware. Google came out with something called BF16, which is another representation of 16-bit floating point operations and has a slightly different configuration of how many bits are in the exponent and how many bits are in the fraction. And Google probably is in a better position for doing computations. It gives up a little accuracy, but it has a wider dynamic range over what the IEEE is providing. And then NVIDIA in its hardware has, in its more recent generations, has 8-bit floating point arithmetic. So we went from 64-bit to 32-bit to 16-bit to 8-bit floating point arithmetic.

27:27And that lower precision is being driven by AI. It's being driven by how the AI process, how the neural networks work. So in the neural networks, there's a forward propagation and the back propagation. The forward propagation through the neural network requires slightly higher precision for doing the weights and the activations. And the back propagation, the gradients in the back propagation require a high degree of dynamic range. So that's as the exponent needs to be changed. So that NVIDIA has two formats for 8-bit floating point arithmetic as a result of that. And those operations in 8-bit and 16-bit run very fast compared to what we can do in 64-bit floating point operations.

28:14So that's caused us to rethink how we do our algorithms and can we leverage those lower precision computations in our algorithms themselves. So again, our algorithms have traditionally been written with 64-bit and 32-bit And now we're looking at using 16-bit and maybe even 8-bit floating point operations to help our computations work through their process. And the way to think of this is we're trying to leverage the lower precision to get an approximation for the solution and then use the higher precision to reinforce or to increase the accuracy of the solution that we obtained. So we do something very fast in lower precision and then do something slower in higher precision to refine the solution to get up to a point where it's acceptable for our computations.