Competition makes for better chip design with AMD CTO Mark Papermaster

0:05Hi listeners, for potential AI founders, my early stage AI fund Conviction is accepting applications for its Embed Accelerator for two more days. Embed offers$150 ,000 in an uncapped safe, more than half a million of free compute and API credits, a hand-selected set of peers, and access to leading founder and research mentors. Apply at embed.conviction.com by March 1st. Hi, listeners, and welcome to another episode of No Priors. Today, we're excited to be talking to the CTO of AMD, Mark Papermaster. Mark has had a storied career in chips and hardware with previous leadership positions at IBM, Apple, and Cisco.

0:43We're excited to have Mark on to get into GPUs and the competition that's been driving this industry. Welcome, Mark. Thanks, Sarah. Glad to be here with you and Elad. Can you start by telling us a bit about your background? You've worked on all sorts of interesting things from the iPhone and the iPad to the latest generation of AMD supercomputing chips. Oh, sure. I've been around a while. So what's really fun is my timing was pretty good getting into the industry. I was an electrical and computer engineering grad, University of Texas, and got really interested in chip design. And so it was back at a time when chip design was radically changing.

1:19The kind of technology everyone uses today, CMOS, was just coming into production usage. And so I got on IBM's very first CMOS projects and created some of the first design. So I got to get my hands dirty and do just about every facet of chip design and had a number of years at IBM and took on different roles, took on driving the microprocessor development at IBM across first their power PCs. And that was, you know, meant working with Apple and Motorola, as well as the big iron, the big computing chips that we had in the mainframe and the big risk servers. So I really got all facets of technology there and included working on some of their server development, but then shifted over to Apple.

2:12Steve Jobs hired me to run the iPhone and iPod. And so I was there for a couple of years, but it was a time of a great transition in the industry. And for me, it was a great opportunity because I ended up in 2011, and fall of 2011, taking the role here at AMD of being both CTO and really running the technology and engineering and right at a point where Moore's law is starting to slow down. And so, you know, tremendous innovation was needed. Yeah, I want to get into that and sort of what we can expect in terms of computing innovation if we're not just jamming more transistors on chips or were unable to do that.

2:56Every one of our listeners, I think, has heard of AMD. But can you give like a very brief overview of the major markets you serve there? Sure. So AMD is a stored company. It's been around well over 50 years. And it started out really being, you know, a second source company, really bringing, you know, second source on key components and x86 microprocessors. But you fast forward to where we are today, and it's a very, very broad portfolio. When Lisa and Sue, our CEO, and I were brought into the company just over 10 years ago, it was with a mandate to get AMD back into very, very strong competitiveness.

3:42And so we started with the CPU line, brought the CPU, very, very competitive, and then really across the portfolio. and just in February of 2022 acquired Xilinx. So that expanded the portfolio further. So AMD creates the world's largest supercomputers. It's got a massive install base now in the cloud. So many of your cloud operations that you're running are running on AMD EPYC x86 CPUs. Gaming, we're huge. were underneath all the Xbox, all the PlayStation, as well as many gaming devices that you buy when you buy your add-in boards. And then across embedded devices with all of that rich Xilinx portfolio, as well as embedded x86.

4:30And we acquired Pensando. So it extends that portfolio right into a networking interconnect that we need as we scale out these workloads. So very, very broad portfolio. Yeah, AMD has had a pretty amazing run over the last decade plus since you joined. One of the things that you folks have really emphasized over the last couple years as well as AI, and there's been a big shift both in terms of the adoption of AI over the last decade or so in terms of the traditional CNN, RNN, and other types of neural network architectures, but also in terms of this shift to transformers and diffusion models and everything else.

5:04Can you tell us a little bit more about what initially caught your attention in the AI landscape and then how AMD started to focus more and more on that over time and what sort of solutions you've come up with? You bet. Well, we all know the AI journey has been going since really the race began when the application space for AI opened up and GPUs were obviously pivotal there. When you look at the key work that, you know, Henson had done in terms of showing how GPUs could drastically improve the accuracy of image recognition, natural language processing. And so that's been known for some time. And so what we did at AMD is we right away saw the opportunity.

5:56The question was plotting our course to be that strong player in AI. So it was a very thoughtful and deliberate strategy because AMD, we had to turn around the company. So if you look at where AMD was in 2012 through really 2017, it was largely all of the revenue was based on PCs and then gaming. And so it was about making sure that the portfolio, the building blocks were competitive. Those building blocks had to be leadership. They had to attract people to get on that AMD platform for high performance applications. And so first, we actually had to rebuild the CPU roadmap. And that was a Zen microprocessors that we released in 2017 in both PCs with our Ryzen line, as well as Epic, our x86 server line.

6:52So that started the revenue ramp for the company and started extending our portfolio. And so right about that time, in parallel, as we saw where heterogeneous computing was going, we had called the ball on heterogeneous computing before myself, before Lisa ever joined the company. AMD had made a great acquisition of ATI that brought GPU into the portfolio. It's one of the big reasons I was attracted to AMD in the role is that, wow, it was really the only company that had a very strong CPU portfolio and a very strong GPU portfolio. And to me, it was clear that the industry needed that powerful combination of the serial, the scalar competing of these traditional CPU workloads and the massive parallelization that you get from a GPU.

7:48And so we started with that heterogeneous compute, created an architecture around that. So we've been shipping CPUs and GPUs combined for PC applications longer than anyone, started shipping those in 2011 with what we call APUs, accelerated processor units. And then for big data applications, we started with HPC, the kind of high performance compute technology that's in national labs, that's in oil exploration companies. And so we focused first with big government bids that ended up leading to supercomputer wins that we now have AMD CPU and AMD GPUs under the world's largest supercomputers. But that work started years ago and it was equally a hardware and a software effort.

8:37And so we've been building that hardware and software capability, and it really culminated in December 6 of 2023 of last year when we announced our flagship, the MI300, which just is a beast for both high-performance compute with one variant we have and takes high-performance AI for both training and inference head-on with a variant which is optimized for those AI applications. So it's been a long journey, And we're really pleased to be where we are, where our sales are taking off. Now, it's fantastic. I mean, I guess when you launched the MI300, you had public commitments from Meta and Microsoft, for example, to purchase that.

9:20And you just mentioned that there's a series of applications that you're pretty excited about there. Can you tell us more about which applications and workloads you're most excited about or most bullish on today? Sure. So if you think about where the bulk of AI is today, you're still seeing just tremendous capital expenditures and building up the accuracy of the capabilities for large language model training and inference. So it is the likes of ChatGPT, of BARD, and the other LLMs that you can ask at anything because it's trying to ingest the vast of data that is out there and it can be trained upon.

10:02And it's with really an ultimate goal of artificial general intelligence, an AGI type of capability. And so that is where we focus the MI300 is to start with that halo product that could take on the industry leader. And in fact, MI300 has done that. It's competitive on training and it leads in inferencing. It has over 2x. If you look at FP16 VLLMs, which is a metric that generally everyone can run that, it's got a tremendous performance advantage. And we did that very purposely. We created very efficient engines for the math processing that you need for that training or inference processing. But we also brought the memory that you need to have more efficient computing.

10:54So that's more computing at less power, less rack space than you need with competition. A big front of competition is, as you just pointed out, there's performance, like overall performance. There's efficiency. And then there's like the software platform like CUDA, RockM, et cetera. How do you think about the investment in the optimized math libraries and like how you want developers to understand your approach versus competitors? Yeah, you're so right, Sarah. It's multifaceted to be able to compete in this arena. You see many startups going after the space. But the fact is the bulk of inferencing done today is done on general purpose CPUs, not the huge LLM inferencing, but just general inferencing for AI applications.

11:43And then for large language model applications, it's almost all on GPUs because that is the software and developer ecosystems out there. And so we've been competitive on CPUs. We've been gaining share at a rapid clip because we've got a very strong CPU generation after generation that we've been releasing on schedules we've laid out for the industry. But for GPU, it did take us until now to develop really world-class hardware and world-class software. And what we've done is ensured that because we're a GPU, it should be easy to deploy. And so really making sure that we leverage the fact that we have all the GPU semantics.

12:30So if you're a coder, it's just easy to code if you're using the lower level semantics. But also, we support all of the key software libraries that are out there. When you think about the kind of frameworks, whether it be PyTorch or a founding member of a PyTorch foundation, whether it be Onyx, whether it be TensorFlow, flow, we are out there very closely working with developers. And so what we've now gotten to now that we have, you know, competitive and leadership offering is what you'll see is that when you're deploying with AMD, very facile if you're, let's say you're using Hugging Face, any of the, you know, thousands and thousands of LLMs, open source LLMs out there on Hugging Face.

13:17Well, we've partnered with Clem and his team. They test as they release any of those language models. They're testing on AMD with our Instinct GPUs equally as they're testing on NVIDIA. So we've really done the same thing as well with PyTorch, where we're one of two qualified offerings on PyTorch. And so all of that testing is being done, you know, routinely with the regression testing that's run literally every night on any software release. The other thing that's key is to learn from deployments. And so we've had early engagements like Lamini, who's running on AMD, and they've been offering services of getting on AMD and running your LLMs on their cloud, on their rack configurations they have.

14:09And so they've already been working with customers. And now, as you saw other people on stage with us at our December event, you can see that we're in there with a key hyperscaler. And we're also being sold to many OEM applications. And we're directly working within customers. So there's nothing like that feedback from key customers that are running on your platform to speed us, you know, ensuring that we can just be easily deployed and make sure that it's a seamless process. Yeah, yeah. Lamini is a portfolio company for me, and Sharon and Greg are great. I think it's an indication of you guys having a big ecosystem of software developers and machine learning people that want to see competition and more heterogeneous compute out there for these AI applications.

15:02Sarah, you cannot underestimate that. It tells you that it was a very constrained environment. There was a lack of a competition. was bad for everybody, by the way, if there's not competition, because you really end up with a stagnant industry. You can look at the CPU industry. Before we brought competitive and leadership, it was really getting stagnant. You're just getting incremental improvements. And so the industry knows that. And we've had tremendous pull and partnership. And we're very appreciative of that. And in return, we're going to keep providing generation after generation of competitive product out.

15:39For such a huge like software stack, like Rockham to be open source, like talk about that philosophy. Oh, it's a great question. It's very near and dear to us because we are, as I mentioned, all about collaboration. That's just such a strong part of our culture. And what open source does is it opens up technology to the community. And so if you look at the history of AMD, it's been very focused on open source. Our compiler for our CPUs is LLVM. It's open source. The LLVM is underneath our compilers on our GPU. But more than just the compiler and the GPU, we've opened up the Rockin' stack. It is our enabling stack.

16:25It was a huge piece in our winning supercomputing with such large installations we have. Why is it our philosophy? And by the way, Xilinx had exactly the same philosophy. And so bringing Xilinx and AMD together in 2022 did nothing more than even deepen that commitment to open source. But, Sarah, the point is we're not about locking in someone with a proprietary wall garden software stack. What we want is we want to win with the best solution. And we're committed to open source and we're committed to giving our customers choice. We expect to win having the best solution, but we're not going to lock our customers in.

17:12We're going to win on merit generation in and generation out. I guess one of the areas that I think is evolving very rapidly right now is sort of the clouds for AI compute. And so there's obviously the hyperscalers, the Azure from Microsoft and AWS from Amazon and GCP from Google. But there's also other players that have been emerging, you know, Base 10, Together, Modal, Replicate, et cetera, et cetera. and one could argue that they both are providing differentiated services in terms of different tooling, API endpoints, et cetera, that the hyperscalers don't currently have, but also that in part they have access to GPU and there's a GPU shortage.

17:51And so that's also driving part of their utilization. How do you think about that market as it evolves over the next three, four years and perhaps, you know, GPU becomes a bit more accessible and maybe shortages or constraints fall away? Well, that's definitely happening. I mean, the supply constraint will go away. We'll be a part of that. We're ramping up and shipping as we speak on our instinct line, and it's going quite well. It's going according to plan. But moreover, to answer your question, I think the way to think about it is that it's just breathtaking how the market is expanding so rapidly.

18:27I said earlier that most of the applications today that started on the degenerative AI with these LLMs, that's been largely cloud-based and not just cloud-based, but hyperscaler-based because it's such a massive cluster that's required, not just for the training, but frankly, for quite a bit of that type of generative AI LLM inferencing also is on these massive clusters. But what's happening now is we're getting application after application that is just taking off non-linearly. And what we're seeing is a proliferation is people are understanding how they can tailor their models, how they can fine-tune it, how they can have smaller models that don't have to answer any question you have or any application you need to support, but it might be just for your business and your area of exploration.

19:21And so that allows a tremendous variety of the size of compute and how you need to configure that cluster. So a rapidly expanding market, application-specific configurations you need for your compute cluster, and it moving even further, not just from these massive hyperscalers to, you know, I'll call it, you know, kind of tier two kind of data centers, but it just keeps on going because when you think about applications which are really bespoke and they can be run on the edge, right on your factory floor where, you know, very low latency, put the inferencing and, you know, right at the source of data creation, right to end user devices.

20:02So we've added our AI inference accelerators right onto our PCs. We have been shipping it throughout all of 2023 and actually at CES this year announced already our next generation of AI accelerated PCs. And then, of course, with our Xilinx portfolio across embedded devices, we're getting a lot of pull from industry that has bespoke inference application right in a plethora of embedded applications. So with that trend, we're going to see more of that, more tailored compute installations with an attempt to service this ballooning demand. Yeah, that makes a lot of sense. I mean, I guess a lot or a subset of inference is going to push to the edge.

20:50And obviously, we'll have things on device, but on laptops, as well as phones in terms of, you know, where certain small models will be running. And then it seems like there may be some ongoing potential set of constraints for larger models or larger data centers, at least in the short run. What are the main drivers of the constraints on the GPU supply side? I've heard things around packaging. I've heard things around TSMC capacity. I've heard sort of a mix of potential drivers of constraints. Some people say the next constraint after that is, do you have enough power into data centers to actually run these?

21:20I just don't know what's real in terms of all this stuff. And so I'm a little bit curious how to think about what are the constraints and how do we think about when the supply-demand things come a bit more into balance? Yeah, supply demand is frankly something that any chip manufacturer has to manage. You have to secure your supply. You look during the pandemic, we had actually a tremendous run on our devices that stretched our supply chain because the demand for PCs went way up. People were working from home. The demand for our X86 servers went way up. And so we were in a scramble mode during the pandemic, and we did very well.

22:04We worked. We had shortages of substrates, and we secured more substrate manufacturing capability. We worked closely with our primary wafer foundry supplier, TSMC. We have such a deep partnership with them. We've had it for decades that if we get out ahead of it and we understand the signals, we are generally able to meet the supply, or if there's a shortage, it's generally well-contained. And so what's happening with AI is, yes, it is clear that we're seeing this massive increase in the demand, and the fabs are responding, and you're having to not think of it just as a wafer fab, but you're absolutely right.

22:48It is the packaging. Our cells and our GPU competitor both use advanced packaging. I mean, I'll show you. I know if in the camera it will come across here. But that is our MI300. And what you see is a whole set of chiplets. So smaller chips with either a CPU function, an IO and memory controller. It can be the CPU for the version we have that focuses on high-performance compute. We literally drop our CPU chip that's right in that same integration and all the high bandwidth memory that you have around it to be able to feed those engines. And those are connected laterally. And on the MI300, we connect those devices vertically as well.

23:33So it's a complex supply chain, but it's one of which we are very, very good at. We're a fabulous company. We've been fabulous for, you know, coming on 18 years now. And so we've got it down. Hats off to the AMD supply chain team. And I think overall as the industry, you'll hear that generally we're going to move beyond those type of supply constraints. Now, you mentioned power. This is, I think, ultimately going to be certainly a key constraint. And you see, you know, all the major operators looking for sources of power. And for us as a developer of the engines which are consuming that power, it brings tremendous focus for energy efficiency and that we can drive into each generation of our design.

24:23And we are committed to that certainly at very top priority. One thing you said before, Mark, is that you were actually excited about the innovation of the end of Moore's Law and that being a reason that you actually wanted to go to AMD. Like what directions of innovation should we expect investment in? I don't know if it's like too deep to ask you to give us a layman's understanding of like 3D stacking, but I think it is really interesting to think about it at a time when it's not obvious where to go. Well, no, Sarah, it's a great question. And the reason that I was so attracted to AMD is one, it had a storied history of being a disruptor in the industry.

25:03And I certainly felt very strongly that AMD could disrupt with very strong CPU and GPU. But more importantly, putting the pieces together, the idea of chiplets was just coming together. There was early exploration of that around that time. And the engineering team here at AMD, we were able to, you know, really get the team rallied and the key leadership rallied around it and drove that innovation. So the reason it's so important is when Moore's Law slows down, the easy way to think about it is it used to be that the chip technology itself, the foundry, going from one generation to the next, did most of the heavy lifting.

25:53So you could just bank on that new semiconductor technology node, shrinking your devices, giving you more performance. It'd have less power and it'd be at the same cost. So that was what Moore's Law was about. And with Moore's Law slowing, it means you still get those device improvements, but it costs more. Your power is not coming down as much as it used to. And you are still getting that integration. You're still certainly being able to pack more devices. But it demands more innovation. It demands what I call holistic design. So you're going to rely on those new transistor devices, new foundry nodes.

26:32But how you use heterogeneous computing, meaning bringing the right compute engine for the right application, a CPU, a GPU, a dedicated engine, like we have super low power AI acceleration that we have in our PC devices and our embedded devices. So it's about getting tailored engines for the right application, leveraging chiplets that you combine them, put them on what is the best technology node you want each of those chiplets, each of those functions to be on. And then, frankly, holistic design means you've got to keep going right up through the packaging, how you package it together, how you interconnect it, and how you think about the software stack.

27:15And so it's literally got it. The optimization has to be the full circle of transistor design all the way up through the integration of your computing devices and equally with a view of the software stack and applications. And what I'm thrilled about, along with all the engineers that I work with at AMD, is that we we have that opportunity. We have the building blocks and we are built on collaboration. It's just such a part of our culture that we don't need to develop the entire system. We don't need to be the ones developing the application stack and the end applications. What we do is partner incredibly deeply and ensure that the solution is optimized end to end.

28:04I think everybody is very suddenly interested in the chip industry from a strategic perspective as well. I think everybody's thinking more about the supply chain from the, you know, TSMC near monopoly to the idea of fab security in an increasingly complex geopolitical environment. How does AMD prep for this or think about these issues? You know, you have to think about these things. We are very supportive of working with certainly the U.S. governments and other governments across the world, which have exactly that question. How, you know, our country is running now on chip design that powers such essential systems that it becomes a matter of national security to make sure that there will be continuity of supply.

28:52And so we build that into our strategy. We build it in with our partners, and so we've been supportive of fab expansions. So you see TSMC building fabs in Arizona, and we're partnering with them. You see Samsung building fabs in Texas. But it's not just in the U.S. They're actually expanding as well just global facilities in Europe and other parts of Asia. And so it goes beyond the foundry. It's the same thing with the packaging. So where do you as you put those chips onto carriers and you need to interconnect it, you need that ecosystem to have geographic diversity as well. So the way we think about it is it is a matter of importance for everybody to know that that there will be geographic diversity.

29:43And we are heavily engaged in action. I'm quite pleased with the progress we're making. It doesn't happen overnight. That's the difference between chip design versus software. Someone come up with software, you can come up with a new idea and get that product out very, very quickly, get that MVP design, get it out there, and it can go viral. But it does take years of prep in expanding the supply chain. The whole semiconductor industry was built up as historically as, well, this is a global industry and will create geographic pockets of expertise. So that's how we got to where we are today. But when you have more volatile macro that we're facing today with political tensions, with economic tensions, it's just imperative that we spread out that manufacturing capability and it's well underway.