AWS CEO Matt Garman on Amazon's Big AI Chips Bet, Working With OpenAI, and Nuclear Energy

0:00AWS CEO Matt Garman is here to talk about AI, the state of the economy, and Amazon culture. That's coming up right after this.

0:38Welcome to Big Technology Podcast, a show for cool-headed, nuanced conversation of the tech world and beyond. We are here in Las Vegas, Nevada at Amazon's Reinvent Conference with the CEO of AWS, Matt Garman. Matt, so great to see you. Welcome to the show. Yeah, thank you for having me. Let's talk a little bit about infrastructure. You're the kings of building data centers, right? There's no one that does it better than AWS. But there are headlines in the AI world. Elon Musk took 122 days to build a 100 ,000 plus GPU data center. Does this show that scaling data centers is now core to competing in AI?

1:15Is it validation? What do you think about it? Well, look, we've been building data centers for almost two decades now. And so this is something that we spend a lot of time on. And it's less that we were out there kind of bragging to the press about. But what we do is we provide... You can brag. Let's hear the size of yours. Well, more as what we do is provide infinite scale for customers. And so our goal is for largely the customers not have to think about these things, right? And so we want them, across their computer, across their storage, across their databases, to be able to scale to any number of size.

1:46And so take something like S3 as an example. It's an incredibly complex, very detailed system that keeps your data, keeps it durable, and scales infinitely. And customers largely just put data in there and don't have to think about it. And so today S3 actually stores 400 trillion objects. That's an enormous number that's hard to even get your head around. But it's just something where we just keep scaling and we keep growing for our customers. As you think about AI now, these are power hungry massive data centers for sure. And AWS is adding tons and tons of compute all the time for our customers.

2:18Largely what we think of though is less about how fast can you build one particular cluster. or the absolute size of AWS is dwarfed by any other particular cluster out there. But we're focused on how do we deliver the compute the customers need to go build their applications. Take somebody like Anthropic as an example. Anthropic has what are widely considered to be the most powerful AI models out there today in their Claude set of models. We're building together with them what we call Project Rainier. And so it's using our next generation Terranium 2 chips. And this cluster that we're building for them in 2025 will be five times the size, the number of exaflops that they use to train the current generation of models, which are by far the most powerful ones out there.

3:02It's going to be five times that size next year, all built on Tranium 2, delivering hundreds of thousands of chips in a single cluster for them so they can train the next generation. That's the type of thing where we work with customers, understand what's interesting for them, and then help them scale to whatever level they need. And that's just one of our customers, of course. We have hundreds and hundreds of other customers as well. Here's my point. You're so good at this. Look at what you just talked about in terms of Anthropik, being able to help them scale the way that you are. And that would lead me to believe that Amazon would have its own cutting edge, state of the art model, one that would lead, and be better than the open AIs and the Anthropics.

3:38This is your core competency, and this is what makes these models run. So why hasn't that happened? Our core competency is about delivering compute power for all of the people that need it. And for a long time, we've been very focused on how do we build the capabilities to let our customers build whatever they want. And sometimes there are areas that Amazon also builds and other times they're not areas that Amazon builds. And so you think about whether it's in the database world or you think about in the storage world or you think about the data analytics world or you think about the ML world.

4:06We build this underlying compute platform that everybody can go build upon. And sometimes we build services that compete with others out there in the market. think about a redshift competing with a snowflake who's also a very important partner of ours and a big customer of ours somebody that we do a lot of partnering together on and then there's other times where there's applications that people build on top of aws that amazon doesn't go and build and so we we uh uh operate across that whole swath of area and um and sometimes we'll build and sometimes they don't but that's the the kind of the beauty of aws is that our goal is to build that infrastructure so that sometimes we can build those sometimes we won't build them but we want this platform that everybody can go build the broadest set of applications possible out there.

4:46But I'm thinking about it for AI specifically and in the world that you play in, you have Google, they have their own model, they sell cloud services. You have Microsoft, okay, they don't have their own models necessarily, but they have this deal with OpenAI. Pretty sure that OpenAI is exclusive on Azure. Now this is where a lot of the growth is coming from. And so... And I think it's a mistake actually. So the interesting thing there is, and this is where a lot of people started, And I just think it's fundamentally the wrong way of thinking about it. Because a lot of times people are thinking about there's just going to be this one model.

5:15And I want to have the one model that's going to be the most powerful and the one model to rule them all. And as you've transitioned, as you've seen over the last year, there isn't one model that's the best at everything. There's models that are really good at reasoning. There are models that are great for, that provide open weight so that people can bring their own data and fine tune them and distill them and create things that, and create kind of completely new models from that that are completely custom for customers. And on that, you may want to use a llama model or you may want to use a Mistral model.

5:40There's customers who really want to build the world's best images and they might use something like a stability or they might use something like a Titan model. There's customers that need really complex reasoning and they might use an anthropic model. There's a whole ton of these operating out there. And our goal is how do we help customers use the very best? It doesn't have to be one thing. It's not just one. And we don't think that there's one best database. We don't think there's one best compute platform or processor. We don't think that there's one best model. It's across that whole set, and that's been our strategy.

6:09And customers have really embraced that strategy. As you get to them and they're thinking about how they go building production applications, they want the stability, the operational excellence, and the security that they get with AWS. But they also want that choice. It's incredibly important for them. And I think choice is important for customers no matter if they're building generative AI applications, no matter if they're picking a database offering, no matter if they're picking a compute platform. They want that choice. And that is something that AWS has, from the very earliest days, really leaned into.

6:38And I think it's an important part of our strategy. And it's maybe not the strategy that others have. Maybe others say it's just this one, and this is the one that we're gonna lean into, but it's not the strategy that we've picked. Our choice is around choice. And it's part of why we have the broadest set of partner ecosystem as well. It's why, as you walk the halls here on reInvent, it's filled with partners who are building their business on top of AWS. We're leaning in and helping our joint customers accelerate their journeys to the cloud, their modernization efforts, their AI efforts. And it's because of that, I think, that is a lot of what makes AWS special.

7:07Okay, and I'm going to move off this in a moment. But the reason why I'm asking these questions is because you do have at least a bet that big foundational models are going to matter. That's the$4 billion you just invested in Anthropic. And I think that the strategy that AWS has makes a lot of sense. This Bedrock strategy, there's a lot of different models in there. People have their data in the cloud. They're going to build with their data they have within AWS using bedrock picking models. But you also are limited in the fact that OpenAI is not there. I don't think Google's there. So wouldn't it make sense in parallel to the bring your own model strategy to also use this capacity that you have to scale infrastructure to get in the game yourself?

7:45Look, what I will say is never say never, right? I think that there's an interesting idea and then we'll we never close any doors. I think we're always open to frankly a whole host of things. We're always open to to having OpenAI be available in AWS someday, or having Gemini models be available in AWS someday. And maybe someday we will spend more time focused on our own models for sure. I think all of that is open, and part of what I think makes AWS special is we're always open to take our announcement earlier this year about partnering deeply with Oracle, about making Oracle databases available in AWS.

8:14Lots of people would have said, oh, that's never gonna happen, and it's against your strategy. Our strategy is to embrace all technologies, because we want anything that customers can use, we want them to be available and to be able to use it inside of AWS. And look, sometimes it happens today, sometimes it happens tomorrow, sometimes it happens weeks from now, months from now, years from now. But that is our goal, is to make all of those technologies available for our customers. Okay, I'm going to parse your language a little bit, because you said that you're always, you might be open to having OpenAI on Bedrock within AWS.

8:44Sure. Are you talking to them? Would you want to ask them to come on? There's nothing to announce there today, but I'm saying if customers want that, that's something that we would want and we'd love to make it happen at some point. Okay. Well, maybe they're listening and they want to make that move. Sure. Yeah. But let's speak to the one that I think is the biggest challenger to them, the one that you have all this money in, which is Anthropoc. So what does the$4 billion that you just invested in Anthropoc get you? And how does that make you differentiated from other cloud providers? Well, there's a couple of things I'd say.

9:13One is, you know, we, we, we make the investments in Anthropoc because we think it's a, it's a good bet. They have a very good team. They've made some incredible traction in the market. and we really like where they're innovating. Yeah, we're definitely clawed heads on the show. Yeah, it's a fantastic product, right? And Daria and team are very good and they continue to actually attract some of the best talent out there in the market today. You know, the other thing that we get from that is a deep collaboration on Tranium. And, you know, we've made a big bet on Tranium as an additional option for customers.

9:46You know, the vast majority of... We should define it. Yeah, I was going to get there. Oh, sorry, go ahead. I'll just say it. the chips that people can, that companies can use to train their own models with. That's right. At AWS. That's right. And so today, the vast majority of AI processing, whether it's inference or training, is done on NVIDIA GPUs. And we're a huge partner of NVIDIA. We will be for a long time. And I think that, and they make fantastic products, by the way. And they continue to do that. And when black hole chips come out, I think people are very excited about that next generation platform.

10:13But we also think that customers want choice. And we've seen that time and time again. We've done that with general purpose processors. We have our own custom general purpose processor called Graviton. And so we actually went and built our own AI chips. The first version was called Tranium. And we launched Tranium 1 a couple of years ago in 2002. And our just-yaing Tranium 2 here at reInvent. So that's news that's happening this week. That is happening. That will be announced in my keynote. Yes. However, this is filming. It may have already happened. We're going to release the transcript as your keynote hits in the podcast a day later.

10:47Great. But this is brand new news, fresh off the press. Fresh off the press. And so we'll have Tranium 2. And Tranium 2 gives really differentiated performance. We see 30 % to 40 % price performance games versus our instances that are GPU powered today. So we're very excited about Tranium 2. And customers are really excited about that. And what Anthropic gives us, back to your question, is a leading frontier model provider that can really work deeply to build the very largest clusters that have ever been built with this new technology where we can learn from them, right? And just learn what's working, what's not, what are the things you need accelerated so that training three and training four and training five and training six can all get better as we continue to go.

11:27And the software associated with GPUs gets better or the accelerators gets better as well. I think that's one of the things where people who've tried to build accelerated platforms before have fallen down is the software support has not been as good as NVIDIA software support is fantastic. And so that's a big area where they're helping us as well as we help iron out the creeks and the kinks and try to figure out how we make sure that developers can start to use these Tranium 2 chips in a very seamless way and high performance way. And so we learn a lot from them as big users that are really leaning in and help us learn.

11:58And they get benefits from they get that scale and cost benefit of running on this price performance platform that gives them a huge win. and we think then from that investment, we can both benefit as they deliver better and better models over time. There's an interesting thing that happens when I speak with people who are working in cloud or working to train models or working to build their own chips. There's always a preface. We love working with NVIDIA and we're also building chips that compete with what they do. So how does that relationship work out? They don't get upset that you're trying to build the same.

12:30I mean, they have a supply issue, but how does it work with them? No, I have a great relationship with NVIDIA and Jensen. And this is a thing that we've done before. We have a fantastic relationship with Intel and AMD, and we produce our own general purpose processors. And it's a big world out there, and there's a lot of market for lots of different use cases. That's not one that's going to be the winner. There's going to be use cases where people are going to want to use GPUs, and there's going to be use cases where people are going to find training to be the best case. There are use cases where people find that our Intel instances are the best choice for them.

13:04There are ones where they find that the AMD instances are the best choice for them. And there's increasingly a large set where they find Graviton, which is our purpose-built, general-purpose processor, is the right fit for them. And it doesn't mean that we don't have great relationships with Intel and AMD, and it means we'll continue to have a great relationship with NVIDIA. Because for them, and for us, it's incredibly important for NVIDIA processors and GPU-powered processors to perform great on AWS. And so we are doubling down our investment to make sure that NVIDIA performs outstanding in AWS.

13:34We want it to be the best place for people to run GPU-based workloads. And I expect it will continue to be for a really long time. What's the buying process like with NVIDIA? Because you want as many chips as you can get, I would imagine. You have Elon who buys them by the truckloads. You have Zuckerberg who has been buying lots. And I think he wants to power them with a nuclear submarine or something like that. So do you have to jostle with the other companies to get NVIDIA chips or do you get every exact quantity you want? NVIDIA is very fair about how they go about it. How do they do that? I mean, you can ask them about how they internally allocate.

14:07That's not really a question for me. It's for them. But they're very fair in dealing with us and we give long-term forecasts and they tell us what they can supply. And we all know that there's been shortages in the last couple of years specifically as demand has really ramped up. And they've been great about ensuring that we get, you know, enough to support our joint customers as much as possible. What about your inference chips? Inferentia? Yeah. Because last time I heard you speak, you said that the activity within AI right now, Gen AI, is 50 % training, 50 % inference. Does that ratio still hold?

14:43And how are you going to put the chips out there to allow companies to be able to do cheaper inference? Because that's the issue with generative AI. It works well. But it's so expensive that companies take proof of concepts and only one fifth actually make them out into production. Yeah, it's absolutely the case. And I think we're still probably seeing about that ratio of 50-50. I think more and more it's more inference than training. And increasingly, we'll see more and more of the workload shift that way. Cost is a super important factor that many of our customers are definitely worried about and thinking about on a daily basis.

15:17And, you know, if you think about where a lot of people were, they went and did a bunch of these Gen AI tests, right, where they did proof of concepts and they launched hundreds of proof of concepts across the enterprise without really paying attention to, like, what was the value going to be or anything like that. And now they're looking at them and they say, well, the ROI is not really there. They're not really integrated into my production environment. They're just kind of these POCs that I'm not getting a lot of value out of. And they're expensive, as you mentioned. So two things that people are thinking about is one, how do I lower the cost so that I make that the cost much lower to run?

15:47And that's the point about cost of inference. And two, how do I actually get more value out of that? So the ROI equation just completely shifts and makes more sense. And it turns out it's probably not all hundred of those. It's probably two or three or five of those that are really valuable. And so there's a couple of things. Number one is on the cost side, as there's a few things that we're doing to help people lower costs. Number one is I think Tranium 2 will be a material impact there. And as these models have gotten bigger and bigger, you mentioned Inferentia. Originally, we had a small chip called Inferentia that would run really fast, lightweight inference.

16:19Now, as you're running models that have billions, tens of billions, hundreds of billions, trillions of parameters, they're way too big to fit on these small inference chips. And effectively, they're running on the same training chips. Like, they're the exact same things. And so you run inference today on H100s, H200s, or you run inference today on Tranium 2s or Tranium 1s. And so we may come out over time with other Inferentia chips, as you will, but they're really using a lot of that same architecture, and they're still really large servers. And so we actually expect that Tranium 2 is going to be a fantastic inference platform.

16:50Our naming is not necessarily always our suit as to what these chips are for, but it's going to be a fantastic inference platform. We actually think it'll be, as you think about that 30 to 40 % price performance benefit that customers are going to get. Now, if you can run inference at 30 to 40 % cheaper compared to the leading GPU-based platforms, that's a pretty big price decrease. And then there's a couple of, there's also announced at reInvent, we're launching automated model distillation inside of Bedrock. And what that makes, lets you do is you can take one of these really large models that's really good at answering questions.

17:22You can feed it all your prompts and things for the specific use case you're going to want. And it'll automatically tune a smaller model based on those outputs and kind of teach a smaller model to be an expert only in the area that you want with regards to reasoning and answering. So you can get these smaller, cheaper models, say like a Lama 8b model as opposed to Lama 405b model. Cheaper to run, faster to run, and you can still get it to be an expert at the narrow use case that you want it to be. And so that combined with a cheaper infrastructure, we think is one of the things that is really going to help people lower their costs and be able to do more and more inference and production.

17:57Yeah, those small models seem to be the cost solution. Sounds like you're a believer. One more question about NVIDIA. You've tested the new Blackwell chip. Is it the real deal? We have, you know, they're working on getting the yields up and getting it into production, but we're excited about that. And also at reInvent, we're going to announce that the P6, which is the Blackwell-based instance that's coming early next year. And we're excited about that. I think customers, I think we're expecting about two and a half times the compute performance out of a Blackwell chip that you get out of an H100.

18:30And so that's a pretty big win for customers. So you're in with Jensen's. The more you spend, the more you save. That's right. That team has executed quite well, and they continue to deliver huge improvements in performance, and we're happy to make those available for customers. Okay, should we talk about ROI? Sure. All right, two-year anniversary of ChatGPT. All these companies have rushed to put generative AI in their products. Yeah. To this point, there's a couple of things that I've heard that have worked well. Yeah. AI for coding. Mm-hmm. AI that is a customer service chatbot. with a little more juice.

19:09AI that can read unstructured documents and make little sense of them. Those are the three big ones. I haven't heard much more outside of that. We're talking about something that's added trillions of dollars potentially to public company market caps, something that has had the largest VC funding round and then probably the subsequent three after that. Yeah. Are the three examples that I listed enough to make this worth the money? No, definitely not. But they're super valuable right now and they're just the tip of the iceberg. And that's the thing. It's like you just have to look at the rest of the iceberg to realize how big the opportunity is.

19:41And on those three, look, I think those are actually massive opportunities by themselves. We have a number of announcements here at RandVent around QDeveloper and making developers and their whole lifecycle more valuable. And you think about the first generation, just using this as an example, the first generation of even developers was just code suggestions, right? It was in like code suggestion. Super valuable, actually. It made developers much more efficient being able to code. It turns out also developers on average code about one hour a day. The rest of their day is spent doing documentation, it's spent writing unit tests, it's spent doing code reviews, it's spent going to meetings, it's spent doing upgrading existing applications, doing all that stuff that's not writing code.

20:17Maybe some ping pong in there. Yeah. And so as part of that, we're actually launching a bunch of new agents that do all of those things for you. You can just type in slash test, and it'll actually automatically write unit tests for you as you're sitting there coding. You can have a Q developer agent write documentation for you as you're writing code. And so you can have really well-documented code, and you're done, and you don't have to go think about it. It'll even do code reviews and look for where you have risky parts of your code, where you maybe have open source, or your parts that you want that you should go look at and think about what the licensing rules are around, how you think about where, even from deployment, where you may want to think about how you're deploying stuff.

20:52Things that you would expect out of somebody doing a code review for you before you go do a deployment, Q can now do all that for you. Same on the contact center side, right? We're doing a ton of announcements around Connect, which is our contact center in the cloud offering, making it much more efficient. So for customers to get a ton out of that contact center, all powered by generative AI. And to your point, that's just, you know, those use cases, I think, get more and more valuable as you add more capabilities. And, you know, I think if you think about where things are going, it is a lot more about, you know, if you think about how I talked about code generation moving to a bunch of the value, it's adding agents in there so they can do a bunch of these things, right?

21:34Now it's not just giving you code suggestions. It's actually going and doing stuff for you, right? It's writing documentation for you. It's helping you identify and troubleshoot where you have operations issues. And it says, ooh, you have an operations issue. And it can look and understand your whole environment. You can interact with it. And you and Q together can go and look and say, ooh, it looks like some permissions over here were broken. And if you go fix those, maybe this is something that you can automatically, you know, it'll fix your application. So saving tons of time across that whole development lifecycle.