Amazon Reveals Its AI Master Plan — With Matt Wood

26:42You put the constraints on and constraints are probably the single largest force that we have to improve the capabilities of these LLMs. Give a practical example. Practical example would be in my$1 ,000 to$2 ,000 example. you constrain it to a set of markets. You constrain it to a set of stocks. You constrain it to a set of financial products. You constrain it to a set of operations and buy and sell operations that you would do in a particular given time. And every layer of constraint that you add reduces the chance that the language model will just create a spurious, erroneous output, but also just keeps the whole thing grounded and keeps the whole thing focused.

27:24And when we've looked at these approaches inside the company, you know, they end up kind of acting like humans, like they argue, and sometimes they get stuck in a loop, and you need to go in there and intervene. And other times you need a tiebreaker, because there's two equally good ideas and two equally good options, and you need some, you need another agent or person to come in and tiebreak. And so it's very interesting watching these very tightly constrained ants run around trying to build things on your behalf. And you think this is going to be useful? Absolutely, because it drives levels of automation for solving complex problems, either completely automatically in cases where you would want that, or in tandem with one or more people where you would want that.

28:08Yep. Let's talk a little bit more. You said there's some chat applications. You helped Bloomberg work on Bloomberg GPT, which is their chatbot that queries financial data. So talk a little bit about that process. Like, is that the same? Are they coming in and saying, we're going to pick our model, but we're going to use your software to fill in the blank? Yeah, that's right. So Bloomberg has, like a lot of customers, actually, they have huge amounts of text information. And so they were able to take all of that text data, their natural language data from market reports and analyst reports and everything that they've, news, everything they've accumulated over however long Bloomberg had been operating, 50 years, I don't know.

28:48They were able to take all of that and then load it into the cloud on AWS and then use a machine learning algorithm to build their own chatbot, which is Bloomberg GPT. And they ran all of that inside our cloud computing infrastructure. Interesting. So where does Amazon get paid in that loop? We get paid by providing compute capacity to actually do the model training and for providing the access to the large amounts of storage that are needed to store the data and then get that data into the machine learning models. And so we provide that capability on a pay-as-you-go basis as if it was a utility.

29:25And so you only pay for what you use, and we meter it by the second. So for one second, you pay us a certain amount. Yeah, so this sounds pretty expensive to me. I'm curious, I mean, and it does seem like it's the province of companies that have more money. I mean, the fact that we're talking about Bloomberg, I mean, is sort of, okay, that's sort of indicative of the companies with the resources to build these models. So, tell us a little bit about the cost factor and, you know, who can actually afford this stuff. Yeah, I think training net new models is not going to be very common. It's very complicated.

30:01It is expensive, to your point. You need a lot of compute capacity, a lot of data, a lot of expertise. Some folks that have differentiation in one of those three things will want to invest there. I think it makes sense. But the vast majority will not want to invest there. Now, that said, we want to, from the Amazon side, make it as cheap and easy as possible to train those models in the first place. And so we've been investing in custom silicon in order to be able to accelerate that process with chips that are specifically designed and built for large-scale machine learning training. And then once you've got the model, you want to be able to operate it in as low cost as possible.

30:41And whilst a lot of focus is put on training, if you think about it, you may train a model once a month, once a week, let's say. But you're going to be running predictions and inference and chatting with that model hundreds, thousands, tens of thousands of times a day. And so if you're not careful, actually the vast majority of that cost isn't in the training, although it can be expensive. it is in the operationalizing of the model for actually doing the chat. And that's why we have a second chip, which is specifically designed for low cost, low latency inference. We call it inferential. And I'm paying for the compute on that.

31:15What do you think the cheapest is for someone who wants to build their own bot with AWS? Like what's the entry level price? If you use an existing foundational model, I don't know, 10 cents. Really? just to build it. 10 cents to build it, and then the cost of running it is priced per token. Yeah. I was speaking to Michael Shmulek from Bernstein. He's a financial analyst. He follows Amazon closely. And I was like, Michael, what should I ask? And he said, well, look, this is going to cost a lot of money. Microsoft just said that they're making something like a$3 billion infrastructure investment in this.

31:56Is Amazon thinking about making an investment anywhere in that range or has it already? Well, I don't think we're going to release the size of the investment. But when you're thinking about the size and scope of possible investments, they don't get much larger than custom building and fabricating chips. And so that is a huge investment that we've been making at AWS for nearly a decade now. You know, we're on our second generation of our inference chips. We're on our first generation of our training chips. We're going to keep investing in those and we're going to see, I don't think we're anywhere near the point of diminishing returns in terms of the capability and price performance improvements that we can provide through those chips.

32:35And so I say, I don't know what they, I don't know that the raw number is actually all that interesting. What's more interesting is what's the outcome and is that outcome truly benefiting this broad democratization that we're seeing? Okay, so you've mentioned chips. We've talked about your own model. So I want to take a break quickly and then come back to talk about those two things. And I have another post-it that I'm holding with me and the headline is fun. So stay tuned. It sounds great. I'll be back right after this. These days, it feels like every dollar should be working a little harder, but figuring out where to put your cash can be confusing.

33:08That's where Wealthfront comes in. Wealthfront is a tech-driven financial platform built to help you grow your savings into long-term wealth. Their high-yield cash account through program banks offers a 3.5 % APY on your uninvested cash as of November 7th, 2025. And there are no monthly fees, no minimum or maximum balance to earn that rate. And you can even make free instant withdrawals to eligible accounts in just minutes, 24-7. So your money can always be within reach. Right now, Wealthfront is offering new clients an extra 0.65 % APY over the base rate for three months on up to a$150 ,000 balance.

33:44That's a total of 4.15 % variable APY when you open your first cash account. Go to wealthfront.com slash big tech to sign up today. This is a paid testimonial for Wealthfront. It may not reflect the experience of others, and there's no guarantee of future performance or success. Wealthfront brokerage is not a bait. Rate is subject to change. Promo terms and conditions apply. For more information, see the episode description. Capital One's tech team isn't just talking about multi-agentic AI. They already deployed one. It's called Chat Concierge, and it's simplifying car shopping. Using self-reflection and layered reasoning with live API checks.

34:18It doesn't just help buyers find a car they love. It helps schedule a test drive, get pre-approved for financing, and estimate trade in value. Advanced, intuitive, and deployed. That's how they stack. That's technology at Capital One.

34:39And we're back here with Matt Wood. He's a VP of product at AWS focused on AI. So, I mentioned that you have your own models, your own LLMs, and that's actually something that's available if people want to build within Bedrock. They can pick, it's Titan, or they can pick something from OpenAI or Lama. Anthropic or AI21 Labs. We had a co-here this morning. So, why build your own? I mean, it seems so good up until the point where you start building your own model. And now all of a sudden, you're running into the same problem that we talked about in the first step that Microsoft has, where like, if I'm building something, you know, I don't know if Amazon really is neutral.

35:25So talk a little bit about why you built your own one. And Shmuel said the same thing, too. He's like, you don't need an 81st model. So why did Amazon build it? Well, on the 81st model, I think you do need an 81st model right now. It would be completely arbitrary to decide right now, at this point in time, that we need to limit the model or that we've got enough. There is so much opportunity. It is so early. I think there's going to be no end of invention in the foundational models going forward. So that said, that's why we took our approach of making all of them available because who knows which one is going to have a breakout capability.

36:03Who knows which one is going to be the best fit for a particular use case. Our approach has been that we have training, we've trained a set of our own foundational models. We have a language model and we have a vector embedding model. And each of those models is actually a family of models. So the customers can choose the right model for their use case, not just for the capability, but also for the latency and also for the price. And so you may have a use case for a very simple, small model. that you want to operate with very, very low latency. And so that's an option that you have with Titan.

36:42You don't have that option with some of these extraordinarily large models, particularly that are hosted in who knows where, where the latency is just what you get. But with Titan, you can choose the right trade-off for capability and latency. You can choose the right trade-off for capability and price. And for each of those models, you can add your own data to the model to improve it privately. Yeah. Now I'm going to get to chips, but it's always at this point in the conversation, Like we're a little bit more than halfway in, or there's always a thought that pops up in my head, which is we've been talking about generative AI, wanting to talk to computers as a given, as if we actually want to interact with them in natural language.

37:19We want to have them as chatbots. We want to talk to them. But even the Alexa example shows that like there was all of us, there originally was this whole range of things we wanted to do. And then the test narrowed. And even, I don't know what you're, actually, let me ask you, are you using LLM, like ChatGPT and consumer bots as much now as you were at the beginning? I use, we have an internal tool that we actually built for ourselves, primarily so that our engineers and our own builders could get familiarity and practice with prompt engineering. And so it's super simple internal tool. What's it called?

37:58It's called the LLM Playground. and it is literally a playground. You can build little one-page mini apps where you can provide a prompt and you can chain that prompt into another prompt and you can play with the parameters and the different models and then you can arrange the widgets on the screen to build out little applications. You can build a chat application that way. You can provide it a URL and it will fetch the website and then use that as the part of the context for the prompt so you can reason and ask questions about the website. So you're using this stuff. So, okay. It's great. That's good.

38:33It's the most fun I have all week, honestly. Really? It's awesome. Yeah. How many hours do you spend doing it? How many hours? I don't know. I probably spend at least 30 minutes a day just exploring what's capable and exploring what the team is identifying as kind of emerging capabilities. Okay. So now that I've effectively sabotaged my own question for like the last minute and a half, I'm going to ask it. Hit it. Which is, is this something that people actually want to do? Like, do they want to talk to computers? I mean, it sounds good. it's really freaking cool when you use it. But even now, people are saying chat GPT is getting dumber.

39:04Largely, people believe that's because the novelty has worn off. So we talk about this generative AI moment that you're saying it's going to be bigger than the internet. What makes you so convinced that this time is for real? What makes me so convinced is that the level of invention and efficiency and automation that we've seen inside the company and that our customers are experiencing, chat being just one modality. So what else do we have? Well, we have the other ones that I mentioned earlier, like the generative pieces, the search pieces, but the collaborative problem-solving pieces, the automation pieces, completing complex tasks.

39:44That, I think, is where the majority of the value is going to be. And I think that chat is a great user interface. It's a great way to explore some knowledge and a domain. It's great for new users that are getting up to speed on a particular product, all of that. So it's a really useful use case, but it is very hard for me to imagine that we nailed the use case first time right out of the gate with chat. I think that there is going to be all manner of model improvements and supporting engine improvements that allow us to deliver customer experiences that we just haven't even imagined yet. And we are doing that imagination and going through that process inside the company now.

40:23I like a lot of our customers at AWS, and it is inspiring. Anything cool from inside Amazon you can share? I think some of the early stuff that we're looking at, we announced today around generative BI. Business intelligence. Business intelligence, thank you. To be able to ask and interact with your data just using a chat interface, one. But also to be able to create dashboards, two. To be able to understand and find insights, number three. And then number four, when you found those insights, to be able to quickly just summarize your narrative, immediately create a business report that includes all of the summaries and all of the reports and charts that you might need.

41:04And then email that around to your colleagues. If you imagine the level of what you would have to have done before these capabilities were available to be able to enable that, you would have needed to have teams of business analysts to connect to the data. They would have had to spend time investigating what to build and then building the dashboard and setting it up. And then you have to train everybody in order to be able to do the work with the data. And then you have to find the insights, which is very, very difficult. It just shortens that whole path to discovery through automation in a way which is unprecedented.

41:39So who's learning from who here? And please don't say both. Is it AWS learning from the rest of the workflow inside Amazon or people inside Amazon learning from AWS?

41:52I think it's honestly true that, I mean, Amazon is a very big company. Right. I think we're taking inspiration and we're kind of organizing ourselves internally, deliberately to take inspiration where we find it. And so, number one, we're enabling all of our builders, all of our software engineers, in which we have a pretty large number, to be able to experiment and try out large language models through Bedrock. So, everybody has that capability. And then, we're finding ways that they can show their thinking and their invention to each other with something as simple as a demo day. So internally, we have multiple different teams, not a lot of them, but multiple different teams.

42:34And they will proactively reach out. And we have a schedule of people that are bringing their demos. And sometimes it's just slideware. Sometimes it's an idea. But more often than not, it is running software that we can take a look at. And they share their thought process and their implementation techniques. And that gets everybody else excited. And then the next iteration, we're building on top of that. And round and round it goes. And so those kind of idea nucleation points across the company have proven to be very inspiring to our developers. And number two, enables us to share our knowledge and our discovery and our thinking very, very broadly.

43:09And number three, honestly, prevented us from building the same thing. Let's say we've got a thousand development teams. You start them all off from the start line at the same time. Come up with the same idea. They're going to come up with the same idea. A lot of them will. So we've avoided that as well. Let's talk briefly about chips. Sure. it's interesting because I think that there's minimal awareness that Amazon has a its own LLM in Titan. And there's even, I think, less awareness in the general public. I'm not talking about the folks sitting here. I'm sure they all know about it, but we hear so much about Nvidia chips.

43:42I swear, I feel like every day I hear Nvidia, like it's just a marching, you know, chant in my head, Nvidia chips, Nvidia chips, Nvidia chips, but you have your own chips. How are you making them and are they serving the same purpose or something slightly different? Well, we acquired a... This is all for chips for training AI. That's right. Yeah, that's right. We acquired a chip design company called Annapurna probably about 10 years ago now. And since then, we've been on a path to build ARM processors for general purpose computing. We have ARM processors specifically for high performance computing.

44:20and to find very specific, not a large number, but very specific use cases that we could accelerate in silicon. And the machine learning use cases very quickly rose to the top. And so we've been investing there in terms of building out custom silicon that you can deploy on AWS today for building out your own large language models and for running the inference against large language models. And you design your own chips as well? That's right, yeah. And it's not just ARM that builds it, right? Arm's just a blueprint. It's a starting point. But with Tranium and with Inferentia, those are totally custom designed.

44:56Who's building it? I think they're constructed in Asia somewhere. I don't exactly know where. Taiwan? Most likely, yeah. Okay. Let's go to the fun post-it because I feel like people are getting restless. Okay. Now I'm nervous. No, it's all good. I think a fun post-it for you is a nervous-inducing post-it for you. That is how it should be. Oh, ethics. All right. So you have your own LLM, Titan. Yep. What was it trained on? And how can I be sure as a writer that it wasn't trained on my work? Titan was trained on publicly available data. That's a very squishy phrase. It's very precise. Okay. And proprietary data that we had licensed specifically for the purpose of training.

45:43Okay. So can you say definitively that there's no chance that this model was trained on, like, for instance, Substack articles? If they're publicly available, there's a good chance that they were part of the web crawl that we would have used. If they were part of or had been licensed to a large proprietary set of natural language, we would have licensed that with the right permissions to be able to use them for training. So my Substack stories are publicly available. They're available on the Internet. I guess I didn't really opt in for them to be used as part of training. Should I have the ability to decide whether or not they're going to be part of LLM training or not, even though they are live on the web?

46:22It seems like, I mean, it's your content. You own it. You can do as you please. But I think it seems like a strange use case to identify and single out. Who's to say what this data can be used for when it's publicly available? You chose to make it publicly available. If you want to put some permissions around it, or you want to take it private, it's totally up to you. You still own the data. Right. But it is public. And as a result, it can be used for things that you may not have thought through initially or that weren't possible early on. Yeah. And I'm not sitting here and saying, you know, how dare you take it.

46:59No, I understand. I understand. But it is interesting. It's something that people who are producing content are having. We always thought it would just be Google, for instance, that crawled our stuff. But clearly, it's going to be more. Yeah, I mean, there are open source, open web crawls available as open data, for example. You can just go and look at what's in there. It's maintained and kept up to date. It's called the Common Crawl. You can check that out. Sergey Brin is back inside Alphabet. He's called generative AI something like the most exciting technology or moment of his entire life. Jeff Bezos, do you know what his feeling is about this stuff?

47:41Should you? I do. I think he feels that it is, I wouldn't want to speak on his behalf, of course, but, you know, I think he feels the same. This is the single largest transformational step in how we interact with data and information and each other, you know, since the very earliest web browsers. And so I think I probably stole that from him at some point. How do you think he would feel if people inside Amazon started using generative AI for their six pagers? that ship has sailed, I can tell you. People are doing it? For sure, yeah. What? Of course. Okay, but hold on. It's a starting point. Wait a second, because the whole point, if I have it right from Bezos, is that when you write something, you have to think it through super deeply and make sure every idea connects one-to-one.

48:28If you turn that over to AI, you're not really going through the process. Well, I don't think that's true because what you're getting back is just a first draft. and so that's actually a really good way to explore your idea you can get a gut check as to whether your idea kind of tracks whether it's got legs and you can start to poke and prod at the idea and all of our ideas get better through that poking and prodding and the discussions that we have around their ideas and so to be able to do more of that early on you actually front load a lot of the product development work and you can do some of that with your team you can do some of it on your own, do some of it with an LLM.

49:10I think it makes perfect sense. It's a huge efficiency gain. When you read one of these six pages written with an LLM, can you tell that it's been involved in the process? I don't know if I've read one that was completely written autonomously with no edits. Even with a little bit of help. I think for sure, for sure, that I have read paragraphs, maybe even pages that were automatically generated with probably some pretty heavy editing that I did not notice. That's good. It's very encouraging. Two more for you. Okay. Amazon's culture. The whole point, I mean, I wrote a book, the title is called Always Day One.

49:53So the point of the book is that the company operates as if it's a startup on its first day. and the culture has been extremely intentionally built that way by bezos and there was a story recently about how amazon has more of a big company feel uh lately and you even have um adam from aws i want to make sure i get the language right saying uh you know we're going to be insurgents and you only say this word we're going to be insurgents when you feel like you need that rallying cry. What's the story? I see your theory. It's an interesting theory. I personally have not seen any, well, number one, I haven't really known what big companies start.

50:40I've only ever really worked at Amazon. And so I've been here for a long time. I'm pretty well entrenched in the culture. But I haven't seen many elements of big company slowness. I haven't seen many elements of big company politics. I haven't seen many elements of big company infrugality or wastage. And so I think, I'm sure there's many more that you could list off that would be qualities of big company-ness that would be negative. What I have seen is a continued focus on working backwards from the customer. And what I have seen is a continued focus on scrappiness. and a continued focus on doing what we need to do in order to be able to solve real problems on behalf of customers.

51:27I think you'll see that in our approach to generative AI. You'll see it in our approach to analytics and satellites and all sorts of things. Yeah, because you really need that sort of scrappiness if you're going to be able to compete. I mean, I feel stupid even saying this out loud, someone who's worked at Amazon for as long as you have, but this is going to be a fight, man. It's going to be a very, very interesting time for sure. And, you know, I would also say that the sort of cultural norms that we'd have, they don't exist and they're not maintained without some energy and without some effort.

52:04And anyone that has had a conference call with me from my office will have seen that behind me on my office wall, I fly a pirate flag. Partly an homage to Steve Jobs. You have the pirate? I have the pirate flag. When I was writing the book, I heard about this pirate mentality. Yes. And I could never nail it down. Talk a little bit about it. Yeah, this is good. It's part homage to Steve Jobs and the early Mac team. Very early day one, drove tons of transformation. I'm a huge fan of Apple, huge fan of the Mac. I've used Macs all my life. So part of it is an homage to that. But part of it is in periods of discontinuous change, you just can't operate like a big super tanker.

52:44You've got to operate like a small merry band of pirates. that are just cruising and adventuring around every cove that you can think about and just staying scrappy and nimble. And so for my part, such as it is, I fly the flag in my office as a reminder to myself and any of the teams that I'm working with that this is a period of discontinuous change. And this is a time in which we need to be scrappy and resilient and explorers and missionaries. And the people are listening? So far, so good. People seem to light my flag. All right. This is not the last one, but I have to ask you about this. The news is gearing up for the FTC to bring up a lawsuit to break up Amazon.

53:32Obviously, it hasn't happened yet. It's all speculation. But it seems like it will. And it's going to be potentially the biggest government action against a U.S. company since Microsoft, maybe even Mavel. So do you think about that at all? Is it even something that you pay attention to? That one is above my pay grade. Okay. Last question for you. You know, you have a very interesting position within Amazon because you're like really working industry by industry and helping them imagine how they're going to transform with the latest technology. But we're in the middle of this really unbelievable moment in technology where we're starting to really get a chance to imagine things we couldn't before.

54:15Or one example, you guys have released a medical note-taking generative AI application, HealthScribe. So I'm a son of a foot doctor. And my dad spent too big of a chunk of his life writing notes. And just think about all the hours he could have had back. I went to med school before I joined Amazon. So you know. And it's just, think about how much better care you can provide to patients if you're actually focused on that versus doing these things that generative AI applications can do. So why don't you take us on a top three interesting things in different industries that you could imagine generative AI having a real impact in?

55:00I think that's, the medical one is interesting. That's a really good one. What else, where are some other examples that we're just not looking at yet? Well, number one, I am sure that everything I'm going to touch on, that there is a startup or even a large organization out there already working on it. And they can, they'll probably be getting ready to ship as we speak. There's just so much investment and activity happening on this area. I think that a couple that spring to mind. The first one is cybersecurity. There seems like such an opportunity to employ these language models in the identification of the very subtle signals that have become harder and harder to identify, which indicate some sort of vulnerability or threat.

55:46And so being able to identify those threats across multiple different sources with better precision is going to be better for everybody. I think that's one. It's not really an industry, but I think it's one that's going to be important. That still counts. Okay, good. I think another one is just going to be developer productivity, like code generation. We didn't really talk about that yet. Such a large accelerant. I think there's going to be... Whenever I talk about that with CEOs, they're like, yeah, we're doing it, but they haven't really seen the productivity increase. Now, I know you've seen it internally, but...

56:16We for sure have seen it internally, and we've heard from our customers that... There's definite. Yeah, that there's... It's also early, and I wouldn't be at all surprised if customers are still trialing it out and getting a sense for it. that developers are very used to a particular workflow and changes to that workflow can take time to get right. And they should be thoughtful about it. But I think that's another one that's really going to drive change. And then just more generally, I think any industry that has access to very, very large volumes of text is going to be the first places that we see this sort of change.

56:49And what's interesting about that is there's some areas, like healthcare, that are steeped in natural language. but they don't usually have like the best reputation for being at the vanguard for technology adoption we're seeing so much interest and so much excitement health care lawyers legal health care life sciences clinical trials drug discovery all these areas where financial services insurance like the oldest stodgiest industries that you can imagine they've got so much natural language, it's such a large opportunity. I think that's where we'll see the earliest returns potentially on generative AI.

57:29Matt, can you believe this? I mean, what a great crowd. I love these guys. So for listeners at home, what we're looking at is it's a silent disco type of conversation where everybody is wearing headphones. We're not even using any amplified sound at all. And just been looking at this crowd as we've gone and they've sat here and hung on every word. So thanks to you guys. This is awesome. Thank you so much. Appreciate it. Thank you for being here.

57:59And I'm going to thank Matt in a second, but I'd be remiss if I didn't mention that the Big Technology Podcast airs every Wednesday and Friday, Wednesday, a flagship interview. Like this Friday, we cover the news. All right, everybody. Thank you so much. Thank you to you. Thank you to Matt. Thank you. And I hope you enjoy the rest of your day. Thanks.

58:20Thank you.