#753: Amazon Bedrock Mantle and Developing at the Speed of AI

0:00This is episode 753 of the AWS podcast, released on January 26th, 2026. Hello everyone and welcome back to the AWS Podcast. I'm Les here with you. Great to have you back. I'm joined by a super special guest. I'm joined by Joe McGarimov, who is VP and Distinguished Engineer at AWS. Joe, welcome so much to the podcast. Thanks, Simon. I'm really excited to be here today. It's amazing to have you here. You've been at Amazon for 20 years. Not many times I meet Amazonians that have had more tenure than myself, but you're well into there. Yeah, you've seen some stuff and you've written so much code and led teams and helped teams write a lot of code that I'm sure many of our customers use each and every day, whether they know to or not.

0:44Just to give us some context, I mean, 20 years at Amazon, what are some of the things you've worked on as an engineer during that time? Oh, wow. It might be easier to say what I haven't worked on, but it's kind of interesting. My time at Amazon kind of has two halves. So I spent the first half, the first 10 years working on all the systems behind our Amazon.com retail website. So I worked on some of our shipping systems, some of our payment systems, a number of systems behind our marketplace. And about 10 years ago, I transitioned to work in the cloud where I worked on our computer networking services.

1:22So I've worked on services like VPC or load balancers, NAT gateways, and all other types of gateways you would have in networking. And then I also worked a lot of time with our container and serverless serverlesses, so ECS, Lambda, UKS. And now last year, I've been working on Bedrock, an inference platform for Amazon. And we worked on Project Mantle, which is something we released this year and something I've been super excited and passionate about. Yeah, just got mentioned and called out in reInvent as well. I guess just to spin, I mean, all that subset of things you worked on, as you mentioned, I mean, I think the characteristics there are they're very much about scale.

2:06They're about robustness and reliability. I mean, you're not working on stuff that, well, if it's an error, there's an error, it's okay. You know, the user will retry. This is like, you know, trillions of packets, real-time processing, mission like this is the stuff that keeps you up at night if you get it wrong i'm guessing that's right yeah it's uh you know reliability and scale is at the heart of a lot of what we do at amazon clearly scale makes a lot of things more interesting and more challenging so you constantly have to think about not just only how do you get the scale right but also how do you get it right in a way that doesn't add too much complexity because complexity tends to be the enemy of reliability and so there's this constant tension behind engineering for scale versus engineering for reliability, and you have to kind of navigate that tension, navigate getting things right, but not too complicated, not too complex.

2:55Yeah, it's one of those classic things if you know it when you see it, but it only takes a whole bunch of years of experience to figure that out. But also, something else I want to call out that our listeners can't see, but I can see, because I can see Joe on video here. Behind Joe is, I don't know what the collective now would be, but I'm going to call it a wodge of patent puzzle pieces. So at Amazon, if we get a patent, we get a puzzle piece, which is very exciting and great. And so it's always good to do that. And I've always been proud of my own patents. I have nine, a small number of nine.

3:26Joe does not have nine. Joe, how many patents are sitting behind there? There's a lot. It's hard to say. It's about, it's a cube of about, let's see, six by six by two. So I'm guessing around 72. Yeah, it's a lot. It's impressive. Yeah, the funny story at Amazon, for folks who don't know, at Amazon, patents are you get a puzzle piece, like an actual puzzle, a three-dimensional acrylic puzzle piece. And while people have different opinions on software patents, what truly gets me is that I love puzzle pieces. And so this whole concept of stacking things together and building out of them, and you hear a lot about it, is I'm a builder at heart, I'm building stuff.

4:12And so the whole concept of stacking those pieces and building it is really what it feels to me. And that's, I'm afraid to admit, that's by far the most fun part of getting a software pattern. I think you've picked the hardest possible way to get puzzle pieces to make a puzzle, but I love the dedication. I think that's fantastic. So we talked about this thing called Mantle, which is part of Bedrock. Just help us understand, I guess, what problem are we solving, what it's for. And then we're going to get into the guts of how you built it, which is, I think, really a great unpacking of the current way of using AI for software development.

4:47But before we get to that, let's just talk about Mantle and give it its due. Yeah. And so Mantle is a new inference engine underneath Amazon Bedrock. So, of course, starting Bedrock is an inference service for Amazon. It's a public AWS service. and it's a service that's seen tremendous growth, tremendous amount of customer adoption and we've learned a thing or two operating bedrooms. So as we've operated it for a couple of years, we've learned how customers use it, we've learned how inference behaves, we've learned how inference scales. And so with Mantle, it's our realization that at the heart of its inference is not quite a web service, but more of a scheduling system.

5:33And so a request comes in, and there's a lot of concerns you typically see in a scheduling system. Things like prioritization, things like fairness, things like placement, things like call placement, where you want to have multiple requests placed at the same time. And so there's a lot of those concerns come in. And so if you look, if you step back and look at the whole ecosystem, and if Bedrock, you realize that it's, well, what it does is a giant scheduling system. It's our, in an effort to accept that reality, to make sure that Bedrock can offer the best customer experience, we've built this new inference engine underneath the Bedrock.

6:09So if you're using Bedrock today with models like Minimax, GBT-OSS, or Mistral, Mantle system is what's actually placing and executing those requests. And of course, from the customer's point of view, the benefits are twofold. The first one is we want to offer customers the best possible customer experience. So the best latency, the best performance, the highest number of features that the customers want to. But on our side, we also want to do it at, we want to operate this fleet efficiently because it's a large fleet. And as everybody knows these days in France, it's a demand constraint space where there's enough demand for the customer.

6:48And so if you can't quite utilize your resources efficiently, then you're not serving all the customers who want to who want to use the servers. And so on our side, we also want to make sure that we do it as efficiently as possible. And so Mantle was a combination of kind of observations of how inference behaves and us building the inference system that allows us to serve our customers, serve them well, and keeping our utilizations and our costs as good as possible because they don't want to be able to pass them. And then we pass them to our customers. That's right. Exactly. Exactly. And I think one of the great things about that challenge is because of the proliferation of different models and being able to cater for all these kinds of models and kind of being able to run inference efficiently on that is a non-trivial problem.

7:38And I think what's interesting is that you and the team decided when you were going to build this particular service or part of the service that you were taking an AI first development approach. and just before we got into the show, we were having a quick chat and we try not to chat too much before the show because then all the good conversations happen off the show. But we're talking about the fact that what is the way to develop using AI today is not what it was three months ago, it's not what it was 12 months ago, it won't be ordered to use in six months. It's changing all the time. But what really appealed is you and the team have taken an approach and then you've blogged about this approach and used data to talk about what happened, what you've learned.

8:19So we're going to get into the guts of this because I think there are folks who are hungry to hear about this. So firstly, you're a distinguished engineer. As I said, you've got the wadge of patents. You've written more code than most would have, yet you've taken an AI-first approach. What was the thinking here? Let me kind of tell you a little bit because I think everybody had a slightly different path in how they started trying to use AI-based at all. Let me walk you through my path. So, of course, five years ago or four years ago, ChatGPT comes out, LLMs become a mainstream, and one of the capabilities they had was writing code.

9:00So folks, a lot of interest in that industry, a lot of folks trying it. And probably roughly about two years ago for me, I started trying to write code using LLMs. And of course, the results were pretty mixed back then. You can try, you can maybe have it write a little piece of code and occasionally would have to get things right. Twitter is full of all the jokes and all the comments making fun of the code produced by Lollams. And slowly they were getting better and better. And for me, there was about an inflection point where I started being able to use it on actual prototypes. So I could literally have an idea, I could use a model to write a little prototype, try it out.

9:42But yet, I wasn't quite convinced yet whether this is something that could be used in production. So I would totally take the prototype and then turn it into something that I would build. It was usually maybe more as a gimmick or as a tool to help me learn rather than something that would make me more effective. More productive. Yeah. And then maybe about six to nine months ago, I started noticing that I was doing that a lot less often. All of a sudden, the code produced by the model was getting good enough to where I still needed to make modifications. I still needed to constrain it, but all of a sudden it was solving the problems in a more robust way.

10:20And so you could have this steady march of progress. And at the same time, we were also trying to figure out how do we can apply models? How can we apply this learning to actually turn it into real production code? What needs to change in our industry? One is the change in our approach is to where we're actually not just using it for tour exercises and POCs, but actually using it for real production. For real stuff. And so within Manful team, we had a couple of ideas, a couple of thoughts. They're not that fancy. And one of our realizations was that at the end of the day, the model is a tool. And it's a tool that accelerates an engineer.

11:07And so one concrete rule we have, and I think that rule worked out really well for us, is that at the end of the day, any line of code committed into the repository has a human name attached to it. And a human is ultimately responsible for the quality of the code. And so if you look at it that way, it's probably more analogous to a compiler or to a programming language other than to a fully autonomous agent that runs around and modifies the code. And I'm not saying there aren't patterns that could benefit from full autonomous agents, but in our case, we made a decision that a human is the ultimate author of the source code.

11:45Well, that's kind of the continuous extension of you build it, you run it type thing. It's like accountable for all your code, however you produced it. You know, you could have got a barrel of monkeys to make that code. It doesn't matter. That's right. It's still Joe's code. That's right. That's right. And once you have that accountability, you now start a lot of write tensions happen where the engineer responsible for writing code has to figure out how to make the model produce the right quality code. For me, everybody has a slightly different approach, a slightly different pattern. What I do is I give a model a prompt, and we can talk a little bit about it, and that process was interesting.

12:23and then I let the model produce the code and I review it. I review it. I decide my first decision is like well did the model solve my problem in the right way or not or do I agree with the solution and more often than not I do and then I decide did the model you know did the model solve the problem to my liking. Does it have that did it use the right practices? Did it use the right libraries um is it overly complicated or did it miss edge cases and then i iterate on that on that project the way i would almost do it myself i i fix bugs i fix issues uh and at some point you're fixing the code directly at that point or are you prompting the ai to do the coding what's the balance there for you uh both um what i found out works the best for me is that if the model is almost there, but missing maybe a line or two, or maybe missing an edge case or two.

13:16I'll just take over and finish it. Just quicker, faster. I don't mind doing it at all. Occasionally I find that, well, it's not quite how I would have done it. And then we go for a second round. And no different how you would do it with an engineer. You provide feedback, perhaps change data structure here, perhaps let's use a different algorithms. And we kind of iterate on it until we go. Earlier on, though, and this is part of the learning experience, I would actually, and this is me getting into a little bit of kind of how my approach evolved. In the early days, I would literally stay glued to my screen.

13:56And I would look at lines of code appearing, and I would even stop the model in the middle and say, no, no, no, wait. yeah yeah you're going off on a tangent yeah let's change the direction let's try something different as i've gained more better intuitions and gained a little bit more confidence about how the model works i started going becoming more and more in asynchronous where like it's literally i i send the task over the fence i wait for the results i evaluate the results and i repeat Google. And almost universally, it's a multi-step process. We iterate, but I've got to the point where I've also calibrated my prompts enough to actually find out what's just the right level of complexity to ask the agent to do that results are going to be something to my liking.

14:44And I think at the moment, that's one of the sort of techniques or tweaks that's really important. And before we get into that, I'm going to call out what we're going to talk about in a minute because we're talking about this, but we have data that shows 10x development velocity. So we'll get into that. So I want folks to listen carefully to what we're doing here because this can really help. So the prompting is a huge thing. I think at the early stages of quote-unquote vibe coding, people are like, you know, write me Twitter and expect the thing to do stuff. And we've come a long way in terms of understanding, well, you know, the better you prompt, the better the outcome can be, how big should something be, etc.

15:19It's almost like the old microservices conversation we'd have is how big should a microservice be. So tell us about the prompting approach you're currently using. And I'll preface this on your behalf by saying, this is the approach you used today. It doesn't mean it's going to be the best approach tomorrow, but it's what you've learned. Yeah. And I think it hit something important when you said, you know, don't ask the model to write me Twitter. Because the reality of it, like just like anything else in life, you need to calibrate yourself on how to most effectively use the tool, right? Like when I first started software development, I learned, I started with C and I'm terrified to look at my first code.

15:58I didn't know what I was doing. I was likely, I was likely using wrong idioms, bugs galore, asking the language to do too much or something that it wasn't meant to do. And as you hone your skills, as you practice, you kind of get this intuitions about which approaches are likely to work and which approaches are not. And one of the intuitions that I find that is super helpful to build is finding the maximum supportable request from the model, meaning that something, you know, you ask too much and the model is likely to fail. It might run out of the context window or it may too much ambiguity. It does weird stuff.

16:37Yep. Yep. Ask it for too little. And well, you're not getting quite the speed ups because yours still has to be constantly in the loop and asking. And so one of the intuitions you build up and one of the reasons I think that actually doing and trying things is so important is that you build up that intuition of what is that maximally supportable request? And changes over time. It's not a static thing. It changes with the model's abilities, projects, domains. But having that intuition has been incredibly, incredibly helpful. And so for me, I've learned a couple of observations. One is the observation of having that sweet spot of the request helps me get the maximum acceleration, maximum speed up.

17:25The second one is ambiguity. And this is where one of the interesting places is where, you know, we work in a field where many different approaches can solve the same problem. And sometimes you don't care and you can just pick new ones. Other times you may have strong opinions based on your past experiences as an engineer, based on your kind of what you're trying to accomplish. And so helping the model disambiguate also what you want to do turns to be super helpful as well because it sets up the guardrails for what you want the model to operate, which is, by the way, no different than dealing with junior engineers where you want to set up guardrails, you want to set up shared expectations and work from that.

18:07And so a large chunk of what I kind of do, how I operate, is just trying to think through what are the appropriate guardrails, what are the appropriate constraints, what are the high-level approaches I want to do. And for example, it's super common for me to actually brainstorm a problem with the model first before we even start implementing. And so just recently, I was working on something where I didn't quite have a good intuition as myself yet what I wanted to do. And I literally started with like, here's the problem I have. List solutions. The model listed one. Immediately became obvious that one of those was really not going to work.

18:46So we went the other way. We worked back and forth to the point where I finally felt confident that this is going to work. The model understood what I wanted to do or at least the context was there. And then I flipped from the brainstorming mode to, okay, let's go make it happen mode. and so it is you know it is this the tool is extremely flexible it can be used in different ways and you kind of want to take the maximum advantage of it you want to not just um and not just tell it what to do but sometimes also use it to help you figure out what to do and how to do it i think i think that's a really important insight because it's you know if you think about how these models are trained they're trained on huge corpuses of code some good some not so good but lots of code and so it it has it doesn't have an opinion it's a statistical model but it has access to way more code than we can fit into our heads and there's a lot to be said for hey here's the problem domain i'm trying to work on and simply one of the things i've found is really useful with the models are saying ask me questions you know like prompt me to tell you what you need to do to get to a better point and you as you say you start that dialogue and you're still driving, but it's taking you around lines that you may have not even considered and different approaches that just weren't in your mind because, you know, you hadn't had your first coffee yet, you weren't sort of really thinking clearly.

20:06It's fascinating to see, like you said, it's like working with a colleague to some degree of having an interaction. Yeah, and that's exactly right. The model has seen every single implementation of B-Tree out there. I have not. Sometimes it's just brainstorming. I still want to be in the driver's seat. I still want to be the one that makes the ultimate decision the way we go but using the model as that sounding board uh on on problems is oftentimes i found it as a useful first step and a lot of times helps me help helps me decide what i want the model to do before i even go to the because you're still deciding this comes back to you own the code you're you're it's not you know hey model figure something out and then go implement something i have no idea about it's like you still you're and even if it's proposing something maybe you're not that familiar with i'm assuming you probably do a deep dive yourself and And so, well, actually, what is in this for me?

20:57Is this an algorithm I haven't seen before? Is this an approach I hadn't considered? What do I need to know to understand the risks? Yeah, absolutely. And you still review the code. And you still have to keep your judgment on what complexity you want to introduce and when you want to go the tried and tested way versus trying something more performant or more experimental. And you have to still own the decision. And at the end of the day, you have to verify what's being produced. But having that conversation is, I found it often useful to just, because at the end of the day, the clarity of thought, clarity of what you want to do is what provides the acceleration.

21:42The more you have clarity in your own head, the more you kind of dealt with ambiguity, the faster you're going to go. That's been true before. That's true even more so now. And so the models are fantastic tools to just help you gain that clarity as well and kind of brainstorm ideas and try new things. And then, you know, in the old days, we would use Google, we would use Docs. These days, it's just a new way of doing research and a new way of kind of learning. Very, very true. Well, you know, even things like the fact that, you know, there's the AWS MCP with access to all the documentation just saves us time on looking up the document.

22:17I mean, even we at Amazon look at the documentation. Oh, absolutely. In fact, I found out the models, they know AWS services better than me sometimes. Which is crazy because you wrote them. Asking questions and asking about behavior has been a tremendous time saver. Let me touch on context windows briefly because certainly what we're seeing is that if you're running a heavy full context window, So, things start getting squirrely. And so, certainly in my own personal workflow, I'm using the frequent intentional compaction approach and finding great results. I've almost become resolute about, you know, once it gets over 40 % to 60%, I'm compacting because weird stuff happens.

22:59Are you seeing that? Do you manage your context window particularly closely? Oh, like there's no tomorrow. And for listeners who are not familiar, models have context windows, which is the the ultimate kind of limit, constrained resource when dealing with the model. Typically, this range from 100 ,000 to a million tokens. And as soon as the context window reaches the maximum, the model cannot do work anymore. So you have to use techniques like compaction or reduction or kind of long-term memorization to just start working around that. And so it is one of the things that you as an engineer need to actively model and manage.

23:44And what I found out, what I do, I do a couple of things. And there's an interesting conversation we can go into what happens, how the tools need to evolve. But I clear the context window between every request. And so what I try to use, I use a lot of files as a long-term memory for the model. So concretely, some requests fit into a single context window where you want to fix a bug, make a change, and you work on it, you finish, you clear the context window. A lot of things I work on, they take more than, you know, they require multiple iterations. And there, what I find works really well is starting with a file that describes a high-level approach we wouldn't do.

24:29No different than what I would do if I was working on a large problem myself. yourself yeah yeah and then we break it down right you break it down um uh cure that some of that for you with their spec driven development i tend to use the command line tools a lot and i would start with the model i start a prompt with okay here's our here's the design we agreed on we're now on step two out of seven or two out of however many um and here's what we're going to do in this step and this step we're going to be doing blah blah blah and then Just these things. Just this thing, right. And then when we finish it, we commit the code and then we clear the context so we forget everything we've done.

25:08Except that durable file that keeps the track of our intent and the last commit and we continue from there on. And so, currently a lot of it is things I do manually by hand and it's just that's the match really well how I work myself as well. I start with the end-to-end goal but then I break down problem and it just generally how humans tend to work, right? You want to break down the problems, the smaller problems, then go after those. I can imagine longer term, this is going to become part of the built-in tools and this is going to become, this workflow is going to be a lot more natively built in.

25:44But yes, the context window is something in your, kind of in the back of your head, in the back of your mind, and you are effectively managing to it as a constrained resource and breaking down the problems as much as you can to the point where you take maximal advantage of that resource. And the funny thing is, both of us being quite, you know, experienced, let's say, practitioners in the field is that there'll be a time in the next few years where we'll sit back together and go, do you remember when you had to manage context windows? You young kids these days, you don't know what it was like.

26:16Yeah. So, Joe, you mentioned in your blog post 10x development velocity. Now, that's a classic, you know, well, that's got to be marketing gump. There's no way you can prove it. You know, come on. Yeah, and anyway, anytime somebody says use a round number, you have to. Yeah. You should have gone to 12.7x. Yep, yep, yep. So tell us about it. Tell us what that looks like in your team, in that team. Yeah, well, so I don't know exactly what the number is. Is it 9x or 11.3x? I'll tell you from my personal experience. and personally you know our our field when we think about measure productivity that's a that's a topic for many other podcasts because it's such a deep topic and so many strong opinions but at the end of the day the way i view it like well we we engineers we we build software to solve customer problems and then at the end of the day to build software you have to write code and you have to write high quality code but you have to still write code and for me i've always I've always enjoyed writing code.

27:24Time's always been a challenge as I've become a senior, but I would always find an hour or two a day to write some code, to do a little bit of engineering work because I just enjoy doing it.

27:39And with the kind of switch with agentic-first development, I just find that I just accomplish so much more in the same amount of time. And for us, you know, for our team, again, And commits are not the whole story. They're just the slice of the story. But we've written on average close to 10x, close to maybe even more than 10x number of commits across the whole team. And it's not even just one team member. It's not just B. It's every team member on the number team. It's the whole team. The whole team. And in your blog post, just for folks to understand, there is a great commit graph that shows the velocity of the team pre and post.