#716: Concrete, Cooling, and Compute: Reinventing Data Centers for the AI Age

0:00This is episode 716 of the AWS podcast, released on April 14th, 2025. Hello, everyone, and welcome back to the AWS Podcast. I'm going to share with you. Great to have you back, and I'm joined by a special guest. I'm joined by Stephen Callaghan, who's a senior principal engineer at Amazon, and he works in the infrastructure services space, and he's been here for a long time. In fact, one of the few Amazonians I get to meet that's been here longer than me, almost 15 years. I'm only 13, Stephen. Welcome to the podcast. Yes, thank you very much. Yes, it has been a roller coaster. When I think back of how things started those 15 years ago and what we were doing then compared to now, it may be 15 years in one company, but it kind of feels like we've been through three or four different variations since then.

0:47Yep, absolutely. Every year is like three to five over here. But that's a good thing for our customers because it means we've done lots of stuff and done it at scale. So today we are talking about infrastructure. In fact, we're talking about data centers. Now, we don't often talk about data centers at AWS and at Amazon because that's the undifferentiated heavy lifting that supports what our customers do and customers connect via an API and get all the good stuff. But servers have to run somewhere. IT has to run somewhere. Services are created with technology. Technology is made of hardware and software.

1:19And those things have to come together. And at Amazon, they come together at enormous scale. And because of that, we think about things very deeply. maybe more deeply than if you've run your own data center many of you have um that you've had the opportunity to and before we get into sort of some of the things we've been doing differently and how we've been changing things i guess let's talk a bit about scale steven because i think you bring a really interesting perspective on that topic given what you've seen and um help folks understand the different mental models that you and your team apply um yeah so i suppose kind of if I look through the different scale aspects, you know, there's a joke I have kind of around my organization that every couple of years, we just kind of change what the exponent was.

2:04And so we started in gigabits, we kind of moved to terabits and then we're talking about petabits and it just kind of moves up as we do it. It's just, it's just a different letter. You know, it, you know, it'll, it'll always evolve. And yes, like one, you know, 800 gigabit or 1.6 terabit network interfaces. Um, you know, that they were, fantasy worlds, you know, not too long ago. And I suppose when the data center space, you know, a lot of the changes that we're seeing now, it's, you know, it was a data center. It was like one thing that we were talking about. Now we're kind of planned in, you know, in clusters or we'll get onto a little bit later on.

2:44But one of the things in kind of preparing for this, I was thinking was, you know, in the non-ML world, we're adding a building. It's like we add a building to AZ1 and then AZ2 and AZ3. and we're trying to balance them out. That's no longer the case now. Now it's a case of I want six buildings in this location as quickly as possible. And so it's just a different way of looking at it. And like you said about the way that we think about it is a bit different is we go to much deeper things or much deeper levels of ownership in these data centers than I think most people would think is reasonable to do.

3:17Well, I think it's interesting. We'll get onto some of the challenges of scale, particularly some of the newer technologies. But one thing you said to me when we were preparing for this call is nothing is uninteresting at scale. What do you mean by that? Because I think it's a fascinating way to look at things. Yeah, there's a set of kind of core tenants that drive most teams here at Amazon. And that's one of the ones that I advocated for within our team. And that's a case of we're trying to balance, you know, speed of delivery with, you know, availability, sustainability. You know, there's lots of different things that we're doing.

3:51And so we're all looking for those trade-offs, but I don't want to just sit here and think about it. And so when it's at small scale, you can kind of make a faster decision. But as scale starts coming into it, then that's when the more interesting things start coming about. And maybe you'll reevaluate a decision you made in the past because now suddenly we're 10 or 50 times larger than we were before. Gotcha. Gotcha. Now, you and the team have been building lots of data centers for a long time. Surely it's just cookie cutter, you know, set and forget, stamp them out, away you go. What's, why are we rethinking things?

4:27What's, what's changing? Well, I, I think that's one of those things that if you're not continually moving forward, you just get into stasis and that kind of, you know, ultimately gets to, as we say here at AWS, that gets into day two thinking, which ultimately leads to like just being frozen and ultimate death in terms of your innovation ability to deliver. And so, you know, we're continually rethinking some of these things. But I suppose what's different with the pace in the data center space is there's a, you know, obviously there's a huge market interest in Gen AI. And those things, you know, those data centers, they're different.

5:04You know, we've gone to much higher density, much more power intensive processing. And also, you know, I look at these things as much more correlated workloads. when you do have a general purpose cloud data center, you've got 100, 1 ,000 customers, they're all doing different things and you kind of gain some benefits by everyone doing something different. But when the entire data center is correlated and on or off, you know, new trends come up. And anytime that happens, you get a new opportunity to just rethink things. And so you talked about Gen AI and what we know is that, you know, it uses lots of power power needs cooling as well talk us through the whole the whole thing because because i guess from for the uninitiated you know it's easy to think about say oh it's just a bunch of gpus and you throw them in a rack and how hard could it be what's so different what is so different i i would love if that was the case well you wouldn't have a job but you know yeah that that that's one thing um so yeah a lot of the things that we're talking about here with gen ai like when you get to, you know, generating tokens, which is ultimately what these machines are doing, they need to do so at as fast a rate as possible.

6:21And when you start looking at that, it's how many other machines are working and like what is the scale of the model that you're kind of working with, whether it's training the model and creating it or inference where you're using it. You know, there's a lot of correlated compute that's happening. And so how much concentration of compute can you get? and ultimately the distance is a factor like you know i did a talk not so long ago where i said uh you know the young first version of myself never realized i'd be obsessed upset that the speed of light on earth is so slow and so where we change man you change i know where we can get things closer to each other and where we can um ensure that we're you know under a couple of microseconds latency, that makes a material difference to these systems.

7:11And so if you look at, you know, example of what NVIDIA puts out or what we put out with our Tranium Ultra servers, we're talking entire racks are now systems. And so we used to have servers and instances which were slices of servers. And now we're talking about the rack is the entity, or in some cases, clusters of racks are the entity. And so having that concentration helps these models. And then even within that, we don't talk about, you know, five GPUs for this purpose. You know, there are some customers out there who want tens of or hundreds of thousands of GPUs all working together on the same workload.

7:47And that's where the data center innovations comes into it. And so if we did want to get 20 ,000 GPUs all in the same building because we want high speed network between them, we want low latency between them, we want a lot of bandwidth between them. that's a data center full of gpus um and accelerators in general whereas a few years ago before this boom we were using a room of gpus or a you know a corner of a building now it's multiple buildings together for a singular purpose so help us understand what that means from a power consumption and a heating and cooling perspective because i think there's you talked earlier on about the numbers being very different the numbers are scary different and we have to tackle it differently, don't we?

8:31Yeah. Yeah, the one that's kind of, you know, people talk about the power that's consumed by these data centers and the power that goes into it. But the other thing to think about is all of that power gets converted to heat. And so if I look at just the innovations that have to happen on the cooling side, like that's kind of a representative of, you know, what are we doing with all of this heat? And so if I look at, you know, the previous world, before we got to these levels of concentrations, we had a bunch of cooling technologies. And so we do things in the data center. One is called hot and cold aisle containment, where your chilled air is on one side.

9:11It goes through the servers, gets heat, is then in a hot aisle, and then it gets exhausted out of the room, processed by the facility, and then the cool air kind of comes back in. And that processing could be, you know, for a given data center, maybe there are chillers that are there. And so, you know, your traditional HVAC sort of system, or maybe it's evaporative cooling. So there's a bunch of technologies that we would have chosen. But if you look at what the heat load of the data center was, it was CPUs. Like, you know, it's fairly spread out compared to today. Yeah, that was where it was coming from.

9:44And as you said, non-correlated necessarily. Yeah. Yeah. Whereas today, what we're looking at is, you know, the entire building is going to be outputting multiple megawatts of heat. And so this is an opportunity where we said, well, you know, it's not, I'm from Ireland and it's a relatively temperate climate. A couple of times a year, it gets up a little bit in the temperature and maybe we need to do something. But it's not great to size your data center cooling to the hottest day of the year. because in Ireland, we get a lot of rain. We get a lot of like cool weather. And so sizing it for the middle of June or July when we are in January is not great.

10:23And so one thing that we can do is we've been moving to systems called multimodal cooling. And so that's where we've got different modes available to us at the data center and we're able to balance the systems. And so sometimes we're going to use chiller. We're going to augment that maybe with some evaporative cooling or shut down the chillers. And so we can move through different modes depending on the needs of the data center at that time. And then when you add in kind of the future machine learning racks, we know pushing liquid cooling all the way to the GPU or the accelerator itself. And so we're instead of going pure air in the entire building, now we've got either all of the racks are liquid or maybe only some of the racks are liquid.

11:07and similarly I don't want a data center that's only going to be suitable you know I have to wait for a liquid cooled data center before I can launch this rack it is never a statement infrastructure are going to make and so we've had to come up with methods of having some liquid cooled racks in a typically air cooled data center and so this is kind of the the multimodal piece that we're doing to handle this massive amount of heat load. It's interesting like the concept of a of an adaptable data center or set of data centers versus, as you say, just, you know, I build it for one case. I guess the other wrinkle you have besides, you know, dealing with the fiendishly challenging Irish weather is building data centers in different countries and different climates altogether.

11:51So I think, you know, something like, you know, Australia is a little bit different to Europe in terms of its weather envelope. Yeah. And, you know, we look at some of the locations that were you know there's a ml data centers in stockholm and that's very different to what we're going to have to do in mississippi where it's far more humid and free air cooling and swamp cooling is not going to be as effective as something where you've got chillers um and then even regions in the middle east in the desert uh that's a very you know completely opposite thing of you've got cold you've got humid you've got dry and so every building and i suppose one of the things that I think of in overall data center innovation is we're far more attuned to not just stamping out the same thing everywhere for every use case, is we need to have a number of options in our tool belt to go and say, this is a machine learning data center with liquid cooled racks of this many megawatts in this particular climate.

12:51And I have access to cold water, or I don't, or I have nearby access to which power source that I have. And so is there a local wind farm? Are we talking about some form of nuclear plant nearby? All of these things go into the recipe that builds this new modern data center. And the big thing that I think of as super innovation forward that we are is having the right number of tools in that tool belt to build the best data center for the scenario that we have. Gotcha. Gotcha. Now, we talked about the fact that new technologies, new workload use sequencing is changing the demand on power. And clearly, sustainability has always been important to us.

13:38How are we dealing with that in this new evolved world? In a traditional AWS region, we have all of the compute in that location. And so, So U.S. East 1 is in Northern Virginia. It's near, you know, Dulles Airport. We kind of have this concept of that's where the compute is. But it is also known as Data Center Alley. Like there's lots of data centers there. And so when we look to extend this, having more and more power in that location is not necessarily going to be the best thing. And so we do look at places like Mississippi where there was a solar and a wind farm. That kind of comes there. There's a site in Pennsylvania that's adjacent to a nuclear power plant.

14:16And so these are kind of looking at the different renewable energy sources that are available and locating the data centers there. And so rather than pull the power from the source and drag it all the way to Northern Virginia, we're able to put our data centers in those locations. And so that's different for us, right? We've had N regions before, and now it's a different approach where we put them. um and then the sustainability you know we we get we can get into a lot more things like in things like embedded carbon in the concrete and you know analyzing the steel production and like going through our supply chains and trying to find a way to decarbonize that uh there's a lot of wins that we kind of have in that particular area and i think it's interesting because we have hit our goal of pairing our data centers with 100 renewable energy so that was hit in 2023.

15:05But like you said, the team's still diving deep. That's not done. There's more efficiency and economy to be found. There is. And I think every year for the past few years, we've been the world's highest consumer of renewable energy. And so we're kind of pushing that. We're at 100 % renewable. We want to keep going on that. And so we just keep pushing more and more in that area. And I even believe there was an announcement not so long ago about you know, really long-term things in terms of an agreement for a small modular reactor, that's not going to, like that's order of a decade away by the time we get there.

15:43And so, you know, yes, there's more things to be done now, but we also are planning eight, 10 years ahead to see where we want to be and kind of move on that direction towards power. And I guess, look, there are big lead times with things like this, you know, it doesn't, You can't just magic up a huge set of chillers or a massive building structure, et cetera. It takes physical time. It does. And it takes time. But it's also a case of, like, we iterate upon this at a, you know, there's a new building in construction at any point somewhere in the world all the time. And so we're also using this opportunity to say the concrete that we've been using.

16:22And so it's a case of, like I was mentioning before, about in a particular region, you know, there's indirect carbon emissions that we can have that are in our supply chain. And they're not exactly within our control, but we can kind of push the industry to go along with us. And so there, you know, there's things that we've done with a process called trial batching. When you're coming up with the concrete, what are the different mixes that you can do? And so, you know, if water to cement ratio, air content, how much shrinkage there is. And so we look at in a particular area, like the cement is not the same everywhere in the world.

17:00As much as we would like to think it, what you get in Brazil is not going to be the same as what you get in Virginia. And so what we've been doing is looking to see, are there other substitutes we can put in this while still meeting our performance standards? but, you know, replace some of the carbon pieces from the cement with other materials. And so if I remember right, there's a supplier we worked with in Virginia that replaced 40 % of the cement with slag. And so that's a byproduct of refining metals. And that meant that we reduced the amount of carbon in the cement mix by 35%. Also by, you know, encapsulating this byproduct that normally would have gone to something like landfill.

17:44um and so it's like a double win it's like an offset benefit yeah yeah it's one of these things that if we have a single win it's great but there's lots of single wins out there but it's these double and triple wins that we get where ideally it's more sustainable and more efficient and we can go faster like that's that's where the fun is and that's where the you know i get the energy to kind of keep going on this it's finding those double and triple wins so let's unpack that a little bit more because obviously we're constantly thinking about scalability and reliability and efficiency and it's all about customers but you're sort of touching on I guess some of the the mental models you're using about how to get these these wins and these multiple wins but what's what's the philosophy here what's really coming down down in terms of why this happens away to us well anyone who's been around Amazon would kind of know there's a set of leadership principles that kind of guide most of our decisions and and the one that kind of dictates a lot of what you're talking about is this customer obsession.

18:42And so we're trying to obsess about where the customers are going and we're looking to anticipate their needs of the future. And so how we're looking at this is what I call extreme ownership. And so we're asking questions of ourselves and of our suppliers that we think will meet the needs of these customers. and so we've kind of already mentioned that nothing is uninteresting at scale and you know that is that a decision made two years ago may not necessarily be the right decision for now and so we kind of we we always go through this trade-off you know we want those win-wins like we want that reliable and efficient and readily available and cost efficient and highly performant um and so anytime we change a scale effect like we're building a new data center type or we're even we're in a new region we want to re-look at some of these things and kind of quickly go through these trade-offs to see are we obsessing about the right things uh are is there anything that was a win-win that's now a win-lose um and is there anything that we should we should do different um and so like one thing that i i'll bring up is uh some of the sensors that we had in the data center like environmental sensors whether they were humidity or um you know temperature vibration there's lots of different things that we can do um and just the supplier and the equipment that we had was was giving us telemetry but but not quite at the rate that we wanted um or not quite at the fidelity or, you know, we looked at it and went, yeah, let's go and do that.

20:26And so we went so far as to write our own embedded software for these devices in these data centers that not only natively interacts with all of the systems that we have ourselves, but it's giving us the data that we need. Like maybe we don't need the temperature sensor because we have a different set of temperature sensors and we're looking at IR cameras or something. And so we're kind of optimizing those things because we want this level of ownership that we can go down all that. What I talk about going down the stack as deep as we like, we can then find those win-win scenarios on something like a sensor, which means it's more efficient.

21:02Potentially it's cheaper because we don't need the more advanced version and we get greater control over it. That's fodder for Amazon people. And it's interesting because, you know, thinking about, you know, being able to take the time and to dive deep into, you know, sensor telemetry and those types of things, even if it's not your own, if it's third party, is, as you say, something you can't necessarily do if it's just a one-off type thing or not at scale, but at scale, it actually makes sense to do that. Tell us more about some of the other electrical and mechanical controls that you've made changes to and how they work and what the benefits you're seeing are.

21:40Because it's fascinating to hear these little pathways down things that maybe most practitioners wouldn't have the time to look at. And you've gone deep into it. And what are some of the results? Yeah, I suppose, like if I look at a data center design, you know, the standard one's going to have an electrical room, a battery backup room, you know, generators, switch gear. There's a set of like components or modules that necessary to build a data center. and in a world where every area kind of is its own little kind of island anyone will make a decision that's best for them um and then kind of pass it on to the next team but some things that we can do is we well maybe we want to look at the entire package and so you know the complexity of the electrical distribution system is always something that's going to be top of mind right There's a, it's kind of seen as an opportunity, lots of opportunities for us to improve.

22:36And so, you know, we've done things like look at the number of connections that we have from the transformer to the rack and examine each one of them and ask the question of, is this providing the value of for the complexity that it brings us? And so, you know, we, we went and we reduced it. I think it's from seven to five. And so it's a case of we took two of them out. We're also able to look at the switchgear that we have within the data center and again, analyze that. And I think we, again, we took out 50 % of it because it was a case of the value was not there for the complexity that it was bringing.

23:12I'm not talking about like dollar value, but like the business, the operational. The effectiveness or the impact it has. And then even things like UPSs and redundancy. You know, traditional data center will have a UPS room with these batteries. And we kind of look at that as that's a very large effect radius or, you know, blast radius. And so if we were to have a problem in that location, well, all of these racks get impaired by it. Similarly, if we want to take it offline, then all of these racks no longer have a battery backup. And maybe we need a redundant battery backup room and everything just kind of stacks in terms of complexity.

23:50and so we brought those power systems much closer to the rack and so now a lot of racks actually have battery backups in the compute racks themselves um and so that means we don't have this central large blast radius point where it's going to affect a lot of compute or a lot of servers if we have problems um and it means the system is has kind of higher efficiency because we're able to kind of push all these down and we see lower likelihood of failure and if there's a failure, it's a faster time to recover. And so it's one of like, you know, it took a long time and a lot of effort to kind of basically get rid of these rooms.

24:28But in doing so, we have three or four wins. And so that's what makes it a lot, you know, worth the benefit. We're worth the effort. That's really interesting. Really interesting. And you touched earlier on as well around cement, which is, you know, it's funny talking about data centers. You know, for most people, I'm not going to overgeneralize, but most people think about data centers, their mind goes to the racks and the servers and the compute and the battery backup and stuff like that. But there's a lot to be done with the concrete and the building elements of that. You touched a little bit on the sort of use of slag and replacing some of the cement, but there's some other stuff you guys have been doing that is really interesting around the building side of things.

25:10Can you unpack that for us? Yeah, it's one of the, I would say it's one of the trigger words uh if you know we're always asking questions and if someone's why do we do that and if the answer is well it's the way we've always done it that's just a red flag you know let's go have a conversation on this and so it's things like um you know we have the uh when we're building the steel when we're building the frame of the building the mezzanine will have a steel floor you know well there's a steel beams that make up the mezzanine and then we cover that with concrete, you know, to give a cement floor up on top.

25:47Why do we cover it with concrete? Like the cement or the steel is pretty strong. Does that need to be covered in concrete? Is that, you know, going to be, what benefit is that bringing to us? And it turns out that we can actually just not do that. And per data center, you know, someone went and did the math and it works out at 115 metric tons of CO2 that we're able to not create by just not putting a concrete topping on the mezzanine floor. And so there's no practical effect to the customer in doing this, but we're able to, you know, again, win-win. It's going to cost less. It's going to cause less CO2.

26:28It's a one step fewer in our building process, which means it's going to go faster. And it's one other thing that, you know, could go wrong that we don't have to do. And so we're avoiding all of these things by asking these questions, all the way from, does that concrete need to be there? Yep. Yep. That's a pretty fundamental first question that, as you say, is often missed because we just always did it that way. I think it's interesting hearing some of this stuff because as regular listeners of the show will know, I always say the phrase undifferentiated heavy lifting. And in fact, if you make that a drinking game, you're going to be in trouble.

27:03But undifferentiated heavy lifting doesn't mean it's not important. That's the heavy lifting part. And it doesn't mean it can't be better. And I guess that's the message here is that, you know, there's a whole team and many teams at Amazon who are working hard each and every day for our customers who this stuff should be completely opaque to them. They don't know about it. They don't need to know about it, but it's kind of nice to know it's happening, isn't it? And I guess from your perspective, you spend time with customers and hearing what they want from a macro perspective and able to execute this on a micro perspective.

27:36It's a great, I guess, story of the concept of ownership and what it means to provide something for our customers. Yeah. Yeah. Like the, I know we're talking about the physical data center piece today, but I spend a lot of my time in the network space. And when I do talk to those customers in depth about something as trivial as updating the firmware on a transceiver in a network, like that's way down in the stack and customers don't think about it. it's not something that's done in the industry, but I'm able to say that because we do this at such a rate across the entire data center, your workloads are interrupted less.

28:13We have fewer failures because of this. And so that's one that because other customers, you know, at some point, most customers have had a network switch somewhere and they've had to deal with optics or something. And so when I can equate it to something like that, that's the depth that we can go to that we can update all of the transceivers in a network switch at the same time. I can apply that to anything in the data center, how we manage the transfer switch, how we do the cold plate cooling, all of these aspects that do affect more of the data center and the servers, we're able to go to that depth of detail as doing this in what seems like an inconsequential thing on the network switch that actually provides a huge amount of value.