AI Inference Is a Data Movement Problem: Microsoft's Saurabh Dighe on Maia 300

0:00Hello listeners and welcome to another episode of Semi-Doped. I'm Austin Lyons with Chip Strat and with me today is a special guest, Saurabh Dighe, Corporate Vice President of Azure Hardware Systems and he's an architect at Microsoft. So welcome, Saurabh. Oh, thank you. Thank you, Austin, for having me over. I really enjoyed your YouTube videos, so I'm glad to be doing this one as well.

0:28Awesome, awesome. I appreciate that. So for newer listeners, you may not know, but Sarabh and I talked back in January and we talked about the Maya 200 from Microsoft. And today he's back 10 months later and we're talking about Maya 300. So I guess with that, like remind us very high level again, maybe for new listeners, like why is Microsoft making the why did you make Maya 100, 200? And then what are we talking about today with the 300? Oh, yeah, absolutely. We'd love to do that. And it was a great conversation in January. So here we are 10 months later. Before I just go deep into Maya itself, which I'm sure we'll do for the next hour, I think it's important for us to explain why Microsoft is building custom silicon-in systems.

1:11And we have this privilege of running a planet-scale infrastructure. And we run a lot of workloads, both SaaS, PaaS, IaaS, businesses, where we are able to understand what our customers' needs and our customer requirements are. and really to understand what infrastructure requirements do they need. So we're not here to just build a chip for chip's sake. We're here to really differentiate and deliver value to our customers. And that philosophy for us is really not a chip story. It's a how does the whole system work together to deliver that value to our customers? And so we have been on multiple years of silicon and systems journey inside Microsoft.

1:57You may have heard about our compute silicon, which is Cobalt. We're doing Maya, of course, for our AI. We do Azure Boost for our front-end networking. We do Azure DPU for storage processing and data processing. And then we have a lot of custom security silicon chips. So we have been embracing this philosophy of really building our infrastructure that delivers value to our customers and gives us that freedom to innovate so that that infrastructure can then become something that the customers can rely on at scale. So now the Maya journey really started for us in Maya 100, as you mentioned before, but this was before the world of LLMs, right?

2:39We were starting to look at, hey, what does it take to really accelerate speech and sometimes convolutional neural networks because that was kind of the growing trend that time. and we quickly entered the regime of LLMs when we decided to build Maya 200 for how do I build the best AI infrastructure for LLM inferencing. And this is a bet we took because a lot of that time period, people were talking about training, training large models. And we looked ahead and we said, even though training is going to be super important, really the distribution of that intelligence is going to be equally important, if not more.

3:20And so this is where we took an infant-first approach for our Maya system. Again, I want to say it's not just the chip story, but it's really about how we kind of build the best AI infrastructure system and take the software, the orchestration that goes around it, the model families that need to be ported on, the custom silicon itself, the memory subsystem, the networking and data center, and really wrap the whole thing into how do we maximize, you know, the token throughput, the intelligence we can deliver for every watt or every dollar that we deploy. So that's really our philosophy behind the Maya architecture.

4:02And so we disclosed Maya 200 earlier in the year. Today we are unveiling the Maya 300 architecture. And in the Maya 300 architecture, we will talk about kind of the fundamentals or the foundations that we built on top, but then how we develop a generational, you know, leap in delivering the performance per dollar and the performance per watt with this new system and architecture. Yeah, this is all great framing. I love that you started with planet scale because I think, you know, it's easy to forget just how massive, you know, Microsoft Azure's reach is. And therefore, especially when it comes to sort of the defining workload of our time, LLM inference, how big the opportunity is to co-optimize, co-design the workload from the software to the serving layer, you know, all the way down to the infrastructure, to the custom silicon.

4:57And to your point, not only the accelerator, but the memory decisions, the interconnect decisions, as I know we'll get into scale up here after a bit. And so, yeah, I can definitely see how this is a huge opportunity to co-design. And then, you know, we'll talk about workloads too here in a little bit, but I think just to remind listeners, you know, even just first-party workloads for Microsoft of, you know, GitHub Copilot, Microsoft Copilot, Microsoft Foundry, you know, obviously you guys are already deploying inference at scale, and that's not even, let alone, you know, maybe third-party workloads.

5:33But maybe talk really be quick at a high level about the workloads, and then we'll get into the architectural differences with the Maya 300. Absolutely. Absolutely. Yeah, I think, as you know, since we talked in January and now, the shape of the workload and the computational profile has definitely changed, right? We are moving from predominantly a chat-based experience to now agents with long-running tasks. And that has definitely kind of rethink how we build infrastructure, how we deliver infrastructure for this new agentic computational profile. And we are starting to see the number of tokens that every agent generates is a lot more than what humans can generate, right?

6:16We're starting to see more persistent KV because the context is getting longer. We are able to see that, hey, you know, the agent is planning, then doing tool call, then ingesting new documents, then running an inference and iterating that loop. And so KV is getting more persistent. We're also starting to see how important it is for memory bandwidth and memory capacity to be kind of a first-order design principle. We're also starting to see how important it is to pack more and more efficient compute in a given power and capital envelope or energy constraint and a capital constraint. So all of these changing trends and changing factors, we are able to see firsthand with the workloads deployed at scale and how customers are using it and being able to get that telemetry out of the systems.

7:09At the same time, design the entirety of the system to then maximize the intelligence you can deliver in that constraint. So yes, workloads are definitely, definitely changing. But I think with Maya 300, we feel very confident that some of the decisions we made are now we are able to extract the right cost point, the right latency point, the right performance, and think of it as the right class of service for that particular workload. And like we talked about, we tend to talk workloads, but they're not monolithic, right? It's almost like we talked last time. There is this frontier efficiency curve.

7:50And we do believe that one size does not fit all. And so I think when we talk about Maya and some of the architectural decisions we made, we make sure that the system was balanced, but it was also leaning towards where we believe is longer context, you know, memory bound, wrong learning workloads. That's where I think we can extract the maximum value out of the system. And there are other systems that we will deploy. We will deploy a heterogeneous fleet to make sure that we cover every point on that frontier efficiency curve. Yeah, yeah, yeah. No, that's good. And feel free as we talk through these architectural decisions, like you mentioned, the workload now demands with the agentic, like a long-running kind of persistent KV cache, lots of memory bandwidth, memory capacity.

8:36But continue to sprinkle in your thoughts on how the workloads are shaping your design decisions. Because, you know, when we see that Pareto frontier with throughput on the y-axis and interactivity on the x-axis, I'm always interested in what the actual deployment and the companies are choosing for – obviously, they're not necessarily, like, filling out that whole curve. So I'm always curious, like, what is the class of workload, like you said, you know, that would land at a certain point? Is it, you know, higher throughput or lower latency? So, you know, again, we'll go through this here, but definitely feel free to sprinkle that in because I don't think we get a lot of, you know, detail about exactly which points are people sort of optimizing for.

9:20No, that's absolutely will do. I think one of the things that, you know, we believe in as we look at kind of the first principles of how you're designing an AI inference system, AI inferencing to me is not a compute problem only. It's actually a data movement problem. And so we decided that instead of trying to attack the compute problem, let's see how we can create a new class of architecture that really focuses on the data movement energy. Like the industry has done an excellent job over the last few years with new microarchitecture, new process technology, and new nodes of any new data types to actually be able to reduce the energy per flop.

10:08But what has not kept up is how do you reduce data movement? Because that is a significant, sometimes two orders of magnitude in terms of the energy that you could consume to move data and keep the compute engine fed. And so one of the fundamental principles of our architecture is what we call the software-defined data flow architecture, which we introduced in Maya 200 and we continue to enhance. And the underlying thesis is that with AI inferencing, we actually can plan for the data movement ahead of time. And because we can plan for the data movement ahead of time, because we know what these data movement patterns look like, we can have distributed memories, we can actually pin the data at the right location at the right time, We can hide communication and computation and actually continue to improve the efficiency of data movement in this architecture.

11:07And now with AI-assisted kernel development, it has become even more efficient. And we don't need armies of software people to be able to go and program this system, right? We can have agents optimize kernels. And so to your point, the workload, if it's LLM inferencing at the heart of it, it is actually the data flow architecture we believe gives the best energy efficiency for inferencing per token. So that's where we started, right? At the same time, we said, okay, that's great, but we do need to be able to now optimize the compute engine and the data engines together, right, in one system. And we then took an approach where we had distributed memories on that, so we do use a lot of SRAM, and we use it as a scratch pad.

12:04and so we expose the programmability and the explicit data movement instructions back into the compiler of the software layer. We're also able to create much better computation and communication patterns on DAI. So we have like the SRAM with our tensor unit, our vector unit, but then also we have a global NOC that distributes the data within the chips inside the package. And so that really helps us, again, reduce the data movement. We have gone towards a more NUMA-based architecture, which means that, you know, once you've placed the data close to your compute, you don't have to drag it from another memory controller or PHY sitting from the other side of the chip.

12:51So a lot of thought has gone into how do you continue to keep the data movement, the data storage, the data computation at the lowest energy possible. And then as we move outside the SOC, then we look at it and say, oh, okay, how do we connect these chips together? And a lot of times people think of this as, hey, I'm just going to put a network and connect these chips together. But we thought of the network as an extension of the data flow architecture. What that means is that specialized data movement engines are DMAs and the communication protocol that goes on top with chips in other nodes or other servers and other racks can talk through the same communication paradigm as what they're doing on the chip as well.

13:38So it's just an extension of the data flow architecture. And then finally, we also made sure that we drive a lot of rack scale efficiency. Because at the end, what matters is what is the deployment? People don't care about just the chip. People care about, hey, how does the entire deployment deliver more tokens at a lower operating cost? And that's where our Rackscale architecture really, really helps us build that into a more efficient deployment scenario in our data centers. So, yeah, lots of layers of optimization, right from starting from the silicon itself all the way to how do we deploy and allocate power in the data centers.

14:19This episode of Semi-Doped is presented by Crusoe. They build the power, the data centers, and the cloud behind AI. Did you know Crusoe can help you tune an open source model on your own data for just a few bucks? Frontier models are very good, but they don't know your company's terminology, your part numbers, or how you like your reports formatted. So you keep stuffing instructions and context into the prompt of an expensive frontier model to get the job done. But for simple repeatable work like reports, an open source model trained on your data can actually match a much bigger model. Think of a new grad with three months of on-the-job training.

14:57You don't always need the PhD. And the open model is cheaper and faster to run. Fine-tuning used to mean renting GPUs, working on orchestration and Kubernetes, writing training code, and babysitting the entire setup process. But Crusoe's serverless fine-tuning removes all of that. Upload your datasets, pick an open model like GLM, Llama, or DeepSeek, and they handle the training. You pay per million token used, not per GPU hour. And on the smaller models like QN38B,$2 can take you a long way with millions of training tokens available. Once the training is done, you can then deploy them on Crusoe Cloud in just one click or download the weights and run them anywhere.

15:41Your data only fine-tunes your model. And it doesn't have to be work data either. You can try it on how you write emails. Pick a task, compare the tuned model to your current prompt, and see how it does. Crusoe's giving all new accounts$5 in credit to get started today. That's enough to start tinkering with fine-tuning right now. Links in the description. Okay, so the framing is all, with Maya 300 and with Agentic AI, it's all about minimizing data movement. And so you're co-designing all the way down from the amount of SRAM on the chip to the local HBM to the software for developers to manage this, all the way to the networking and how you're communicating across chips because of course on a scale up fabric they're sharing memory so to speak with dma all the way to rack scale how many accelerators can we fit into this rack and so on and so forth so let's see maybe for listeners who haven't like read the recent announcements can you just zoom out and give us the high level like what is the Maya 300 look like like how many chips are in a rack and maybe so on and so forth but then and then we'll dive into the memory specifically, I want to dig into the SRAM versus HBM trade-offs and decisions you made, and also maybe comparing against Maya 200, like HBM 3E to HBM 4.

17:00But yeah, maybe just zoom us back out and kind of remind us at a high level some of the specs for the Maya 300. No, absolutely. So what we've announced today is really our conviction of the Maya inference system architecture. We have Maya 200 in production, you know, outperforming some of the, you know, GPUs that are in our fleet today. And with that evidence and the production proven, we are saying that we are going to be scaling Maya, you know, to multiple gigawatt deployments starting in 2027. So that's basically a pretty significant strategic positioning for us as Microsoft. And it'll be running, and it'll be a foundation element of all the Microsoft product and services in the future.

17:46Now Maya 300 specifically, what we're saying is that over Maya 200, we are expecting this rack scale architecture to deliver 7x more tokens throughput per rack than what we could do at Maya 200. And it will be delivered at 2.5x more performance per watt. So the interesting story now is how do we get there? I think that's what we're excited to talk about. And as you look at it, we talked a little bit about the software-defined data flow architecture. That's really a foundational element. We also went to multiple chiplets. So we went to TSMC N2P process. And with the chiplet architecture, we were really able to balance how much flops we want, how much HBM do you want in the system, and how much IO or scale-up bandwidth do you want in the system.

18:41So we've kind of tuned the system a lot more to delivering, again, not stranding flops or not stranding resources in the system, but delivering that useful intelligence through every walk. So the multi-chiplet architecture really helped us there. In addition, we have introduced much better arithmetic, so low-precision arithmetic, and we can talk about how with agentic use cases and with more persistent KV and growing KV, these MXFP4-like formats really help us optimize our memory, like HBM memory. So, like you said, our HBM memory is a constraint today in our environment. And you use it for storing weights as you're storing for a growing KV.

19:26And with lower precision data formats, we have optimized weight storage, model weight storage for our lower precision. At the same time, we can optimize the KV with some of these lower precision. On the compute side, we can have computation that's FP4 by FP8 so that we can continue to keep the accuracy and the quality of the model output. At the same time, pack more and more concurrency and higher KV in the same HBM. So a lot of thought in how do we use load precision math. A lot of thought in how do we actually design the memory subsystem. So like we said, we have the Undy SRAM, but at the same time, we've also gone to HBM4, and we have got eight stacks of HBM4 on Maya 300, delivering a total of 18 terabytes per second of HBM bandwidth.

20:19That's almost 2.5x of what we did on Maya 200. And if you look at the rack scale level, that's 5x of what we could do on Maya 200. And so that really was a focus for us, again, going back to how do we manage these longer contexts, more persistent KVs, you know, decode-bound or memory-bound, decode-heavy workloads, right? Our scale-up network is the same scale-up network that we removed right from a server all the way to SuperPod, which is now 12 ,288 accelerators in a SuperPod configuration, or 96 accelerators in a pod, which we call as a rack. So in a rack, we will be now 96 accelerators, which is double from what we did on Maya 200.

21:10And even though we double the number of accelerators, we can deliver this at 167 kilowatts per rack, which is extremely energy efficient if you compare it to anything else that you'd be getting from other vendors next year. So 96 accelerators at 167 kilowatts and then be able to scale up with one scale-up fabric, right, from rack to superpod to about more than 12 ,000 accelerators. It's really how we have spent our time creating a really unique fabric, scale-up fabric. Yeah, fascinating. Let's get into the scale-up fabric here in a second because it's very interesting to go so hard into the scale-up fabric and make a big bet there.

21:57So I want to unpack that. But first, I want to hit on the HBM. So, you know, and you've basically alluded to this, but I just want to hit it again. you went from six HBM3 eStacks to eight HBM4 stacks, and the bandwidth went up, I think you said, maybe like two, two and a half times, and capacity went up per accelerator. So, and I think, you know, zooming out, the amount of bandwidth and capacity went up more than the compute. So, maybe again, you know, it sounds like you were really making decisions here around the decode side of things and the memory side of things for KB cache? Like, how do you decide, you know, obviously, if you add more bandwidth and capacity, you can actually sort of unlock more of the flops.

22:44If you're not keeping the flops, the compute sort of just waiting on the memory, how do you thread that needle and decide, you know, how much more of your like die area you want to commit to flops versus to HBM. And also, I guess, relatedly, I think, remind me, do you guys have just one compute die with the eight HBM stacks around it? And then maybe contrast with others who are doing like two compute dies? Yeah, I know. I think this is a great question. And it really comes down back to, you know, what problem are we trying to solve and why, right? And like I said, I think we continue to focus on inferencing as our primary.

23:22And so for inference based workloads, we want to make sure that we had a balanced system where we are able to expand our memory bandwidth and capacity, like we talked about, while keeping flops continuing to go increase. And so we did move to N2P because we believe that that's the most efficient process node for computation flops. We went to a full reticle size compute. But at the same time, we felt that going towards two compute dies would have not been the right architectural choice for the workloads and the shape of the workloads that we are seeing. So instead, what we did was we still wanted the eight HBM stacks on HBM4.

24:05And so this is where our chiplet architecture really came into play, where we were able to create this memory interface die to extend the shoreline on either sides of the die. So we were able to extend the shoreline with chiplet architecture, put four HBM and four HBMs on either side of that, while still keeping it as one compute, very efficient, but one compute tie. And now going from HBM3E to HBM4, you get those two multiplier effects. On one end, picojoule per bit is super critical for HBM data transfer. And so with HBM3E to HBM4, the industry has moved to kind of a wider interface, but kept the pin speeds kind of relatively the same.

24:46And so what happens is you end up with a much better data transfer rate and energy profile for the data transfer rate. So that's one multiplier you get. And then obviously the second multiplier is when you, in Maya 200, we went from six stacks, we had six stacks of HBM3E, so now we get to eight stacks of HBM4. So if you put those two things together, along with our multi-chiplet architecture, you go from something that's delivering seven terabytes per second with a certain amount of flops to now 2.5 times that, so 18 terabytes per second, and with another around 35 % to 37 % of flop increase. So I think the bandwidth per watt significantly improved over mile 200.

25:35That's an interesting figure. And the chip went from something like 750 watts to 950 watts, so just with a 200-watt increase. Again, we were extremely cognizant of the fact that when we put in the power into these SOCs and into our systems, it needs to translate to, you know, significantly better intelligence outcome. And we did not want to strand any of that resources and those resources. So that's all the thinking that went behind really kind of balancing the machine towards these workloads. Got it, got it. But yes, increasing your bandwidth per watt is a very interesting figure of merit. And it makes a lot of sense and aligns with your earlier framing of your thinking a lot about data movement.

26:18Okay, so for listeners, I want you to explain the need to expand the shoreline a little bit. So you talked about obviously increasing more stacks. So instead of maybe having three on each side, now you have four. And then on top of it, you said the move to HBM4E doubled the number of pins. So instead of like 1 ,024 lanes, it's 2 ,048 or something. Is it just a geometry problem, essentially, where now with four and with more pins, you just don't have enough room to route it all? Or why do you need the little extra memory die or whatever you call it to extend the shoreline? Yeah, I think there are just quick corrections.

26:57So we're going from HBM3E to HBM4, not 4E. But everything else you said, absolutely correct. I think it's really two things, right? One is we are trying to optimize both for energy efficiency and cost. And so when you look at energy efficiency, you get a much better, you know, picajoule per bit kind of overall transfer bandwidth per watt is a better profile. When we go to kind of the wider interface, you get a lot more bits coming out. At the same time, you also wanted to make sure that we are using the right process technology for the right, you know, function, right? So your compute, you want to get to N2P, you get a really bad process technology.

27:37For some of the HBM and HBM-5s, we decided that we'll have a different process technology and to be able to manage the cost and the power profile of going to HBM-4 as well. So I think that really drives, again, our philosophy on, hey, how do we make sure that we are more cost-efficient over generation and more energy efficient over generation. And so that's where some of the, along with the geometry, the decisions to go chiplet really helped us there. Got it, got it. That makes a lot of sense. Okay, so you're talking about a lot of decisions that are very beneficial for decode for the memory-bound part of the workload.

28:20Maybe talk to me a little bit about pre-fill. I mean, within a rack of 96 accelerators, will you still ultimately disaggregate, pre-fill, and decode? And does that live within the rack? And you must have enough compute where you feel like, you know, with enough chips you can still do pre-fill efficiently. Absolutely. No, we did a lot of modeling on understanding what our pre-fill to decode ratios are inside the fleet. And then we are pretty confident, you know, even with what we have sent right now, like what we have in our chips and over the rack, we can absolutely, you know, deliver the pre-fill to decode ratios.

28:59I think you bring up a good point is what is the pre-fill desegregation, like pre-fill decode desegregation strategy. And I think this is where kind of our strength in networking comes along because we have one scale-up fabric, right? And so we can absolutely take instances of pre-fill that have multiple GPUs and position them in one rack or spread them across rack and then instances of decode in the same scale-up fabric, right? And so this really helps us because we don't really have to go over a scale-out network to actually pass the KV from a pre-fill to a decode position. We can do that all over the same scale-up.

29:36And so we do run our fleet with desegregation, and we are able to take some of those mayas and run it as a pre-fill instance and some of them as decode. And when we kind of maturate them, So depending on how many pre-fill machines you have and what is the token output on those machines, we try to match that with our decode side as well. Nice, nice. Got it. Okay, so yeah, let's go into networking topology. I know yours is two-tiered, but tell us, you know, I think maybe zooming out, you know, people like to think about like, where is it copper? Where is it optics? How fast are the links? How many accelerators does it connect?

30:14And maybe even at a high level, like if you can fit 96 accelerators in one rack and you're doing a mixture of experts, inference at scale, like why do you even need to go beyond one rack? Why need super pods? Yeah, yeah. No, I think it's a great question. So let me kind of build it up from the bottom and then kind of talk about it. At the lowest level of our network technology, we have our integrated NICs. We introduced them in Maya 200. We continue to advance the capability of those NICs. They are on Dynix, extremely power area and power efficient. And they're really built for scale-up, right?

30:52So we're not trying to solve, you know, millions of endpoints, talking about, you know, general, you know, cloud network NIC kind of thing. We're trying to basically look at it as, hey, what does a scale-up domain look like? It needs to be extremely high-performant, right? It needs to be lower latency. You know, we are going to continue to increase the bandwidth. That's kind of what a tensor parallelism or some form of workload that's spread across nodes in the rack would need. And so looking at the communication patterns, we have optimized. Now, from there, we have used standard Ethernet, you know, switches and cables, because we do believe, again, with the philosophy of maintaining the cost and not trying to go into more exotic sometimes implementations, They've stayed on Ethernet to really benefit from the scale economics of that industry.

31:45And then we have built a node topology that keeps some of those links within the server itself. Because, again, we believe that neighboring mayas would have a lot more higher communication, especially if you're doing a tensor parallelism. So we were taking those tensors and just spreading them across a few mayas, a couple of mayas in the node. I think that's where a lot of the bandwidth stays within the node, and then half of that bandwidth goes to the switch links. So that's the first tier of switch. And then that 96 accelerators all have connections to the first tier of switch. And then we also decided that, hey, what if we wanted to, as the models grew bigger or as we started to look at better and higher and higher sharding schemes, or we thought of flexible job placement so that the allocator's job, actually, the software allocator's orchestration starts becoming a lot more easier.

32:43We felt that a second tier of network would really help us scale into those use cases. And we have our own transport on top of Ethernet. So we were able to then use some of those transport features to be able to say, yeah, I can actually scale now and have a second tier. And so we can have accelerators from one rack talk to another accelerator from a rack using the same stack of communication, communication stack that's been developed for in-rack as well as cross-rack. So that really helps us now get into this place where, oh, you can start expanding the cluster size to about, let's say, 128 racks in the same scale-up domain.

33:28You know that copper has a reach, right? And so we stayed with copper inside the rack. But as soon as we went from, you know, the switch in the rack to the next level of switch, we started using optics. And you'll start seeing some of those systems come next year as well. You'll start seeing, you know, more and more optics being used in the scale-up domain really to find that reach and to build these larger and larger pods or super pods. Got it. Got it. Okay. This is really interesting. So you said at first, you don't really need to design around scale out because you're not trying to do, you know, a million accelerators for a training job.

34:07You're just focused on inference, as we talked about before. So you're really focused on scale up. You said there's like four accelerators that are like tightly coupled to each other, like directly connected. And you're trying to think about splitting up the workload so that neighbors talk to each other a lot, you know, directly over copper. All the accelerators in the rack are connected via the first level of switch to each other, also copper. And then the switches connect to other switches, like rack to rack, using optics. Using optics. Awesome. Interesting. And it sounds very powerful and flexible.

34:43Do you have any two cents? Are you guys saying anything about how Microsoft feels on near package optics versus CPO, NPO versus CPO? Yeah, no, I think the story is developing. We do believe that optics and scale-up is something that everybody will eventually move towards. At the same time, I think we need to bring the reliability and the cost of optics along with the right supply chain, you know, motion to make sure that the industry kind of everybody moves together in some ways. And I think in terms of NPO and CPU, there are real trade-offs there, considering, again, talking about, you know, what do we do with reliability and how do we make sure that with NPO you have a less disruptive, you know, much streamlined path into the server and how you design it.

35:38But over time, we also know that CPU gives you the power efficiency at extremely high bandwidth rates. And so I think the industry is going to kind of start with a little bit of an NPO, trying to get a feel of how do we manufacture this? How do we make this very serviceable? How do we improve the reliability and the cost? And then move towards a more integrated approach, which is CPU on the XP side. And similarly, on the switch side, we are already starting to see some of those motions as well. Yeah, yeah. Makes a lot of sense. So then talk to me at the rack scale. Let's talk about power and cooling.

36:15I saw a sweet picture with this big thing sitting next to a rack of the 96 accelerators. What was going on there? And when you're talking about cooling, are you thinking about deploying this only in places that already support liquid cooling? Or can you go into air-cooled data centers? And why did you make these cooling decisions? Yeah, no, this is, again, really what I think is the co-design aspect of where we play out really well, right? I mean, we have designed Maya to be deployed in both an air-cooled and a liquid-cooled data center. So we know that, you know, as we expand this to, you know, global and planet scale, there are going to be opportunities to say, hey, there's a data center here and it needs AI infrastructure.

37:06and sometimes you do not have access to liquid-cooled facilities because they are still being developed. And so this is a great opportunistic deployment for us to say, hey, we can actually pair a Maya with a heat exchanger unit. Again, going to the power, the energy efficiency of the rack at 167 kilowatts, we are able to actually design what we call a heat exchanger unit. And it's really what it is. it takes the Maya chip itself is liquid cool, right, along with other components inside the server. But then when the warm water comes in, it can be put next to a rack that has fans in it, and it cools down.

37:47It takes the ambient temperature of the air cool data center, takes the air, close it through this liquid, and brings it back at a lower temperature so that Maya systems can run at the full 167 kilowatt rack capability. So we have designed this heat exchanger unit with our data center deployment in mind and are able to then look at it and say, if I need to scale this to different parts of the world where air-cooled data centers are still available, I can easily scale it there. At the same time, if it's a natively facility water-cooled data center, you don't need to use the heat exchanger unit.

38:22You can just take the MyARAC and then connect it directly to the plumbing that's available in the data center and be able to just deploy it in a next-generation liquid pool data center. So that really gives us, again, the deployment flexibility, and it has to be architected up front. This is not a decision you can make once the server and everything is built. You have to architect these decisions up front because this is how your deployment and data center operations and where you deploy these at scale all have to kind of play in at the level that we are deploying these systems at. Sure. Yeah, that's really cool.

39:01I like that you keep the optionality to be able to deploy these wherever you want. And if it can't support liquid cooling like normally, then you have this heat exchange unit to make it happen. But that makes me wonder then, how do you guys decide where to deploy these? I mean, are you putting them in the same place where you've got the Maya 200s? Are you trying to put them near for inference, like near densely populated places to have the inference as close as possible? or like how are these decisions? That is definitely a complex equation. But I think going back to, you know, the kind of starting with, we are seeing our customers use a lot of our Microsoft products and services.

39:46And so we talk to our internal product teams and services and understand where the demand is and where the growth is and where the capacity needs to land. And then based on, you know, the demand and growth, We have a pretty well-oiled process inside the company to make sure that the systems are ready for those data centers. And at the same time, we want to make sure that these systems are delivering that class of service that this thing wants. So we do a lot of work internally to make sure that those services are first kind of, in some ways, they run, let's call it shadow traffic on it. Just make sure everything is correct.

40:29Everything is hitting the right customer experiences and expectations. And when the services are ready to scale, we have them deployed in the regions that they want. That makes sense. Okay, so when you're talking about your internal customers, how much are they using Maya 200 today? And then as far as Maya 300 goes, from a software perspective, you talked about the software-defined data flow architecture. Is it going to be a big lift for those internal customers to move to the next version of Maya? No, absolutely not. In fact, I think what this is, is it's really at the kernel level. where all these optimizations play in.

41:13And we have a very robust programming stack that's already ready so that we can build the right kernels. But I think what I'm really, really excited about and what has changed since maybe the last time we talked is the ability to actually program these machines really well and extract almost peak performance out of this machine. And that's because of how we can now take agents in the loop and use agentic kernel optimization to build and understand, put the right optimizations in place even before the customer or the application sees it, right? So for them, this is mostly an endpoint, an API endpoint kind of a thing where they don't have to worry about the underlying architecture.

42:01That's kind of abstracted away from them. And all of this data flow architecture, the placement of the data, How do we continue to increase the performance of those specific kernels? That's all hidden from them. That's part of the library that gets built along with these systems. Gotcha, gotcha. So it's not a problem for your customers. And as far as internally needing to upgrade the kernels, because those specific kernels are just the things running over and over and over and over and doing all the math. you're saying that agentic AI coding capabilities have progressed so quickly in the last six to nine months that you can also throw a swarm of agents at it, and that helps you even more quickly optimize these kernels.

42:50Exactly. So day zero model availability is not going to be a problem. Nice. I think you're going to be able to, like you said, really form these out. If there are new models or new kernels that need to be written, these agents are able to continue to run in a loop and able to create new versions of optimized kernels that we can then populate back into our kernel libraries that can then expose to just LLM calls. Mm-hmm. That's amazing. So as you stand these up to serve internal customers, essentially, you know, tokens as a service, do you ultimately configure them differently to meet different, like, SLAs?

43:34Like, does someone want, you know, five milliseconds of latency? Someone else is okay with 20 milliseconds, but they want more throughput or cheaper tokens? Or how are these getting configured? Yeah, and I think it really, the answer honestly really depends. Like you said, and you mentioned it, that there's kind of an interactive versus throughput charts that people look at. And when you actually look at the charts, there are different points of sharding and different points of optimization of the same system. And so depending on where your workload is leaning towards, like if you have chat or reasoning heavy, you will have a certain kind of aspect of what the workload is stressing.

44:16Then you also want to understand what your interactivity is. Is it like, hey, I just finished a Teams meeting and I just want to summarize what it is? Well, fine, we could just batch that off and make it a throughput equation. Whereas this is a super critical agentic loop. It has extremely low latency requirements. Yes, you have to reduce the concurrency and batching of what you can do in that system so that most of the resources are given to that particular task, right? And so I think the answer really is the systems are flexible to some extent. Now, there is model infrastructure co-design that can definitely happen, right?

44:54So when we look at, for example, working with our own model teams, we can expose the strengths of the system to the models team. And what they do is now they say, oh, great, now I know what the systems are capable of. I can write some of the model infra decisions can be made in a better way. And so I think that can also help continue to extract more and more out of the system. But I would say there are like those two layers of optimization you can do. And how much are you really looking at model infra co-design? And then how are you looking at the task that's coming to you? And then finding the right optimal configuration and sharding scheme and the right interactivity point.

45:35or again, going back to what is the class of service and capability versus price that I have to deliver for this particular request? And how do I do that in these systems? So that's really, in some ways, you can optimize this, make that optimization decision at runtime. Think of it that way. Sure, sure. Okay, so zooming way out, last question. We've covered so much ground here. This has been awesome. How should we think, when we look at Microsoft and the Maya 300, Microsoft Azure, how should we ultimately think about the benefits to Azure? Is it that Azure can now run internal workloads more cost-effectively?

46:17Is it, oh, here's a vector where actually maybe there's an opportunity to rent these out or sell them to other customers? Or how should we think about the big picture? Yeah, I think the big picture is how do our customers really benefit from the investments we are making. I think that's the real story here. And like I said, it's not about the chip. It's not about Maya itself. But building an AI-inference system that really maximizes the token throughput or the intelligence you can create out of a given power envelope or a given cost envelope or even a capacity constraint. And so I do believe that with Maya 300, We've had this significant leap in architectural performance where we can deliver these services.

47:08You know, the tokens can be, you can generate new business models, so you can create new feature set at a different cost point and capability points. And that's what really we are trying to see is how do we make sure that we can now diffuse this intelligence through not just frontier models, but frontier infrastructure. I think that's the play here. We have to build infrastructure that can serve our customers. And Satya mentioned this, when we can actually drive down the cost point of delivering these, the usage and the consumption and the productivity, it should show up in productivity stats and GDP stats across the world, when more and more people are able to take some of these AI features, and they are scalable, they are much more efficient, they are much more affordable, they are much more sustainable from a data center perspective.

47:57And so all of that kind of cruised towards how the world is going to use AI in their day-to-day lives. So I hope that really as we talk about frontier models, it's really the frontier infrastructure that also kind of dives into diffusing AI. Yeah, definitely. Okay, so to summarize, it sounds like you're saying, hey, the world has a fixed amount of power and a fixed amount of dollars, and we're just trying to help customers get more intelligence out of their budget, out of their power budget, out of their financial budget. And the Maya 300 is a step in the right direction, co-designed with thinking about minimizing data movement and co-designed from accelerator to memory tradeoff decisions to network to cooling and racks, all the way co-designed to just it's for the customer trying to do more for them, essentially.

48:55Exactly. All right. Well, I'll be a Microsoft salesperson for you. There you go. You got it. Thank you, Austin. It was such a pleasure. Okay. Yes. Always a pleasure. And let me know when you're ready to talk about the Maya 400. That will be soon, too. Okay, excellent. All right, thank you. You too.