Microsoft Reveals Maya 200 AI Inference Chip

0:00Welcome to the podcast. I'm your host, Jayden Schaefer. Today on the podcast, Microsoft has made a huge announcement when it comes to AI chips. They've announced a really powerful new chip for AI inference. So today on the show, I want to break down, it's called Maya 200, what it does, why it's a big deal for what we're going to be seeing with AI in the future. Before we get into the podcast, I wanted to mention if you want to build AI tools without knowing how to code, without being a developer like myself, I would love for you to try out my platform, from AIbox.ai. We have a vibe tool builder where you can describe a tool that you'd want to create, whether that is, I just created one that creates profile pictures for people, you upload an image of yourself, and it has this right kind of lighting, and it kind of creates all these different things that you're looking for.

0:49And you know, these are great for business portraits, or all sorts of other headshots for LinkedIn or other platforms as well. But I just created this tool without being a developer, I put in a prompt and it linked together a whole bunch of different AI models to create this perfect tool for me. So if you want to be able to build tools like this without knowing how to code, go check out AIbox.ai and give it a try. We have over 40 of the top AI models, everything from Anthropic to DeepSeek to Google, Meta, Mistral, OpenAI, Perplexity, XAI, Quen, tons of image, audio, and text models on there. You can build some amazing tools without knowing how to code.

1:22So go check it out. Now let's get into the episode. Like I was saying, Microsoft, they just launched their newest custom AI accelerator. It's called the Maya 200. So this is their very purpose built. It's a silicon platform, and they are aimed at one of the most expensive. And also it's one of the most complex parts of modern AI systems. If you're looking at this from kind of like an operational perspective, and that is large scale inference. The Maya 200 is the successor of the Maya 100, which Microsoft, they actually launched that one back in 2023. as it was kind of like their first serious in-house AI chip that they were creating.

2:00This new generation, now that they've made the 200, is a really big step forward. So there's a couple things that it does. Number one is just raw performance, and then also how tightly the chip is integrated into Microsoft's kind of broader cloud and also AI stack. So according to them, the Maya 200 has more than 100 billion transistors, and it's capable of delivering up to 10 petaflops of performance in a 4-bit precision, and roughly 5 petaflops in 8-bit, which is a massive increase over the last generation, and I think it's really trying to optimize for just running larger language models efficiently and doing this in production.

2:37It's interesting to me seeing Microsoft get into the chips game. There's a lot of competitors in this space, but not a lot of competitors that could really compete at this level, And Microsoft, I think, sees just how much money they'll have to spend, let alone, you know, not to mention just how they're not able to customize everything the way they like if they're if they're using outside suppliers for this. So it's interesting for me seeing them get into this. And for those that are curious, right, inference is essentially just the process of executing a training AI model to generate outputs as opposed to training, which involves teaching the model in the first place.

3:11Right. So we have inference, which is getting it to generate for you. And what's interesting is I think we talk a lot about the GPUs involved from NVIDIA if you want to train an AI model and just, you know, how intense that can be. And yes, it does cost a lot of money. It is very intense. But I think it's also important to remember there are millions of people around the world using these AI models. And we also need to optimize the tech stack for people that are generating stuff. So I think while training oftentimes gets a lot of kind of like the headlines and people talk about it a lot because it's basically this kind of massive upfront compute demand, right?

3:44Like in order to train one of these models, you're spending millions and millions of dollars. I think inference is quietly becoming a really dominant cost center for a lot of these AI companies because their models are, you know, getting deployed to millions of users. That's chatbots. And then if you look at Google, that's like all of the search tools. You have copilots from Microsoft and a bunch of others and a lot of the enterprise software. So every query, autocomplete or, you know, generated paragraph, every bit of that is consuming compute power and cooling. So as a result, even like a very small efficiency gain at the chip level can translate into some really big cost savings at cloud scale.

4:19So it's interesting because this is, you know, obviously something Microsoft's concerned about, but every other AI company should be and is concerned about this as well, because they need to make those, you know, they need to make the cost savings, not just when they're training the model, but when they're actually generating stuff. Microsoft right now they're betting that this new kind of Maya 200 it's going to be a really big shift in that financial equation they said that the chip is going to be designed to essentially run today's largest frontier models so you can imagine the ones that they partnered with like open AI and they're going to be able to do that on a single node while leaving enough headroom to you know accommodate larger and more demanding architecture in the future which is kind of interesting right they're not just looking at what is open AI what do what do our AI models need today they're looking at what is it going to need in the future so I think because they kind of have this design, it's very forward looking.

5:04I think this matters because model sizes are continuing to grow. And I think as companies are increasingly, you know, expecting lower latency, they're expecting kind of this always on AI service rather than a batch style workload, right? Like you're not going to, I think in the olden days or olden days, but like in the past, it was kind of like, hey, we need like an AI model that's going to go and run through and do this massive project, it's going to get this huge batch done for us, and then we're going to be done. When you're looking at like how consumers and how the enterprise is using it today people are pinging this all day every day all it needs to always be on no one wants latency and so it needs to kind of accommodate for that not just like this huge fluctuating big usage and then a lull it's it's kind of like we're we're getting this constant steady usage so beyond just like the performance i think the power efficiency is a really key part of all of this data centers right now they're already straining against energy constraints we have even to the top levels of the government talking about um look you You guys need to be building power generation or power creation in some way alongside your data centers because there just isn't enough.

6:06And that cost gets passed on to the consumer. Like if you live in an area with a lot of data centers and they're all getting subsidized, you are paying for it. Essentially, your power is going to be more expensive. So I think the AI workloads are getting really intense right now. but essentially by designing this chip and creating its own silicon, Microsoft can tune Maya specifically to its data center layouts, which is a really interesting thought. Microsoft being, you know, one of the biggest players buying and building these data centers, they could build a chip that's specifically designed for how they structure and run their data centers.

6:40And that's, you know, that's like the cooling systems and it's kind of the software framework and they can do all of this to reduce any sort of wasted power and they can smooth out the deployment at scale. So I think that's a really interesting vertical integration. It's difficult to achieve with any sort of off-the-shelf GPU alone that you might get from an NVIDIA or anyone else. And so I think this kind of Maya 200 chip is also reflecting a really big shift in the whole industry, right? The whole world's largest cloud providers are there. More and more, they're getting into designing their own chips to try to reduce their reliance on NVIDIA.

7:12And let's be honest, NVIDIA's GPUs have become basically the backbone of the AI boom. boom. But I think it also like remains that they're very expensive. They're very supply constrained. It's hard to get them. And so I think Google was kind of pioneering this whole approach years ago. They had their tensor processing unit, their TPUs, which are now on, you know, they're offered as a cloud service rather than kind of a standalone hardware. Amazon also followed up and kind of copied Google. They did Tranium and Inferentia. It's kind of their in-house accelerators for training and inference. And then recently they also rolled out a new generation that they were kind of aiming at improving some of the price performance and, you know, for like larger models and stuff.

7:52So we see Google doing it. We do see Amazon with AWS doing it. So it kind of only makes sense that we're seeing Microsoft get more serious about this. And I mean, they already had this, the 100 version of this chip. Now this is the 200. I think Microsoft is now really solidly positioned with Maya kind of as like a peer for some of those other alternatives from Google and Amazon. And so in their big announcement, they said that it delivered roughly three times the FP4 performance of third generation Amazon Tranium chips and exceeded the FP8 performance of Google's seventh generation TPU. So I think while those types of comparisons often depend on, you know, specific workloads, if we're being 100 % honest, I think they do show that Microsoft's in like, they're really trying to be competitive, not just internally, but also across the broader AI kind of cloud market.

8:40They know that this isn't just them that it's going to be using these for training. They're going to have other customers and other people doing this. So I think it's really important to remember Maya is not, you know, being treated as sort of an experimental side project. Microsoft says that the chip is already powering internal workloads, which includes models developed by its super intelligence team and also some of their core features of Copilot. So saying that right now their AI assistant that is, you know, on like open or on office on Windows on all of their enterprise tools, it is using this.

9:10So by deploying this internally first, I think Microsoft can kind of validate the performance in the reliability, the cost savings, and then they can go and kind of roll this out to other people. And it's honestly, I mean, that's the greatest validation. Microsoft's a massive company. They have millions and millions of users on their products and their co-pilot is used by millions of people every day. So, you know, if they're like, look, if it's big enough, if it's good enough for us, it'll definitely good enough for other AI companies. I think as of this week, they started inviting internal developers and some academic researchers or some frontier AI labs to experiment with it and experiment with their software development kit.

9:43And I think that's kind of just basically showing that Microsoft is putting out the groundwork for MyEd to become a first class compute option within Microsoft Azure, their cloud platform. And they're going to do this alongside GPUs and other accelerators. So if this is successful, it's going to give a lot of different customers that they have more flexibility in how they run AI workloads, going to give Microsoft a lot more control over one of like, this is really one of the most strategic kind of important layers of the AI stack. They're not going to have to rely on their competitors to give this to them.

10:12And so I think all of that together, it's going to be less about winning some sort of benchmark war with the Maya 200. And it's going to be more about kind of this long term leverage as the inference workloads continue to scale, and the margins are getting tighter. And so if you want to own the silicone beneath all the software, I think that is going to prove to be one of the most, you know, one of the best advantages in the next phase of the AI race. So I think Microsoft is really well positioned for that into the future. Thank you so much for tuning into the podcast today. If you enjoyed the episode, if you learned anything new, it would help the show a tremendous amount.

10:46Honestly, like it'd be a huge help if you could leave a rating and review if you have not left one already. I read them all, I appreciate them. But most importantly, it really helps the show to be shown in the algorithm to more amazing people like yourself that are learning about AI. I'm trying to get the word out and help everyone learn together. So if you wouldn't mind doing me a huge favor in helping everyone else learn about AI, drop a review on the show, and also make sure to go check out AIbox.ai if you want to build AI tools and if you are not a developer. All right, thanks so much for tuning in, and I'll catch you in the next episode.