Microsoft's AI Server Revolution: Bypassing Nvidia with New Tech

0:00Microsoft is developing their very own AI server gear to essentially become less reliant on NVIDIA So today on the podcast, we'll be diving into what they're developing, why they're developing it, and why I think this is important for the future of AI. Before we get into that, I wanted to say there are four days left in the crowdfunding campaign for AI Box. If you're interested in investing in a cutting-edge AI startup, I'll leave a link in the show notes to republic.com slash ai-box, where you can check out our crowdfunding campaign. We've raised over$300 ,000 there and$100 ,000 off platform to build a no-code AI app builder and marketplace.

0:36And we would love to have you as part of the journey with us. But let's get into the episode. In a really strategic move to essentially lower its reliance on NVIDIA, Microsoft is in the process of developing their very own networking card, which essentially is looking to help enhance data transfer speeds between a lot of its servers. So the initiative which they're doing right now is essentially looking to kind of give an alternative right to NVIDIA's networking solutions. This I think it could do a couple things. I think it could lower the cost for Microsoft. But at the same time, I think they could actually boost the performance of their servers quite a bit right now.

1:13All of them are currently like using and running some NVIDIA chips. And I think they're thinking by essentially creating their own in-house chips. They save money and they increase the speed. So right now, all of this is really kind of looking to alleviate some of the network congestion, which is essentially encountered when AI is in kind of the development phase. So all of this, I think, could actually help companies like OpenAI, for example, speed up some of their AI development. All of this was spearheaded by Pradeep Sindhu, who is the co-founder of Juniper Networks. And so the project is part of Microsoft's kind of, they have this, I guess, kind of a broader effort right now to optimize their Azure infrastructure at every layer.

2:00So they're really focusing and looking at the ways that they can, you know, make this more optimized. And this is one of the big initiatives that they're taking. Right now, trying to create their own networking chips. This is not an easy task, I would say for sure, but it aligns with the recent developments of the Maya AI server chip, which I think right now is really showing how comprehensive of approach they are taking to this and really to reducing their dependency on NVIDIA's ecosystem, which right now is, of course, the absolute behemoth and is, you know, just the it's the elephant in the room everyone's trying to compete with and take market share with and perhaps be less reliant when you're one of these really big companies.

2:46So the urgency to get this developed is underscored by OpenAI CEO Sam Altman. So he said he has concerns over computing power disparities, particularly in comparison to Google. So Microsoft's right now, their independent networking solutions, I think are really, really critical for OpenAI who really is quite deeply tied to Microsoft. And if they're able to lower the costs, make this faster, this is going to be a really big boost to OpenAI. So Microsoft, I will say though, is not alone in trying to do this. There are some other cloud giants, including Amazon's AWS and also Google Cloud who are trying to do some similar things and develop their own AI chips and kind of server networking hardware.

3:33So the trend definitely, I think, is showing that there is a huge desire right now among a lot of these top cloud service providers to diversify their hardware from just being dependent on NVIDIA and what they have to offer. Now, to be honest, like NVIDIA obviously is doing a great job, you know, as evidence, a phenomenal job as evidenced by their stock price and how they just have an unlimited line of customers. But I think that might actually be their downfall, right? When NVIDIA has an unlimited line of customers, companies like Amazon and Google and Microsoft are concerned that, you know, should NVIDIA not have the capacity, should they become, you know, you know, should they decide they don't want to work with you, all of a sudden they could give their chips to your competitors and not yourself and you'd be in a really bad position.

4:23And so right now, of course, NVIDIA, I think, has been doing a great job of trying to be fair and equally dole out their chips to everybody. But I think there's always, you know, no one wants to be too dependent, have all their eggs in one basket. And so people are trying to really build out diversified, diversified sourcing for some of their chips. And this means that a lot of them are actually looking at making them themselves. So Microsoft, I think right now, and this new initiative they're doing, it poses like a direct challenge to NVIDIA. And NVIDIA really has a stronghold in the kind of server networking gear area.

4:58I think this has the potential to impact NVIDIA's revenue streams, assuming they're able to scale this and get this out. However, I will say creating these chips is something that does take quite some time building up the manufacturing facilities and all that like this, this all does take time. So it's not immediate. And I think NVIDIA is going to continue to enjoy a healthy lead from that. So all of that could definitely take years. And I think that it's, you know, far from being an immediate threat to NVIDIA. What I will say is there's something really interesting said by Baron Fung, who's a researcher at Del Oro Group.

5:35He said, quote, Microsoft and other cloud providers don't want to be tethered to NVIDIA's ecosystem. And then I think, you know, in kind of response to this and some other questions, there was a Microsoft spokesperson who recently said, quote, as part of our system approach to Azure infrastructure, we're focusing on optimizing every layer of our stack, we routinely develop new technologies to meet the needs of our customers, including networking chips. So, you know, the Microsoft spokesperson really kind of made it seem like, hey, this is just us optimizing, while you see, you know, other researchers saying, look, they don't want to be tied to NVIDIA.

6:14So you're kind of hearing two narratives here. But I think at the end of the day, you know, maybe both can be true. And I think that they're both, you know, very relevant points. So all of this, I think, reflects a kind of a broader industry dynamic where you're seeing a lot of these leading tech firms that are increasingly investing in proprietary technology, right? They're trying to build stuff in house, they're trying to do it themselves, not be super reliant on other companies. And all of this, you know, they got a couple of motivations. One is enhancing performance, right? So they're building these things to specifically be more performant on, you know, their tech stack and on their cloud provider.

6:50And then of course, reducing costs, right? If they don't have to pay Nvidia for these chips, they can significantly reduce costs. I think Nvidia has something like a 70 % profit margin. And that might just be broadly across their company. So, you know, maybe it's a little bit lower on chips. Maybe it's a little higher on chips, but they do have a healthy profit margins. And so there's definitely room if you build these yourself to cut costs significantly. So NVIDIA right now is really continuing to expand. They have a bunch of big new initiatives like the DGX Cloud, which integrates closely with cloud providers data centers.

7:25And I think this again might be something that scares, you know, the Google, Microsoft, Amazon, who all have their own cloud providers and are worried about the competition there. So overall, a fascinating story, and I'll keep you up to date on how this continues to evolve.