Inside Microsoft's AI Superfactory with Scott Guthrie

0:07Scott Guthrie:Hey everyone, welcome to The Neuron. I'm Corey. We're here again today at Microsoft Ignite 2025 in San Francisco. And today we're joined by a special guest. I'm here with Scott Guthrie, who is the EVP of Cloud and AI here at Microsoft. How are you doing, Scott? Doing well, Corey. Thanks for having me. Good. I sure appreciate it. It's great to have you on. You've been at Microsoft a long time now, haven't you? Yeah, 28 and a half years. So it's been a while. It feels like just yesterday, but it's been a while. That's a, you've seen a lot of change over the course of that, I imagine. Yeah. You know, that's still early consumer internet even at that point.

0:44It was before, you know, Netscape had just come out. Yeah. Oh, wow. The internet was just starting to get big. I think, you know, IE3, I think, came out this summer. I was an intern at Microsoft and so it was all still pretty new. And yeah, it's amazing how the world's changed since then.

1:01Scott Guthrie:And today, the word of the day is agents, for sure. Agents in Co-Pilot and Windows in Azure, everywhere, I guess. Yeah. I think that's, you know, the fun thing of technology is there's always something new. And, you know, at the same time, the new thing builds on the things of the past. And so, you know, with agents, obviously, they have the similar dynamics of the early days of the Internet. We're all still trying to figure out what you can do with them and how they work and what the patterns are and how you run reliably at scale and You know a lot of what we're doing and ignite and talking about is is just that you know kind of like how do you build them with things like get hub and co-pilot studio and how do you?

1:43Integrate them into employee workflows things like Microsoft 365 co-pilot. How do you? run them on infrastructure with things like Azure? How do you connect them to custom data with things like Fabric and then introduce your own custom models with Foundry? And then, in particular, one of the things we talked about at this event and announced today is how do you kind of like run them and operate them

2:06Scott Guthrie:and observe them as they're doing things inside your company? And with our new Agent 365 offering, we now have a way that you can sort of even register the agents, have them have an identity watch what they're doing, and then ultimately be able to govern and control and orchestrate them at scale so that you can really focus on delivering business value. And there's still a lot of things to figure out across all those things. And there will be for years. And there will be for years. But if you just look even at this event versus our one last year, it's similar to the early days of the Internet.

2:40It's amazing how much progress has happened. And the other big announcement we made today was obviously we do all that with OpenAI, But we also announced our expanded partnership with Anthropic. And so we're also using Anthropic's cloud models inside GitHub. We have it for a while, but then also inside M3C5. And then now it's available on Azure as a model you can use with our AI Foundry.

3:05Scott Guthrie:Natively right there. Natively running inside Azure and being able to leverage and work across everything I just mentioned. Well, that's an interesting approach you guys have taken, too, in my opinion. I like that, you know, there's the partnership with OpenAI. There is we're building models on our own. There is we're going to go and we're going to build our own tools. We're going to look at other people's tools. We're going to work with third-party model providers. And I think I've always wondered kind of what is the reasoning behind there? Is it a matter of choice for users? I think, you know, I think there's a, you know, different users have preferences.

3:41I mean, I think we saw that pretty clearly with GitHub Copilot, where a lot of people just really prefer Claude for developer scenarios. Other people have a preference for codecs, and other people have a preference for other mechanisms. And so I think we're very much about how do we provide choice and flexibility. And I think especially given just how dynamic and how fast the AI space is moving You know can't leave not having not forcing you to use something you don't want to use and how you have Flexibility where you can bring an existing tool chain or existing model and integrate it with everything else Yeah, I think is the name of the game for people to get the most value and and you know We're doing a similar thing on the infrastructure side when you think about you know, we support Nvidia GPUs We support AMD GPUs.

4:29We support our own first-party Azure Maya AI accelerators as well. You know, that diversity of choice and diversity of options, it's good for customers. And it also, you know, ensures that we have the most competitive offering because we have a complete offering that allows you to also pick and choose the pieces that you most want.

4:52Scott Guthrie:All right, so if you're building anything in AI right now, whether it's models, tools, workflows, you're going to want to hear about this. Dell just dropped something that's honestly in a league all of its own, the Dell Pro Max with GB10. That GB stands for Grace Blackwell, which is NVIDIA's next generation architecture, and here's why you're going to care about that. This machine looks small, but it's a powerhouse. you're getting 128 gigs of unified LPDDR5X memory, super low latency, and the brand new NVIDIA GB10 module, which lets you run local inferencing on models all the way up to 200 billion parameters, which is insane.

5:28Scott Guthrie:No cloud queues, no sky-high compute costs, just your own personal AI sitting right there on your desk waiting for you. And here's the wild part. If you need to go bigger, you can connect two Dell Pro Max units together using the Kinectex 7 smart neck and 200 gig networking to scale your workloads bigger. It's seamless, fast, and designed for serious AI development. Everything stays local and secure, whether you're building at the edge, handling sensitive data, or just trying to push the limits of what your models can do. Plus, it comes ready with the full NVIDIA AI software stack and DGXOS, so you get right to work.

6:04Scott Guthrie:If you want cutting edge AI performance and a compact form factor, check out the Dell Pro Max with NVIDIA GB10. It's the future, and it's already here. Check out the link in the description here to go get one today. You know, we see a lot of that in AI right now with just the conversations we have with the Neuron. And it means things like a lot of people don't realize sometimes you're building an agent. You might have a task that runs through eight different models. And the fact is, this model's great here. This model's maybe cheaper here, where you don't need as much of the inference and the cost.

6:39Scott Guthrie:So I think that flexibility really builds out something that is developer-friendly, for sure. Yeah. And I think in a given quarter, a given month, a given year, I think you are also going to see this dynamic, which is healthy for the market, healthy for consumers, of this model pulls ahead on this scenario, this model pulls ahead on that scenario. And to the extent that you can also have common tools that let you do evaluations, safe rollouts of new models. In some ways, it's like the DevOps, if you will, of AI development, where you want to run your tests and your evals and know that you're not regressing on a quality in one scenario while you're also making another one scenario better.

7:22Scott Guthrie:Watching out for model drift and all of the things. And then having kind of safety and security that works sort of independent of a specific model It means that you can try out new things and become either convicted that it's going to be better or realize, well, it's better here, but it's worse here. Maybe I won't use it. Yeah. And, you know, that's a lot of what we're trying to build is sort of the tool chain that lets you do that and do it repeatably and do it reliably. because as more and more organizations, whether they're startups or enterprises, put these models into production and use them for mission-critical apps, you need that level of hygiene and that level of control to avoid having a problem that impacts your customer experience or impacts your business.

8:10Scott Guthrie:Yeah, because time is money when it comes to apps. They're running 24-7. You know, something that has been in every discussion I feel like I've had this year has been observability. And you all have really leaned into that and come up with solutions that I think are simple, clear, offer that vision inside the box. To the best extent, we can have vision inside the box. And seeing agents laid out in a simple dashboard where it's very monitorable, where you can adjust permissions and all of those things, I think that was excellent timing and a really interesting thing to bring to the table right now.

8:46Scott Guthrie:Was that a high priority in bringing all of this out? Yeah, I mean, I think it's definitely, you know, we work a lot with businesses in particular. And I think, you know, for all of the businesses that we talk to, I think there's sort of two big conversations that people want to have with us. The first is, how do I demonstrate business value so I can get my CFO to continue to invest in AI? And so, you know, people are like, I need to move from prototyping, from demos, from experimentation to I can measure customer support costs went down 12 % or, you know, sales increased by N % or, you know, business operations improved by Y%.

9:33You know, and so like, you know, I think people are increasingly focused on not just for the potential of AI, but really like, okay, let's demonstrate how we use AI in business.

9:43Scott Guthrie:Yeah. to really drive tangible results. That's one aspect and you need some degree of observability to know if you're doing that. And then there's the, okay, now that we're seeing the big results and now that we're using it for mainline tasks that are really driving business value, how does it not break? And I think a lot of people go through an experience where something goes wrong in their AI workflow. Maybe someone rolled out a code fix without it being fully tested or someone rolled a new model and it improved on eight dimensions, but regressed two. Yeah. And that's when people suddenly realize also, wow, I've really taken a dependency on AI.

10:20Scott Guthrie:Yeah. And I now need the maturity level that I would have when I'm going to touch one of my core systems. And that's where you need observability as well. And that's why I think observability just pops over and over again in our conversations. And, you know, I think people are also realizing as models are smart and as employees can ask the model to do things, How do you know that the model doesn't give an answer because it's been given access to maybe more information that the employee shouldn't know? Things like financials or, you know, internal HR or legal, you know, things that are going on inside an organization.

10:58Or, you know, if a customer asks in a support bot, like, hey, what's the maximum discount that my salesperson is allowed to give me? you might not want that support model to give you the answer. And that's where also I think things like governance and things like the right compliance levels really come into play. And again, the more you have a really good story for that, the more organizations can bet on AI and get real value and are willing to invest to do even more. And so it is a great unlock, and I think it's also a great need. And we're excited with Agent 365. I'm sure there's more we need to add to it, but I think it's the most complete solution out there today.

11:42It is kind of multi-model. It is multi-open. And it's also integrated inside the number one enterprise AI solution out there, which is Microsoft 365 Copilot. And I think we're really excited. We're using it even internally. And when we turned it on, step one is even just figuring out how many agents do you have in your organization? We were having a discussion even on our leadership team at the company level on Friday and The number of autonomous agents that came back with was about an order of magnitude more than we thought we had And a lot of its people, you know individuals and departments that had set up little tasks and You know once it sort of showed us how many we were like, whoa That's a lot Right, and then you know, and I think that's also gonna be a similar dynamic when people start using it is it will help people realize they're using the AI a lot more than they thought.

12:33And again, it's a good way to make sure you're using it responsibly and appropriately and safely and securely.

12:39Scott Guthrie:And it's done in a way that I think non-technicals will be able to understand as well and be able to, you know, because I think that's a real key here is that when you have subject matter experts who are doing a specific job, for example, they know that job better than someone building an agent for them is going to. And there's a certain level of nuance that I think that really unlocks, and I think you guys have done a fine job with it. I think they're also going to work. The other thing that we're seeing with AI adoption and agents is, if you have a suboptimal business process today, like you're doing everything in spreadsheets and email and handoffs across eight different teams to make a decision, you can automate that with AI.

13:24Yeah. But you often are automating a suboptimal process with AI. And, you know, I think a lot of what you want to do is re-engineer that process. And to your point of enabling more people to use these tools as opposed to just developers or just, you know, the security office or CISO office, you know, if you can get the people that actually own the process to be able to use the tools and be successful, it really enables people to step back and say, okay, why don't we change our process?

13:54Scott Guthrie:And the people that can most change the process are often the people doing the process. Yep. And so to the extent that we can kind of democratize those tools, hopefully it really enables organizations to kind of do deeper business process reinvention and use technology to solve it. And yeah, I think that also is gonna be a great unlock for AI. Yeah, it sure will. Well, now that we've talked about agents and all that's happening at Microsoft, I would love to talk about infrastructure. Because the other end of what you're doing right here is what makes the rest of this possible, I would say. you know, without data centers and your attention to GPUs, the rest of this doesn't work.

14:34Yep. Yeah, it's, that's the part I love the most as well, because that's the part I work on. And it's definitely fun. And, you know, especially the last couple years is kind of the AI buildout has happened. It's not easy either, in terms of just the scale that we need to kind of do it at. But, you know, we talked, you know, it's been fun at the event. And then the previous couple of weeks leading up to it, we've also disclosed a bunch of details about some of our new Fairwater AI center designs that we're particularly proud of. And they're in production today. And so, you know, if you're using an AI app, there's a decent chance, you know, tokens are running through it.

15:13And, you know, it's fun to be able to kind of talk about some of those cool innovations.

15:16Scott Guthrie:You know, there were a couple of call outs you made in your portion of the keynote today that I wanted to bring up. The first one was that you have the largest concentration of GPUs in this new center. Is that correct? Yeah, the new Fairwater data centers that we're building. We're building some in Atlanta and Wisconsin are two areas that we've both kind of disclosed. And there's lots of other sites around the U.S. and the world that we're doing stuff to. But those are two that we've done now. Aerial photography, we showed a video in my keynote this morning for the first time of the Atlanta facility.

15:52They are the densest concentration of GPU power in the world and by densest I mean the most GPUs and in the most GB grace Blackwell GPUs look at the latest All liquid cooled, you know, which means that just from a pure token perspective You know a grace Blackwell 300 is roughly 12x the throughput of AWS tranium 2.

16:15Scott Guthrie:Yeah So it's you know, there might be one GPU each but one is 12 times more powerful than the other It's like having a case of the other. Yeah, exactly, exactly. And so if you pack them all into one data center, and the data center is a flat network. So it's basically, I think it was a two-tier flat network, as opposed to a traditional cloud data center, it would be usually three tiers. And so that means that when you want a GPU to talk to another GPU, you're going through more switches, which means lower throughput, more latency. You know we've tried to keep a very flat network so that each of these GPUs can interconnect with the others Incredibly high volumes and then you know part of what we've done to kind of optimize speed even is these data centers in the video You'll see our double decker meaning they're two floors And the reason we do that is we put the network core the data center in the center of the building on the second floor And that means that no cable in the data center is more than about 230 meters more than a quarter of your building Yeah, yeah, and and or 230 meters in depth in length and that minimizes Latency because you're only fastest the speed of light but it also means that like from a retransmission perspective You're also really minimizing the number of retransmissions at the at the literally at the wire level There's no over subscription to the network.

17:31So it's line speed and you know that combination. It's just a screamingly fast data center It's fun. It's running. It's real production now you know open eyes using it for a bunch of workloads already and so it's it's been fun to kind of see all of that come together and it's you know put in perspective of that network I mentioned there's five million individual cables in that data center it's just an insane amount of connectivity and GPUs and then it's all liquid facility water so in other words the data center itself is All the GPUs are cooled by water as opposed to traditional air chillers.

18:12And so it's outside the data center. In our facility in Wisconsin, we have a chiller plant, and it's the second largest chiller plant in the world. There's only one more that's bigger. It's Qatar in the Middle East has one, but it's the largest chiller plant in the United States and in North America or anywhere else around the world outside of Qatar in the Middle East. What's cool about the liquid cooled, because people often ask me, like, what about water consumption? It's about 20 houses worth of water to fill the data center. So 20 American homes. So it's, you know, the grand scheme of things, it's not a lot of water.

18:43And then once we fill the data center, we will not replace the water for at least six years. So there's no evaporation. There's no water loss. It's continuously recirculated.

18:53Scott Guthrie:Working like a radiator system in a car. It's a closed-loop system. Closed-loop system. It's about five feet per second of water flow into a GPU and out. And so, we flow the water in, the water temperature goes up by 20 degrees Celsius. Runs out and cools somewhere. Shoots out the building, we cool it, shoot it back in the building when it's chilled again. So it's incredibly efficient from a sustainability perspective. It's also one of the commitments we talked about in the keynote this morning is, all of our data centers in Azure will be 100 % renewable energy by the end of the calendar year.

19:30So we're pumping more renewable energy in the grid than we're taking out.

19:34Scott Guthrie:I guess my question would be How You know, we made that commitment was before the AI push so it became a lot harder As we started building you're hanging in there with it. We're hanging in there and you know It's a lot of it is you know, these are not carbon credits. This is not an accounting trick. We're doing this Yes, this is real a renewable energy that we're pumping into the grids and it's you know combination of you know solar hydro Yeah. Wind, nuclear in many places. Yeah. And so, you know, we feel good in terms of that sustainability promise plus the innovation. Wow. That's serious innovation.

20:13Scott Guthrie:And, you know, it's interesting to talk about, like, sustainability in relation to data centers because obviously a lot of people have concerns. And there are definitely elements where it can grow. But I personally have felt like for a long time, all of the incentive is there for companies to want to make these more efficient. Because you're looking at not just the environmental cost, but you're looking at inference and the cost of these resources and building out gigantic farms for windmills or solar or whatever. It just makes a lot of sense to me that the good business decision would be for these to operate more efficiently as well.

20:50It's the right thing from an environmental and sustainability perspective. Hence our commitment. It's also just, candidly, if you are operating in the U.S. or Europe or Asia or pretty much anywhere in the world, if you want to get permitting to build something, you need to get the local county or the local district to give you that permitting process and that permission. They're ultimately accountable to voters.

21:19Scott Guthrie:And it's not always easy with data centers. It's not always. And if you don't do the right stuff with sustainability or on emissions or water waste or electricity, you know, at the end of the day, the voters aren't going to let you build it. And so it is one of these things where you've got to take a long-term view. And, you know, part of where I feel good about our approach is on sustainability, which I'm talking about. You know, people don't have to worry about a bunch of diesel emissions wafting into their community. You know, these are, you know, these are highly sustainable, great environmental facilities.

21:54And then the other thing that we've really invested in is how do we hire great craftsmen that are building this and trade craft? And, you know, these are good jobs. And then, you know, we're often building sites now where when we finish one day, we go to the next one. And so to the extent that, you know, the Atlanta facility, I think, will have over 6 ,000 people that we've hired. Wow. You know, many of them.