Microsoft's Head of Cloud & AI on the AI Buildout's Risks and ROI — With Scott Guthrie

0:00Microsoft's head of cloud and AI joins us as we ask, is this AI build out going too far? That's coming up right after this. Welcome to Big Technology Podcast, a show for cool-headed and nuanced conversation of the tech world and beyond. We're joined today by Scott Guthrie. He's the head of cloud and AI at Microsoft, and he is the perfect guest to give us some context on the massive and some would say insane build-out of AI data centers taking place today. What does it mean? Is it going too far? What will it lead to? Scott, I'm so thrilled to have you on the show. Welcome. It's good to be here, Alex.

0:34Thanks for having me. All right, let me take you through the headlines over the past couple of weeks. It's crazy that this has just been over the past few weeks, but here we go. NVIDIA agreed or announced that it would invest up to$100 billion in OpenAI, starting out with$10 billion. Oracle announced it would invest$30 billion in OpenAI or a$30 billion build out with the company. Anthropic raised$13 billion. So, you know, we're just talking about a cool$143 billion, no big deal. So is this crazy? Is this overinvestment? Well, I think there's a great question. I'm sure that's top of mind for everyone.

1:11I think stepping back for a moment, I would say if you look at AI and the impact I think it's going to have in the economy, it's going to be, I think, the most profound technology shift in our lifetimes. And so I think if you look at the long-term trend, I don't worry about overinvesting. I think there will be a question on the horizon of different companies or making different strategies in terms of their investment and how they get their return in the one, two, three-year horizon. So am I going to say that every company is perfectly timed? I'm not going to make that assertion. But at the same time, I do think the long-term secular trend of AI is going to be that we're going to need more infrastructure.

1:58There's going to be more ROI from it, and it's going to be more widely used. And so I think directionally, from an industry perspective, the investments do make sense and will ultimately yield pretty profound results. So you think that this level of build-out is healthy? I think we definitely are not nearly at the point at which there is too much AI infrastructure, given, I think, the number of AI workloads that are coming for the world. And I think we're seeing over the last couple of years, as people use AI, they get value, they use it more, the models get better, and people then use it even more for new use cases.

2:42And I think at this point across the industry with AI, we're still more supply constrained than we are demand constrained. And I think I expect that to continue over the next couple of years as the technology continues to evolve and as people start to integrate AI into more and more workflows. OK, so let me put it bluntly then. Microsoft has a partnership with OpenAI, has invested 13 billion thereabouts. has the capacity to build big data centers. Why did Microsoft make the decision not to do the$100 billion level build out with OpenAI or even the$30 billion that the company is doing with Oracle and leave it to other partners to do that?

3:32Well, we have a great partnership with OpenAI and it's gone back many, many years and continues going forward. And we are building out and doing a lot of projects with OpenAI. And across the Microsoft Cloud, we're building out AI data centers all over the world. And at the same time, we are balancing our investment to make sure that it maximizes the AI infrastructure for both our first-party Microsoft offerings that we're investing in, our customers, AI offerings that we're investing in, and obviously OpenAI's offerings that we're deeply enabling. So, you know, we are very invested. I don't think it's a binary.

4:19Are we building out for OpenAI or not? We definitely are building out for OpenAI. And at the same time, the way our partnership works is we're supportive of others participating in that as well. Okay. But I just want to put a fine point on it because, Because again, like if you believe that this technology is going to be massively transformative, which you stated, and that we're not at the sort of optimal point of the AI build out yet, that there's room to continue to do more. And again, it's the partnership with what is the consensus leader in the space. They needed more infrastructure. there must have been some calculation within your group or your company to say, is it worth it for us to be the one that goes out and builds this massive, massive footprint, you know, in partnership with them or somebody else?

5:15So I definitely understand there's multiple stakeholders. But what made Microsoft pause on that front? Well, we have a balanced view. And so we take a long-term view in terms of making sure that we're building out in all the locations that we want to build out, that we're being thoughtful in terms of kind of the investment spend and the infrastructure that we're building, and also recognize that we don't have to do it all. And so I think we're always trying to kind of take a continually balanced view of that. And as you've seen from our CapEx and as you've seen from our earnings calls, we are investing a lot in infrastructure and building out like crazy.

5:57But again, at the same time, you know, we're always constantly reevaluating and watching closely, you know, which data centers in which markets to what specifications and making sure that we keep, you know, good discipline as we're doing it. that optimizes for both the long-term, near-term, and mid-term horizons. Okay, I'll just ask one more follow-up and then we can move on. Good discipline. What about this would have been undisciplined to have gone to this level? Well, I don't think it's so much the volume level. I think it's one of the things that we do when we add new data center capacity or AI infrastructure is making sure that we can use this infrastructure for a variety of different AI use cases.

6:49I think one of the things that's really going to differentiate AI infrastructure companies in the future is that ability to kind of maximize yield on the infrastructure. Like, how are you driving down the cost of tokens per watt per dollar? And part of what makes the Microsoft portfolio so unique is the fact that we have a lot of our own AI products, Microsoft 365 Copilot, GitHub Copilot, the work that we're doing with Nuance and Dragon and healthcare. We've got the world's largest consumer application with ChatGPT that runs on top of Azure and we have thousands, hundreds of thousands and millions of businesses that are also building their own AI applications on top of us.

7:39And so as we think about like what market are we going to build a new data center? Is it for training? Is it for inferencing? And, you know, how do we make sure that that infrastructure is going to be maximally used? You know, we feed in kind of each of these different customer scenarios into our calculus. And there are certain tranches of capacity that we're happy to build out because we can see very clear line of sight in terms of how we're going to maximize the usage and the revenue from it and there's others that were maybe less likely to see the immediate or that the ROI that we'd like and so we try to be disciplined about it as we've kind of shared in our blog post you know we do kind of look at every request first and we do have an opportunity on that and and as you've seen from our capex we are swinging in a lot of opportunities but that doesn't you know we're not going to be undisciplined and say, blanket, we're going to do everything.

8:36You know, we know that, you know, some opportunities will have more certain returns than others. And we're trying to make sure that we maximize our focus around those. You know, as we're talking, I'm kind of laughing at myself because, you know, my question is basically boiling down to you've like talked about your CapEx. Isn't Microsoft expected to spend like 80 billion on infrastructure this year in the neighborhood? I think that's what we shared in our last year. And I'm like, well, why aren't you doing another 100 billion? But the fact that you're not is actually very interesting. And it goes to a point that you just made.

9:14And I'm trying to read between the lines. And you tell me if I'm getting this right. You're talking about where you invest and where you're pretty sure you're going to get an ROI. And to me, if I'm sitting in your shoes, the question I would be asking is, is it worth spending all that money on training where there's been a lot of noise about diminishing returns of training larger models with these unbelievably massive data centers? Now you have the startups like OpenAI and Anthropic, you know, their belief in the scaling law seems unabated. And so the numbers get bigger and bigger. And they seem to believe that they'll continue to get an exponential return from training these bigger models.

10:03But is your decision in terms of being disciplined based on a belief that you're not sure if scaling up will continue to work? And therefore, it's too big of a risk to make such a large bet on training an even bigger model in a bigger data center. Well, I think there's a couple different elements of that. I think one is recognizing that you want to have the best models. So, you know, training is super important because if you don't have the best models, then, you know, your actual ability to monetize AI goes down. You know, at, you know, and, you know, part of what makes our partnership with OpenAI unique is the fact that we do have access to the best models, frankly, whether they're trained on our infrastructure or anywhere else.

10:50You know, that's part of our partnership that's really important. and i also think when you think about training training is evolving uh from maybe where simplistically we think of training a couple years ago of you you do training in one place and then you do inferencing where you are uh executing the models and building applications you know there's now multiple types of training there's pre-training there's post-training uh there's reinforcement learning there's fine-tuning um there's a lot of new techniques that uh both sometimes require lots of contiguous infrastructure and sometimes requires lots of infrastructure, but sometimes it's smaller sizes that can be used for very specific tasks.

11:34And so when we think about the investments of our infrastructure, we're trying to think about all of this and compose it all end to end. You know, for us, that means, for example, we want to make sure that we have lots of inferencing capacity because that ultimately is how customers pass and how ultimately you make money from any product, AI product that you build. And I think increasingly on the inferencing side, you know, one important element is the geopolitics of the world have gotten complicated over the last many years. And, you know, customers in Europe want to make sure that their AI is in Europe and the customers in Asia are going to care about their AI in Asia.

12:17Obviously, the customers in North America and the United States are going to care about their AI being delivered in North America. And so even as we build out our infrastructure, we want to think about it not just narrowly as we want to have one giant pool all in the U.S. We need to kind of be distributed around the world to kind of meet those geopolitical needs and to make sure that our AI is as close to the customers that are going to be using the AI as possible and can meet all of the data residency and data sovereignty needs. And so even if you look at our infrastructure builds around the world, we have regions in more countries in more locations than any other structure provider.

12:58And again, as we balance out the investments we're making on AI Infro, we're trying to keep that in mind versus narrowly put it all in one location. I totally understand that. But I have to go back to the diminishing returns of training question. Where do you stand on that? Well, I think if you look at training broadly, I think you're going to continue to see more value from the models by doing more training. But kind of going back to my answer earlier, I don't know if that's always going to be pre-training. I think increasingly lots of post-training activities are going to significantly change the value of the model.

13:37And so by post-training, I mean take the base model and how do you add financial data or healthcare data or something that's very specific to an application or a use case. What's nice about post-training is that you don't have to do it in one large data center in one location. And so part of the technique that we've been focused on is how do we take this inferencing capacity around the world? And a lot of it is idle at night as people go to sleep. You know, how are we doing increasingly post-training in a distributed fashion across many, many different sites? And then when employees come to work in the morning, we serve the applications.

14:15And so having that kind of flexibility and being able to dynamically schedule your AI infrastructure so that you're maximizing revenue generation and training, ideally in a very swappable dynamic way, I think is one of the things we're investing in heavily and I think is one of the differentiators for Microsoft. Okay, but you'll forgive me for going back to this scaling pre-training question. I'm just trying to see what you believe here. And you haven't said it outright, but from your answers, it does seem to me like you believe that spending wildly on scaling pre-training is a bad bet. I wouldn't necessarily say that.

14:58I think we've definitely seen as the scale infrastructure for pre-training has gotten bigger, we are seeing the models continually improve. And we're investing in those types of pre-training sites and infrastructure. We recently, for example, announced our Fairwater data center regions around the U.S. We have multiple Fairwaters. And we did a blog post recently of one of our new sites in Wisconsin. And these are hundreds of megawatts, hundreds of thousands of the latest GB200s and GB300 GPUs. and are, you know, we think the largest contiguous block of GPUs anywhere in the world in one giant training infrastructure that can be used for pre-training.

15:49And so we're investing heavily in that, as you can see kind of from the photos from the sky in terms of massive infrastructure. And, you know, we do continue to see the scaling laws improve. Now, will the scaling laws improve linearly? Will they improve at the rate that they have? I think that is a question that everyone right now in the AI space is still trying to calculate. But do I think they'll improve? Yes. And the question is really around what's the rate of improvement on pre-training? And I do think with post-training, we're going to continue to see dramatic improvements. And that's, again, why we're trying to make sure we have a balanced investment, both on pre-training and post-training infrastructure.

16:30sure. And yeah, and just to parse your words here, it's you can see improvement by making by doubling the data center. But that's why I use the word bet, because are you going to get the same return if it doesn't improve exponentially and just improves on the margins? And that, I think, is the big question right now, right? It's a big question. And, you know, the thing is it's not like a law of nature that's immovable. And so there could be one breakthrough that actually changes the scaling laws for better. And there could be a lack of breakthroughs. That means, again, things will still improve, but do they improve at the same rate that they historically did from a raw size and scale perspective?

17:21And that is the trillion dollar questions. Okay, great. I do want to get to the ROI of AI spend in a moment. You know, it's always great to have a chance to speak with someone who's in a position like you are within Microsoft, because we get a chance to, like, take some headlines and which might paint a portion of the story and then ask you what the truth is. There were some stories over the past year talking about Microsoft had, like, canceled options to build data centers in certain locations. And people took those headlines and they read into it that there was no demand for AI or that it wasn't going as well as Microsoft's telling us.

18:04But what is the reason for why those data centers, there were the options and they were canceled? What happened there? Well, we're constantly, I think in general, the headlines were focused on things that we canceled as opposed to all the things we signed. And so if you look at a given. It's amazing how news works that way, right? If it bleeds, it leads. If you look at kind of the overall investments, and certainly if you look at the overall CapEx spend, it has been going up and up and up. And so, as has, again, the revenue that comes from it. And so, I think I would kind of focus on the overall picture as opposed to individual tranches or individual projects that we potentially made decisions on.

18:49Now, the thing that we did do and we continually do is look hard at every single investment decision we make. We don't take this level of investment and this level of project and infrastructure lightly. It's critical that we invest wisely. It's critical if we invest that we make it successful and that we bring it to market on time with the right quality and the right security. And it's critical that we have the right go to market to monetize it. And so, you know, part of our calculus that we do as a leadership team is constantly looking at the variables for all of those. And there are places and times when we slow down or pause projects.

19:29And there are times when we accelerate projects somewhere else. And kind of going back to my comment around the world, you know, also the regulation, geopolitics of how AI is going to be used going forward has changed quite a bit. And what Europe thinks about where GPUs can be based has evolved quite a bit, I'd say, in the last 12 to 18 months. And I think it's going to continue to evolve around the world. And so even as we think about the investments we're making, we're also being very, very thoughtful in terms of where geography-based are we investing. So that we can, again, maximize the AI tokens we can serve in real production applications.

20:15and then ultimately use that maximization to ensure that we're delivering a good return on investment for every capital dollar we spend. Okay, and I have some technology questions for you, but just to keep on speaking about the financing of this stuff, because it's so important. So there has been some interesting reporting about how the AI infrastructure build out has begun to be funded by debt, not just profits. Great story in the Wall Street Journal this week. It says debt is fueling the next wave of the AI boom. I'll read the beginning. In the initial years of the AI boom, companies comparisons to the dot-com bubble didn't make sense.

20:56Three years in growing level of debt are making them ring truer. Early on, wealthy tech companies were opening their wallets to out joust each other. For leadership in AI, they were spending cash generated largely from advertising and cloud computing businesses. There was no debt-fueled splurge on computing and networking infrastructure like the one that inflated the bubble two and a half decades ago. However, that is starting to happen. Now, OpenAI's deal with Oracle has been pushed Oracle to start taking on debt. They say, this is according to the story, analysts at KeyBank Capital Markets estimated in a recent note that Oracle would have to borrow$25 billion a year over the next four years.

21:44Obviously, you guys are not Oracle, but you're watching this happen as it plays out. And see the parallels to the dot-com boom, I'm sure is not fun. You've been at Microsoft for, I think, 27 years? 28 years? 28. Sorry, I don't want to miss that last year there. So you've seen it, Scott. This seems to be an issue, at least from the outside. What do you think about it being on the inside? Well, I think obviously there's a tremendous amount of spend from lots of different companies. And I would say, yeah, the thing I can speak most to is what we're doing. And kind of per my comments earlier, I think we're trying to make sure that we have a smart investment play and a long-term strategic play that allows us to ride the AI revolution that we think is going to transform the world and do it in a way that leverages some of the strengths that we have at Microsoft, which is we have very good cash flow.

22:51We have a very diverse portfolio of businesses, in particular in the commercial enterprise space, whether it's cloud infrastructure, productivity applications, business applications, security, et cetera, all of them are going to be transformed by AI. And, you know, if you look at, say, to your comment earlier on the Wall Street Journal post, I think if you can read further in the post, you know, it does show the ratios for different companies. And there are some companies that are 400 % debt to equity ratios. And then there are other companies that are much smaller, and that would be Microsoft.

23:28And I think, you know, we want to make sure that we're not, and I think, again, based on our CapEx spend and the rate at which our CapEx spend is going up, you know, we're not going to sit on the sidelines and not be bold as we invest. And at the same time, you know, I think the thing that our investors expect, and ultimately, I think every investor of every company will expect, is to see that revenue growing in terms of AI services and products that are being delivered in terms of net revenue recognized in a quarter and making sure that the proportionality of that to the spend and in particular to the obligations that maybe are being undertaken with debt are balanced.

24:09And yeah, that's the thing that we've been focused on. I think, you know, if you look at our last quarterly earnings, I think people were pretty pleased with the getting the balance right there. And, you know, every quarter going forward, People are obviously going to be looking at making sure that that balance is right so that they see us investing for the long term and going to win. And at the same time, doing it in a way that is sustainable and allows us to kind of ride through, you know, the ups and downs that inevitably will happen over the next many years as this technology transforms the world.

24:51What are the consequences if this goes wrong with the debt? uh well you're obviously not taking on the same amount of debt so there's a rationale behind it what happens if yeah it breaks uh well i mean we have the ability we're not constrained i mean our borrowing costs ironically right now are yeah but industry-wide big picture industry-wide not microsoft specifically well i think the thing that um uh that we as an industry i think you know again, you need to have that thesis of how you're going to use the infrastructure. And is it, do you have, I would focus less on the megawatts that sometimes get reported in the press and more at where are those megawatts and what are you going to do with those megawatts?

25:39Is it going to be ultimately capacity that you can use to serve customers? Is it to build better models that help you serve customers? and what is the line of sight in terms of the product services and revenue that comes from it. And I think that's a place where, again, between ChatGPT, which is the number one AI app in the world, between Microsoft 365, which is the number one enterprise AI app in the world, and between GitHub, which is the number one developer AI app in the world, I feel good that we have applications using our infrastructure and maximizing it. And I feel good about the investments we're making in terms of capital spend and build out in the right locations to kind of continue to do that.

26:20And I think not every company probably has that level of game plan. And I don't think that maybe not every company is probably doing the same level of thoughtfulness of that. And, you know, at some point, you know, different companies will probably be hit by it. But, you know, we're very focused on what we do. And how do we make sure that we stay aggressive yet disciplined and make sure that we get that balance right. All right. I want to take a quick break and then I'm going to ask you a couple technology questions about the state of the build out, GPUs, custom silicon, and then maybe we can get a little bit into this ROI question.

27:00In fact, we will. We have to talk about the ROI of AI. We'll do that right after this. Did you know your credit card points and miles can lose value to inflation? Credit card companies often reduce the redemption value of your points and miles. Now, imagine a credit card with rewards that can grow in value. With the Gemini credit card, you can earn Bitcoin or one of over 50 other cryptos instantly with no annual fee. Every swipe at the store or gas pump earns you instant rewards deposited straight to your account. Plus, sign up now for a$200 Bitcoin bonus to kickstart your rewards. visit gemini.com slash card today check out the link in the description for more information on rates again if you're looking to invest in bitcoin but don't know where to start the gemini credit card makes it easy the gemini credit card is issued by web bank in order to qualify for the 200 crypto intro bonus you must spend three thousand dollars in your first 90 days some exclusions apply to instant rewards in which rewards are deposited when the transaction posts.

28:01This content is not investment advice and trading crypto involves risk. The Gemini credit card cannot be used to make gambling related purchases. What the hell is going on right now? And why is it happening like this? At Wired, we're obsessed with getting to the bottom of those questions on a daily basis. And maybe you are too. I'm Katie Drummond, the Global Editorial Director of Wired, and I'm hosting our new podcast series, The Big Interview. Each week, I'll sit down with some of the most interesting, provocative, and influential people who are shaping our right now. Big interview conversations are fun.

28:37I want a shark that... That eats the internet. That turns it all off. Unfiltered and unafraid. So in a lot of ways, I try to be an antidote to the unimaginable faucet of reactionary content that you see online, to the best of my ability. Every week, we're going to offer you the ultimate luxury of our times. Meaning and context. True or false, you, Brian Johnson, the man sitting across from me, one day, at some point, as of yet undefined in the future, you will die. False. Tell me more. Listen to the big interview right now in the same place you find Wired's Uncanny Valley podcast. Subscribe or follow wherever you get your podcasts.

29:19And we're back here on Big Technology Podcast with Scott Guthrie, the head of cloud and AI at Microsoft. Scott, we have a Discord here, a big technology, and I asked some of our members what they would ask you, and we got a flood of excellent questions. And I think they were great because they focused on the technology, some questions that I don't think you hear too often in the common conversation about this technology. So you're the perfect person to ask. I'm going to ask them to you. One of our members asked, what is the working life of a GPU and how long until they burn out? Are there use cases for GPUs once they are no longer top of the market?