What it will take for AI to scale (energy, compute, talent)

0:00What I'd like to do today is talk about what we need to look out for over the next 24 months with the AI build out with all of the things that are going on in the deployment of AI. how people are feeling about it, all of the tensions, all of the potential crises and the potential wins. Now, two years is a particularly difficult time to shoot for. So I am really hanging myself out because everyone can predict the next week and you can kind of predict the next 20 years, but I'm going to be brave and talk about those two years. Readers will know that I've talked a little bit about this in the newsletter.

0:39So go back and look at those essays. In short, This is all about absorption. This is all about the extent to which AI can be absorbed in the economy, in our world. Are companies absorbing artificial intelligence fast enough? Can the electrical and power systems absorb all the new demand from the data centers? Is the economy absorbing any benefits at all? And do people, do society, do us, do we want to absorb AI, all the things it brings with us at the speed with which it is emerging? So let's just step through each of those questions of absorption. You know, on the firm, I've written a lot about this over the last few months, weeks and months.

1:30Are companies really making use of this technology at all. And I think we can even step back a layer and say, are people really using these technologies in any meaningful sense? You know, we talk about 800 million people using chat GPT. We talk about a couple of billion using these chatbots globally. But are we using them the way that we've used other technologies when we talk about them being deployed, like the flush toilet or electricity. Is it really use of one of these technologies if you just put the odd query into it rather than Google, but your life remains largely the same? I mean, it's almost trivially easy to put a product in the hands of tens of thousands, hundreds of thousands, millions of people because of the app store, because of iPhones.

2:18And perhaps when we think about where we are in absorption, we need to go beyond someone downloaded the app and played with it a little bit. And I'm sure within those 2 billion people, there are many people like me for whom our ways of working and ways of living have changed because of these tools. But we need to be a little bit thoughtful about that. And that also, I think, reflects back on companies. You know, companies have a lot to do when they want to transform their businesses. API access is the easy bit. It's the institutional metabolism that is quite hard, very hard in some cases. That's why you hear stories of, and I speak to bosses a lot of the time, they're very happy, largely with how their AI deployments are going, but they also recognize that really deep and meaningful change is going to take quite a lot of time.

3:06Now, I don't think this is going to be a multi-multi-decade process that it was with electricity. I just think companies are more adaptable, people understand the technology better. We've spent years thinking about how to manage large-scale change, that horrible management consultancy word, transformation. And firm appetite is very, very clear. And the adoption will get easier now that there are standard operating procedures, there are playbooks over the last couple of years that at least the leading firms have developed. But there is a real constraint. And that real constraint is electricity.

3:40It's the physical limitations of this software and whether existing systems can absorb the demands being put on them. Now, a data center can be built in a couple of years, maybe a bit faster if you're Elon Musk. But getting that grid connection, as we know, can take several years. So you can have a data center, but you can't power it up. And Satya Nadella, a few weeks ago, said that they had that very situation. So there is a scramble for getting energy into data centers. I've spoken to data center developers in the last few weeks and what I'm hearing are really quite staggering stories. Multi-billion dollar offtake deals for electricity, not just in the US, but in Europe as well.

4:28The demand is really high that the hyperscalers are somewhat price insensitive in order to be able to build capacity to meet that demand. This isn't just about large language models. This is actually also about the shift of businesses, processes into computation, into digital processes. It's also about the growth of digital services. Earlier this year, AWS, which is Amazon's hyperscaler business, they put in tens of billions of dollars, nearly$100 billion in capex this year, had to turn down business from from fortnight a multi-million dollar contract i didn't even know fortnight was still a thing it's that you know that game where people fly around and wear skins they had to turn down that deal because they didn't have the capacity and if you follow the news and the analysis you'll see that that story comes up time and time again so this is a long-term cycle this is not just about large language models this is not just about whether open ai can grow and can become profitable this is a fundamental shift in the economy, as fundamental as going from 1880, where nobody was really using electricity in the economy, to the 1930s when the US, it was the prime move of the bulk way in which factories and offices were getting their power.

5:48More and more economic activity will move into computational systems, even when LLMs look completely long in the tooth and who would use an LLM in the same way that not many of us use penny-farthing bicycles to get to work. And so it's really fascinating, again, to see how that is changing the narrative. Now, some of this, I think, is just expediency. It's just, let's take advantage of the changes. But some, I think, is real. We know, for example, that Google has done a deal with Commonwealth Fusion Systems for a 400-megawatt power tranche when their fusion reactor goes live in a few years. And Helion Energy, which is another fusion company, has ties with Microsoft to power data centers.

6:28So this is a really, really significant problem. It's a big issue in the US, much less of an issue in China, where they've mastered the ability to deliver clean electrons at scale. There's also this squeeze coming in between inference, which is the bit of the AI activity that makes money, and training, which is when you're doing your product development for your next model. Model companies will be battling between where do they put their resources into training the next model or into serving company customers for revenues today. They have lots of resources, but even those resources are not infinite.

7:02And earlier this week, Brookfield, an asset manager, lined up with our estimates that in a few years, about 70 to 75 % of compute cycles will be used on inference. So that's going to be a tension, right? Do we pay bills today or do we build the next big thing in some different way? And you see the labs. I mean, I think the contrast between Anthropic and OpenAI is most marked in how they approach that, right? Anthropic appears to be rather more focused in thinking through the economics of that particular trade-off between training and inference. There are levers to address that, efficiency gains being one that is an obvious approach.

7:40We are starting to see more and more companies routing requests to cheaper, which means models that use less electricity and cost less models within their application. So you put in the query and the query figures out, oh, maybe I should send this to DeepSeek rather than to an OpenAI model to get the result. And we've made real technical progress, both across the algorithms and across the chips that serve them over the last few years. If you look at a sort of GPT-4 level class of inferencing back in 2022, it would take one watt hour to generate 50 tokens. So what is a watt hour? If you've got a 10 watt LED bulb, one watt hour is sort of leaving that on for six minutes to get your 50 tokens.

8:24Today with the, you know, the latest NVIDIA chips and more efficient optimized language models, we're getting to about 600 tokens per watt hour. So that's a 20x improvement just over four years. Of course, the amount of tokens we want has increased significantly. And so on the other hand, you have these new reasoning models that might burn 10 to 100 times more tokens per query. So what you have there is these firms, the hyperscalers who are really hungry for compute, they're hungry for compute, not just for AI, but for other workloads. And they're hungry for that because we as consumers and as businesses want those types of services.

9:02So you've got those on one hand, the grids can't keep up, the GPUs are being rationed between training and serving. this is a this is a short-term squeeze is my sense and that over as the industry matures the trade-offs will become much more apparent we'll get through some of the the blockages around providing power we often see that in markets that you get these these squeezes and ultimately the the market industries have take one or two years to reconfigure and to be able to deliver what is required it's just not going to happen tomorrow i think the third thing is about the economic engine and the question about whether we are going to see results from all of this.

9:45Now, I went into this in some detail in last week, but there's just one thing that I wanted to come back to. There's a lot of uncertainty in this market right now. And it's amazing that such huge decisions are being made in the face of such uncertainty. One good example is that Arvind Krishna, who is the CEO and the chairman of IBM, was speaking last week. And he said, look, it costs about$80 million per megawatt, because in the last year, tech companies have started to think of their computing in terms of watts and megawatts. So$80 million per megawatt for a data center. Now, that's quite a high number.

10:22Most people were using a number that Jensen was talking about, Jensen Huang from NVIDIA, of$50 to$55 million per megawatt. A few years ago, a couple of years ago, a really, really high-end data center was running at about$20 million. Now, if Arvin's right, and perhaps he is, perhaps that's IBM's experience, that changes the payback economics quite significantly. Another thing I want to just bring up is that I talked about the middle of 2026 as being a really key point for evidence to emerge that this technology is more than helpful and seems to be quite positive, but actually is starting to deliver results.

10:58And I just want to unpick why the middle of 2026 is so important. It's important because ultimately, many companies really started their enterprise buildouts in early 2024. So the enterprise tooling from Microsoft and from Google was available in late 2023. You need to put a project team together. You need to hire external talent. You need to get going. So you might get started in early 24. You should give an enterprise project a couple of years time to prove itself as a pilot. And that two-year clock will start to be reached in the early summer through to late summer and early fall of 2026. And at that point, we should start to see more and more companies talking about the results they're getting.

11:43If they don't talk about them, they might still be getting results and just don't want to share, which is not unheard of in this market. So that's the third layer. Then the fourth layer is really about the politics of absorption. So are we really able to absorb this politically? And there are complexities. The idea of sovereign AI, sovereign technology, which I wrote about in my first book as well, is becoming incredibly, incredibly real. It's a race for the US and for China, but for middle powers, which is really everyone else, right? So if you're not US or China, you're kind of bunked in as middle powers.

12:22It's a challenge, right? How much control are you going to actually have on this absolutely critical, critical infrastructure? And it creates this strategic dilemma for states. There are concrete signals of these smaller companies doing things, of course, the UK, of course, the Gulf, Brazil and India both have new AI data center projects running into the tens of billions. But it's going to be really complicated for those middle powers when they think about not so much the models, because you can always get an open source model, but they think about the chips, they think about ultimately the serving infrastructure.

12:52And that is going to play out significantly, certainly over the next couple of years. But the final one, I think, is this. And this is the most paradoxical part of this AI wave, where 2 billion people are using these tools. People like NanoBanana, and they like image generation, and they like Sora, and they like other things. And yet, we know from surveys, Edelman Trust Barometer being one that I use as a barometer, frankly, showing that roughly 70 to 75 % of Americans are pessimistic about what AI might bring to them while they even use the tools. It's almost like you're forced to use them somehow because you need to participate.

13:33It's an uncomfortable place to be. And I think that that problem is going to become more and more acute. We're already starting to see resistance in the US to the build out of data centers. There's an analyst pressure group that tracks this and they had identified 142 data center projects of total value over$64 billion stopped in the US since 2023, whether that was happening in Virginia or the Midwest or Pennsylvania or the South, these groups are organizing and trying to resist the buildout of this infrastructure. And it's fanatically bipartisan. So you've got traditional landowners and rural communities connecting up with environmentalists to ensure that the I's get dotted and the T's get crossed, but also at some point that you can resist the build out of this.

14:27And that, to me, is going to be a really interesting and important tension that will grow over the next year or two. It does introduce the idea of a legitimacy crunch and how AI companies need to talk about what it is they're doing, which I think they're going to have to do over the next year. Talking about replacing every job, talking about scientists in a data center, thinking up everything. It may not be the messaging that is going to be appealing to anyone. It also may not be how we think about this technology in the most sort of human, beneficent type of way. Okay, so that's what we've got to.

15:03This is what I think was important to think about over the next couple of years. And the models aren't the bottleneck. There's a series of absorption challenges. A quick note, if you want to support us in bringing more of these conversations to the world, please consider subscribing to the show. But let's turn to some questions and take disagreements as well that you might have. I'll be looking over there to see what's come in. And I'll start with, can you provide any insight on whether it makes sense for an organization to jump in and not be left behind? Should we wait until after this kind of crux moment when things settle or not?

15:40You know, should you wait? Should you settle? I mean, this is the should you upgrade your iPhone problem multiplied by a factor of a thousand. The challenge is that nobody really knows how to make the most out of AI. The only way you're going to learn how to do that and learn about how dynamic it is, how it changes the way teams work, how individuals progress, is by actually building that capability, by building it yourself, by doing it yourself. If you choose to delay for there to be stability, well, I can't tell you when there will be stability. I mean, normally it takes a couple more years.

16:13But if you do wait that long, then you've got this issue with your people. You haven't trained them up on the technology that we know is going to be really important. You also haven't established the capacities within your company. So you're going to start from a cold start. And so to that extent, I think you do have to start now. You should start now. You should have actually started a year and a half ago. So if you haven't, you might want to just drop off this call now and get going. Another question has just come in. And it sounds as if OpenAI is going through an inflection point in balancing expansion with model training.

16:45What do you see as its USP that could help stem the loss of market share to Gemini? We know that Gemini has done well. We know that OpenAI's traffic to chat GPT has declined a little bit. We don't know whether that's not seasonal. We also don't know whether beyond a few early advanced users, people have actually started to use chat GPT less. But I think there is a really important thing in the heart of that question. which is, does OpenAI have a really distinct way of thinking about what their position in the market is? And, you know, I think that they went out to conquer a lot of ground and they've done that really successfully with sovereign deals all over the world, with special products for the Brazilian, the Indian market and some other countries with, you know, an enterprise offering with ChatGPT obviously doing very, very well.

17:36And now big enterprise deals with companies like Accenture and Emirates and others, and they keep on rolling down. I think there was one with a company that provides stock market data yesterday. It's a lot to ask though, to chase lots of different spaces. It's not what a Y Combinator startup would do. You know, you would find your beachhead, you would get on your beachhead, you would understand the customer, and then you would grow from there. And I think one challenge was that nobody knew ChatGPT would be successful. You got product take up before you'd understood why you had product market fit.

18:08And that's why the product itself has felt a bit experimental and perhaps not improving in the ways that we might think. You know, I have a lot of respect for the quality of thinking in the senior team in OpenAI, and they will have seen data much deeper than I have. So as an outside observer, you know, it's just a simplistic mantra, which is, you know, find a thing that you can do really much better than other people and do that thing and use that to grow out. And I think that's what a company like 11 Labs or a company like Anthropic has chosen to do. Why do you think the stock market was not negatively impacted by the OpenAI Code Red announcement?

18:46Yeah, well, I also was surprised by that. But the market is a complicated beast that's got lots of other levers that are moving around. And frankly, they are flying with quite poor quality data. You know, it's not like they have really, really great sources of analysis. These are leaks in trade in the trade press. I can't really tell you why it didn't react that way. I mean, I would also just add that I simply wouldn't, you know, count that team out and I wouldn't count a short wobble over the last two weeks and turn that into a long-term, long-term trend. I think the sensible thing to do as a boss is to say, see that and say, I'm not going to be, you know, I'm not going to be lackadaisical about this.

19:27I'm going to prepare for the worst and sort of drive the team as if this really, really is real. Where do I think value will accrue in the value chain? And, you know, the mantra this week, Mark Benioff said it as well, has been, you know, if you own the workflow, that's great because you can always swap out the model. And I'm definitely seeing stories of companies preparing to swap out models and building the tools they need. So it may be interesting, your firm may have this, if any of your companies are building this, just say so in the chat, but a model routing layer within your company, company X, that allows them to switch models in and out alongside all of the other prompt changes and guardrail trade changes that you might need based on availability or cost.

20:11So in some sense, there's a downward pressure on models that don't differentiate themselves. And I think that's where, you know, you see HSBC, which is a reasonably sized European UK bank, doing an enterprise deal with Mistral earlier this week, which makes the open source models out of France. And I think that's where, again, you see someone like Claude with Anthropics Claude being really, really good at one thing, which is particular. I mean, it's a good, great generalist model and we use it a lot, but it's really good at code generation. And therefore, it's much harder to unpick it from those particular workflows because people get used to it.

20:51What's the best strategy for middling powers? Ah, it's a good one. I mean, I'm working on a project with some other people on this question. they do need to establish at the very minimum some class of sovereign stack but what that looks like varies country to country so you probably will see some mini lateral arrangements between companies that might might be sort of formally tied together like in Europe in the European Union or just our neighbors and you're certainly seeing that in sub-Saharan Africa where you start to say okay what are the things we actually need at each stage and what can we what can we share in terms of provisioning compute and provisioning the power and the data and the data governance that we might need and even the talent.

21:36So there it's going to have to be collaboration. But I think the really difficult question, and I don't think this necessarily shows up in the next year or two, is that that's only sovereignty by vibes. Like it kind of feels like sovereignty. It's not real sovereignty in the way that an international lawyer might think about it. Because ultimately, you know, if you are building on the China stack, you're building on the US stack, one of those two countries can say, actually, we don't want to support you anymore. And you don't have that lever of control then at that point. But I think you can do quite a lot by building capabilities and just making your own capacity better.

22:15It gives you more room at the bargaining table. Cybersecurity risks seem to be growing. Any chance that security problems will call AI growth to grind to a halt, you know, there's always a chance that something can happen. You know, we live in a world of fat tails, right? And odd things can happen. We've been through moments of really, really deep security vulnerability in the past, which felt, you know, felt almost existential. So Microsoft had all of those issues 20 plus years ago, which is why they then started to emphasize the idea of trusted computing. There's clearly an emerging set, a new set of risks that come out.

22:55But every technology wave has those new sets of risks. And if you'd gone to someone and said 25 years ago and said, oh, there'll be a trillion cyber attacks a year on the internet, you might've said, well, let's not build the internet. But actually the reason there are a trillion cyber attacks is because the internet is much more useful than any individual cyber attack. But the seriousness, I think, of what these systems as they get progressively more agentic can do. And I think, you know, Claude is discovering all sorts of weaknesses in the red teaming that the Anthropic team runs is getting progressively more serious.

23:30But the thing to note is that we are, what we hear is absolutely state-of-the-art, on-the-edge results. What we don't hear are the mechanisms of defense that are being built up by the cybersecurity companies. And, you know, they are, of course, not sleeping on all of this. Thanks for listening all the way to the end. If you want to know when the next conversation is released, just hit subscribe wherever you're listening. That's all for now, and I'll catch you next time.