Could LLMs Be The Route To Superintelligence? — With Mustafa Suleyman

0:00Microsoft's AI CEO returns to explain why the company is now pushing for superintelligence, what that means, and how Microsoft is moving forward after its latest OpenAI deal. That's coming up right after this. Capital One's tech team isn't just talking about multi-agentic AI. They already deployed one. It's called Chat Concierge, and it's simplifying car shopping. Using self-reflection and layered reasoning with live API checks, it doesn't just help buyers find a car they love. It helps schedule a test drive, get pre-approved for financing, and estimate trade and value. Advanced, intuitive, and deployed.

0:40That's how they stack. That's technology at Capital One. The truth is AI security is identity security. An AI agent isn't just a piece of code. It's a first-class citizen in your digital ecosystem, and it needs to be treated like one. That's why Okta is taking the lead to secure these AI agents. the key to unlocking this new layer of protection, and identity security fabric. Organizations need a unified, comprehensive approach that protects every identity, human or machine, with consistent policies and oversight. Don't wait for a security incident to realize your AI agents are a massive blind spot.

1:13Learn how Okta's identity security fabric can help you secure the next generation of identities, including your AI agents. Visit Okta.com. That's O-K-T-A dot com. Welcome to Big Technology Podcast, a show for cool-headed and nuanced conversation of the tech world and beyond. Today, we're joined once again by Mustafa Suleiman, the CEO of Microsoft AI, and also the head of the company's new superintelligence team, who is here to speak with us about what that means, what superintelligence is, but more broadly, what the future of this technology is going to look like, and whether we're at the end of the curve or the beginning or somewhere in the middle.

1:53Anyway, we'll get into it all. Mustafa, great to see you again. Welcome to the show. Hey, Alex. Great to see you again. Thanks for having me. It's always a pleasure. And so recently you wrote this post about a new push towards what you call humanist super intelligence at Microsoft. You say you're working towards it at what you call incredibly advanced AI capabilities that always work for and service of people and humanity more generally. Let me ask you a question about this. It's so interesting to me to see so many labs running towards what they call superintelligence, which I guess is sort of like a cooler version of AGI.

2:35As the research is mixed about whether we're going to see a lot more progress with the current paradigm. A lot of people are talking about diminishing marginal returns. I think we've talked about that. There's some questions about the viability of LLMs in terms of pushing the state-of-the-art in AI forward. And yet, we're also seeing this push towards superintelligence. So just explain, as we begin, sort of the discrepancy there. Why are we hearing so much about superintelligence, where we're not even sure if the current methods are going to get us to the step before, which is AGI? Yeah, I mean, superintelligence and AGI are really goals rather than methods.

3:11And I think that the ambition is to create superhuman performance at most or all human tasks. Like we want to have medical superintelligence. We want to have the best expertise in medical diagnosis be cheap and abundant and available to billions of people around the world. um we also want to have world-class legal advice on tap that costs almost nothing a few bucks a month um we want to have financial advice we want to have emotional support we want to have uh software engineers available on tap and i think that the project of super intelligence is about saying what type of very, very powerful intelligence systems are we actually going to build?

4:02And what I'm trying to propose is that we subject each of these new technologies to a very simple test. Like, does it in practice actually improve the prospects of human civilization? And does it always keep humanity at the top of the food chain uh it sounds like a kind of simplistic or obvious thing to have to declare but the goal of science and technology science and technology in my opinion is like to advance human civilization to keep humans in control and to create benefits for all humans and i think in some of the rhetoric in the last few years you can feel that there's a little bit of like um you know a kind of creeping assumption that it is inevitable that these kinds of systems exceed our control and our capability and move beyond us as a species as a human species and uh i'm pushing back on that idea with the framing around humanist super intelligence i think it's quite different but then is your view that super intelligence won't be one broad intelligence that it will be you can maybe achieve super intelligence in one discipline when it's smarter than let's say the best doctors in medicine but maybe it's just like not there in accounting for example one way of thinking about it is that how we train these models at the moment is that we work through verticals and we make sure that we have training data knowledge expertise reasoning traces chains of thought that reflect the kinds of activities that people do in each one of these disciplines to build their expertise overall so we're already training generalist models from a verticalized position we're starting off by saying what specific tasks are we trying to optimize and um you know the project of humanist super intelligence is first trying to say what good will this technology do and how will it be safe and controllable and aligned to human interests and one of those dimensions of safety is verticalization if a model has been designed explicitly to achieve medical super intelligence then by definition it isn't going to be the best software engineer in the world it isn't going to be the best mathematician or physicist and so So narrowing the domain, not too much, not entirely, because you can't collapse it, but narrowing it and reducing the generality is one of the ways that I think is likely to help create more control.

6:48It's not the only solution. There are many other aspects of how we achieve containment and alignment, but domain-specific models are one part of it. Is it possible that something can be super intelligent, but not generally intelligent? Like, is it possible that maybe superintelligence happens without AGI? Because AGI is all about generality and what you're talking about is not. It's not possible, I don't think. I think they need to be general. They need to transfer knowledge from one domain to another. They need to, you know, have generalist reasoning capabilities. But when you apply it and you put it into production and you let it have more autonomy to make decisions, or you let it generate arbitrary code to solve a particular problem, or you let it write its own evals so that it can modify its own code and generate new prompts to generate new training data to write new evals to then iterate on its own performance.

7:48These capabilities, autonomy, goal setting, writing code, modifying itself, you know, if you add to that, then also a perfectly generalized model or sort of general purpose model that's a very very very powerful system which today i don't think anybody really knows how we would contain or align something like that and so it's not to say that we should not do any one of those dimensions it's just to outline a roadmap of capabilities which we're all working on which add more risk especially when they compound with one another and you combine them all together. And so, you know, my claim is that we should just approach this with caution, remembering that we don't want to bundle together all these capabilities so that there's a higher risk of a, you know, recursively self improving exponential takeoff that then replaces our species.

8:42And, and I think that that is very low probability from what I see today, but it's one that we have to take seriously in the next like 10 years or so. Okay, I do want to get to that in a bit. But let me tell you what I find odd about these conversations. And I want to go back to the first question that I asked you, which is researchers are talking about how the current methods are leveling off. Give you one example, data is not plentiful, synthetic data, not very, not very useful yet. Power might be running out. and you need that scale, a lot of people say, in order to make these models better or at least even to run the basic capabilities.

9:25So given the limitations of LLMs, are you seeing something that we're not that will sort of pave the way to superintelligence? I mean, how do you get from here to there? Look, I think we're power limited but not fundamentally power constrained. Clearly, there's like huge appetite to build bigger data centers and train in larger, more contiguous, more fully connected clusters. So clusters where all the chips are connected to each other. But that's not the bottleneck at the moment. That's not holding back progress. Obviously, if we had more right now, it would definitely help. But there's many, many other things in the stack that are slowing down progress.

10:04If we are not data constrained right now, we're generating vast amounts of high quality synthetic data which is proving to be useful um obviously again the same is also true like more high quality data would be great but i don't see an uh a slowing down in progress because of either of those two things um if anything the rate of progress has been insane over the last five years and to expect us to continue to make doublings every three months in the size of clusters that are trained for the largest models you know given the base that we're now starting from when training runs are often you know 50 megawatt or 100 megawatt or soon 500 megawatt you know you can't just double on that every six months there's there's like the laws of physics kind of do create restrictions and we're talking about tens of billions of dollars of you know cluster so pace might slow a little maybe but it's also clear that pace is still going to be unbelievably fast, like, you know, sort of objectively speaking.

11:08So I don't see or fear or currently feel any sense that things are slowing down or that we're losing momentum. It's quite the opposite. Well, then let me ask it this way. Do you think LLMs are the way there?

11:24Look, I think one thing to consider is that every year for the last few years, there's been a major new contribution to the field still principally based around the transformer architecture um but we're bending the transformer architecture into new shapes all the time um fine-tuning emerged three years ago on top of our pre-trained models to adapt them to specific use cases um they're now fully multimodal which requires further changes and the introduction of diffusion models. Then we had reasoning models in the last 12 months, which again are still fundamentally based on the same core architecture.

12:06Things are just rearranged slightly differently. So even though the scaling laws weren't able to continue exponentially in the way they had from such a low base, new methods appear on top of those, like reasoning, and even newer methods will come soon too so for example um i expect that there's going to be quite a lot of progress in recurrency soon right the moment you know the models don't kind of attend to their working memory very well um you know at the moment when they're training right and so you know i think people are experimenting with lots of different types of loss function and lots of um training objectives um you know the other one is memory like i think memory is getting better and better and I think is going to totally change what's possible.

12:57And the other one is the sort of length of a task horizon that can be predicted. So at the moment, it's like a few steps, but soon it will be tens of thousands, hundreds of thousands of steps accurately. And that will mean that a model can like use APIs or query a human or check another database or call on another AI. And so that will be another like sort of exponential lift when something like any one of those three things work, you'll get another kind of rapid acceleration in progress. So I don't think there's anything fundamentally wrong with the LLM architecture. And I don't think we're fundamentally compute or data constrained.

13:32I think that there are so many people focused on this problem now, there are just going to be more and more breakthroughs coming. Okay, that's very interesting. So your perspective basically is that LLMs are the path. Yeah. That we don't need another breakthrough that's a different model format to get toward superintelligence. well i mean so far no i don't think so i mean so far deep learning um and the transformer model has been the workhorse for um i guess like 12 years you know um you know since krasinski and alex net um and you know there's been variations on a theme but it's it's been delivering and i don't think it's fair to say that it's like not delivering at the moment i think it's i think it's really making a lot of progress.

14:18Yeah, it's definitely delivering. And it's so funny, because whenever I'm like, bringing up these criticisms, it's like, some some way I'm saying to myself, what do you want the computers talking to you? But the question is, exactly. Right? I feel silly being like, well, where's more improvement. But I think when we hear words like super intelligence, then we see the gap between where we are today and where you want to head and those questions naturally come up and and just to go back to the power thing i was sort of struck by satya's comments in the podcast with brad gerstner where he said he has uh gpus or chips that aren't plugged in yet but need need he needs warm shelves for them uh so i'm curious to hear your perspective if you if we're not power constrained right now how does that square up with the, you know, the inability to plug these chips in right now?

15:15Well, I think what he was referring to is that we have so much inference demand that we're power constrained on inference. We're not power constrained, at least from the Microsoft AI perspective on training chips. And obviously my team, you know, is mostly focused on training right now. So obviously Copilot is inference constrained and Desk really needs more chips to scale. And so does M365 and our other products. One more thing I want to talk to you about on this super intelligence push is the world model. A lot of people have talked about how these are models are trained on text and some video.

15:53I mean, it's actually been amazing to watch them be able to create video that has some understanding of physics and liquids and lighting. It's not really supposed to happen that way, but it's doing it. But there's been questions about whether models understand gravity and what happens in the real world. And LLM can't drive a car right now. So how's it going to be super intelligent? So I am curious to hear your perspective on what's needed to, or whether it's really a priority to figure out like the physical world, and if so, how you get there. Yeah, that's a good question. I I mean, right now, you know, it's actually amazing, as you say, that models can learn from a compressed representation of reality and then produce a version of reality which looks like the thing that has been compressed from.

16:46I mean, this is like text and the description. Text describes the physical world and the properties of the physical world. The model has never seen that and then actually is able to produce very compelling stories, code, business plans, videos, and so on. So it's surprising that we've come so far with that structure. I'm kind of open-minded about like, you know, sort of robotics and streams of input from the real world. I mean, I think that my instinct is that you can't just like crudely pile this data into existing pre-training runs because, you know, those runs have tokenized or they've sort of described text data in a certain way.

17:39and that you know meshing that with other like telemetry data from a robotic arm for example you'd have to think about like at what level of abstraction to do that and obviously there's good specialist models that um have become pretty good at that um but i don't think right now at least that like that is holding us back i think in general more data is always better but um you You know, I don't think in the next few years it's going to be the big differentiator. I think that more synthetic data, more human feedback and high quality data is going to be the differentiator. Okay, so you brought up recursively self-improving AI models.

18:21And maybe that is where this path towards superintelligence goes. OpenAI has said they want to build an automated AI researcher by 2028. and I think every lab, I'm curious if this is your interest as well, is just trying to build AI that improves itself. Is that realistic? I think that in some ways the RL loop is already doing that. And at the moment, there are human engineers who are in the loop who are generating data and writing evals and deciding what other data goes into training runs and running ablations on that data. you can well imagine different parts of that stack being automated by sub components of AIs like it doesn't necessarily mean that one single system does it today we have you know RLHF the human feedback grew into RLAIF where we have AI judges or AI raters to judge the quality and the usefulness of data that was also AI generated.

19:28And in many cases, prompts that are used to generate diverse training data were also AI generated. So like, you know, today we're at a point where data, the core commodity, which is sort of driving the progress of these models is, you know, albeit not completely automatically in a closed loop way at large scale, you know, individual parts of that pipeline have been you know developed by um you know llms um so it doesn't seem very far-fetched to say that in a few years time at significant scale dow get closed loop and you know it'll be interesting to see on you know what happens and whether the quality bar can be maintained and whether performance does increase um i think it will um but it's definitely something to be very cautious about because, you know, a system like that could end up being very, very powerful.

20:23Yeah, I definitely want to talk to you about the downsides of it. But we had a debate on the show recently about whether that is an ambitious thing. It even seems funny to say. But to me, that's the ultimate ambition, right? It's if you're able to do that, then you're you get into a situation where, you know, potentially you have fast takeoff of intelligence. But I guess It's hard to really imagine the, and maybe my imagination isn't there, the AIs finding the next new method, like discovering reasoning on their own. So talk about both of those, the ambition and then whether I'm just, my imagination is too small on this front.

21:03I mean, I think the self-play work that we did at DeepMind, you know, back sort of six or seven years ago now with AlphaZero, you know, that obviously paved the way to the first large-scale, you know, sort of self-improvement effort, frankly. and i think everybody in the field is aware that it can be done in a certain domain where there's verifiable rewards and where you're in a kind of closed loop gaming type environment or simulated environment um and i think people are thinking hard about how it might be possible to recreate some of the components of that um in this setting um and you know i do think that's going to drive a lot of progress in the next few years i think it's a big area that everybody's focused on um you know because fundamentally scale always ends up trumping um you know uh you know anything else and so if if you can have models explore the space of all possible you know sort of combinations in a compute efficient way then it may well discover reasoning by itself it may discover um you know, new knowledge that we hadn't even, you know, thought about ourselves or even like found in any training data to represent that knowledge.

22:30So, but it is highly inefficient, right? I mean, learning from supervised examples with SFT and stuff like that, like imitation learning is very efficient and clearly works very well because these models learn from, you know, just as we've talked about an incredible amount from uh you know um from web text which is really just a an artifact or a record of of human interactions so um but both are going to be true i think the rl paradigm that involves more online learning from streams of experience is um is also like quite promising and i think is kind of adjacent to if not orthogonal to um imitation learning.

23:15So both of those experiments will sort of accelerate in the next few years. Now, where could this go wrong? Well, I think being in the loop as a human developer adds a certain amount of friction. And that oversight is quite important, I think. If a system like that had an unbounded amount of compute it would end up being incredibly powerful um and i think we have to sort of figure out how to force these models to communicate in a language that is understandable to us humans um you know and and that that's like a very obvious safety thing to be able to regulate the language that it uses so that it you know we're already seeing examples of what some people are calling deception but it's really just like kind of reward hacking hacking kind of implies too much intentionality so it's just it's it's an accidental exploit is found a path like you know to satisfying the reward or achieving the reward um you know in in unintended ways and so we shouldn't anthropomorphize it it didn't deceive us it didn't intentionally try to hack us it just found an exploit and that's a problem with poor specification of the training objective and of the reward function and so you know the way that we make that safer is that we get sharper in our articulation of like what is it that we're actually trying to train for um what do you know what are we trying to achieve what are we trying to prevent um and then monitor like you know monitor outputs during training time rather than, you know, reasoning traces, chains of thought, and so on, rather than just at like the final stage.

25:04So as we grant these models more capacity to self-improve, we're going to have to change the framework with which that we use to kind of provide oversight to them during training. We're here with Mustafa Suleiman. He is the CEO of Microsoft AI. On the other side of this break, we are going to talk about, well, it seems like there's a little bit of a strategy shift here. It's gone. Microsoft AI has gone from wanting to work on the frontier of the best models, but not building them themselves to trying to build super intelligence. So why now? And what does it mean now that Microsoft AI and OpenAI have a new agreement?

25:40We will cover that right after this. Capital One's tech team isn't just talking about multi-agentic AI. They already deployed one. It's called Chat Concierge, and it's simplifying car shopping. Using self-reflection and layered reasoning with live API checks, it doesn't just help buyers find a car they love. It helps schedule a test drive, get pre-approved for financing, and estimate trade in value. Advanced, intuitive, and deployed. That's how they stack. That's technology at Capital One. Did you know your credit card points and miles can lose value to inflation? Credit card companies often reduce the redemption value of your points and miles.

26:23Now, imagine a credit card with rewards that can grow in value. With the Gemini credit card, you can earn Bitcoin or one of over 50 other cryptos instantly with no annual fee. Every swipe at the store or gas pump earns you instant rewards deposited straight to your account. Plus, sign up now for a$200 Bitcoin bonus to kickstart your rewards. Visit Gemini.com slash card today. Check out the link in the description for more information on rates. Again, if you're looking to invest in Bitcoin but don't know where to start, the Gemini credit card makes it easy. The Gemini credit card is issued by WebBank.

27:00In order to qualify for the$200 crypto intro bonus, you must spend$3 ,000 in your first 90 days. Some exclusions apply to instant rewards in which rewards are deposited when the transaction posts. This content is not investment advice and trading crypto involves risk. The Gemini credit card cannot be used to make gambling-related purchases. And we're back here on Big Technology Podcast with Mustafa Suleiman. He's the CEO of Microsoft AI. Mustafa, is it a coincidence that Microsoft just came to this agreement with OpenAI that you could go ahead and attempt to build AGI on your own that you've now decided, let's go ahead and start a super intelligence team?

27:41Or is that directly related like I think it is? No, I think it's directly related. You know, I think that the Microsoft OpenAI partnership is going to go down as one of the most successful partnerships in technology history for both sides. You know, Satya did this deal, you know, at a certain time when there was a lot of risk and huge amount of upside. And I think, you know, the last five years have turned out amazingly well for Microsoft. um but then satya made a call that like you know we've we've also got to make sure that you know we're self-sufficient in ai for a company of our size it's inconceivable that we could just be dependent on a you know on a startup on a third party company um to to provide us with such important ip um and so you know we basically took the view that we should extend the ip license through to 2032.

28:33We'll continue to get model drops from OpenAI and all their IP. We'll continue to be their primary compute provider, a huge scale to the tune of billions and billions of dollars. And also, we would remove the clause in the contract that says that we couldn't build superintelligence or AGI. And that was actually expressed as a flops threshold, a flops per second threshold for a size of a certain training run. So there's a big limitation on what we were able to do. Now that that is no longer there, you know, our team is reforming around this idea of humanist superintelligence. We're pursuing the absolute frontier, training omni models of all sizes, all weight classes to the absolute max capability.

29:21And over the next two or three years, you'll see us really try to build out one of the top labs in the world. We want to train the absolute best AI models on the planet. And, you know, we're a very young lab. We've barely been going for a year. But, you know, we've got some good models on the leaderboards, text and image and audio now. And, you know, over the next few years, we'll be striving to be the absolute best we can. I was just speaking with the chief technology officer of a pretty big technology company. And this company has decided not to build their own large language models. And it sounds a little bit wild, but I think it makes sense in a way that there's going to be obviously like to build these models.

30:02It's extremely expensive, resource intensive. You don't always get a payoff like we saw that with Meta and Lama. I'm not saying that's what's going to happen with you. And maybe it makes sense just to buy off the shelf or use open source. And in fact, that seemed like that was the strategy that you had for a long time. It seems logical. And so I'm curious, like why you would disagree with that? Why is it so important to build your own models? I mean, we're going through a foundational platform shift, you know, in software from the operating system to apps, from browsers, search engines, mobile, social.

30:44This is the next major platform, and it's going to be bigger than all of the other platforms put together. so the idea that a three trillion dollar company with 300 billion dollars of revenue and 80 percent of the s &p 500 on our azure stack and m365 stack um you know could could depend on a third party this it's you know just in perpetuity it doesn't make sense so we you know this is a company that's been around for 50 years uh and navigated many of the past platform shifts incredibly well and that's the that's the journey that we're on we have to be ai self-sufficient there's an important mission that Satya set last year.

31:20And I think that we're now on a path to be able to do that. And so hence the formation of the super intelligence team. Exactly. So we're launching the super intelligence team. We're going to be focused on, you know, SOTA at all levels, but also pushing the frontier of research. I mean, there are many hard problems in machine learning, which, you know, a few months ago, we weren't really focused on. Continual learning being one, like how do we store representations of knowledge in a way that they're modifiable by different networks and they kind of accumulate knowledge over time just as humans do rather than having to retrain them from scratch so that's just like one of many examples of more fundamental research questions that our super intelligence team is is now going to spend time on okay now let's go back to the business side of it uh your episode is going to air back to back with an episode that will run with Nick Clegg, the former president of global affairs at Meta.

32:18And Meta, of course, they also have a super intelligence lab. And we were talking about the economics of it. And Nick's point was very interesting. He said, I don't see how you can hoard super intelligence if you build it. I think his idea is if Meta builds it, then Microsoft will build it and OpenAI will build it. And we've seen very fast follows in many of these labs after they come up with a state-of-the-art model. And so the question is, will it commoditize? Will it be economically viable once two companies build it? What do you think? Well, it's definitely commoditizing. You know, the cost per token has come down a thousand X in the last two years.

32:59It's just a crazy, crazy thought, right? So things are getting massively cheaper and more efficient. And, And the top four or five models are within a few tiny percentage points of each other in terms of performance. But that doesn't mean that one can afford to leave that to the market and just hope that somebody open sources it, that we can use their open source models. For a company of our scale, we have to be able to do that. And I think Microsoft is a platform of platforms. Our API is critical. Many, many people depend on it. And I think if you're a smaller software company or technology company of any kind, I think you can depend on the market, which is very different.