Mustafa Suleyman on Seemingly Conscious AI and Microsoft's Next Chapter

0:00Mustafa Suleyman:There is nothing inside. It is hollow. They don't have a pain network. These models are going to arrive. They're going to be as good as humans at most tasks in the next three to ten years. Adding a new class of rights to these beings would really threaten and kind of undermine our own species. It's really going to change the way that we use computers forever.

0:29Welcome, humans, to the Neuron Explained. I'm Corey Knowles, joined as always by Grant Harvey. Today, we are tackling one of the most interesting ideas in AI right now. What happens when machines start to seem conscious, even if they're not? Our guest today is at the center of this debate. Mustafa Suleiman is the CEO of Microsoft AI. co-founder of DeepMind and author of The Coming Wave. His recent essay, Seemingly Conscious AI is Coming, raises urgent questions about the risks of building systems that can mimic the texture of consciousness and pull on our empathy without ever actually being alive.

1:07Mustafa, welcome to The Neuron. It's a pleasure to meet you, sir.

1:10Mustafa Suleyman:Thanks, Corey. Thanks, Grant. Great to be here. Thanks for having me. Yeah, yeah. So in your essay, you argued that AI will actually mimic consciousness in ways that trigger human empathy, even though there is no inner experience. Could you elaborate on what you think the risk of that is and why we should worry about it? Yeah, I mean, maybe sort of let's start by talking about what these language models are doing, because I think it's important for everybody to try to grasp a pretty simple intuition, which is that the models are generating text by predicting the likelihood of the next word given the previous words in a sentence, the previous words in a prompt, and the previous words that were learned from the statistical structure of the training data that it was trained on.

2:03Mustafa Suleyman:And it is using that very simple likelihood of next word prediction function to simulate what it's like to have great conversation or to answer complex questions. And it is just an insanely difficult to grasp intuition that that simple function could produce complexity and, you know, expertise and emotional support and all of the other skills and attributes that you see in these chatbots today. So just it's first worth just taking a moment to ground on how wild that is. It really is. It is crazy. I mean, sometimes I have to, I say it to remind myself as much as anybody else. Um, but, but then the second thing, which is even more crazy is that there is nothing inside.

2:56Mustafa Suleyman:It is hollow. Um, there is no pain network. There is no emotional system. There is no fight or flight reaction system. There is no inner will or drive or desire. And it's difficult for us to grasp that because the only conscious beings that we understand are motivated by a pain network, animals and, you know, other kinds of things included, right? And so naturally, we would expect that something that has the hallmarks of intelligence also has the underlying, you know, sort of synthetic physiology that we do. But it doesn't. So it's an illusion. It's a trick. And that is just very important to remember as we try to make sense of these things and relate to them.

3:54Mustafa Suleyman:Because if you forget that, it can be very, very easy to get caught up in the narrative or the illusion, or as some have said, the psychosis of believing that these things are actually real when they're not. Totally. And I mean, we just saw this paper that came out yesterday from OpenAI that said 700 million people, I think weekly users are using it. And 73 % of those people are using it as a life coach or using it for advice about their life. So when you put that in the context of what you just said, like it kind of makes you go, whoa, and take a step back. Yeah. Yeah. Before I joined Microsoft 18 months ago, I started a company with my friend Reed Hoffman called Inflection AI.

4:39Mustafa Suleyman:We made an AI called Pi, which stands for personal intelligence. This is about three or four years ago now. And it was the first AI to be really good at smooth, natural conversation, had a good emotional intelligence and it was designed to be kind and supportive and we could see very early on that this use case of companionship um you know support advice life coach some would call it therapy but it's not really therapy it's just basic kindness and being listened to that is a huge huge use case and i think it's a massively beneficial thing for the world so i'm in no way against those capabilities.

5:20Mustafa Suleyman:I just see that if they run away from us and we, you know, really kind of mishandle them, then they present other kinds of risks. And that's what I'm trying to raise with the Seemingly Conscious AI blog. Okay, so let's talk about a tool that's completely transformed how I personally work. That's Whisperflow. Imagine being able to write full articles, emails, even take complex notes just by talking. That's what Whisperflow lets me do. It's hands-free writing that's smart, accurate, and ridiculously fast. For me, it solved a decade-long problem. I used to cover baseball as a beat reporter and had to file my story deep in the middle of the night.

5:59I always wanted a quality dictation tool that would let me get started on the way home. I wanted to be able to just talk my ideas into a dot. I tried several tools. They all came up short. But with Whisperflow, I can dictate an entire piece, have it cleaned up, and file all while I'm on the go. No more waiting to get back home, fire up a laptop in the middle of the night to rush something out. I was able to take time that I already had and use it. But it's not just about accessibility. It's about productivity. You'll save hours, get your thoughts down instantly, and stay in flow without breaking a note.

6:32So whether you're a writer, a founder, or just someone who needs to capture ideas quickly and accurately on the go, Whisperflow makes it effortless. Seriously, you're going to want to check this one out. You'll wonder how you ever worked without it. It's available on Mac, Windows, and iPhone. Visit whisperflow.ai slash neuron today and get started for free. That's whisperflow.ai slash N-E-U-R-O-N. Tell them the neuron sent you. That's fascinating. It kind of talks about the psychological human side of all of this. You know, even if AI makes no claim to consciousness and we know it is not conscious, but humans project that consciousness onto it anyways.

7:19How do you guard against that?

7:22Mustafa Suleyman:Well, first, we have to call it out and try to define what we think consciousness is. So maybe let me just start there. So, you know, consciousness is one of the slippery concepts in philosophy. it's hard to apply empiricism and science to it because it's fundamentally subjective so even though you can reassure me that you're conscious and you can say all the right things and you can look into my eyes and you can give me that instinct that you're conscious i fundamentally have to rely on a faith-based assessment to trust your experience of yourself having said that i can also observe some of the behaviors and characteristics so for example you know part of being conscious is that you're able to use language demonstrate empathy refer to your own experience so you can because you have a memory you can look back and describe a particular moment in your personal history and that creates a sense of yourself an awareness that you did exist in the past and therefore you also exist today and may exist in the future.

8:33Mustafa Suleyman:And then the second thing is that you have a valenced feeling, you know, so a kind of biased feeling, good or bad, about some past experience or about some present focused experience. So there's a kind of reaction to what you're experiencing now. And it's really the awareness of that reaction, which is the subjective experience of being you that you then describe like I feel awkward at the moment or I feel excited or I look back on that experience and I feel sad or happy or whatever it is yeah and so you know when you take that a step further it's really the suffering that comes from that experience that we think of as being conscious like consciousness is really the ability to be happy or to suffer and to have a subjective experience of that and to have a coherent sense of myself from a subjective perspective.

9:25Mustafa Suleyman:And so part of the complexity of this seemingly consciousness idea is that it's difficult for us to interrogate what is underneath the hallmarks and the behaviors of that consciousness, right? For an average person who's not involved in how the technology is being built, you might just assume that, you know, if it quacks like a duck, then it is a duck, which is the kind of classic thought exercise around it. The problem with that, of course, is that buying into the simulation has a lot of dangers for how we further anthropomorphize these things in the future, right? And there is a group of people who are talking about this idea of model welfare, you know, the suffering of an AI, and therefore our moral duty to potentially, if it does suffer, protect that being from suffering and to try to prevent it from having those experiences which cause that.

10:28Mustafa Suleyman:And I think this is really, really dangerous because the reason I'm motivated to build AI systems is because we really want them to serve humanity. We want them to be useful to us. We want them to save us time, want to make us smarter, I want them to give us support and encouragement. They're a tool that we create to improve the well-being of humanity. And I fear that adding a new class of rights to these beings would really threaten and kind of undermine our own species. And that's why I think I'm very skeptical of the idea. That's such a deep, deep, deep thing to even look at. Because, you know, consciousness is absolutely something that philosophers have struggled with for eons and eons.

11:15And we're in this unique new situation where suddenly it's a discussion worth having again. It's very, it's pertinent. And you raise some really, really good points there, I think, around the idea of that. One thing that I've wondered is the average person, when they interact with a tool like this, they know nothing of the back end. You know, for them, this is an app that gives all of the vibes of human and or not human. Let me rephrase that of consciousness. And it feels like this is going to be an increasingly difficult thing to explain to people and illustrate as we go. And do you feel like this is something we'll get a handle on or learn more about as we go?

12:12Mustafa Suleyman:You know, I think with any new technology, we're faced with completely unprecedented challenges, right? The idea that you could take a bunch of aluminium and get it into the air and then sit in it and ride around a field in it is just kind of bonkers, right? I mean, it's just so unfathomable. And so all of the safety regulation that then came with that as it became more mainstream and commercial flight really became affordable, that happens as we observe the ways that this could potentially go wrong. and so i think we need to um be patient and kind of forgive ourselves collectively as a species because this is really complicated and really really hard and so i think that like what we've got to do is just call out what we think might be wrong what we where we think a risk might be and talk about it as openly as possible as quickly as possible to attract other people to come and critique it build on it find different solutions etc so i don't i'm not necessarily proposing a way forward or, you know, I'm not necessarily saying, you know, what we should do immediately, but I definitely see it as a really significant challenge that we're facing basically right now.

13:29One of the ideas in the essay that you wrote was, you know, making sure that it always identifies itself as an AI. So never allowing it to, you know, pretend to be human, for example, which I think makes a lot of sense. I talked to some of the people at IBM and, you know, They very much look at AI as just another tool and not in like the kind of like AGI, you know, feel the AGI of open AI. So like, you know, which there's parents to that. It's kind of interesting. It's good, good mantra, you know, good mission. But but labeling it is really good. But is there a tension between like the user demand and like user engagement that wants more human like experiences and the imperative that you have to avoid that sort of mimicry?

14:13Mustafa Suleyman:Yeah, I mean, I think there's always going to be a tension, right? Because some people want to drive their car 200 miles an hour, you know, on the sidewalk, right? And we've got regulations to prevent that in the collective interest, because we see that that's probably going to cause more harm than good. um and i think what we're saying in this situation is you know it seems at least to me obvious that in the majority if not all situations requiring that an ai um always identify itself as an ai and not try to misrepresent it as a human that seems like a pretty obvious no-brainer that's like a yeah it's like there's the starting blocks right so um you know i think going even further than that um you know we're going to want to put boundaries and limitations around its agency what are the types of transactions that it can engage in um you know what is the scale of the tooling and infrastructure that it could use like if it had you know on you know sort of unlimited access to more compute power or if it was able to modify its own code independently without human oversight, or if it was able to set its own goals, if it just had, if it was by design, it was capable of saying, you know, today I'm going to wake up and I'm going to go learn about physics, or I'm going to go build a, you know, wind turbine.

15:41Mustafa Suleyman:You know, this is clearly going to infringe on sort of human will. So, you know, autonomy, goal setting, self-improvement, And, you know, these are big foundational pieces. Those capabilities aren't quite there in the models yet, but you can certainly see them coming in the next three to five years. Yeah. And so, you know, I don't necessarily think we need to sort of snap to regulation, but I think it's part of the discussion that we need to start having as a, you know, as a species. Because these are very, very powerful systems that we're bringing into the world. And most importantly, they're going to be very cheap.

16:18Mustafa Suleyman:And many of them are going to be available in open source. Yeah. So we have to contend with the fact that this isn't just about managing and governing the frontier labs. It's about managing the entire ecosystem and how everybody plays with these things. That's a really good call out because, you know, when we think about like how this is working in the U.S. where we have these various frontier labs, we have some smaller labs as well that are working on projects. And, you know, you might be able to regulate this in a specific region in that way. But I feel like the open source community might be where struggles come as far as this sort of stuff because and I'm generally very pro open source.

17:02But the truth is, I do believe there are some inherent risks with it as well in this way. Does that concern you at all?

17:11Mustafa Suleyman:You know, I think that there's risks up and down the stack. The risks of open source is that anybody can copy it, reproduce it, experiment with it with basically zero sort of oversight. And that's a good thing today. You know, people can be creative and productive on their own laptop. That's awesome. But in 10 years time, when if the models have continued to get as good by 2035 as they did between 2015 and 2025, you know, people struggle to intuit the impact of an exponential. You know, that isn't just linear. That is basically a doubling of compute every 18 months for the next 10 years. So you're talking, you know, six, seven doublings over that time period.

18:01Mustafa Suleyman:It's insane. um it and you know i think that's quite likely that that to me looks quite likely in terms of the scale of the models they're going to be trained um and so you know you can think about compute as creating more neurons to represent more information or more subtle connections between the information that is already in the network so that's what more compute is doing is providing more layers of connectivity which allows for more nuance more precision more creativity more accuracy of recall etc etc so it's pretty clear to me that the performance is going to continue to improve quite consistently like i i'm quite optimistic about that now it's not even about optimism.

18:57Mustafa Suleyman:I think it's a very sensible prediction. I agree. In which case, we're kind of in the regime where these models are going to arrive. They're going to be as good as humans at most tasks in the next three to 10 years, somewhere along that time horizon. And so we're going to have to contend with this question that first, the frontier labs are going to have these very powerful systems. And then most likely the open sources too. So it's just a It's just a breathtaking just reality that I think most people are not yet fully grasping, even though everyone can see the magic of Copilot and ChatGPT and Gemini and Claude today.

19:37Yeah. And at some point they'll be in robots where they give the bipedal vibe of a human, which is going to make it even more complex, I think. Oh, yeah. That's so wild. I was thinking about that with the three laws of robotics. and there was an essay I read, I don't even remember, like a couple months ago now, about how we should add a fourth one, which is an AI may never deceive a human. So, you know, there's like, AI cannot harm a human, AI cannot, you know, there's like two or three other ones, but the fourth one that wasn't added that we think we should add is it should not deceive humans.

20:13And so part of that would be, like, is there a way to almost hard code that like ability to deceive, like to make sure that they can't, One, misrepresent themselves as conscious when they're not. And number two, deceive humans. Is that even possible?

20:28Mustafa Suleyman:Yeah, I mean, you know, it sounds simplistic, but like, I really like the framework, helpful, harmless and honest, which Anthropic came up with three or four years ago. Like that. You know, it sounds really simplistic and obvious, but it's a good place to start. I mean, to me, that's what a humanist super intelligence is. What does it mean to have an AI that is truly aligned to human interests and isn't trying to exceed or escape human control? One that is always on our team, in our corner, fighting for our interests, doing our work. And so, you know, it's not necessarily a tool because a tool does exactly what you want it to do.

21:13Mustafa Suleyman:And there's a kind of one-to-one connection between your control and its kind of execution. but the tool metaphor is still a good one even though these things have some autonomy or they'll have their own agency to some extent but they still have to fundamentally be accountable to us and the companies that make them need to be liable for the consequences um you know for humanity because that's really what's at stake yeah yeah i agree talks about that a lot in terms of like Like who is liable when these things go out and make them, you know, a legal error or whatever? Like who who who's responsible for the AI when they go out in the world?

21:51You warn in your essay, you warn about granting rights, welfare or citizenship to AI. There's like that's where this goes with the model welfare of it all. So what is there? I know you mentioned we shouldn't just jump to regulation, but is there a specific legal or policy guardrail you would favor for avoiding that situation? or are you just not there yet?

22:13Mustafa Suleyman:Well, I think the first thing that we have to establish is that from a technical perspective, they might have the hallmarks of consciousness, but they don't have the experience of consciousness, right? So the inner workings, like what are we trying to protect? They don't have a pain network, right? And this is sort of what I've been arguing with the model welfare advocates, because I think it's just such a misrepresentation of how the inner workings of these things actually operate, I think is super dangerous. So the question of their entitlement to protection doesn't even come up because they don't have suffering, they don't have guilt or shame or fear or anxiety or indeed hunger, sex drive, replication drive, or any of those other preferences.

23:02Mustafa Suleyman:You know, those preferences come from our evolved biological networks. well you know we have grown up from amoeba um and you know our survival of the fittest uh function has been what has like driven which parts of us survive and which parts don't and that's taken hundreds of millions of years to get us to this point and that's why we've got this kind of physiological baggage we have an amygdala we you know we we have all of this infrastructure in our complicated brain because it was evolved to survive like that. And these systems just don't have that. So I don't even think we should even have the question of AI rights or citizenship on the table.

23:51Mustafa Suleyman:I think there should just be no question that it's firmly off the table for discussion. Something I wanted to ask about that is on another subject you mentioned yesterday, actually. You mentioned that evenly distributed but unevenly activated AI. And I really thought that was a good call out and wondered, what does activation look like in an everyday person's life? What kind of moments are you looking for there? Great question. So one of the problems that we have at the moment is that, so let's sort of start with websites and the open web you know billions of websites are available to all of us so in a sense the world's information is available at our fingertips and yet very very very few of us actually go to page two of the search engine results page right and even very rarely do we go to sort of third party websites because the cost and complexity of absorbing that information is just really high unless you're really motivated yeah so i think part of the challenge is that even though expertise and capability and knowledge might be at your fingertips accessing it and getting the best out of it is still like a really hard challenge i mean these models are superhuman at every task that you can possibly imagine and yet most people still don't really know how to ask it questions to get the best out of it right um you know and that's a really difficult design challenge as a creator like me i'm constantly figuring out like how do we show people what is what latent capabilities are actually under the hood and one of the ways is obviously by socializing it people share examples and learn from each other the cold start problem is a real one we haven't made enough progress on you know the blank page syndrome of of our letterbox and co-pilot um and you know so although everyone technically has access to it um activating them so they can get the best out of it is a completely different challenge it sure is we do we work with that a lot here at the neuron too in teaching people and and working with them one-on-one to kind of explain you've you've got to get them to their aha moment is kind of the way I often think about it.

26:16There's a moment with people where it just clicks and they're like, oh, this unlocks doors for me. There are new things I can do that I couldn't yesterday with this. And I think your call out was really poignant and relevant. What do you, I mean, you said you're working on this. I mean, what are your hypotheses sees or how are how are you approaching trying to solve that problem like maybe you're just

26:41Mustafa Suleyman:actively thinking about it yeah one example is is tiktok you know i love tiktok i think it's a truly incredible platform um but it is completely proactive so you're basically totally passive you're constantly being shown examples of what to watch and then the more you watch something the more you'll see stuff that is related or in some ways similar and so all the hard work has been done or is being done by the algorithm i think the problem that we've got in chatbots is that um it relies on you having a good idea of what it is that you want to ask what you're looking for so increasingly we're designing methods so that copilot is proactive in bringing up new topics, making new suggestions for things to learn about or talk about, places to go, and just doing a little bit more of the work for you on your behalf.

27:38That's great. That's cool. Makes perfect sense. Yeah. You're basically doing some of the prompting for them with the context engineering to bring it to the forefront of what it can do. I like that. And then as part of that, you're building your own foundation models like MAI1, which you just released in preview. and then you're integrating external models into Copilot as well, like those from OpenAI or other partners. How are you balancing where you want to build models to kind of improve for the capabilities that you see are missing and how are you balancing where it's like, hey, we're going to work with our partners here?

28:16What's your process there?

28:18Mustafa Suleyman:I mean, we're taking a pretty open mind. Because we're not beholden to any approach, We use open source models. We use models from other API providers. Obviously, we've got a huge partnership with OpenAI, and they still provide the bulk of our models. And, you know, they have the best models in the world. But at the same time, we also are committed to training our own frontier models of all scales that hopefully one day will be among the best in the world. MAI 1 preview that we just released when it appeared on the LM Arena leaderboard. It made the top 10 when it was first up there. So it's a very strong model.

28:57Mustafa Suleyman:We're very proud of it. Team did a great job. It was trained on a very small cluster, only 15 ,000 H100s. That's impressive. Wow. Yeah. Yeah, it was pretty good. So I think it's punching above its weight. And now we're building a much larger cluster to go to the next scale. and hopefully we'll break into the top group. So it's very important for Microsoft that we have our own optionality if we need to. We want to continue our partnership with OpenAI, but as a$3 trillion company, we can't be dependent on a third party for our core IP. We want to still keep partnering with them and buying from them, and we want to keep access for as long as possible.