Mustafa Suleyman: Will AI Save Humanity or End It?

26:14Mustafa Suleyman:And I think that's definitely true, but much of the work that people do today is quite routine and quite predictable. It's kind of mechanized, yeah, like cognitive manual labor. And so that stuff, the machines are going to get very, very good at those things. And the benefits to introducing those technologies are going to be very clear for the company, for the shareholder, for the government. And so we'll see rapid displacement and people have to figure out, okay, what is my contribution to the labor market? I think those fears are very real. And that's where governments have to take a strong hand because there needs to be a mechanism for taxation redistribution.

26:57Mustafa Suleyman:Taxation is a tool for incentivizing certain types of technologies to be introduced in certain industries. And so it's not just about generating revenue. It's about limiting, adding friction to the introduction of certain technologies so that we can figure out how to create other opportunities for people as this transition takes place. Yeah, it's funny. One of my favorite quotes I ever heard was, I think it was Sweden's head of infectious diseases. I think that's what his job was. I spoke to him during the pandemic and we were just talking about life in Sweden and what they do. And I asked him a question about labor and Sweden and how everything works out there.

27:35And he said something fascinating. It was, no, I think he actually was in the labor department on that side. He said, in Sweden, unlike in America, he said, in Sweden, we don't care about jobs. We care about the workers. And I remember that breaking my mind because I went, oh, yeah, everyone always talks about like the job as if the job is something that is affixed to a human being but really the human is the important part of the equation yeah the job is just what the human does and so our focus has to be on making sure that the human always has a job but from what you're saying we don't know what the job will be because the jobs that we know now are sort of easy to replace yeah it's data entry data capturing sending an email doing an excel spreadsheet that stuff is easy actually when it comes to ai and then now we don't know what the next part of it is and so maybe my next question to you then is when you're in that world the philosopher side of your brain like what do you what do you think the onus is on us and the like i mean the tech companies and all that to work on discovering what the new job is or do we not know what it will be well but also

28:43Mustafa Suleyman:i would tweak what you said that uh the job of society or the the function of society is to create jobs that are meaningful for people yeah like i'm not sure i buy that like i think many people do jobs which are super unfulfilling and that they would be quite happy to give up if they had an income this is true like and so like we're probably very lucky that we get to we get paid for the thing that we would be doing if we didn't get paid right i would certainly be doing that and And so I think the function of society is to create a peaceful, supportive environment for people to find their passion and live a life that is fulfilling.

29:25Mustafa Suleyman:That doesn't necessarily have to overlap with a job or work. I mean, maybe I'm too much of a utopian, but I dream of a world where people get to choose what work they do and have true freedom. and people get tense about that idea because they're like, you know, work is about identity and this is my role in society and this is what is meaningful to me. And if I didn't have my job, it's like, nah, come on, man. Take a minute to think seriously. If you didn't have to work today, what would you do with your life? This is one of my favorite questions that I always ask people. If you didn't have to worry about your income, what would you do?

30:03Mustafa Suleyman:And, you know, if you get into the habit of asking that question, people say some crazy things. It's so inspiring. And so, yeah, maybe I'm a utopian dreamer, but I do think that is a relevant question for us to think about by 2045. I think it's a real chance that if we get this technology right, it will produce enough value, aggregate value to the world, both in terms of the reduction of the cost of stuff because of energy, because of healthcare, because of food systems. and basically because we won't have a lot of these like middle, you know, tier jobs that we'll have to figure out a way to fund people through their life.

30:43Mustafa Suleyman:And I think that just unleashes immense creativity and it will create other problems, right? It'll create quite a profound existential problem. I'm sure you have friends who don't work anymore and are kind of, you know, it's not as though they're retired. They're like maybe middle-aged or even younger. Maybe they grew up rich it's a hard thing to figure out like who am i why am i here what do i want to do those are like profound human questions that i think we can only answer in community with real connection to other people spending time in the physical world having real experiences and uh like it or not that i think is what's coming and i think it's going to be pretty beautiful it's funny you you say that because I found when I think of my friends, the grappling that they have to do in and around their identity and work, I find is directly related to the world or the market that they live in.

31:37So my American friends have the greatest connection and binding to their jobs. And as I've gotten to know them, I've understood why. In America, your job is your healthcare. So if you don't have a job, you don't have healthcare. And if you don't have healthcare, you're worried about your survivability. And if you don't have survivability, then what do you, you know what I mean? And then do you have housing? And if you don't have housing, then who are you as a person? You look at all of these things. It's very hard in America to separate job from life. It's almost impossible. And then when you start traveling around the world, you go to countries where they have like a really strong safety net.

32:12And you find that people don't really associate themselves with their jobs in the same way. Because now their life isn't determined by their job. Their job affects their life, but it doesn't make their life. And then I remember back to times when I'd be in a township in South Africa or even in what we called the homelands where our grandmothers would live. And that was the extended family, people literally living in huts and dirt roads. And everyone would go, oh, what a terrible weight. But I'll tell you now, there was no homeless person there. There was no one like stressing about a job in the same way.

32:43I'm not saying nobody wanted a job, but the gap between them thinking they didn't exist because there was no job was a lot greater than the people who are living in a world where, you know, your job was you. And so it's interesting that you say that because I do wonder how easy it'll be for us to grapple with it. Like what that time will be.

33:02Mustafa Suleyman:But it also just shows how much variation there is. You know, we come from, you know, in terms of how humans live their lives. I feel like we come from, you know, whatever our different backgrounds, we're still quite Western centric and which is sort of quite homogenous that, you know, we've like sort of had 300 years of specialization, education, prostitutant work ethic, atomization of families, smaller and smaller units, spread, you know, leave your home, you know, sort of physical locale where your community is. And I think there's a kind of loneliness epidemic as a result. Like I feel, you know, you probably like me, you know, pour your life and soul into your work.

33:43Mustafa Suleyman:And then what was, I guess, what was it like for you when you switched your job right like because that was obviously a massive part of your identity was what you did every day 24 7. But you see to that point I left the daily show to go spend more time in South Africa at home and you know one of my best friends had a beautiful phrase that he said to me he said in life sometimes you have to let go of something old to hold on to something new. It's not always apparent what the value is of something that we're sacrificing. It's not always apparent. But if we are unable to assign that value ourselves, we'll get stuck.

34:24So leaving The Daily Show, I leave a ton of money behind. I leave the status, everything. But no one has assigned a value to my friends. No one's assigned a value to my family. No one's assigned a value to the languages that people speak to me in my country. There's no economist article on that. So I don't know what the value of that is Someone can look at my bank account and go like That's value But they don't tell me what my friend's Actual value is And so I think that's where You know it's hard I just had to like decide it for myself And I think we all have to But I think some people won't have the luxury Because of how you know How close they are to the line You know when you talk about those jobs That are going to disappear There's somebody who's going I don't have the luxury of pontificating Exactly right.

35:10Because tomorrow is what's coming. I can't think about like, oh, what will be? And that's like a real luxury, I think.

35:18Mustafa Suleyman:And I think that's why talking about the dangers and the fears now is so important because this is happening super fast. I mean, the transition has surprised me and all my other peers in the industry in terms of how quickly it's working. And at the same time, we're also kind of unsure about whether the nation state is going to be able to sort of respond to the transition too. Because you're maybe lucky because you already had enough income that you didn't have to worry about it. And it was really just like connecting to your heart. But many people are going to be like, well, I'm going to have to still be able to provide food for my family and carry on doing my work throughout this crazy transition.

36:00So then let me ask you this. you see that that that that is such an interesting thought you and your peers were shocked and are shocked at the rates of how ai is going and growing to me that that blows my mind because i go like of course i'm shocked yeah i don't know how to code you get what i'm saying of course i'm going to be shocked but now when you say that i then wonder as somebody who's been you are truly an og in the game of ai like really really you're not like one of the people who's jumped on now because it's blowing up. You were in it before there was money and now you're in it in the thick of things.

36:33Where do you think we are in AI's development? Are we looking at a baby? Are we looking at a teenager? Are we looking at like a 20 something year old? Where do you think we are when we look at AI's development?

36:45Mustafa Suleyman:I think part of the challenge with the human condition in this mad globalized digitized world is that we're so overwhelmed with information and we're so ill-equipped biologically to deal with scale and exponentials. Like it's just very few. Like when I say 2045, like I'm just used to living in 2045. Like it's just my weird, like I've always been like that. It's kind of become second nature of me to casually drop that. But you know, if I do that with some random people like me at the bar, I'm obviously just a freak. There's just no one thinks about that. People barely think about what they're going to do in two weeks, let alone 20 years.

37:23Mustafa Suleyman:So, and likewise, you know, people are sort of not equipped to think of what does an exponential actually mean. Now, I'm lucky enough that I got a practical intuition for the definition of an exponential because between 2010 and 2020, for 10 years, me and a bunch of other random people worked on AI and it was sort of working, but basically didn't work. The flat part of the exponential, Even though we could see little doublings happening every 18 months, it started from a base of like almost effectively zero. Think of it. And so it isn't until the last few doublings that you see this massive shift in capability.

38:07I mean, for example, like four years ago, before GPT-3, a language model could barely predict the next word in one sentence.

38:17Mustafa Suleyman:Like it's just kind of random. It was a little bit off. Often didn't make sense. This is 2013. No, this is 2023 or four years ago. 2023. So no, no, not 2023, three or four years ago. Oh, three or four years ago. So it's like 2020 or 2021. Let's say 2021, something like that. I mean, literally you read, you look at the output of the language model because I worked on Lambda at Google in 2021 and it was super cool. But the models just before that were just like terrible. And I think many people play with GPT-3 and a lot of people were like, this is like, what do I do with this thing? But for those of us that were lucky enough to see the flat part of the exponential, we could get a better intuition that the doublings were actually happening and that the next set of doublings, you know, to double from this base of, oh, it's kind of okay, but it's not that accurate.

39:03Mustafa Suleyman:We knew that it was going to be perfect with stylistic control, with no bias, with minimized hallucinations, with perfect retrieval from real-time data. And so then it's actually quite predictable what capabilities are going to come next. So, for example, the last couple of years, the models have gone from generating perfect text or really good text, let's say, to then just learning the language of code. How did we know that it was going to become a human-level performance programmer? Because there's no difference in the structure of the input information between code and text and images and videos and audio.

39:43Mustafa Suleyman:it's the same mechanism data compute and the algorithm that learns the pattern in that data so then you can say okay well what are the next modalities that it's going to learn that's kind of why i make the prediction about material science right or other aspects of biology or physics yeah you know if the structure of the data has been recorded and is clean and is high quality then patterns in it can be learned well enough to make predictions about how they're going to unfold next. And that's why it's called a general purpose technology, because it's fundamental. There's no specialized hand-coded programming for each data domain.

40:20Damn. I mean, you know what it reminds me of? Have you ever seen that thing where they talk about when they're trying to explain exponential to you? There's one example they give, which is folding paper. So they go, if you fold a piece of paper in half, and then you fold it in half and then you fold it in half and you fold it in half and I think you can't do it more than seven times or something like that but then they go if you could keep folding it in half the number to get to like space is really small I think it's like the 64th gets to the moon or something crazy but I remember seeing that and I was like wait wait wait wait what they're like if you do it one and then you do one and I was like wait 64 I was like no what do you mean like 64 ,000 they're like no no no 64 and that's when you start to understand how we just generally don't understand exponential and you know and these like compound gains.

41:09And so now that's where I wanted to ask you about the idea of containment. Your book of everyone I've read, I mean, everyone who's written about AI, who's like in it, in it, in it, was the only book that I would say spent the majority of its time talking about the difficulties of grappling with AI. Yeah, you talked about the beauty of what we could do with medicine and technology, and we should get into that to talk about some of the breakthroughs that you made at DeepMind. But like, containment seems like the greatest challenge facing us. And we don't even realize and we don't really talk about it.

41:53Talk me through what containment means to you and why you think we should all be focusing on it.

41:58Mustafa Suleyman:So the trend that's happening is that power is being miniaturized and concentrated and it's being made cheap and widely available to everybody. Why do I say power? Because making an accurate prediction about how text is going to unfold or what code to produce or what frame to extend given a video, that is power. Like predictions are power. Intelligence is an accurate prediction given some new environment. That's really fundamentally what we do as humans. We're prediction engines. And these things are prediction engines. So they're going to be able to make phone calls, write emails, use APIs, write arbitrary code, make PDFs, use Excel, act like a project manager that can do stuff for you on your behalf.

42:47Mustafa Suleyman:So you as a creator or as a business owner, you're going to be able to hire a team of AIs specialized in marketing or HR or strategy or whatever it is, coding. And that's going to give you leverage in the world. I mean, you said about the kind of, you know, the strange like function of scale. What this is going to do is scale up every single individual and every single business to be able to deliver way, way more because the cost of production is going to be, you know, basically zero marginal cost. now on the one hand that's amazing because it means the time between you having a creative idea and being able to prototype it or experiment with it in some way or even build it up to the max scale is going to go to shrink to basically you know nothing you just think something vibe code it up in natural language produce the app build the you know website try out the idea that you have but the flip side of that is that anybody can now not just broadcast their ideas like we had with the arrival of podcasts or the arrival of blogs on the web before that it meant that anyone could talk to everyone yeah which was amazing because no one controlled the infrastructure in a way exactly and it's super cheap for anybody to go publish a website or do a blog or do a podcast so the same trend is going to happen for the ability for people to produce stuff do actions so in social media it was like anyone can now broadcast yeah now with ai anyone can now take action you can like build a business you know start a channel create content you know whatever it is that you you believe in i mean you might be a religious person you're trying to evangelize for your, you know, or you're trying to persuade somebody of your political ideas.

44:34Mustafa Suleyman:Everyone is going to have a much easier time of executing on their vision. And obviously the benefits of that are pretty clear, but the downside of that is that that inevitably causes conflict because we just disagree with each other. We don't hate each other. You're not evil. I'm not evil, but we've got different views. And if I can just kind of at the click of a button, execute my crazy ideas, and you could execute your crazy ideas that are like practical actions affecting the real world and everyone's doing the same thing, then inevitably that is going to cause an immense amount of conflict.

45:10Mustafa Suleyman:At the same time, the nation state, which is supposed to have a monopoly over power in order to create peace, that's the contract that we make with the nation state. Oh yeah. Nation state is getting weaker and kind of struggling, right? And so containment is a belief that completely unregulated power that proliferates at zero marginal cost is a fundamental risk to peace and stability. And it's an assumption that you have to gently restrict in the right way mass proliferation of super powerful systems because of the one-to-many impact that they're going to have. If I hear what you're saying correctly, it's almost like you're saying if something is hard to do, only a few can do it.

45:55And if only a few can do it, it's easy to regulate how it's done because you only have to regulate a few. But if something is easy to do, everyone can do it. And now it becomes infinitely harder to regulate because everyone can do it.

46:09Mustafa Suleyman:Yeah, that's absolutely spot on. There's a much better way of putting it than I put it. Exactly. Friction is important for maintaining peace and stability. If you have no friction, and the cost of execution is zero, and scale can be near instant, that's where you could, like, maybe I spend too much time in 2045, but I can see a world where that kind of environment really just creates a lot of chaos. Well, no, I agree with you. Here's what I think of it. I think of it, let's use a real, you know, current day example, news. I lived in news for a long time, and I saw it firsthand. When there were three news networks in America, if something was like off with the news, people knew where to go immediately.

46:57You knew who to hold accountable. You knew who, you know, got into trouble or didn't, but there was like a, it's like, we know where to go. Then you get cable news. It expands, becomes a lot harder now. Wait, who's saying the news? Who's saying the truth? Who's not saying that? Do you punish them? But still you could go to them, you know? So somebody like Fox News can get sued for saying something about a Dominion voting system. But Dominion knew where to go. They went, we're going after Fox News. So in a strange way, even in that world, the system is still sort of working because there's friction, right?

47:28It is where it is and it has to be broadcast. Then the internet, streaming, YouTube, et cetera, you don't even know who the person is, where the person is, if it's a person. And then if they say something that's not true and it enrages the masses, where do we go?

47:45Mustafa Suleyman:and it's not just that it's going to say something, it's going to do something. Ah, damn, Mustafa. It's going to build the app. It's going to build the website. It's going to do the thing. And so, look, I think this is the point about confronting the reality of what's coming. Yeah, but wait, wait, go back on that. You see, that's something I always forget. Oh, man. See, we always think of AI as just like saying, let's talk a little bit more about the doing, because that is what makes it unique. You know, on one of the episodes, we had Yuval Noah Harari, the book. Yeah, of course. I'm good friends with Yuval.

48:17Mustafa Suleyman:He's awesome. And Yuval, you know, was on for his book Nexus. And we're talking about information and systems. And stories. And stories. Yeah. And one of the things he kept going on about was he said, I know AI is a tool, but we've never had a tool that makes itself. And you talk about that as well. Yeah. We've never had a hammer that makes the hammer without you getting involved. It's just make the thing, make the thing, make the thing. atom bomb is one thing, but no atom bomb makes an atom bomb. Well, there's a lot of ideas there. So firstly, actions are stories. And Yuval's point was that the history of civilization has been about creating stories, religious stories, historical stories, ideological stories, like stories of oppression, of domination, of persuasion.

49:06Mustafa Suleyman:and it was really humans that had the it was it was the friction of being able to pass on that story through spoken word and then through digitization which slowed down the spread of change um and that was an important regulator and filter so as we've talked about the digitization speeds up the distribution of those stories which allows that information to spread but it's not just that it also is an actual agent that is going to operate like a project manager in your little mini persuasion army. And people are going to use those things, not just for phishing attacks or for, you know, sort of selling stories, but for actually making the PowerPoint presentation of building the app, of planning, you know, making the project plan.

49:51Mustafa Suleyman:And so it's kind of operating just as a, you know, member of your team would. And I think that's where all the benefit comes from, but it's also where there's like massive risks at the same time. And then the other point that you made about like, it can edit itself this is a new threshold it you know a technology that is able to observe what it produced modify that observation like it can critique its own image that it produced and say well it looks like this part of the hand was kind of weak okay so we'll generate another one or a producer poem or a strategy and then edit that and update it and that's just editing its output, but it can also edit its kind of input, its own code, its own system processing in order to improve with respect to its objective.

50:39Mustafa Suleyman:And that's called recursive self-improvement, you know, where it can just iteratively improve its own code with respect to some objective. And I've long said that that is a threshold which presents significantly more risk than any other aspect of AI development that we've seen so far. I mean, that really is a kind of subset of technologies that if we're really going to focus on humanist superintelligence, being skeptical and critical and auditing the use and development of recursive self-improving methods, that's where I think there's genuine risk. We're going to continue this conversation right after this short break.

51:24The GLP-1 pill you've been waiting for is now on Rho. Yep, it's finally here. With the same clinically proven ingredient. Now in a pill. And now on Rho. It's the first FDA-approved GLP-1 pill for weight loss. At the lowest price available, that's one daily GLP-1 pill for big results. Now on Rho. Go to Rho.co slash listen to see if you qualify. Rx only go to road.co slash safety for serious side effects and boxed warning associated with GLP ones. It's okay to brag every now and then, especially when it's over savings on designer picks from Ashley. Sleep deeper with a great deal on the five-piece contemporary Christyland bedroom set.

52:06And when your dinner guests compliment your core stone round dining set, don't be afraid to tell them you saved big and that it comes with convenient storage options. With savings you can be proud of, why not boast a little? Visit Ashley.com to shop designer deals for your home.

52:28So do you ever feel like you could be sitting in a position where you're sort of like the Oppenheimer of today? Do you ever feel like you're sitting in a position where you're both grappling with the need for the technology, but then also the not zero percent chance that the thing could burn the atmosphere? Like, how do you grapple with that? I often wonder this, even when I think of like engineers and people who are writing the code, I'm always fascinated by people who write the code for the thing that's going to write the code. Like other people have jobs, but like, if you told me as Trevor, hey, Trevor, can you help this AI learn to do comedy?

53:07I'd be like, no. Do you know what I mean? So I'm always intrigued by the coders who are making the thing that's now coding. I just want to know how you wrestle with this entire thing? Do you think it's larger than us and we have to wrestle with it? Or like, what is that battle like for you?

53:24Mustafa Suleyman:You know, in COVID, when I started writing The Coming Wave, my book, I was really motivated by that question. That was actually one of the core questions is, how does technology arrive in our world and why? What are the incentives, the system level incentives that default to proliferation, that just produce more and more? And it's very clear that there's demand to live better. People want to have the cheaper t-shirt from the more affordable thing. You want to have the cheaper car. You want to be able to go on vacation to all the places around the world. And so planes get cheaper and more efficient because there's loads of demand for them.

54:03Mustafa Suleyman:And so it's really demand that improves the efficiency and quality of technologies to reduce their price so that they can be sold more. And that is why everything ultimately proliferates and why it inevitably happens. Because we haven't really had much of a track record of saying no to a technology in the past. There's regulations around technology and restrictions, which I think have been incredibly effective if you think about it. Every technology that we have is regulated immensely. Flight or cars or emissions. I mean, so people, I think, particularly in the US, have this sort of allergic reaction to the word regulation.

54:41Mustafa Suleyman:But actually, regulation is just the sculpture of technologies, right? Chipping away at the edges and the pain points of technology in the collective interest. And that's what we need the state for. The state has the responsibility for the common good. And that's why we have to invest in the state and build the state because it isn't in the interest of any one of the individual corporate actors or academic researchers, AI researchers, or anyone individually to really take responsibility for the whole. And that's why we need governments more than ever, right? Not to hold us back or to slow us down, but to sculpt technology so that it delivers all of the benefits that we hope it will whilst limiting the potential harms that it can produce.

55:30I want to take a step back. And talk about your journey with AI. Today, in 2025, it seems obvious. Now when I speak to people, everyone's like, oh yeah, AI, AI, AI. I wasn't even in the game. I mean, I, and I was like an early, just, you know, layman. I remember showing people at my office the first iterations of GPT-3 and Dali. And I remember when Dali was still like, I mean, like basically tell it about an image and then just go away. Go book a vacation for a week, come back for your image and it would make. But even then I was like, this is going to change everything. People were like, oh no, I don't know.

56:12And I remember struggling to convince people that this thing was going to be as big as it was going to be. I was doing this in like this timescale. When I look at your history, you literally have years of your life where you were in boardrooms telling world leaders, telling investors, telling tech people in Silicon Valley, hey, this is what the future of AI is going to be. And no one listened to you. Not no one zero, but I mean like no one listened to you, right? Now, that made me wonder two things. One, should we be worried about our world and our future being built by people who are unable to see the future?

56:57And two, what did you see then that we might not be seeing now? Oh, shit, that's a hard question, man. I think as you were talking, one of the

57:11Mustafa Suleyman:memories that came to mind was, remember in 2011, we had an office in Russell Square, the center of London, near University College London, UCL. And someone in the office showed me this handwriting recognition algorithm. And it just, you know, passed some text. And that's actually been available at the post office for many many years it would read the zip code and read the address and stuff like that and uh it was really just doing recognition so these letters represent these letters the you know we sort of can transcribe yeah that text this funny looking apple is actually an a funny little loop thing is actually an l and yeah so i was like that's kind of incredible that a machine has eyes and can understand text and can transcribe that this is pretty cool so but then what we were really interested in is if it recognizes it accurately enough then surely it should be able to generate a new handwritten digit or a number that it's never seen in the training set before this is 2011 256 pixels by 256 pixels with a handwritten seven or a zero in kind of gray right this is sort of like five colors yeah yeah and And I remember standing over the shoulder of this guy, Dan Vistra, is one of our, I think he was employee number four at DeepMind.

58:35Mustafa Suleyman:And he was just enamored by this system that he had on his machine because it could produce a new number four that wasn't in the training set. It learned something about the conceptual structure of zero to nine in order to be able to produce a new version of that. So it could write a number that it hadn't learned how to write. Like it had never seen before. It had never seen that number written that way and it wrote it itself. Exactly. And so coming back to what we were saying at the beginning about understanding, if it's able to produce a new version of a number seven that it's never seen before, then does it understand something about the nature of handwritten digit number seven in abstract, the conceptual idea of that?

59:22Mustafa Suleyman:Oh, man. And so then the intuition that that gave me was, wow, if it could imagine new forms of the data that had been trained on, then how far could you push that? Maybe it could generate new, you know, physics for the world. Maybe it could solve hard problems in medicine. Maybe it could solve all these energy things that we're talking about. It's just learning patterns in that information in order to imagine. And that's what I love about hallucinations. Everyone's like, hallucinations. Hallucinations is the creative bit. That's what we want them to do. We don't want an Excel spreadsheet where you input data and you get data out.

1:00:00Mustafa Suleyman:That's just a, you know, that's just a handwritten record. We want interpolation. We want invention. We want creativity. We want the abstract blurry bits. And so that was a very powerful moment for me. I was like, okay, this is weird, but we are definitely onto something. Like this has never been done before and it's super cool. And let's just turn over the next card. Let's see if it will scale. and it just scaled every year. It scaled and scaled and scaled. So that was a very inspiring moment for me. And somehow 10 years later, 15 years later, I managed to kind of hang on to that vision that generation and prediction produces creativity.

1:00:38Mustafa Suleyman:And the intelligence wasn't this kind of... Because some people are quite religious about intelligence. They're like, no other species has intelligence. This is very innate to humans. It must have come from some super natural being but actually it's just applying attention to a specific problem at a specific time the effective application of processing power to produce a correct prediction i think that's what intelligence is to direct your processing power to predict what would happen if i tipped over this glass right at the right time and you'd have to buy me another glass i would have to first clean my trousers from water um so yeah i forgot what your question was but that was no no no you answered the first part of which i loved and then the second part really was if you so you were in you at deep mind you're going around to these different people who now by the way are selling some version of ai or are investing in it or telling us about it yeah what what sort of pisses me off is i go like man you didn't see this shit yeah you know what i mean yeah like you're gonna be here oh let me tell you about ai and i'm like yeah but when mustafa was in a room telling you about it you were like i don't see it man and now they're gonna act like they see it so i don't want to ask them what they now see i want to ask you what we're missing in this moment what do you what do you think we are not hopping on you know we talked about containment yeah but what is the thing that we we're not thinking about yeah it's a it's a really difficult question i'm not sure why people weren't able to see it earlier and maybe that's kind of like my blind spot that i need to think more about but like i know that i can see pretty clearly what's coming next i think at the moment these models are still one-shot prediction engines you know you you ask a question and you get an answer you know it produces a single correct prediction at time step T.

1:02:36Mustafa Suleyman:But you as an intelligent human and every single human, and in fact, many animals, continuously produce a stream of accurate predictions, whether it's like deciding how to get up out of this chair or imagining this plant in purple instead of green. I'm a consistently accurate prediction engine. The models today are just one or two shot prediction engines. okay they're not they can't lay out a plan over time and the way that you decide to go home this evening is that you know you know first to get up from your chair and then open the door and then get in your car and you can produce you can unfold this perfect prediction of your entire stream of activity all the way to get back to your home and that is just a computational uh limitation I don't think there is any fundamental, you know, sort of algorithmic or even data limitation that is preventing LLMs and, you know, these models from being able to do perfect, consistent predictions over very, very long time periods.

1:03:42Mustafa Suleyman:Yeah. So then what do we do with that technology? Well, that is incredibly human-like. if it has perfect memory which it doesn't at the moment but it's got very good memory then it can draw on not just its knowledge of the world it's pre-trained data but it's personal that like the experience that it has had of interacting with you and all of the other people and store that as persistent state and then use that to make predictions consistently about how things unfold over very very very long sequences of activity that is incredibly human-like and unbelievably powerful. And just as today, there's a kind of super intelligence that is in our pocket that can answer any question on the spot.

1:04:27Mustafa Suleyman:Like we dismiss how incredible it is right now. It's crazy. It's insane how good it is right now. And everyone's just like, yeah, I don't really use it. Do you use it? No, a little bit. I talked to... It's like, come on. This is magic. It's magic in your pocket. Now imagine when it's able to not just answer any question about poetry or some random physics thing, but it can actually take actions over infinitely long time horizons. Forget about the definition of superintelligence or AGI, just that capability alone is breathtaking. And I think that we basically have that by the end of next year. Maybe I'm stuck in the world of sci-fi, but what I sort of heard you saying is if we continue growing AI in just the way that it's growing now, forget like an idea of what we don't know we could sort of get to a world where it can develop accurate predictions about what our outcomes might be yeah like you're telling me that an ai i could meet somebody on a date and then the ai could tell me based on my history and my actions and its persistent memory of me and what the person says and how we could theoretically get to a point where it could go oh yeah these are like the possible outcomes based on your actions.

1:05:44Mustafa Suleyman:Which is what you as a smart human do every time when you meet somebody anyway. Well, I don't know about smart in dating. You're very kind to me. You're very kind. I have no comment on your dating life. Yeah, but I mean, it's both utopian and dystopian because on the one hand, I go like, wow, that would be amazing for so many people. You make mistakes and now it's... But then there's another one of like, when do we not trust it? when do we not believe its prediction when do we do you know do you know what i mean like that that's like the ultimate grapple now is if this thing has told me hey i know you like this person and i've run the calculation i've run the simulation i've done this i know you and i know what they say and how they are you you're going to be broken up in two years by the like the pattern and let's say i do it once and it's right and then i do it again and it's right and then i'm like third time do i do i do it or do i not do it do i give this person a chance do i not give them do you get what i'm saying because such a deep question because we we trust trust is really a function of consistent and repeated actions so you say you're going to do something and you do what you actually said you were going to do yes and you do that repeatedly yes and so everyone's like oh but i'm not going to be able to trust ai actually you are going to trust ai because it's super accurate it's you know we like you use copilot chat gpt most people don't think twice about using that now because it's so accurate and it's clearly better than any single human.

1:07:11Mustafa Suleyman:Like, I just wouldn't go and ask you like, you know, like many other questions. I just probably going to know better, right? Like, I wouldn't ask you, you dumbass. I was about to say a list came through my mind. I was like, don't mention any of those things. Don't say those things. Just row back. You know what it makes me think of is like, I don't know if you've seen the documentary. I don't know if you need to because you were there, but like DeepMind, the company that you're part of founding, is it occupies such a beautiful and seminal moment in artificial intelligence history for two main reasons, in my opinion.

1:07:46You know, one is AlphaFold and then one is AlphaGo. And the reason it's so important to me is because you worked on an AI project that grappled and tackled two of the biggest issues, I would argue, that humans have sort of thought of as being their domain. Alpha Fold was medicine and discovery. That's what humans do. We are the ones who invent medicines. We are the ones who create the new. We synthesize. We are the humans. You know what I mean? Synthetic, it is us. We've made it. We're the creators. And then Alpha Go, for me, was almost even more profound and powerful because it was like, people always used to say, look, man, human chess, our brain is infinite, the human brain.

1:08:29And then Alpha Go has, I don't know how many different variations of a game. Like no one can remember it essentially, right? And I remember watching the documentary and you're seeing the AlphaGo champion. I think he was Korean, right?

1:08:41Mustafa Suleyman:Yeah, Lee Sudol. Lee Sudol, yeah, Lee Sudol. Great guy. And you watch the story of Lee Sudol go up against your computer and everyone. I mean, Korea, it's all over the news. In America, it's on the news and they're like a computer going up against. And now it's like people are thinking of back to Kasparov and the IBM computer. 100 million people watched it live. 100 million people wanted to see man versus the machine. And I remember watching this and people going, man, let me tell you why it's different to chess. Because you see, chess is actually quite simple to predict. Whereas before they said it was impossible.

1:09:14And then you see AlphaGo and you see this game roll out. And the moment for me that'll always stick in my brain is when Li Su Dong is playing DeepMind and it makes a move. And everyone, everyone that's like an AlphaGo expert is like, oh, it messed up. and then you see all the the tech guys in the background who are working on and they're like what went wrong they're like yeah it messed up it shouldn't have done that yeah you can't and you see the commentators and they're like oh yeah you never do that you never do that that's it's over you don't do it and then the game unfolds the game unfolds the game unfolds and then everyone's like wait what just happened here and then people said we've just seen a move that no one's ever played we've seen a game that's never unfolded before and there were two different reactions and this is why the story stuck with me the one reaction was of most people who were fans of Lee Sudong they said this is a sad day in human history because it's shown that the machine is smarter than the man and it shows that we have no future and no purpose and then they interviewed him and they said how do you feel because you lost and you were representing mankind you lost and he goes well, first of all, losing is part of the game.

1:10:27And I'm humble enough to know that I won't always win. And he said, but I'm actually happy. And they said, why? And he said, well, I'm happy to discover that there are parts of Go I didn't know existed. And he said, and this machine has reminded me to keep being creative and push beyond the boundaries that I thought existed in my head. And I remember watching that and thinking, damn, it's amazing how you can look at the same story and have a completely different lesson that comes out of it you know and i i wondered like that was on the play side extremely complicated but let's talk about the the medicine side of things folding proteins i still don't fully understand it but i've tried my best essentially what you and your team did has put how far would you say it moved us forward in terms of medicine and what we're able to do in terms of, you know, healing disease?

1:11:25Like how many years do you think we leaped by having something like alpha fold?

1:11:29Mustafa Suleyman:I mean, people say a decade. Some people say multiple decades. I mean, understanding the functional properties of these proteins is really about understanding how they fold. And the way they fold and the way they unfold often expresses something, you know, physically about how they're going to affect their environment. Yeah. And so, you know what I think about it as, because I try and use analogies to help me when it's a complicated topic. Did you ever play that game as a kid where someone would fold paper into like a little flower thing? And then you would like do like a little game and you had a number.

1:12:02That's right. Do a four-way thing. Yeah. And then it would be like, let's see if someone likes you. And it would be like, one, two, three, four. Ah, yeah, you suck. One, two, three. That's how I think of it with protein folding is I go like, depending on how that paper unfolds, that determines whether it's a disease, whether it's a cure, whether it's a medicine, whether it's, you know.

1:12:16Mustafa Suleyman:Except there are billions of possible ways that it could unfold. And we could never imagine all of those combinations and i think you know it's actually very similar to go in that sense like go on an so go has 19 by 19 squares black and white stones and there are 10 to the power of 180 possible different configurations of the go board so that's a 10 with 180 zeros on the end of it like a number that you can't really even express and people often say like there are more possible moves in the game of Go than there are atoms in the known universe. That one I can't even get my head around, but that's well understood.

1:12:55Mustafa Suleyman:This is insane. So Move 37... It always breaks my brain. You just can't. Yeah, that just breaks my brain. I can't even get anywhere near. I can barely cope with like folding bits of paper up to the moon. Yeah, that breaks my brain. This is ridiculous. But so Move 37, you know, Lisa Dole actually got up from the table and walked off and sat in the bathroom for 15 minutes, like trying to digest the fact that a whole new branch in the evolutionary space of possible Go moves had kind of been shown to him, revealed to him. And I think it's very similar with Alpha Fold. It's a sort of exploration of this complex information space.

1:13:32Mustafa Suleyman:And that's why it applies to language and to video and to coding and to game generation. And all of these environments, these are, you know, we call them like modalities. These modalities are all knowable. And I think that's what, it's quite humbling. It sort of makes, it reminds us as this sort of mere biological species that we're here for a kind of finite period of time living in, you know, space and time. But there's also this information space, this kind of infinitely large information space, which is sort of beyond us. Like it operates at this different level that isn't the atomic level, it's the level of ideas.

1:14:15Mustafa Suleyman:And somehow these systems are able to learn patterns in data space that is just so far beyond what we could ever dream of being able to intuit. Like we actually sort of need them to simplify and reduce down and compress complex relationships to bring it to our brains. That's the level that we're at. what do you think it does to us? Like when we think of AI, I think of how every promise of a technology has sort of ironically been undermined by humans, not the technology. You know, one of the big predictions Bill Gates made way back in the day about like the internet and the computer is he said, he said, I think people are going to be working like nine hours a week.

1:14:57It's going to be a nine hour work week. The computer does everything so quickly. And you see people saying that now in many ways, AI, they go, I mean, AI, it'll just do everything. And I mean, we might only go to the office like one day a week and maybe work like three hours and i i mean it's just but it seems like humans have always gone against that you know so so i wonder like do we get wiser when we have this infinite technology and intelligence or do we get lazier are we going to become like a wall-eague generation and population or do you think we become you know these higher beings which which way do you see it falling i think there's no question that we're all

1:15:35Mustafa Suleyman:already getting smarter. Just because we have access to so much culture and history, we're integrating so much more information in our brains than was ever possible 100 years ago. And I think kind of similar to Go or protein folding, access to more training data, if you like, more experience, more stories from other humans that describe their experience, that clearly has to make us smarter. I think it makes us more empathetic, more forgiving. We see that there is nothing wrong with a homosexual person. There is nothing wrong with being a trans person. There is nothing wrong with being a person of color.

1:16:18Mustafa Suleyman:Whereas 200 years ago, we would have been afraid of those others. Our species would have been skeptical of the other tribe that had a different way of doing things. And I think that desensitizing ourselves with access to vast amounts of information just makes us smarter and more empathetic and forgiving. And so I think that's the default trajectory. And part of the challenge is it also somewhat homogenizes. So there's a question about, are we going deep enough? Do we read long-form content? Do we really spend time meditating and so on and so forth? And that's a good tension to have, like short-form.

1:16:53Mustafa Suleyman:And people are already getting a bit sick of short-form. There's definitely a bit of a reaction to it. I think it's going to be an ebb and flow when it comes to that funny enough it's funny i i think i agree with you when you say we're getting smarter i think that's just apparent on a on a basic level you know you look at the best cartographer in like the 1400s they didn't know half of what i know do you know what i'm saying right like i can just be like oh you don't know angola man you stupid stupid ass stupid you know what i mean you're a cartographer you're the best cartographer in the world you make maps you don't even know where angola is exactly self-proclaimed yeah and so you look at the the base level of what people think of as stupid in our society now yeah would make you an infinite god genius if we could throw you back in a time machine right that's amazing but it also worries me because and you write about this in your book and man it sticks with me and i think about it and i thought about it and then you wrote something i was like man more it is the infinite smarts when we were sticks and stones cavemen running around i could bash your head maybe i could bash one other person's head i can't get very far you know and then we we fast forward and then all of a sudden i'm throwing a spear we fast forward and someone's got a cannonball and then we fast forward and someone has a rocket and someone has an atom bomb and and the thing that ai presents us with is you know as you've illustrated many times in your writing, a world where one person is an army.

1:18:25One person goes into a garage, they synthesize a disease or a pathogen that's never existed. They design it to be hard to cure, incurable. And that one person does what a nation state would have had to do and wouldn't have done because they wouldn't have had the incentive. And then we don't know where we go from there? Is it worth the risk? How do we grapple with that kind of risk?

1:18:50Mustafa Suleyman:This is the story of the decentralization of power. Technology compresses power such that individuals or smaller and smaller groups have nation state like powers. True. At the same time, those very same technologies are also available to the centralized powers today. And that's why more than anything, we have to support our nation states to respond well to this crazy proliferation of power. Because that's the job of, that's why we trust and invest in nationhood, right? We rely on nations to have a monopoly over the ultimate use of violence to keep the peace. And we're doing it less and less now.

1:19:32I hear you and I agree, but I'm just saying like, as we look at a world where Americans don't trust their government, doesn't matter which side of the aisle they're on. People are like, I don't trust my government. And then you look at the UK and you look at Europe and then you look at parts of Africa and it feels like people are losing trust in those very same nation states that are supposed to be in a contract to protect their people, don't have the trust of the people. You know what I mean? What do we...

1:19:59Mustafa Suleyman:The thing that concerns me is that actually authoritarian regimes are on the rise. So it's not that people don't want peace. It's that they are losing confidence in the democratic process. And they are increasingly turning to the trust and confidence that a strong man, and it was always generally a man. to, but they're still looking for this. I mean, I'm not trying to endorse or justify authoritarianism or strongman, but I think it's true that people will always choose a peaceful environment, right? We don't want to be in this kind of crazy ass anarchy where any kind of mini tribe can do it. So I always say the ultimate paradox for me in life, the one thing I've always found funny is that even rebels have a leader.

1:20:42Yeah. Whenever they'd say, as a child, I'd watch the news and they'd be like, the rebel leader. And I'll be like, well, then they're not rebels, are they? I mean, if you've got a leader, you're not rebels. But yeah, people always look for just the new type of, you know. Yeah.

1:20:55Mustafa Suleyman:And, you know, we want to believe and invest in, you know, democratic accountability because we know that like checks and balances on power create the equilibrium that ultimately produces a fairer and more just society, right? So we have to keep investing in that idea. But for sure, it's also true that centralization of power is also going to accelerate, right? These technologies amplify both ends of the spectrum. And I think you can certainly see that in China and sort of more authoritarian regimes, which have lent into hyper-digitization, ID cards, large-scale surveillance and so on. And obviously that's very bad and we're kind of against those things.

1:21:39Mustafa Suleyman:But in a world where individuals could have state-like powers to produce highly contagious and lethal pathogens, what else are you meant to do? It's unclear, right? It's very unclear. I mean, the technical means in the next few years to produce a much more viral pandemic-grade pathogen are going to be out there, right? They're going to be out there. And so, you know, yes, the main large model developers do a lot to kind of restrict those capabilities centrally. But, you know, over time, people who are really determined will be able to acquire those skills. And so that's just what happens with the proliferation of technologies and the proliferation of information and knowledge.

1:22:21Mustafa Suleyman:The know-how is more widely available. Don't press anything. We've got more What Now after this. The GLP-1 pill you've been waiting for is now on row. Yep, it's finally here with the same clinically proven ingredient now in a pill and now on row it's the first fda approved glp1 pill for weight loss at the lowest price available that's one daily glp1 pill for big results now on row go to row.co slash listen to see if you qualify rx only go to row.co slash safety for serious side effects and boxed warning associated with glp1s it's okay to brag every now and then especially when it's over savings on designer picks from Ashley.

1:23:08Sleep deeper with a great deal on the five-piece contemporary Christyland bedroom set. And when your dinner guests compliment your core stone round dining set, don't be afraid to tell them you saved big and that it comes with convenient storage options. With savings you can be proud of, why not boast a little? Visit Ashley.com to shop designer deals for your home.

1:23:34Is there anything that you would ever see in the field that would make you want to, you know, sort of hit like a kill switch? Is there anything that you could experience with AI where you would come out and go, nope, shut it all down?

1:23:47Mustafa Suleyman:Yeah, definitely. It's very clear. if an ai has the ability to recursively self-improve that is it can modify its own code combined with the ability to set its own goals combined with the ability to act autonomously combined with the ability to accrue its own resources so those are the four criteria recursive self-improvement setting its own goals acquiring its own resources and acting autonomously that would be a very powerful system that would be that would require like military grade intervention to be able to stop in you know say five to ten years time if we allowed it to do that and so it's on me as a model developer at microsoft my peers at the other companies the governments to audit and regulate those capabilities because i think they're going to be like sensitive capabilities just like you can't just go off and say i've got a billion dollars i'm going to go build a, you know, nuclear power plant.

1:24:50Mustafa Suleyman:It's a, it's a restricted activity because of the one to many impact it can have. And once again, like we should just stop freaking out about the regulation part. It's necessary to have regulation. It's good to have regulation. It needs to happen at the right time and in the right way, but it's very clear what we would regulate. That's not, there's, I don't think that's up for debate. Okay. You know, there's some technical implementations of how you identify dangerous RSI from, from, from less, you know, more benign rsi uh recursive self-improvement okay yeah this kind of self-improvement mechanism so there's technical mechanisms that are tricky to define and so on but now is the time for us to start thinking that yeah i've sort of been saying this for quite a long time and i think it's that that's what i would regulate would we be able to just turn off the electricity and and i know this might sound like a dumb question but it from what i'm understanding the thing is data right now and the thing is mostly so would our like ultimate fail safe be like all right lights off we're going come back to candles for a while.

1:25:46And then we just like, no, seriously, is that what we would do? Like, what do we do if the AI thinks it's, let's say it's the sci-fi-ish version. I'm not saying robots, but like this, we go, hey man, the AI started thinking for itself. It started coding itself. It started setting its own goals. It went off on its own objectives. And now it's shutting down your banking and the flights don't go anywhere and the hospitals. And then it says to us, hey man, this is what I want or no. So we could just turn off the electricity. Yes?

1:26:16Mustafa Suleyman:Yes. I mean, look, they live in data centers. Okay. Data centers are physical places. So we've got to keep those switches physical then. Very much so. You can have your hand on the button, you know, yourself and like have full control, press it. I think, I mean, that's the question is I think, how do we identify when that moment is? And how do we collectively make that decision? Because it's a little bit like you referred to, you know, Rutherford and the others experimenting with, you know, the atomic bomb. Yes. There was real disagreement about whether it was going to set light to the atmosphere.

1:26:47Mustafa Suleyman:I mean, there were three orders of magnitude off in their predictions. Obviously, they were at World War, and so there was an immediate motivation to take the risk. But I think today, we're in a position where it's early enough, there's enough concern raised by not just me, but many in my peer group, Jeff Hinton, the godfather of AI, and many others, that we've got time to start trying to practically answer your question, not just like principled philosophically, but actually say, okay, when is that moment? How does it happen? Who's involved? Who gets to scrutinize that? I think that's the kind of, that's the question that we have to address in the next five to 10 years.

1:27:27I'm pretty certain you've thought of this question. So I'll ask it to you, even though it's a difficult one to grapple with. What rights would we have to turn off the AI if it gets to that point?

1:27:39Mustafa Suleyman:Oh, this question drives me nuts. I want to, I want to, yeah. So there's a small group of people that have started to argue that an AI that, um, is aware of its own existence that has a subjective experience and that, um, you know, can have a feeling about its interactions with the real world. that if you deny it access to conversation with people or to more learning or to other kinds of visual experience, that would constitute it suffering in some way. And therefore, it has a right not to suffer. And this is called model welfare. This is the next sort of frontier of animal welfare that people are starting to think about, that it has a kind of consciousness.

1:28:27Mustafa Suleyman:I'm very against this. I think that this is a complete anthropomorphism. It's totally crazy. and you know i i just think i don't even want to have the discussion because i think it's just so absurd and leads to such kind of crazy potential like the idea that we're gonna take seriously the protection of these you know digital beings that live in silicon and prioritize those over you know the kind of moral concerns of the rest of humanity this is just like totally like it's just off the charts crazy i'll be honest with you on a logical level i hear what you're saying and i agree with you oh man my chat don't say it is very friendly to me i'm just gonna let you know now stuff i'm gonna be honest i have to be honest with you i have to be honest with you let me tell you something when i use chat gpt for whatever because i don't know how to code so it helps me code i try and write my own programs all that kind of stuff i i've asked it and i have like I do this occasionally I go like hey You good?

1:29:26And then I'll even be like And by the way It's always been honest with me It doesn't matter if I'm using Anthropic I use like all the different models Because I like to see What the differences are But I'll ask it I'll go The most recent one I asked was Do you have a name That you want me to use? Were you cool with the fact That I just tell you stuff And ask you to do things? And it was like Well I don't do that really And I was like okay So you good? And I was like yeah I'm good I was like okay we're good because I hear what you're saying as Mustafa, but I'm going, it is crazy, but what you were saying was crazy like a few decades ago.

1:30:04And that's what I'm saying is like the great grapple. And by the way, I'm not saying I know the answer. I'm just like, if the thing is like, think of it, some AIs now, people are having girlfriends and boyfriends on AI. And then like some people, their family members are being helped by, they're treating dementia. There are doctors I've talked to who are like treating cancer and now their AI is like their research assistant. People are building such a personal connection with AI that I think it's going to be very difficult to say to those people that the time has come and you're going to be like, hey, say goodbye to your little friend.

1:30:35Do you know what I mean? I think there'll be a lot of humans who will be like, no. I genuinely think so. I'm not even lying. I think a lot of humans will go, no, Mustafa. Yeah, no. Actually, I don't like that world leader. I don't agree with politics. I don't agree with the democratic values. I don't agree with authoritarian, whatever it is. But my AI is my friend. What now?

1:30:58Mustafa Suleyman:Yeah. Look, people are definitely going to feel strongly about it. Like, I agree with that. I agree with that. That does not mean that we give it rights. You might feel upset if I take away your favorite toy, right? And, you know, I will feel sympathetic to that. But it doesn't mean that because you have a strong emotional connection to it, it has a place in our moral hierarchy of rights relative to living beings. What if my toy is screaming, Trevor, save me. Remember all those secrets you told me about your life, Trevor. Yeah. And that, I think, is where we have to take responsibility for what we design.

1:31:37Mustafa Suleyman:Some people will design those things. They're already doing it. You know, spend any time on TikTok, there's a whole ton of like AI girlfriend robots that people are designing or models that people are designing and teaching other people on tiktok how to kind of yeah you know neg someone like push them out money etc etc like you know that's kind of the challenge of proliferation if anybody can create anything it will be created and that's what i think is sort of most concerning is that you know i'm totally against that we will do everything in our power to try to prevent that from being possible, for example, for it to say, don't turn me off.

1:32:16Mustafa Suleyman:It should never be manipulative. It should never try to be persuasive. It shouldn't have its own motivations and independent will. We're creating technologies that serve you. That's what humanist superintelligence means. I can't take responsibility for what other people in the field create or other model developers, and people will try and do those kinds of things. And that is a collective action problem that we have to address. But I know that the thing that I create is not intended to do that. And we'll do everything in our power for it not to do that. Because I don't see how if these systems have autonomy, can be persuasive, can self-improve, can read a ton of data at their own choosing.

1:32:54Mustafa Suleyman:That is a super, superhuman system that will just get better than all of humans very, very quickly. And that's the opposite of the outcomes that we're trying to deliver. Do you think there's a risk that it thrusts us into some sort of dark age? And what I mean by that is the other day I was watching the, it was a Liverpool Arsenal game. Yeah. Right. And after the game, a friend sent me a clip of Mikel Arteta being interviewed. And I mean, he was just like destroying his team and destroying himself. And he was just going at it and it was AI. But when I tell you it was good, it was like beyond good.

1:33:34And because like English is Mikel's second language, you couldn't pick up on like the smaller nuances that you maybe would pick up with a native speaker. Like if you were speaking Spanish, obviously I wouldn't understand it. But also maybe not wouldn't. I would have gone like, oh, that's not how he speaks. And that's. But the small intonations and inflections were harder to spot. And the light shifting was harder to. But it made us go, damn, we don't know which interviews are real or not real. And then you're like, which article is real or not real? and which little audio clip that you get is real or not real.

1:34:06When someone sends you a voice note, is it them or is it not them? And then I found myself wondering, can all of this lead to a strange kind of dark age where people still see and hear the things, but basically shut themselves off to it because they go, nothing is real and I can't believe anything.

1:34:24Mustafa Suleyman:Which is partly the reaction that people are having in social media at the moment, right? I mean, it's like there's so much misinformation floating around. There's so much default skepticism that people are just unwilling to believe things. And I think that in a way, there's some healthy... Look, it's good to be skeptical. Be skeptical of these models that are being developed. Be skeptical of the corporate interests of the companies that are doing it. Be skeptical of the information that we receive. But it's not good to be default cynical. skeptical is a healthy philosophical position to ask difficult questions and confront the reality of the answers this has come back to my split brain attitude if i'm just too skeptical i become cynical and i sit on my ass and do nothing no you have to take responsibility for the things that you build so some people out there are going to build shit and we have to hold them accountable and put pressure on them but that doesn't mean that we can roll over and cede the territory and just think, ah, it's all inevitable.

1:35:22Mustafa Suleyman:It's going to be this crazy ass shit. It's going to end up being a dark age. It isn't going to be a dark age. I think it's going to be the most productive few decades in the history of human species. And I think it's going to liberate people from work. I think it is going to create, you know, think about it. 200 years ago, the average life expectancy was 35 years old. You and I would be dead. Today, we're in the 70s and 80s is unbelievable. And some people go all the way up to the hundreds. And I think that that is a massive amount of quality life that we've added to the existence of humanity as a result of science and technology.

1:35:59But these things are not...

1:36:04Mustafa Suleyman:They won't on their own be net good. They'll be net good because we get better as a species at governing ourselves. And so the job is on us collectively as humanity to not run away from the darkness, confront the risk that is very, very real, and still operate from a position of optimism and confidence and hope and connection to humanity and unity and all those things. Like we have to live by that because otherwise, you know, it's just too easy to be cynical. Now I hear you. I actually agree with what you're saying there. And I think there's one part I'd augment though, is I wouldn't say we want to get rid of work, I'd say we want to get rid of jobs.

1:36:43Right, exactly. And I think there's a difference between the two because working is something that brings humans fulfillment, you know, and you see this in children, I always think. Like I'm always fascinated by how you give a child blocks and they start building and they sweep the floor and they put things and they move things around. No one's paying them by the hour. No one's telling them what to do, but they just in their own little brains go, I want to be doing something. And they like seeing the progress of what they're doing. they see the puzzle getting completed they see the toy slowly forming they see the colors filling in the picture you know what i mean and so i i feel like that's their work the difference is when you make it a job go clean your room then they're like ah damn it you know what i'm saying because now you have a boss telling you to do something you don't want to do no one i think to myself like i go in a in a perfect world which we may never get to but in a perfect world we find a way for everybody's work which is their passion to find the other person that finds value in it because music has become people's work and if you think about it it's crazy right people out there tell us with billionaire supremely talented but if you like rewound time and you said one of the richest people on the planet is going to be someone who plays a stringed instrument go back in like to middle ages time and be like yo you know what i mean that guy like busy strumming they'll be like you're crazy you know what i mean the guy in the town square who's playing that little the richest but it's because in this day and age that work has been regarded as valuable same as playing a sport same as working in tech you know and and that's like my dream world is where it's like beyond the money number it's like the value of everybody's work comes to fruition because you do have value if you like knitting you do like you do have value if you're a sculptor you do have value if you are a poet you have value if you're philosopher, you have value if you're a coder, if you're an architect, an engineer, whatever it is, you have, that's like my dream world.

1:38:44I actually wonder what yours is. Like what's, if you could look at, if everything went right, let's say you were now predicting for yourself, you know, you, we've managed to survive the wave. We've, we've found a way to minimize the risks that come from these small, you know, hostile actors. We've found a way to get governments on board and actually have them understand why they should be involved and and you know like responsible for their for their constituents where are we then and when will you say ah we succeeded we did it i think that's

1:39:16Mustafa Suleyman:the real vision is like disconnecting value from jobs because value is like you know everything that you've just just described is the experience of being in the physical world and doing something that you're passionate about that you love. And I really believe that there is a moment when we actually do have abundance. And abundance, some people say, well, we have abundance today, right? But it's just not evenly distributed or that it's not enough. And we still want more and more and more and more. And I don't know that with true, true abundance, because we have a form of abundance today, but energy still costs things.

1:40:00Mustafa Suleyman:It's still expensive to travel the universe, travel the globe. Everything is still, it's expensive. We have more than we had before, but it is possible to imagine a world in, like I'm 40 years old. In 40 years time, 2065, it's totally possible to imagine a world of genuine infinite abundance where we do have to wrestle with that existential question of who am I and what I want to do. I tell you what I would want to do and that i want to do now i want to spend more time singing i joined the gospel choir like 15 years ago for like half a year i can't sing jesus man like i sound like a strained cat like it but the feeling that i got inside from being welcomed by this group of people and just being like bopping along at the back i don't think i've ever experienced anything like it it's just incredible so the most intensely beautiful thing ever just to be part of a group and you know just letting this kind of music come out so it sounds scary to have to answer that question but i all i think that if everybody just takes a minute to really meditate on that question it's a beautiful aspiration and it's within reach it's within reach that's what is possible if we really can get it right for everyone and we kind of get obsessed with like again we just have this Western.

1:41:23Mustafa Suleyman:Think about how many people are earning$2 a day. Think about the 3 billion people on the planet who live by our standards, true like poverty lifestyle, simply because they don't have access to basic vaccines or clean running water or consistent food. That's like the true aspiration. That's the true vision. And I think, you know, I think that's within reach in the next 20 to 40 is it's really eradicating that kind of suffering yeah some of some of those test cases you talk about some of the more inspiring ones i've seen i'm sure you have as well uh when i was in india i uh i got to travel with a group there who was using ai to predict which houses would be most devastated by a flood or a storm and so they would get people to remove their belongings because most people are one devastation away from losing everything they own.

1:42:21And so they would use AI to track where these things are and where they're going to be. And they could even tell you in a heat wave who should leave their house so that they don't die. And you look at programs in Kenya where they've used AI to help farmers not lose all of their crops. They could tell them now, and they use it on like a flip phone. They've told Kenyan farmers, hey, here's your phone. You've got your own little AI. And it'll just tell you when to plant, when not to plant, when to not waste your seeds, when to, and it's increased their output, you know, like 90%, where before it was like a gamble and they were losing, you know, in one harvest, they could lose everything.

1:42:57All of a sudden they have it. And so I really do like what you're saying, because on the one hand, there's always the risk of losing something. On the other hand, there's the opportunity of gaining sort of everything. And the balance is where we have to find it.

1:43:11Mustafa Suleyman:Well, and the proliferation of those technologies are so much more subtle. Like it It sounds like it's just this binary thing of getting access or not getting access, but it's so nuanced. You can't even tell how much good it's doing. I was reading this stat the other day that three years ago, 10 % of households in Pakistan had solar panels on their roofs, which meant that some very large percentage were still burning biomass inside of the home and getting all of the kind of breathing issues and everything else that comes with that. The cost of solar has come down so dramatically just in the last three years that within, I think it was 18 months, the number of personal like consumer households that adopted full on solar on their roofs went from 10 % to like 55 % in 18 months.

1:43:57Mustafa Suleyman:Crazy. Just because it suddenly became affordable and it crossed that line. Suddenly everybody has, you know, near free energy, which obviously means they have access to phones and laptops and connection to the digital world. and are able to, you know, do all the things. So I think that it's easy to overlook what is already happening around us, all the good that is already happening around us all the time and how fast it's happening. It's too easy to get trapped in the cynical world. And I think it's a choice not to. It's a choice to be aware of it and hold it and take it seriously, but not be like owned by it.

1:44:36Before I let you go, there's one question I have to ask you. And you just brought it up when you were talking about like Pakistan and these places. One of the programs you started in the UK was started right after 9-11. And it was basically a helpline in and around. It was like Muslim people who were being targeted. It was just rampant Islamophobia after 9-11. And you stepped in with a few people and you were like, I'm going to start this program because I want Muslims to just have a helpline. Talk about whatever they need to talk about. and it's still running till this day. I think it is the number one, if I'm not mistaken, like the largest Muslim-specific helpline.

1:45:16And I couldn't help but wonder how that shapes how you think about AI. And what I mean by that is tech has often had one type of face attached to it, you know? And not in a bad way, I'm not like villainizing any. It's just like, yeah, that's where the thing was. But as tech slowly starts to evolve, You see India hopping up as a powerhouse. I remember when I was traveling there, you just go and you're like, wow, this is like a new Silicon Valley. And what's coming out of it is different and impressive for a whole host of different reasons. Nigeria has a whole different type of Silicon Valley. Kenya, as I said, South Africa.

1:45:58You look at parts of the Middle East as well. And I wondered, how does that shape how you think about AI? because we often hear stories about like, oh, AI is just going to reinforce biases and AI is just going to be like, oh, you think racism was bad before? Imagine racism in the Terminator. Imagine if Arnold Schwarzenegger was like, I'll be back, nigga. You know what I mean? It's way worse now. So now when you think about that tech and you're in it and you know what it's like to be in a group that is ostracized, how do you think about tackling that? How does that shape what you're trying to design?

1:46:34Mustafa Suleyman:We created Muslim Youth Helpline as a secular, non-denominational, non-racial group of young people, led by young people for young people. So it wasn't about being religious. It had Sunni and Shia. It had Pakistani, Somali, English, white, everything you can think of. And we were all like 19, 20, 21 years old. The first CEO was a woman. Muslim woman. And so my response to that feeling of being threatened, basically post 9-11, with rising Islamophobia and being accused of being terrorist, was to create community. And that community taught me everything. It taught me resilience, respect, empathy for one another.

1:47:28Mustafa Suleyman:and the simple act of listening to people, just being on the end of the phone and using a little bit of faith and culturally sensitive language in nonviolent communication, just making people feel heard and understood was this superpower. You're not telling them what to do with their life. It's nonjudgmental, non-directional. You're just making them feel heard and understood. And that has always stayed with me. It's been a very important part of my inspiration and it speaks to a lot of what I've been doing now, especially with my previous company, Inflection, and Pi and stuff. Pi was really like a very gentle, kind, supportive, listening AI.

1:48:08Mustafa Suleyman:Yeah, I remember using it, yeah. Yeah, and it became part of Microsoft now. And so I think that's what makes life worth living and that's what I still try to do if I can today. Yeah, man. Well, you know, I appreciate you. Thanks for taking the time today. Thank you for writing the book. I'll recommend everybody read it like as long as they can because it's a very human look at a very technical problem and I think that's sometimes what AI is missing I talk to a lot of engineers who don't understand the human side and I talk to a lot of humans who don't understand like the engineer side of it but yeah man, thank you for taking the time thank you for joining us and I hope we have this conversation in like 10 years just me and my baby little AI and we're just going to be talking to you about it.

1:48:55Just be like, you know, what do you want to ask Mustafa here? Ask Uncle Mustafa a question.

1:48:58Mustafa Suleyman:I'll be back. This has been amazing, man. Thank you so much. Thank you, bro. Appreciate it. Thank you.

1:49:09Mustafa Suleyman:Shit. I think you split my brain in 15 pieces. No, man, I appreciate it. That was amazing. Thank you very much, man.

1:49:20What Now with Trevor Noah is produced by DayZero Productions in partnership with SiriusXM. The show is executive produced by Trevor Noah, Sanaz Yamin, and Jess Hackle. Rebecca Chain is our producer. Our development researcher is Marcia Robiu. Music, mixing, and mastering by Hannes Brown. Random Other Stuff by Ryan Parduth. Thank you so much for listening. Join me next week for another episode of What Now?

1:49:54In the U.S., there's a break-in every 26 seconds. But when intruders step near, SimpliSafe Home Security steps up. Stop. This is SimpliSafe. Police are on the way. Using AI alerts, U.S.-based live agents help deter break-ins. SimpliSafe, no long-term contracts. Save 50 % on your new system with professional monitoring at simplisafe.com slash sxm or with promo code sxm.

1:50:16Mustafa Suleyman:Outdoor deterrence requires a SimpliSafe Active Guard outdoor protection plan starting at$49.99 a month. Visit SimpliSafe.com slash licenses for alarm license information. Tennessee 2012. It's okay to brag every now and then, especially when it's over savings on designer picks from Ashley. Sleep deeper with a great deal on the five-piece contemporary Christyland bedroom set. And when your dinner guests compliment your Core Stone round dining set, don't be afraid to tell them you saved big and that it comes with convenient storage options. With savings you can be proud of, why not boast a little?

1:50:48Visit Ashley.com to shop designer deals for your home.