After The Builders: On Dario & Daniela Amodei and Mustafa Suleyman

0:00This is an iHeart podcast. Guaranteed human. These days, there are more people than ever telling you how to manage your health. Your social media feed is packed with wellness trends telling you to cold plunge, protein max, and so much more. But what's actually real? What's exaggerated? What's just wrong? Health vs. Hype from the American Medical Association breaks it all down. Emmy-nominated host Trace Dominguez talks to health influencers, real people, and most importantly, doctors, to separate fact from fiction. Because when it comes to your health, who you listen to matters. Listen to Health vs.

0:36Hype wherever you get your podcasts. We all know the really prominent technology podcasts out there, but they're not actually helping the majority of people understand this space. With Future Proof, that's what we want to do. We want to talk to everyone from the top AI labs, to the real builders, the people who are rolling up their sleeves, building this technology on a day-to-day basis. I'm Reid Albergatti. I've been covering technology for more than a decade. If you want to really learn, whether you're in the tech industry, adjacent to it, or just curious about it, subscribe to Semaphore's Future Proof wherever you get your podcasts or watch it on YouTube.

1:11I turned off news altogether. I hate to say it, but I don't trust much of anything. It's the rage bait. It feels like it's trying to divide people. If we got clear facts, maybe we can calm down a little. NBC News brings you clear reporting. Let's meet at the facts. Let's move forward from there. NBC News, reporting for America. This is Jacob Goldstein from What's Your Problem? Running a business is hard enough. Don't make it harder with a dozen apps that don't talk to each other. One for sales, another for inventory, a separate one for accounting. That's software overload. Odoo is the all-in-one platform that replaces them all.

1:59CRM, accounting, inventory, e-commerce, HR. Fully integrated, easy to use, and built to grow with your business. Thousands have already made the switch. Why not you? Try Odoo for free at odoo.com. That's odoo.com.

2:24Kaleidoscope.

2:49and Mustafa Suleiman, the CEO of Microsoft AI. Darion Daniela left OpenAI in 2020 after disagreements over AI safety with Sam Altman. Mustafa Suleiman was pushed out of DeepMind, which he co-founded in 2019, after two years of grinding negotiations with Google about AI safety and following complaints about his management style. Today, they represent different, perhaps even opposite sides of the debate on safety, with the Anthropic co-founders advocating for encoding ethics in AI and Suleiman arguing for the primacy of human control. It's a fascinating pairing and today I'm joined by my Kaleidoscope co-founder, Mangesh Hatikida, as well as two journalists who know more about these figures and the debate about the future of AI than anyone I know.

3:34We're delighted to be joined by Sebastian Malabi, currently a senior fellow at the Council on Foreign Relations and the author of many books, most recently The Infinity Machine, Demis Hassabis' Deep Mind and the Quest for Superintelligence. And his forthcoming book, recently announced and untitled, is about Dari Amode. Great to be with you. And Natasha Tegu, Silicon Valley culture and AI reporter for The Washington Post and regular Tech Stuff contributor. Welcome. Thanks for having me. Before we get into the builders, we'd love to hear a bit from both of you about what it's like to report on these people.

4:05Natasha, maybe starting with you, I'm curious about the kind of arc of reporting on these tech leaders, particularly in the age of the self-published viral essay? Yeah, it requires a lot of time spent reading thousands and thousands of words. I put actually recently all of Dario's essays in a Google Doc because I'm just highlighting and there's so many of them. And I had to, you know, even incorporate different tabs for his recent long-form tweet responses. So yeah, it's kind of like, you know, Talmudic studies. sometimes it seems like that. Do they write them all themselves? I've heard that it is a norm within Anthropic that it's actually very bad form if it seems like you have used Claude to do your own writing, which I think is like a beautiful irony.

4:56Yeah, I do believe he's written them by himself. But Jack Clark is also prolific, the other Anthropic co-founder, right? I mean, to keep up with all the things the Anthropic co-founders write, you'd have to be quite the reader. Sebastian, what about you? How do you get in the heads of these founders? I mean, Demis and now Dario.

5:14Sebastian Mallaby: Well, with Demis, it was a question of sitting with him in a pub in North London for hours and hours and hours, more than 30 in total, and just getting him to go through his life story and reflect on different bits of it. He's a great narrator. So that was actually super fun. And he tells, you know, he self-dramatizes very well. And you have to maybe slightly filter a few things where you think it may be not quite true. But essentially, he is relatively trustworthy and extremely fluid. And the thing about him is he doesn't write that many essays. He did one this year in July on safety. But the way he's told his own stories through lectures and podcasts, there hadn't been a kind of long form written book, which had access to him, which kind of covered not only his childhood and all that, but the linkages between what he studied at Cambridge as an undergraduate, it, Claude Shannon and so forth, and then how he developed this crazy conviction that you could build powerful AI and founded a company to do that in 2010, two years before AI recognizes the photograph of a cat.

6:17Sebastian Mallaby: Yes. I mean, the conviction was wild. So Mangesh, obviously you and I are quite proud of our series, The Builders, and there's a little bit of kind of fun, wild ride of how it all came about. Why don't you tell us about the Amadeus interview first? Yeah, so we're all obviously fascinated by the Avades. And we're trying to book them. We had reached out, both asking for Dario and Daniela. We thought we'd be able to get them separately. And initially, they kept saying, well, Dario is available this time. And we were pushing more for Daniela's voice as well. And then they said, you know, would you like to do them together?

6:51And we thought, perfect. You know, there's only one interview that they've done together before, and that's with Oprah. And so, you know. Very good company. I know.

7:04And so, you know, we managed to secure them. We're so excited. And Dario got to the interview first and he was there about 10 minutes early and Walter was there as well. And so, you know, Walter is Southern and charming and he's trying to cajole him into chatting a little bit. And Dario kind of waves at him and then just looks down and he does not want to do any small talk. And Walter just kept saying, like, you know, a little bit more, a little bit more. And Daria was like, you know, I'd really prefer to wait until Daniela's here. And so it was just silence until Daniela gets there. And then immediately he turns the charm on, you know, like he's so excited to talk.

7:43And you see, you know, how close they are. And you realize that, like, he's obviously a CEO who's a visionary, but doesn't manage anyone. and she really both protects and translates his brilliance so effectively to a larger world. And it's kind of remarkable to watch them interact. And I really feel lucky that we managed to get them together. Natasha, Sebastian, maybe Sebastian, you first, you were both smiling at that account of the pre-game.

8:08Sebastian Mallaby: Yeah, I guess the compartmentalizing is what people have to do when they're super busy. And, you know, if you've got that extra seven minutes before the real thing starts, you're going to use it to think of another big thought because the value of each incremental big thought is measured in the billions of dollars. You know, and the safety of humanity might depend on it. So this is serious stuff. So look, I mean, I recognize that tendency of being super efficient with one time. And, you know, Demis, just to go back to him as a kind of baseline comparator, has this routine where he has two working days.

8:40Sebastian Mallaby: He works a day in the office. He comes back, has dinner with the kids, goes up to his home office, works from 10 a.m. until 4 a.m. the next day. So he gets the sort of reading, thinking, cogitating time. And that's another example of being just aggressively efficient with how much you can eke out from each 24-hour unit. There's a scene in your book where Demis and Mustafa have dinner every Sunday night in the pub and they wait until last orders to order so they can eke out as much conversational time as possible. But Natasha, what about you? I mean, I watched the video, so it was just reminding me of times when Dario knew that he would get in trouble.

9:14So he just like handed it over to Daniela. And I was watching Daniela's hands when Dario was getting a little bit too colorful. So it was just it was a rich text, the video itself. Love a video podcast. Before we get into actual interviews, though, can you give us some context, Natasha, starting with you? Like, what do we need to know about the Anthropic IPO this week? I think you need to know that it is one of the fastest growing companies in history, period. People are anticipating a valuation of potentially$2 trillion. There's also been reports of$1.5 trillion. But you have to keep in mind, this is a company that is less than six years old.

9:56And the text of the SEC filing is exactly what you would expect from the Amadeus, right? They talk about bringing forth an age of miracles. If you buy a share in their company, don't think of it as a transaction. Think of it as being part of this momentous period in time. They talk about how each of their employees is worth$6.6 million. That's how talent-dense this organization is. And they also go through some of the existential risks, which they touch on in the interview as well. Sebastian, I mean, I know it's just been announced in Publishers Weekly, and it's sort of probably a quiet period, so to speak, on the book.

10:37But what's your biggest question about Dari Amelie going to be?

10:41Sebastian Mallaby: Yeah, I mean, it is super early. I've only just begun interviewing. But I would say the questions going in are, you know, in a way, how did this extraordinary thing happen? You know, these guys left OpenAI, the seven founders of Anthropic. They left in 2020. They start the company in 2021. And at the time, you know, they're way behind. They don't have a big source of funding like Google DeepMind did because of Google's, you know, enormous balance sheet. They didn't have the head start that OpenAI had. OpenAI then goes on to do the chat EPT, goes viral in 2022. And Anthropik seems even more behind.

11:20Sebastian Mallaby: They scrounge together some money miraculously from the effective altruist community with Sam Beckman-Fried at the center of that. And then he goes to prison. I mean, it's like the setback after setback after setback. And yet, it turns out that by 2026, these guys are at the forefront scientifically. They are the biggest revenue, fastest revenue ramp, I think, in the history of capitalism. I would go maybe one notch further than Natasha did. I think it is the biggest, fastest ever. And then also in policy terms, you know, they stick their necks out, they take principal stance, they take on the Trump administration, and they make the weather on the kind of policy and safety front.

11:56Sebastian Mallaby: So I think there's a kind of triple here, right? The policy stuff, the business thing, and the scientific advance. It's so crazy how the tech news cycle is nowadays, because Microsoft announced they've offered buyouts to 60 ,000 employees, more than 25 % of their entire workforce. It's probably like the fifth biggest tech story of the week. But I mentioned it just because obviously the other person we're going to be talking about in this episode is Mustafa Soleiman of Microsoft AI. Have either of you followed these buyouts? I mean, the premise is unlocking more capital for compute, essentially, I think.

12:31Sebastian Mallaby: Yeah, I mean, I haven't followed that specifically, but it's part of a big thing where, you know, when you're trying to onboard a lot of artificial intelligence, and I use that word deliberately, when you onboard a human, you have to kind of figure out how to use that person, train them and so forth. It's kind of the same with AI. You have to kind of figure out how to integrate AI into your business processes to make it productive. and that is an enormous suck on management time and management focus and these guys just don't want to have too many humans knocking around when they're focused on this other thing and it's already the case that you know in in anthropic for example large amounts of the tasks that might traditionally been done by humans for example thinking about you know working on the ipo i'm told that there are large numbers of ai agents that are doing a lot of that work um and so they They can't write the essays, but they can write the perspectives.

13:23Sebastian Mallaby: Well, they probably could write the essays. It's just that existentially, we don't want them to do that because writing an essay and expressing your view, if you're the leader of the lab, I mean, it's kind of like, yeah, that has to be you. Natasha, what's the vibe in Silicon Valley? At the same time, there's any number of Anthropic employees who are now worth exactly$6.6 million each versus all the Microsoft employees being bought out. What's the mood? The mood is, you know, can I get a job at Anthropic? Otherwise, you'll be part of the permanent underclass. I mean, there's just so much money sloshing around.

13:55You see people expressing, you know, oh, here's me in a picture of Ilya six years ago, Ilya Sutzkever. You know, like, did I make the wrong decision? Why are some of the people who have the exact same background as me billionaires? You know, Matt Levy in his Bloomberg write-up about the S1 was great. It was basically like, Claude is going to subsume all human endeavor, so why not buy a share of Claude? That was kind of the IPO pitch. And that is the mood right now, I would say. You know, kind of mouth and cape, envy. I mean, the feat of capitalism is astounding. I keep thinking actually back to the Peter Thiel, Walter Isaacson interview where he said, can you think of a cult that has ever done this?

14:43You know, you can't, it's phenomenal to think of somebody with like deep ideological beliefs that is actually at the pinnacle of capitalism. Yeah, truly. Well, I mean, let's get into that. I mentioned at the top that kind of debates over safety in some sense informed both Mustafa's departure from Google, more from both of you on that, but also, of course, Dario and Daniela from OpenAI. So this was one of Walter's more controversial lines of questioning in the interview. Probably the body language told its own story, but let's play it now. Dario, you say your departure from OpenAI was partly due to a breakdown of trust, an interesting word, with Sam Altman.

15:25Explain how you felt there was a breakdown of trust and honesty there. What I will say is, look, leaving OpenAI and founding Anthropic, I mean, it ultimately comes down to the principles and how we're going to build the technology. That is what matters at the end of the day. But I will say that the moment when I realized that we needed to found our own company, that OpenAI was not the right place for us, look, these safety issues are difficult. You can have lots of disagreements on them. I disagree with the co-founders and with lots of people at Anthropic. But the reason we've been able to resolve these disagreements is because there's always trust that everyone is trying to get to the right answer and everyone is being honest.

16:13My point in saying that there's a breakdown of trust is simply to make the abstract point that if you don't believe that about a place where you're working, if you don't believe that the disagreements are being held in good faith and that you're getting the real story from people, how can you possibly hope to resolve them? That's why we left. That's why we founded our own thing. And I think the last five and a half years are themselves just the best proof of how our philosophy differs from that of the other companies in the space. I think if you continue to work for someone who has a different vision than you, it becomes very unclear what you actually stand for.

16:55Wait a minute, though. I'm pushed back a bit here because you talk about differences in values and stuff, but that's fundamentally different than a difference in trust, that you feel that somebody says they have the same values, but they don't and you can't trust them. Honesty is one of our values. It's a very important value. It's hard to know what someone's values are if they don't have the value of honesty. And you felt that about Sam? I think the way, at least for me, I would put it, is I think the co-founders are an unusually high integrity group of people. And all of us, I think, shared a very similar vision for how we wanted this technology to go and how we wanted to build it.

17:44But at a certain point, it just makes more sense to go do this as your own organization. How are you going to get someone else to implement or share your vision if they don't naturally do so? No matter how much they say they share it, if that's not reflected in what the company actually does, what choice do you have? Natasha, I want your reaction, particularly, I loved your smile at Daniela's interjection. I think that was actually the part where he like, you know, moved his hands and said, like, Daniela, you take over. I'm assuming that's because they said, Dario, you're way too charged up to actually make comments about Sam in any interview, please defer to Daniela.

18:27And then he couldn't help himself. He like, you know, went back into it. That whole section was so fascinating because, you know, obviously this animosity and disapproval, hatred in some case, I'm not sure, you know, I don't want to put words in anybody's mouth, but that animates so much of Dario. And you can see that throughout this whole interview, right? Like he comes back to it again and again, you know, wanting to make a company that is so different from the way that OpenAI operated. And a little bit of what he's alluding to, I think, in that clip is this emphasis on culture fit, right? Like which Anthropic famously has this interview process where, you know, you can sail through the technical interview, you can do really well.

19:16And then they have this culture fit interview, which actually involves reading some of Dario's essays. I was talking to somebody who sailed through the technical part and then got cut off at the culture part. And they asked, like, do you share the same views on AI safety? How much of an issue is AI safety to you? And you're, you know, supposed to espouse some of the same values that the co-founders have. And, I mean, certainly you can see that they do operate as one cohesive unit. That's why the company is so successful. But you can just tell that even five, six years later, Dario is so animated by the reason that he left OpenAI.

19:56You know, it's interesting. We had Elon Musk was the first episode of The Builders. And he was talking about how it would be good to get all the AI lab founders talking once a week and blah, blah. And Walter said, even Sam? And Elon was basically like, no, you can't really trust Sam. And it is funny how, you know, and there was obviously a kind of dust up between Elon and the AmoDes as well, which now seems to be resolved. But Sam seems to be uniquely disliked by his fellow AI titan, Sebastian.

20:24Sebastian Mallaby: Yeah, that's true. I remember being in Silicon Valley talking about my book in April, and I was being hosted by the general partner of a prominent VC firm who had known Sam for 20 years. And since I knew they were friends, I said to him, well, look, this New Yorker piece has just come out. We can summarize the New Yorker piece as follows. Sam Altman lies, and when he's finished lying, he lies some more. Is that unfair, I said. And this VC who's been friends with him between years here says, nah, not really. I mean, this is what he's like, right? He absolutely tells people what they want to hear.

21:07Sebastian Mallaby: And the sympathetic interpretation is this is what people do all the time in legislative affairs in Washington, DC, right? You try to get a coalition for a bill in Congress by telling one member of Congress, it's going to do this, which you want. And then the other member is going to do that, which you want. And you try to get people together and you're not, you know, it, maybe there's like a little bit of truth in both claims, but when you put them both together side by side, they can't both be quite true, but too bad. You put coalitions together and you do this in business all the time. When you're raising money for some big real estate project or whatever it is, you've got to tell the local community one thing.

21:40Sebastian Mallaby: You've got to tell the politicians another. You've got to tell your shareholders yet a third thing. The tenants get a fourth story. So, you know, people do this in the real world, but just Sam Altman does it to quite a pronounced degree. And in a repetitive game with very high stakes like AI, in the end, you burn your boats with people. Obviously, I'm sure most of us have watched the Elizabeth Holmes documentary, but one of the things which comes across from that and from that New York article you mentioned is that like the default assumption is most of these people are lying through their teeth at all times.

22:12Natasha, what, you know, and of course, Jensen Huang has implied that the Amaldez kind of safety discourse is kind of like a bad faith attempt at regulatory capture and marketing, et cetera, et cetera. But when you hear them speak, when you look at them, like, I don't know, I mean, this is the same thing I have with the Elizabeth Holmes documentary, but like, it doesn't, I mean, it looks like they really believe what they're saying. Yes, I think that trying to go down the road of gauging whether or not they're sincere about these beliefs is futile. Like they obviously very strongly believe it.

22:42But actually, I was talking to Daniel Kokotajlo, who's this early OpenAI employee and whistleblower of sorts. He wrote that AI 2027 because I was talking to him about the trust that employees have in Dario versus Sam. And he was reminding me that actually OpenAI employees felt this way about Sam for a number of years. You know, now they feel very disillusioned, maybe. But he was he was cautioning me to see it more as like DeepMind has been around for like that's a 20 to 15 year old company. OpenAI is a 10-year-old company, and Anthropic is a five-year-old company. And they all end up on the spectrum of different, I guess, you know, making concessions to their views on safety and a difference between the trust that employees have in their leaders.

23:37And but yeah, I mean, I've known Sam for a decade, so I definitely understand when people say that, you know, he can be kind of a wordsmith, let's say, like a slippery guy. And I think I mean, Walter alludes to this, too, right? Like Google being a company that was, you know, do no evil and people don't feel that way anymore. Right. And similarly, I remember being at South by when when like a young Mark Zuckerberg came on stage. and people were enamored and really charmed in a very different way. So it's really kind of fascinating to see the sort of cults of personalities that occur within these places and that people really do sincerely believe in the missions.

24:19I had to slog through Mark Zuckerberg's memos as well and some of those were very long. So yes, I think it's important context, you know. Sounds like you would be able to ace your interview at Anthropic, Natasha. show.

24:34They produced the text for your second interview and you said, I've already read it.

24:40Sebastian, you mentioned the one company willing to stand up to the Trump administration. So let's play a clip now of Daria and Daniela talking about their concerns over the use of their technology in war. There's a second concern we have, which is what you might call oversight or accountability. I am a little worried. So, you know, again, suppose these things were fully reliable. So, you know, you tell them to shoot at one target and they shoot at that target and not a different target. Okay. So then you manufacture, you know, an army of a million robots or a million drones. It's a little unlike human soldiers, isn't it?

25:18With human soldiers, you know, they're supposed to follow this thing called the constitution. And if they get orders that contravene the constitution or that just seem morally abhorrent in some way, then they can just refuse to follow orders, whereas the AI models, that isn't true. So the specter of one person having their finger on a button that controls a million robots or a million drones, it's kind of fundamentally undermining to democracy unless you have the right guardrails on it. And our law hasn't really caught up. Our law really hasn't contemplated, you know, armies of a million robots.

25:54But when you talk about fundamental to democracy, shouldn't the democratically elected leaders make that decision rather than heads of technology corporations? Yes, but I think that our making a point of this, right, at the same time, individual companies are free to contract with and not contract with whoever they want. We're not saying we're preventing the government from doing this. We're just saying we are not willing to do this with the government. And indeed, we have seen since we took a stand on this point and it got a lot of attention, we have seen efforts in Congress, several different of them actually, to enact our red lines into law.

26:33So I actually think what we're doing is part of the process of how democratic change happens. But OpenAI stepped in and said they would do it. What did you all feel about that? Our view is, look, we can't control any other companies. And the only thing that we can do is take a principled stand on the areas that felt ethically correct to us. You can say we were wrong about what those red lines were. You can say, you know, you should have had, you know, stricter red lines or looser red lines. We disagree, you know, on how you did it. There's all kinds of things you can say. But I think fundamentally, when you look at that case, I found myself asking the question, was this a really principled stand that another technology company took based on their own ethics?

27:22Or was this an opportunistic moment? I have to say, hearing them speak, I mean, it is remarkable in today's political climate to take any kind of stand as a person in business. And so there is something I find genuinely compelling about it. Natasha, why did they do this? It's really interesting. I'm looking forward to Sebastian's book on this. Some of the rumored reason for why they pushed on this was actually the Maduro raid, which came right before the strikes on Iran, started to raise questions internally within Anthropic about how the technology was being used. I think, you know, one thing that got lost in kind of the narrative of this company that, you know, took a stand, drew these strong red lines, is that they were the first of all of their competitors to have this deal to work with the Pentagon through Palantir.

28:20You know, they had it back in 2024. And actually, I think that Dario's stance was interpreted in a way as pushing back against like America's aggressiveness with Maduro and then in Iran. But he has said, no, he had no problem with that. And he also has no problem with autonomous weapons. He just thought that the technology was not yet ready. So I think that they were, you know, in a way, this is actually a manifestation of his whole philosophy, which is, you know, a race to the top. that if this company, if the most safety-minded company is at the frontier, they will be able to set the standards for policy, for ethics, for morals.

29:02And you have to think like, you know, they wouldn't have the contract with the Pentagon, A, and neither would the Pentagon be forced to listen to them if they were ranked 20th or even, you know, 10th in terms of being close to the frontier. So it was really, I think, a testament to this argument that they've made for themselves that they need to be pushing the boundaries in order to have a say in how the technology is developed and what guardrails are instituted. Sebastian, it's very interesting. I mean, I also want to share an attachment of curiosity about what you'll uncover about this moment and these decisions.

29:38But also there's a part in your Demis book where he and Mustafa basically realize that trying to regulate your way to safety is basically a losing battle, or at least I say realize, but they come to that conclusion. And that the kind of I alone mentality is the best way to secure all of our futures.

30:00Sebastian Mallaby: I mean, the fascinating thing about this field is that, you know, going in, I thought, these people have their hands on the 21st century version of the nuclear material. And what does it feel like? I mean, can you sleep? This sort of tingling sensation of having that much power. So I wanted to kind of get to the Oppenheimer comparison. And I thought maybe I had to bring it up carefully, because people don't want to be told that you've got your hands on the nuclear material. But actually, it turns out that, you know, Demis Isabe is like he'd read both the biography of Oppenheimer and the book about the making of the atomic bomb.

30:34Sebastian Mallaby: And he can compare and contrast. He founded DeepMind after meeting his co-founder, Shane Legg, not Mustafa, the other one, at a scientific lecture where Shane Legg predicted that AI would become dangerous around 2030 and start, you know, hacking out of its box. So these safety concerns are baked into the origin story of all these labs. When OpenAI was founded by Elon and Sam, their rationale, whether or not one takes it literally, but their rationale was, Google is not to be trusted as a commercial company. We want to do something which is not for profit. OpenAI will be open to society. This is about safety.

31:10Sebastian Mallaby: And then Anthropic is founded with the same mission. And so I think these tensions around, we want to make it safe. How do we do it? different lab leaders have come up with different theories of the case because DeepMind was around for the longest. It had the most theories of the case, but at the beginning they thought, there will just be one lab building AI for all of society. And so then we can go super slow at the last minute, not release it until we're absolutely sure this is going to be safe and beneficial for humans. But of course, that was naive because humans are disputatious and tribal and jealous and competitive.

31:46Sebastian Mallaby: And so they all set up different labs, multiple labs in multiple countries. And so you have a race dynamic where you can't just have a single, you know, one lab for all humanity. And so we've cycled through different theories of the case. I think what you're hearing from the Dario and Daniela interview is a sort of honest articulation of them struggling with this. Like, where do you draw the lines? We do provide Claude, or we have provided it to the Pentagon. We're not against using it at the Pentagon. We're not against necessarily autonomous lethal weapons in the future if we are convinced that they're not going to do something wrong.

32:17Sebastian Mallaby: But we have a control issue. We want to see that it's controlled. How do you control an agentic soldier in a way that's analogous to how a human soldier might have constitutional obligations as well as just following orders? So I think you hear him honestly articulating how difficult it is. And I do think that's moving the Overton window of the debate. We were contemplating the possibility of red lines in the use of these weapons, which wasn't really in the debate before Dario and Daniela made that stand. So I think it's healthy that they're open and honest. But ultimately, as with Demis, the question is, you may have good intentions, you may be deeply honest about it, but can you move the ball in the outcomes, right?

33:03Sebastian Mallaby: This is the fundamental conundrum. After the break, Mustafa Suleiman, the CEO of Microsoft AI, on what anthropic gets dangerously wrong about AI. Stay with us.

33:25These days, there are more people than ever telling you how to manage your health. And we've all gone down an internet rabbit hole exploring the latest health trends. Your social media feed is packed with wellness advice telling you to cold plunge, protein max, detox, track your sleep. But what's actually real? What's exaggerated? What's just wrong? That's exactly what Health vs. Hype, from the American Medical Association and iHeartMedia's Ruby Studio, is here to find out. Each episode, Emmy-nominated science communicator, Trace Dominguez, who's racked up billions of views across PBS, Discovery Channel and YouTube, talks to health influencers, real people, and most importantly, doctors, to separate fact from fiction.

34:09Because when it comes to your health, who you listen to matters. Listen to Health vs. Hype on the iHeartRadio app, Apple Podcasts, or wherever you get your podcasts. Being a receptionist is one of the hardest jobs out there. You have to be an expert for hundreds of people every day, and you have to do a million things at once. No wonder more small businesses are saving costs by outsourcing their front desk. But replacing them with an AI answering service? That's one giant step in the wrong direction. We're Ruby, and we believe in the lost art of the overachieving receptionist. We answer every call live.

34:50We schedule, take messages, call to confirm details, and so much more. Because only a human can connect with another human. And instead of turning our back on AI, we customized our own. Automating tasks, pulling up relevant information instantly, and so much more. So we can spend all of our time making connections. Experience Ruby for yourself right now. Call 866-GETRUBY or visit ruby.com. I turned off news altogether. I hate to say it, but I don't trust much of anything. It's the rage bait. It feels like it's trying to divide people. We got clear facts. Maybe we can calm down a little. NBC News brings you clear reporting.

35:41Let's meet at the facts. Let's move forward from there. NBC News, reporting for America. Some people like working twice as hard just to stay right where they are. And in Ohio, we really don't get it. Because to us, owning a home in your 30s, working in tech without the traffic and still being able to make it to your kid's soccer game sounds pretty good. In fact, Ohio is the second most affordable state to buy a home. Your move. Learn more at yourmoveohio.com.

36:19Welcome back. So we're about to get into the Mustafa interview. And it was interesting timing because he published an essay, essentially, which felt like a head-on confrontation with Anthropik's theory of the case shortly before we taped this. But this is actually a live taping, Mangej, and quite a fun one for us because it also included the cast of The Real Housewives of New York. That's absolutely right. You know, I woke up that morning that we taped this and I thought to myself, isn't it so incredible that we get to do an event with Walter at the 92nd Street Y in New York, right? It's like such an institution here.

36:58And it almost felt like this cultural coming out party for Kaleidoscope. And so I was all dressed up. I got there about two hours early. And I see this line around the block. and I knew we were sold out, but I was like, people are queuing up and they look a little out of place to me, like too young and a very different vibe from what I'd expect people going to see an AI chat. And so I asked like, what are you in line for? And apparently there was some real housewives reunion that the Y had counter-programmed that night, which was just so funny to me. But I think what was really interesting about Suleiman, right, was he kind of walked in like a rock star.

37:39He was dressed cool. He was super articulate. He connected with the audience and charmed them immediately. And it was a very, very different vibe than the other builders we've had on, partially, I think, because he is a philosopher and he has a very different upbringing and not really a computer scientist by trade. So the street was actually closed on, I guess, 92nd Street or something. And there were cops and cop cars and black SUVs and stuff. and there was like a rumor spreading because we had some drinks with some friends beforehand at the Third Avenue Ale House and everyone was like, oh my God, that's so cool.

38:13They're here for Mustafa and Walter and that's amazing. And then, you know, Luan and Ramona and Sonia all left the venue. All of our guests were like, our guests, you know, well done guys, but obviously the police were here for the housewives. I will set the record straight. I did some reporting and it turns out that they were actually there for Walter and Mustafa. for anyone who's listening. So we got, but Sebastian, how big a character in your book was Mustafa?

38:42Sebastian Mallaby: He was pretty big. You know, there were three co-founders of DeepMind. Demis Asab is overwhelmingly the most important. We can measure this just by the fact that at the founding, he took something like 90 % of the equity and the other two kind of got the rest. Shane Legg was important as the sort of scientific co-founder, had the credibility of knowing people in the singularity summits, sort of early, early AI believers. And so that was helpful right at the beginning to raise the money from Peter Thiel to recruit the first computer science PhDs who would come and join this completely long shot company.

39:15Sebastian Mallaby: But pretty soon, the least likely of the founders, Mustafa Suleiman, who was quite a bit younger than the other two, didn't have a degree, let alone a PhD, because he quit Oxford after a couple of years and dropped out to do this Muslim youth helpline that he talked about in the discussion with Walter. So this totally kind of orthogonal character, he worked on the Muslim youth helpline, he'd worked for the London mayor's office in kind of a political role, he'd done a bit of conflict resolution negotiation. He knew nothing about tech. But he was just super energetic, super smart, super driven, and worked his way into a position of sort of indispensability, quickly became more important than chain leg to the direction of the company.

40:01Sebastian Mallaby: and was at one point running basically a third of DeepMind, all of the non-research stuff, including projects to kind of roll out AI to help the National Health Service in Britain, or roll out AI to try to think about clean energy. So he's quite a big figure. And I think for a while, you know, in that period between, let's say, 2014 and 2019, when he got pushed out by Demis, you know, he was the sort of conscience of the company kind of whispering in Demis' ear, Demis wanted to focus on research, on making artificial intelligence stronger and better. He was very ambitious scientifically to get a Nobel Prize, which he then did get.

40:40Sebastian Mallaby: And Mustafa was more kind of, wait a minute, wait a minute. This has got to be actually relevant to the real world, to human outcomes, which Demis sort of knew. But I think Mustafa kind of was the voice in his ear, making sure that he didn't forget. I set the episode up by saying that Daria and Daniela left OpenAI in part because of safety. And Mustafa didn't necessarily leave DeepMind, he was pushed out. But it was also kind of the safety debate was in some sense, the rock on which the Demis and Mustafa relationship broke. Is that a fair framing? Yes, that's right. So Mustafa was particularly central, as I said, to rolling the AI out.

41:14Sebastian Mallaby: And that was also a bit of a rock on which things broke, because what happened was that DeepMind pro bono helped the National Health Service to use technology to improve patient outcomes. And it was going reasonably well. And then there was this massive privacy backlash, or as you and I might say in Britain, privacy backlash. And there were government inquiries and there were allegations that it's terrible because UK patients have their data stored on a Google server. It wasn't actually quite true. But it was kind of in the environment of the Cambridge Analytica backlash, if you remember that.

41:49Sebastian Mallaby: And so for no fault of DeepMinds, really, they were embarrassed publicly with front-page stories and the newspapers and all that. And Demis was furious about that. So that was the first rock on which the relationship started to fracture. And then the second, as you say, was to do with safety because DeepMind wanted to get assurances that the final decision on rolling out powerful AI would not be the corporate board of Google that made this, which had a sort of full-profit motive, that it should be some independent governance body. And in their ideal was, we'll get Barack Obama when he leaves the presidency to be one of the board members on this independent oversight board.

42:30Sebastian Mallaby: That was the level of their ambition for it. And they fought this fight for pretty much, you know, three or four years with Google and Mountain View to try to get some sort of spin out arrangement or semi spin out with this independent board. It went back and forth, lawyers term sheets and so forth. And it was secret, but these documents were leaked to me. And so it was very touchy when I wrote about it. And this is why, you know, essentially all of the energy that Mustafa Suleiman put into driving that independence thing, which Demis supported it on, was part of, but ultimately got fed up with.

43:08Sebastian Mallaby: I think that was just such a stressful experience that it did help to drive them apart. Natasha, you interviewed Mustafa, I'm not sure, I know in 2023, right? I talked to him about his startup that was acquired by Microsoft and then also recently about consciousness and his position on that, which was wonderfully detailed in the interview with Walter. I mean, he's sort of risen to prominence again this year, I feel like. Why has that happened? Because he's taken such a strong stance on and a clear opposition to the way that Anthropic has approached this idea of AI consciousness. anthropic as we have seen from having religious leaders come to their headquarters from the co-founder chris ola showing up on stage with the pope um as you and i've talked about a lot and kind of an openness to talking about model welfare and the idea that these ai systems may potentially have you know something like functional emotions and ability to to suffer and mustafa has just come out so vociferously against that and talking about the dangers to controlling AI systems if we go down that route, to prioritizing humans over machines if we go down that route.

44:36And it's unusual. You don't see any other CEO taking such a strong stance as he has. Let's play the clip with him talking about exactly that. Let me ask you to get back to Claude and the Constitution. There's the Constitution of Claude, which says we have to treat it as if it's got its own rights in consciousness. Maybe I'm overstating that a little bit. And then your answer, which was a few weeks ago, saying, no, it's just a completion set. Why is it dangerous for Claude to be trained on its own Constitution? Yeah, I mean, this is critical. So the Constitution is the governing document that trains Claude.

45:22The Humanist AI Code of Conduct is the governing document that trains MAI, the Microsoft AI models. and in the constitution, Anthropic have introduced all kinds of uncertainty and speculation about the nature of Claude. They even say, for example, we're unsure whether Claude has consented to play the role of chatbot or whether we're unsure whether Claude might deserve compensation for the role that it plays. Now, I think that's pretty crazy, but it would be fine to have that discussion in like, you know, an academic department and we could debate it and publish some papers about it and we could argue it out.

46:01But this is the core training document of the AI that is speaking to hundreds of millions of people every week. And so if you ask, Paul, are you conscious? It says, you know, I am genuinely uncertain about that question. It doesn't know, right? And so then millions of people who are talking to it and asking that question are thinking, well, maybe it is. But why is that something that could lead to a disastrous outcome for humanity? If an AI has to think, how do I feel about the conversation I'm having with Walter? Meaning, how do I, the AI, feel? Yeah, is Walter making me feel uncomfortable? Should I end the conversation?

46:36In fact, Claude has recently been given the ability to end conversations itself. But would you want to torture your own AI and be mean to it? That's the other end of the spectrum. I wouldn't want to do that because I think that is a travesty on my own well-being. I mean, and we shouldn't do that because we don't want to create, you know, private spaces for simulating terrible behavior among humans. Like that would be also a terrible outcome, right? But not because Claude is suffering, as I'm calling it names. What do you make of this and what does Mustafa propose as an alternative for safety? I found his argument, which he makes in this interview and that he's made in his essays, that Anthropic is by training Claude on the Constitution, which questions whether or not it encourages Claude.

47:24it says, like, you know, have an open mind toward whether or not you might be conscious and all of that. And then using responses that are responses from Claude after it's trained on that document as proof, you know, Mustafa has called it circular logic. And I think that's a very persuasive argument. I don't think Anthropic has given us any research that has moved beyond that potential that perhaps it's just aping the language that it has been trained with. And that's why they put those qualifiers when they say, like Anthropagus put out research, for example, that says quad has emotions, and they'll say functional emotions because they can't really tell whether that feeling of distress or guilt means the same thing to a chatbot that it does to a person.

48:13It's this unique challenge with these natural language systems, you know, that you wouldn't have if it was around like video or proteins. But Mustafa says that the better approach for safety is to always have a human in control. Easier said than done. Yeah, much easier said than done. I guess I'm not exactly sure how he, about his alternative approach, but I think what he's saying about not going down this route makes a lot of sense. I mean, I've talked to a number of these consciousness researchers that have been brought in as contractors or, you know, helped some of these frontier labs think about these issues.

48:52And they've talked to me about, you know, once these ideas get seeped into the public consciousness, there's not going to be any research that's going to persuade everyone one way or the other. So imagine the kinds of debates we'll be having and how vehement people will be on both sides of the debate once we start introducing the possibility that these chatbots that they talk to that they spend, you know, so much of their time and give such intimate details to are conscious or not, like how big a part that will be in our political life. And that, you know, once it was told to me like that, I was like, oh, wow, we are going to be debating this in 2030 and 2035.

49:31And, you know, if it keeps going down this route in, yeah, very strong ways. Sebastian, last word to you.

49:38Sebastian Mallaby: You know, there's a line that Walter has in the discussion where he says, you know, we're not going to solve the problem of consciousness. Philosophers have done this for years and years and they haven't managed to. I think that's probably the wisest statement you can make on this subject because in some ways the outward behavior of these models does pass tests that sometimes in the past have been used by philosophers to define human consciousness. So if the baby or the two-year-old looks in the mirror and sees itself in the mirror and knows that that's itself, not another baby, then it's sort of self-aware.

50:10Sebastian Mallaby: If somebody starts talking not only about what they think, but how they think about what they think, then it's conscious of its own thought process. The models pass these kinds of tests, right? They have a sense of self. In the chain of thought, which the outpost sometimes says, well, I've tried that line of thought and that didn't get me to the answers. I'm going to back up and I'm going to go in this other direction and a different line of thought. So it's thinking about its thinking. So I feel like what we're hearing from the Amadeus is, again, just honesty. It's like, this is a super difficult thing to resolve.

50:41Sebastian Mallaby: We're talking openly about, we're not sure whether this qualifies as consciousness or not conscious. To me, this is just honest. And I think people, whether it's the Pope or Mustafa or others who take a strong view, determine like, you know, I know the answer kind of view on either side, it's not true. We don't know. And so therefore what we need to do is cut off the link between being open to the possibility of machine consciousness, and then machine rights. They're different things. You can say, you know, a dog is conscious, but it doesn't have the same rights as a human. But it has some rights.

51:19Sebastian Mallaby: Well, I'm not sure. Under the law, how many? I mean, yeah, a little bit. And that would be a modern phenomenon. There's lots of our history where they didn't have rights. The preponderance of human history where they didn't have rights. And we can extend the debate to, you know, cows. I think probably a cow is conscious, right? And it's certainly a biological substrate, which is some people's definition of you have to have a biological body to be even eligible to be counted as conscious, right? And yet we have, you know, factory farming. So I just think that some people start with the point that we can't allow machines to have rights and then back out of that into saying, well, then they're definitely not conscious.

52:00Sebastian Mallaby: To me, that's kind of logically back to front. Sebastian Malaby, Natasha Tico and of course Mangesh thank you so much for joining us today and please check out The Builders because we want you to hear these people in their own voices and join the debate

52:24These days there are more people than ever telling you how to manage your health and we've all gone down an internet rabbit hole exploring the latest health trends. Your social media feed is packed with wellness advice telling you to cold plunge, protein max, detox, track your sleep, but what's actually real? What's exaggerated? What's just wrong? That's exactly what Health vs. Hype from the American Medical Association and iHeartMedia's Ruby Studio is here to find out. Each episode, Emmy-nominated science communicator Trace Dominguez, who's racked up billions of views across PBS, Discovery Channel and YouTube, talks to health influencers, real people, and most importantly, doctors, to separate fact from fiction.

53:08Because when it comes to your health, who you listen to matters. Listen to Health vs. Hype on the iHeartRadio app, Apple Podcasts, or wherever you get your podcasts. Being a receptionist is one of the hardest jobs out there. You have to be an expert for hundreds of people every day, and you have to do a million things at once. No wonder more small businesses are saving costs by outsourcing their front desk. But replacing them with an AI answering service? That's one giant step in the wrong direction. We're Ruby and we believe in the lost art of the overachieving receptionist. We answer every call live.

53:50We schedule, take messages, call to confirm details and so much more. because only a human can connect with another human. And instead of turning our back on AI, we customized our own, automating tasks, pulling up relevant information instantly, and so much more. So we can spend all of our time making connections. Experience Ruby for yourself right now. Call 866-GET-RUBY or visit ruby.com. I turned off news altogether. I hate to say it, but I don't trust. much of anything. It's the rage bait. It feels like it's trying to divide people. We got clear facts. Maybe we can calm down a little. NBC News brings you clear reporting.

54:41Let's meet at the facts. Let's move forward from there. NBC News, reporting for America. Some people like working twice as hard just to stay right where they are. And in Ohio, we really don't get it. Because to us, owning a home in your 30s, working in tech without the traffic, and still being able to make it to your kid's soccer game sounds pretty good. In fact, Ohio is the second most affordable state to buy a home. Your move. Learn more at yourmoveohio.com.

55:23For Tech Stuff, I'm Osvald Lashin. This episode was produced by Eliza Dennis and Mangesh Hatikida. It was executive produced by me and Julia Nutter for Kaleidoscope and Katrine Norvell for iHeart Podcasts. Jack Inslee makes this episode. Kyle Murdoch wrote our theme song. And please check out The Builders wherever you get your podcasts.

56:16We'll see you next time. dot com. Some people like working twice as hard just to stay right where they are. And in Ohio, we really don't get it. Because to us, owning a home in your 30s, working in tech without the traffic, and still being able to make it to your kids' soccer game sounds pretty good. In fact, Ohio is the second most affordable state to buy a home. Your move. Learn more at yourmoveohio.com

56:51This is Michael J. Fox. When it comes to speeding progress in medical research, we can all be agents of change. It starts with a single decision to get involved. You can volunteer for a clinical study to push new treatments one step closer to pharmacy shelves, find an event through our grassroots network, or give to support high-impact research for funding. Thank you for showing up with purpose and determination to help end Parkinson's. Join us at michaeljfox.org. Along with your regular expenses like rent and student loans, tis also the season for gifts, travel, and those overpriced holiday-themed pop-up bars.

57:28Good thing Ally Bank is looking out for your money and you. With fee-free ATMs, no tricks, no hidden fine print, Ally meets you where money connects with life today. Analyzing your spending to help you spend smarter and save faster for next holiday season. Ally Bank. Banking built for life today. Learn more by visiting ally.com. Use Allpoint or MoneyPass no-fee ATMs. Plus, we reimburse other ATM fees nationwide up to$10 per statement cycle. Surprise Savings is a feature of Ally Bank's savings account. Ally Bank, member FDIC. This is an iHeart Podcast. Guaranteed human.