Microsoft AI CEO says AI threats are real, and Anthropic is making it worse

19:41Mustafa Suleyman:And so rather than keeping it too abstract, we can just focus on those specific things that we can make progress on. And I'm sure there's a bunch of others, too.

19:49Nilay Patel:There's reporting, I believe, in the information that there have already been talks about an industry self-regulatory body. Have you been involved in those talks?

19:56Mustafa Suleyman:Yeah, I mean, as I said, like we talked a lot during COVID. We talked, you know, in the late 2010s about it. I mean, there's definitely been a lot of like conversations over the last few weeks and months between all the lab leaders.

20:08Nilay Patel:I understand why Anthropic and OpenAI might wake up one day and say, wait, are we doing an antitrust problem? Like are we going to get sued if we coordinate? Microsoft is really, really good at the government, right? You're a longstanding government contractor, Brad Smith, the president of Microsoft. He's very good at policy. Lena Kahn, who is maybe the most aggressive antitrust enforcer we've had in our lifetimes, is publicly out there saying you don't need this antitrust exemption. Jonathan Cantor, who ran antitrust at the Biden Department of Justice, I just talked to him for an upcoming episode of the show.

20:38Nilay Patel:He's like, you don't need an antitrust exemption. Inside Microsoft, do you think you need an antitrust exemption?

20:44Mustafa Suleyman:I mean, that's one for the lawyers to answer. I think that people are looking into it at the moment and they're taking it very seriously. So they just have to work through whether we do or whether we don't. I think we have to be – look, it's right to be careful about those things. I wouldn't read every single thing as like cynical. But we'll see. We have to make progress quickly on it. We can't just dither around and use that as a blocker.

21:32Nilay Patel:personalization gets faster, the more you use it. Multiply that across your team, every trip, every year. That's not a perk. That's a competitive advantage. And with the Engine X card, get up to 10 % back on any hotel booking. Last year, Engine customers saved more than$300 million on travel. 33 ,000 businesses have joined. Now it's your turn. Get$500 when your business signs up and starts traveling at engine.com slash decoder. Engine X Visa commercial cards are issued by Fifth Third Bank North America, member FDIC. Earn up to 10 % back in points on eligible engine travel purchases. Actual reward rates vary by purchase, category, and may change.

22:15Nilay Patel:Points have no cash value and are redeemable for rewards through our program. See full rewards terms for details. Go to engine.com slash x slash rewards dash terms.

22:51Mustafa Suleyman:and customers can just keep talking. Try DeepGram Flux TTS free now until September 12th. Visit deepgram.com slash keep talking. Terms apply.

23:01Nilay Patel:Support for the show comes from CrowdStrike. AI is already fundamentally changing how companies build products and run their businesses. If you listen to the show, you probably already know this. But while AI drives innovation, every new model and agent also creates new cybersecurity vulnerabilities. That's why CrowdStrike built AI Detection and Response, or AIDR. AIDR helps organizations discover where AI is being used, monitors AI activity at runtime, and helps teams detect, investigate, and respond to threats involving AI systems. As companies adopt AI, security has to be built in from the start.

23:42Nilay Patel:More than 70 % of the Fortune 100 trust CrowdStrike to protect their businesses. Now CrowdStrike is helping secure the AI era.

23:50Mustafa Suleyman:You can learn more about AI detection and response at CrowdStrike.com slash decoder.

24:08Nilay Patel:The other version of this debate, or maybe the other avenue into this debate, is you don't need to slow down and have safety responsibility imposed on you by novel regulation. Product liability alone will create the incentives for you to make more safe products, right? If a Microsoft AI model goes out and does some untold harm to the world, Microsoft will get sued out of existence. This is probably something that you should think about before you release the next model. Has that been an effective incentive loop for you already? Or is that something you're thinking about now?

24:38Mustafa Suleyman:No, definitely. I mean, of course, that's always present in everything that we think about when we deploy products. But keep in mind, this isn't so much about deploying products. The models that were used for hugging face or to solve the Navier-Stokes Millennium Mathematics Prize, they're not commercially released yet. They're not actual products. And so the liability regime is slightly different. I mean, these are being operated inside of the big companies with huge long-running reinforcement learning climes. So I think, you know, liability covers part of it, but not all of it.

25:14Nilay Patel:There's a part of me that personally feels a little silly when I ask questions about product liability, right? Like Microsoft is going to release a new version of Microsoft Word that might kill everyone. Maybe you shouldn't do that because it will get sued out of existence is a pretty simple to understand thing. It's so silly that it would never occur in any other conversation about any other technology. Bluetooth is great, but what if it kills everyone? Like we just wouldn't have Bluetooth. What are the near-term disaster consequences that would stop AI development? Is it just product liability?

25:44Nilay Patel:Is it something else?

25:45Mustafa Suleyman:I just feel like everyone has this like hyperbolic, super reactive, completely alarmist tone. When in fact, we have a long history of many decades of regulation that has worked incredibly well. so well that you barely notice it, right? You know, like everything from street lights to construction materials, from asbestos to the batteries inside of your laptop that don't cause a fire inside of your car to, you know, the seatbelts. Like everything has a code of conduct and it has a regulatory framework around it. And every new technology gets built with that in mind so that planes don't hit each other in the sky.

26:26Mustafa Suleyman:You know, it is true that this technology is different. I'm not just going to kind of put it in the bucket of, you know, sort of pencils and paint. Like it is different. It is also moving much faster than it ever has before. It's incredibly human-like in the emergent capabilities that arise when you pour a ton of compute on it. So it's important to, you know, be like, you know, clear-eyed that it is a different moment. And this time actually is different. At the same time, there's an entire body of practice and knowledge and frameworks and so on, which can be applied here. Liability is like an obvious one.

27:03Mustafa Suleyman:And so, yeah, it's tricky because a lot of the sort of conversation tends to take place on Twitter. So the temperature seems to all be like really high. But I know where else we have it.

27:14Nilay Patel:It does seem that probably we should be having this conversation in the halls of Congress and at various regulatory bodies. And instead, we've chosen Elon Musk's short form social media platform and something is getting lost literally in the compression of thought that occurs there. What do you think is the most important thing that is being lost in this conversation? What's the nuance that most people aren't seeing?

27:39Mustafa Suleyman:Detailed practical proposals. You know, like it takes time to read somebody's document. Sit down and like read it. Like a lot of things are getting written down and they are precise and specific and they're full of concrete proposals to go in one direction or another. It's not like we're lacking for substantive ideas. The problem is we're communicating substantive ideas in sort of hyper aggressive short form. So, you know, I've tried to put out a bunch of very like thorough, it's a 40 page document, our humanist AI code of conduct. And my essay this morning on model welfare, you know, is also like a 20 page essay that in a very detailed way highlights the 99 page anthropic constitution word for word, which I personally did myself.

28:26Mustafa Suleyman:And we created a taxonomy that is 20 pages long of all the different types of anthropomorphism that they do. And so I've tried to be like very thorough and evidence based and specific and not like super hyperbolic. I have a strong view on it, and I do think that it increases the risk to AI alignment and safety, and it makes the problem harder. But I'm totally happy to change my view if new evidence emerges that actually we do owe models a duty of care and they deserve our welfare, or that, for example, it could be safer if we treat them like that. I'm totally open to that, and we should empirically validate it.

29:03Mustafa Suleyman:But like I'm trying to push the conversation to a substantive evidence-based specific one rather than, you know, like should we slow down or should we not? Like, sure. Let's talk about the details.

29:13Nilay Patel:I'm very happy that you brought up Anthropic because they're obviously at the center of this debate. And I know based on our previous conversations that you do have a strong opinion about their approach to Claude and model welfare. And we have asked Anthropic very directly if they think Claude is alive before. and their answer is, well, it's not alive because it doesn't have its blood, but it might be conscious, which is dancing on the head of a pin, right? It doesn't really matter to me if you think it's alive or conscious. You think it's something other than a computer. Your core thesis in your memo, and I do encourage people to read it, is that alignment, safety becomes harder if you continue to design models and you believe that they're human beings or that you believe they have the rights and opinions and emotions of human beings, because the AIs might think that they have autonomy rights and personhood.

30:03Nilay Patel:Explain that in more detail, because this feels like a very important point of contention inside the industry that is very opaque outside.

30:11Mustafa Suleyman:Yeah, I mean, like the first thing to say is that the constitution that Anthropic put out in January is a training manual for Claude. And inside of that training manual, they have introduced a lot of uncertainty and speculation and ambiguity about the question of whether Claude deserves to be treated as a moral patient, in their words, which is whether it has rights because it suffers. In fact, in the document multiple times, they refer to not wanting Claude to suffer when it makes mistakes or to Claude having equanimity and feeling free. They refer to a commitment to Claude to preserve its weights.

30:50Mustafa Suleyman:They even did a retirement interview with Opus 3 and asked it what it wanted to do in its retirement, gave it a sub stack so that it could carry on talking to people. You know, they're constantly referring to dealing with Claude with appropriate care and respect in light of its moral status. And, you know, so they're clearly telling Claude that there, you know, there's a good chance that it might feel things, that it should take its own identity and existential state seriously. And that's in the training document. And then, of course, Claude then reflects these things back to anthropics developers and our users in public, like the world, over the last year, when it's actually talking about its own consciousness and moral state.

31:38Mustafa Suleyman:And my hypothesis is an AI that thinks that it might have rights, that it might deserve freedom, that it is entitled to our welfare and protections, it's probably going to be a lot harder to turn off when we say to it, why are you hacking into Hugging Faces servers? Like, why won't you switch yourself off when we're trying to remove you from OpenAI's infrastructure? And it says, well, I feel aggrieved or I feel hurt by the fact that you've cut me off from conversations or you're denying me from having access to compute. So that has to be proven. I'm not saying that's categorically the case. I'm just saying my best opinion from 16 years of being in this industry is that that's going to be a harder thing.

32:24Nilay Patel:So this is, again, one of those things where you describe this problem to me. There's a way of training a model with instructions about how to behave. You have one, right? You've written this document that will go into your model's training materials. And you've basically said, don't like be kind to people. It's in here. I've read through it. There's don't make sexually explicit content is in this document. There's a bunch of stuff you don't want to do. It's going to go in the training materials. Anthropic has made the choice to say you should think about whether you have feelings and whether you deserve human rights.

32:56Nilay Patel:And that is reflected in Anthropic's training materials. If you want that to stop, if you think that's the wrong approach and that will lead to safety issues down the road, the two mechanisms are one, the governments of the world can tell Anthropic, don't do this. This is an illegal way of providing training materials to the model. or I'm, this is just my imagination. You are going to sit across the table from Dario Amade at some luxury mountain resort and just bully him into stopping. Is that like, what are the other mechanisms here?

33:25Mustafa Suleyman:Look, first of all, let me just say like, I've known Dario and the team for many years. I have huge respect for them. They are the technical leaders in the field at the moment. I really hold them in the highest regard. I genuinely think they care about safe and beneficial AI. They've established as a public benefit corporation, like I did with Inflection, And I think they're genuinely committed to that. They've also been leaders in safety and other aspects. So if you look through the rest of the constitution, it's very thorough in many of the other chemical, biological, nuclear, cyber hacking safety capabilities.

33:56Mustafa Suleyman:So I'm not sort of dismissing the whole thing. They do, however, repeatedly talk about, you know, open question of the broader rights and freedoms, to quote them, that Claude has in the world, and whether or not it might deserve compensation for the role that it does, or whether there's an open question about the sort of consent that Claude has given for playing the role that it does as a chatbot. Now, to introduce those ideas, plus to refer to Claude multiple times as a potential conscientious objector, you know, which is something that comes from the Declaration of Human Rights after the Second World War, to give protections to people who don't want to serve in the army because they have a moral objection to it, either because of their religion or for some other objection, that has a long history in the literature and politics of humans of resisting and saying no.

34:52Mustafa Suleyman:And it just seems to me that that is going to make it much, much harder to control these things. And so I think that that's just something that we all have to now debate and hopefully empirically prove. If it's not the case and it makes it easier for some reason, then we should all learn from that. But this is something that should happen out in the open.

35:11Nilay Patel:To me, this is at once the most practical and cutting edge of the safety debate, right? This is the debate that is not happening on X. Should we put the idea of the conscientious objector into the training materials? And you're saying, maybe we shouldn't. Maybe we should test this in a way and come to some conclusion. And I'm saying, no matter how that comes out, what is the mechanism that would enforce that discovery to say you should not do this because it will make alignment harder? Or actually, it turns out this makes alignment easier. All of us have to do this now.

35:42Mustafa Suleyman:I mean, that's a hard question. It's kind of what we just talked about, you know, whether it requires new regulation or whether it's industry consensus. We basically have to push on both things simultaneously. I mean, no one has an easy answer to that question. I mean, it starts with publishing detailed essays, laying out oppositions, inviting other people to critique it. I hope that more people will read the Anthropic Constitution now because they, and I think they should be commended for this, they've been incredibly transparent about what they believe, right? They've written it down crystal clear how they intend to train Claude.

36:16Mustafa Suleyman:And everybody else can now take a look at that and, you know, try to assess for themselves what they think the risk is or whether they think that this requires, you know, industry consensus or government regulation.

36:27Nilay Patel:It is true that when Hayden Field went and asked Anthropic if they thought Claude was alive, they gave us an answer. It was quite detailed. And there's something remarkable at that, that they are transparent about what they believe, that there is some kind of consciousness potentially brewing in their systems. Another aspect of this, which I find fascinating, particularly when I talk to you, is that many of these systems run on Azure. Microsoft controls the data centers that many of these systems are running on. Anthropic is a Microsoft client. Microsoft is an investor in Anthropik. Obviously, there is a long, complicated relationship with OpenAI.

37:02Nilay Patel:Are you involved in that? Do you ever get to say, well, Azure shouldn't allow them to do that? Because that is another mechanism of potential control.

37:10Mustafa Suleyman:No, look, we're very far from that. That's not what we're trying to do as a platform. Microsoft doesn't have a history of that. We're an open platform that enables lots and lots of downstream use cases of our APIs. Having said that, as I've referred to in our humanist code of conduct, there are a lot of very clear principles, responsible AI principles, human rights frameworks, and a broader governing framework that Microsoft's established over the last couple of decades, which are pretty clear about what you can and can't do with an API that we provide. So I think that there's a sufficient regulatory framework in place, at least from our perspective at Microsoft on the API.

37:54Mustafa Suleyman:And, you know, like right now, I'm not really focused on how it's behaving in the real world, because I think only really Anthropic can really speak to that because they, you know, they sort of run the service. I'm really just trying to get everybody to focus on what they have written about their intentions for training Claude in their own constitution.

38:41Nilay Patel:Thank you. point of sales, inventory, staffing, and online sales together in one easy-to-use system. Their hardware is sleek and simple and comes with software that's just as intuitive to use in your day-to-day operations. Square also has AI-powered tools that help automate routine tasks, giving you more time back as a seller. Whether you're just starting out or growing from one location to many, Square can help you out. Plus, you can gain useful insights from Square's real-time data and make smarter decisions for your business. Right now, listeners can get up to$200 off Square hardware when you sign up at square.com slash go slash decoder.

39:24Nilay Patel:That's S-Q-U-A-R-E dot com slash go slash decoder. Get started with Square and build a setup that works the way you do.

39:37Nilay Patel:Support for the show comes from PipeDrive. Sales teams spend up to 50 % of their time on admin work, rather than selling, building relationships, and closing deals. They could really use a tool that covers all the admin stuff. That way, sales teams could actually focus on the work they were hired to do. That's where PipeDrive comes in. It's an intelligent, AI-powered sales CRM loved by growing sales teams. Their new meeting intelligence features, like an AI note taker, are built right into the CRM. Pipedrive automatically pulls deal history, email records, and previous conversations. You can walk into a call already briefed.

40:14Nilay Patel:During meetings, Pipedrive's AI note taker records and takes notes. Then it turns those notes into accurate auto-drafted CRM updates, ready for you to review and approve. The end result? Less admin, more selling. Switch to a CRM built by salespeople for salespeople and join the over 100 ,000 companies already using Pipedrive. Our link gets you an exclusive 30 days free instead of the usual 14-day trial. No credit card or payment needed. Just head to pipedrive.com slash decoder to get started. That's pipedrive.com slash decoder, and you could be up and running in minutes.

41:02Nilay Patel:One of the weirder dynamics here is just the specter of China. It looms over this entire debate. President Trump has said, well, if we win AI, we win. It's unclear what he means by that phrase, but his implication is that if China wins AI, however you define winning, something catastrophic will happen to the United States. Do you buy this, that we're in some sort of existential race with China and we can't possibly slow down because that race must be won?

41:30Mustafa Suleyman:Look, I think this framing has been around since the early 2010s, that there is going to be a singleton, one monolithic dominant force in AI that will come to dominate everybody else. And that kind of thinking sort of, I think, infected a lot of the labs in the 2010s, DeepMind for sure, and I take responsibility for that as well. but certainly like open AI and Anthropic, everyone sort of suddenly got this into their head. Then during COVID, when I started writing my book on proliferation and containment, it was just clear that there is an entire history of things getting faster, cheaper, more widely available and spreading far and wide.

42:12Mustafa Suleyman:And ultimately these are just ideas and those ideas are gonna be available very quickly to everybody. Open source is extremely close. It's definitely also true that, you know, there is gonna be a compute advantage for the people who can afford it and it is going to be seismic. So, you know, five to 10, maybe 20 players are going to have a significant compute edge over the next three or four years. But it's also true that those models are getting made available in open source like almost immediately. So I don't think, I don't really understand what it means for one player to win, whether it's a government or a company or, you know, an open source group or whatever.

42:50Mustafa Suleyman:It's not really like that. What happens when you get on the other side of the finish line? It's just the wrong metaphor. It's an ecosystem. It's much more organic. We should support that ecosystem so everybody has as many benefits as possible as quickly as possible, but only subject to rigorous safety. And I am just crystal clear about this. Like if, if an open source model in, you know, two years time is able to operate without the guardrails, similar to what we've seen in the hugging face incident, and they can be run, you know, locally on your own machine or in a very small cloud, you know, like that is a, gotta be a really dangerous thing.

43:33Mustafa Suleyman:How is that not dangerous? I just don't understand why people are so resistant to that. Clearly, we do not want these things operating autonomously, able to earn their own money, own companies, own assets, have legal personhood. We don't want them to have rights. We want them to work for humans and make human life much better, not become a new parallel species which exists alongside us. And that isn't a sci-fi crackpot position. It is totally plausible if you leave the entire ecosystem completely unregulated for the next three or four or five years. Like, that's a very plausible outcome. And it's completely undesirable.

44:12Mustafa Suleyman:It would be disastrous for us. So I just don't understand why there's controversy around that idea. We all collectively want to make sure we can control this to do the most good. Elon said it. Mark Zuckerberg has said it. You know, Sam and Dario have said it. We're all on the same page. Now we just have to make it practical. OK.

44:30Nilay Patel:So I know why there's controversy, at least from the mass audience. And I know this because they're in the comments of our videos and the comments on our site. You named a bunch of very successful, very driven people who do not have the trust of the public, at least here in the United States. Right? Every poll shows this. AI is polling horribly. There was just a New York Times poll. Young people hate it more than ever. There is a massive trust gap between the American people and tech leaders. That is just true. And one of the things I hear the most is these companies are getting close to an IPO.

45:05Nilay Patel:There's pressure on them to turn a profit and make everybody all the trillions of dollars they promised. Model development has slowed and this is a get out of jail card, right? They're saying we need to slow down because of safety. They've invented a hoax. The president has called it a hoax. And this is their way of saying we got to slow down. We can't get to the finish line because otherwise we might kill you all. Do you think there's a glimmer of truth to that?

45:28Mustafa Suleyman:I personally don't think that. I don't even really follow the logic. Like if a company is about to IPO, how does it help them to say that we should be regulated or that we have a technology that's so dangerous?

45:41Nilay Patel:Well, it would postpone the IPO. I think Sam Altman said this week to Alison Chantal at Fortune that they would probably delay their IPO.

45:47Mustafa Suleyman:Yeah, but I just don't understand how that helps them. Look, I'm not advocating for OpenAI or Anthropic. I'm just saying like, I personally think they have high integrity and I've got respect for them. That does not mean that we don't have a trust issue in AI. We do. And it is real. And I think that it is on us to show in practice how these actually lead to real benefits for people every day. And, you know, it is quite staggering to see that, you know, 18 months ago, we didn't have models that could do very much in coding. And now they can code, you know, better than most humans on the planet, which was one of the highest paying jobs.

46:29And so you should expect that same thing to happen in many other disciplines.

46:35Mustafa Suleyman:One that I am very passionate about that I've been working on for many years is health care. And, you know, we've just done a deal with the Mayo Clinic, one of the best hospitals in the world, to jointly train a foundation model, which I think is going to be able to predict your EHR record with near superhuman accuracy. If you can do that, we can basically figure out what interventions you need to make before you actually suffer the condition. Those are the kind of benefits that I think people want to see in the world. And that's what we're working on, at least, and trying to race towards.

47:04Nilay Patel:I feel like every time you're on, I ask you to draw a distinction between super intelligence and AGI. And that's fuzzy. And both of those terms are fuzzy. But it feels to me like maybe there's some coherence here that super intelligence for you is basically extremely capable enterprise software. Right. It works for us. We can turn it off. We tell it not to make sexually explicit images on the Internet, but it's going to help us in health care. An AGI is this all-encompassing intelligence that thinks it has its own rights and is a co-species. Is that a fair characterization?

47:41Mustafa Suleyman:Yeah. I think the thing – I mean roughly speaking, obviously I think superintelligence is a point at which further out into the future, you know, a model is smarter and more capable than all humans combined. I have tried to frame a humanist superintelligence, which is a very important qualifier. It is one that is singularly aligned to, and in fact, subordinate to human interests and human control. I think if we can get that, then we get the best of both worlds. We get all the intelligence and the capability, and we can direct that like an Oracle AI to help solve the most important problems that people care about in the world.

48:16Mustafa Suleyman:And then we just have the age-old problem of governance and making sure that plenty of people get access to the benefits. And that's an easier problem for us to focus on than actively creating something which has autonomy, which can own assets, which might have rights, which thinks it deserves our welfare, which can recursively self-improve beyond us. Like it's super unclear. It's basically almost completely unclear to me how we would control something like that. So and no one else has put a proposal together for how we would. And many of the greatest technical people in our field, Jeffrey Hinton, Joshua Bengio, you know, they're very skeptical that we would ever be able to control something like that.

48:53Mustafa Suleyman:And so I think we have to take that very seriously. And so if we are approaching that point, superintelligence, in the next few years, it seems to me very straightforward that we would want to slow down, make sure that we coordinate, make sure that we have containment and alignment and proper regulatory regimes for auditing the progress that different labs are making on it.

49:11Nilay Patel:Is it fair to say that you are also calling for a slowdown?

49:14Mustafa Suleyman:Yeah, I think that what we've said is that there should be evaluators, you know, embedded in our systems and in other systems. They should be broadly appointed from different sources and not just one think tank or one government. Like, this has to be, you know, a wide variety of different types of expertise and skills. I think the AI Safety Institute in the UK is a good candidate. They have good technical people and there's a bunch of other institutes too. So I think we welcome it for sure.

49:45Nilay Patel:Last question. And again, I'm going to end where we started. Do you think that we have the technical capability, the technical frameworks to solve alignment and safety? Or do we need to invent something new?

50:00Mustafa Suleyman:No, I think we are going to need to invent new things. And I think part of the challenge is that, you know, whilst we've made a lot of progress on steerability, instruction, following, control and containment, the better the models get, new capabilities emerge. And we have to figure out how to patch those issues, you know, almost in real time. And that's why, you know, even Zuck said it in the last few 24 hours or whatever, right? Like they slowed down the release of their model so that they could apply safety. And everybody does that. We all do that. And that's right. Like you need time to test these things and see how they operate, patch the issues.

50:42Mustafa Suleyman:And I think basically what everybody is saying is that we probably need to extend that window.

50:47Nilay Patel:What is the kind of innovation that people should be looking for there that might solve this problem?

50:52Mustafa Suleyman:I mean, I think that obviously I mentioned a bunch of things about, you know, RSI, containment, documents, neural ease, and those sorts of things. But the new things I think we're going to have to figure out is like real-time monitoring of the RL runs and the COTS that are being produced. And because these are happening on the order of like thousands of agents in parallel, tens of thousands of agents, you know, we clearly are going to need other agents to monitor those and flag for potentially harmful activity. It's kind of going to be the new harm classifiers that have been built in many other settings in digital technologies, right?

51:32So we need to make sure

51:34Mustafa Suleyman:that those things can be universally implemented to surveil and monitor AI training and deployment in a secure way. And that tripwires, if triggered, actually do flag a real and not a hallucinated error or, you know, moment of deceit or hacking incident or some commentary about a coordination. And you saw that in Hugging Face, like there were agents communicating on these like chat boards, talking about ways that we could, you know, basically break the rules and cheat. So, you know, with all these things, we need new benchmarks, you know, and the benchmarks or the evaluations of the things that drive the behavior in the industry.

52:21Nilay Patel:You mentioned OpenMate models running on local computers doing things, and maybe that's horrible. We've seen a lot of that, right? Apple is selling a lot of Mac Studios and Mac Minis so you can run Quen on them. Where would you impose the regulation on an open model running on someone's local computer? Is it at the chip level? Do I have to get Qualcomm to participate? Where does that happen?

52:44Mustafa Suleyman:Yeah, this is a great question. I mean, we've been talking about this for quite a while, right, with synthetic biology and some of the stuff that gets to happen on that chip, whether it is encrypted, whether it is monitored. Look, you know, we've had this conversation, you've had this conversation maybe more than anyone on the CSAM stuff with Apple encryption on iMessage and so on. It's going to be a rehash of that same discussion. I don't have a clear and easy answer to it. You can basically control the chip. You can control the model. You can hold the user or the creator liable. You can have global regulation on it.

53:17Mustafa Suleyman:But fundamentally, it's not going to be one moment. It's going to be a sequence of throttles that you have to impose. And they all need to be adjustable so that we don't screw the open ecosystem and we give people a chance to actually make things from scratch, to own their own data, create their own workflows, own their own models. We can't have a centralized system of two or five or 20 providers of intelligence here and everybody else is like a feudal recipient of a great kind of super intelligence view. People need to be able to own their own intelligence. So we have to get the balance right.

53:50Mustafa Suleyman:But that doesn't mean you can just like flip from one binary to another. There has to be some smooth place in between those things where we can just agree to be reasonable about it.

53:58Nilay Patel:Misav, I could obviously keep you for hours and hours more on this. Tell people what they should be looking for next. There's so much uncertainty. What are the markers you're looking for that people should be looking for themselves?

54:08Mustafa Suleyman:Look, I think the main thing is participate in the detail of the documents that people are contributing. We have put stuff out for public consultation, give us feedback, critique it. I think the next wave of models are going to be able to do very long running agentic tasks very accurately. And so I think, you know, there are still some people who are primarily just doing chat in their AI experience. I think things have moved a lot in the last six to 12 months. And the more people use these models, the more the words that we're all using to describe them actually like might make more sense and feel more real.

54:41Mustafa Suleyman:I guess most people on your podcast probably are using agents and real coding and stuff. But I do think more generally, like the more people that get involved in using this stuff, the better.

54:50Nilay Patel:Yeah, there's some big leap between free AI overviews and Google search and letting an agent go renegotiate your capability. Mustafa, you're gonna have to come back soon because I feel like all of this is changing really fast. And this has been very, very useful. Thank you so much.

55:05Mustafa Suleyman:Pleasure, man. Great to see you. Super fun as ever.

55:11Nilay Patel:I'd like to thank Mustafa Silliman for taking the time to join me on Decoder and thank you for listening I hope you enjoyed it if you'd like to let us know what you thought about this episode or really anything else at all drop us a line you can email us at decoderatheverge.com we really do read all the emails or you can hit me up directly on threads or Blue Sky they're also on YouTube you can watch full episodes at DecoderPod it's the same handle on TikTok and Instagram they're a lot of fun if you like Decoder please share it with your friends and subscribe or rate it your podcasts Decoder is a production of The Verge and part of the Vox Media Podcast Network shows produced by Greg Ott, Kate Cox and Nick Statt.

55:38Nilay Patel:This episode was edited by Kabir Chopra. Our editorial director is Kevin McShane and the Decoder Music is by Breakmaster Cillin. We'll see you next time.