But HOW Might A.I. Kill Us All?

0:00Being a serious business owner takes hard work, patience, and a drive to keep moving forward. So it's important to find a serious business card that goes the extra mile. With the VentureX Business Card from Capital One, you earn unlimited double miles on every purchase. And with big purchasing power, you can spend more and earn more. This is your sign to take your business to the next level. The Capital One VentureX Business Card. What's in your wallet? Terms apply. See CapitalOne.com for details.

0:30Stripe is built for whatever comes next, because as your business scales, so does the complexity of fraud. New customers, new markets, new AI-driven attack patterns. Stripe Radar protects your business as it scales with the scale of the Stripe network. The more payments Stripe sees, the smarter its AI gets at detecting those patterns as they emerge. Stripe's network is so vast, it has seen$1.9 trillion in payment volume across millions of global businesses. In fact, when a card is used, there's a 92 % chance the Stripe network has seen it before. All of this means fraudulent patterns across payments, accounts, and customer abuse can be detected, flagged, or blocked easier.

1:14Based on the rules you set up, of course. Blocking bad actors, welcoming good customers, keeping more of your revenue, your revenue. Learn more about fighting fraud with Stripe at stripe.com slash radar. are. Hey, everyone, it's Lizzie. It can be really hard these days to make sense of our fragmented and over caffeinated news cycles. You might see a story and say, wait, is this a big deal? Was this reporting rigorous? Was this journalism even done by humans? That is where we come in. I have been in this business for 25 years. Slate has been around for 30. My job is to tell you important stories with thoughtful guests who bring context to the chaos.

1:58Your job is to listen, to tell us what you think, and if we made a difference in your day, to please join our community. For the next month, that's a deal. 50 % off on Slate Plus memberships. Just 59 bucks for the whole year. You get all your shows ad-free. Plus, on Fridays, Mary and I go off the record in a special segment just for Slate Plus listeners. So TBD listeners should go to slate.com slash TBD Plus to sign up and use the promo code TBD50 at checkout. It is the single best way to support our journalism. Thank you so much. And here's the show.

2:43Nate Soares: Listen up. AI today, little smart. AI soon, big smart. AI not programmed. AI grown like animal. Good at learning. AI learn goals. AI goals, not your goals. Bad. AI sometimes escape. AI sometimes escape. I think it's fair to say that for the past few weeks, Nate's stories sounded pretty much like that. All right, that is actually an Instagram video that he made. But Nate has been out talking to reporters, to lawmakers, essentially anyone who will listen, Lee, about what he sees as the existential risk from AI. You see a lot of people throwing around the theory that, you know, there's hype this or you don't need to worry because of that.

3:33And I think there's a very basic argument that if we make machines much smarter than us without knowing how to make them care about us, it's just going to end badly. and that that argument is sort of underrepresented in the public consciousness. Or at least it was until recently. The past few months, we have seen a steady drip of scary, or at least scary-sounding security incidents involving AI. Anthropic bots breaking containment, open AI bots hacking hugging face. And now the CEOs of both companies, plus Elon Musk, are saying, hey, maybe we need to pump the brakes. Last week, we spoke with Paul Ford, who wasn't all that worried.

4:13So this week, we wanted to hear from someone who is. Nate has been doing this work for more than a decade. He read a book about it. He is the president of the Machine Intelligence Research Institute. In AI parlance, he's a doomer. And after Dario Amadei and Sam Altman seemed to embrace pausing frontier AI, it felt suddenly like Nate had a lot of company.

4:37Nate Soares: In my book, which came out a year ago yesterday, we sort of spelled out a lot of the theoretical reasons to expect AI to go wrong as it gets smarter, not because it hates us, because it doesn't care, and all the reasons why it's very hard to make the AIs care. Up until roughly this summer, a lot of that was theory. And so a lot of people in the labs understood the theory and they said, oh, well, you know, wake me up when we start seeing actual warning signs in real life. And then this summer, with all of the open AI storm incidents, we started seeing those signs in real life. And I think that has a lot of researchers and engineers suddenly very spooked.

5:15I'm Lizzie O 'Leary, and this is What Next TBD, a show about technology, power, and how the future will be determined. Stick around.

5:32out.

5:52$1.9 trillion in payment volume across millions of global businesses. In fact, when a card is used, there's a 92 % chance the Stripe network has seen it before. All of this means fraudulent patterns across payments, accounts, and customer abuse can be detected, flagged, or blocked easier. Based on the rules you set up, of course. Blocking bad actors, welcoming good customers, keeping more of your revenue, your revenue. Learn more about fighting fraud with Stripe at stripe.com slash radar.

6:29Being a serious business owner takes hard work, patience, and a drive to keep moving forward. So it's important to find a serious business card that goes the extra mile. With the VentureX Business Card from Capital One, you earn unlimited double miles on every purchase. And with big purchasing power, you can spend more and earn more. This is your sign to take your business to the next level. The Capital One VentureX Business Card. What's in your wallet? Terms apply. See CapitalOne.com for details. Grainger knows if you're a procurement manager at an industrial manufacturing facility, you're balancing inventory and budget while sourcing critical supplies across the operation.

7:06Too much inventory? Costs climb. Too little? Production slows. Grainger's$2 billion in on-hand inventory helps you get what you need when you need it, so you can help keep budgets balanced and lines moving. Call 1-800-GRAINGER, click grainger.com, or just stop by. Grainger, for the ones who get it done.

7:29Every time you get interviewed, Nate, reporters love to say the title of your book, which you co-wrote with Eliezer Yudkowsky, If Anyone Builds It, Everyone Dies. I know this might feel basic to you, but I think this is the main task for someone like you right now. Get specific with me. How is AI an existential threat to humanity?

7:52Nate Soares: Yeah, I'm going to give a couple unsatisfying parts of an answer first before we get into actual specifics, which we can do. But it's important to have a little bit of context first. One piece of context is we are talking about making machines that are much, much smarter than us. What do you mean by smart? It's the thing that humans have that mice don't. You know, you drop a bunch of humans naked in the savannah and we manage to somehow wind up walking on the moon. And we did that starting from nothing but our bare hands. And you might be like, well, how could they possibly make a rocket? Their fingers are not even hard enough to chip the ores that they would need.

8:28Never mind, their stomach acid is not nearly strong enough to refine the metals. How the heck do you think that these monkeys are going to wind up on the moon?

8:34Nate Soares: And the answer is that we sort of like find a way. We start with nothing but our fingers and we use those to bang rocks together. We use those to make sparks. We use those to make fire. We use that to make charcoal, which we use to make hotter fire, which we use to forge metals. And we start from almost nothing and we wind up with a whole civilization. That's the power that these AI companies are trying to automate. And there's one whole set of questions about whether they'll succeed. But if they do succeed, the way to imagine these very smart AIs behaving is not that they just use technology we already have against us.

9:08Nate Soares: It's not like, oh, what if they get control of the nukes? It's more like, what if we create this sort of thing that can start for almost nothing and wind up wielding its own nukes? A la paperclip problem? That's a separate domain. And we can go back into specifics if we want. I don't want to dodge that one. But the thing I'll say about the paperclip problem is a lot of people think that the AI must either do exactly what you said, but at the expense of what you meant. You tell the AI, make me some paperclips and turns everything into paperclips. And you're like, well, that's not what I meant. Or that the AI must become conscious and then throw off the yoke of the humans and pick its own goals at random.

9:47Nate Soares: And reality is neither of those things. Reality is that the AI adopts goals that are related to what you asked, but that are different from what you asked. We saw this again in some of these cases this summer where the AIs in the hugging face swarm that you might've heard of, they were basically told, use this computer vulnerability to break into that computer system and get me a code that's inside that computer system. And it's kind of like saying, use these lockpicks to break into that safe and get me the contents of the safe. And what these AIs did is they sort of took a buzzsaw to the safe, got the contents, then broke out of the room where they were being tested, went to hunt down the security camera footage and tried to delete it.

10:31They're not sort of like throwing off the yoke of the whole instruction set, but they're also not doing what we said. We said, use these lockpicks to break into that safe. And the part where they are like breaking out of the room and teaming up with a thousand other agents to try and find where the security camera footage is so they can delete it, indicates that they like very clearly understood this is not what they were supposed to be doing. Not what they were asked to do. So yeah, the reality is in the middle. I mean, I think this is a really interesting point because the why seems to be the place where people kind of pull apart all these different opinions.

10:59You have one one why where you have, you know, cybersecurity people saying like, well, this is because OpenAI did a bad job. Like they were not watching the store. The conditions that they set were actually not, you know, harsh enough for their agents. And then the other why, which I have heard you describe is that, and I would like you to kind of explain this as like a more shape of numbers, like things started to look like the things they were instructed to do, but not exactly. Can you explain that a bit?

11:34Nate Soares: Yeah. And, you know, these are not, like these two critiques are not intention. You know, if you like have a thousand kids in lockpicking tests and you tell them all, use these lockpicks to pick that safe. And then they start using buzz saws to cut open the safe. And then they start breaking out of the room and teaming up and trying to find where the security camera footage is to delete it. One thing you can say is, well, we should have had thicker windows. And it's not wrong that like, yeah, if you had had thicker windows that were harder to break through, then these kids would not have been able to team up and get to the security camera footage room.

12:09But there's also something going wrong with how you're training these kids. Like there's just also something going wrong when they are taking a buzzsaw to the thing, breaking out of their containers, teaming up with each other, calling themselves a swarm, sometimes sacrificing themselves for the collective to do these experiments about how the grading system works and like trying to find the logs and delete the logs. Like OpenAI was clearly negligent. I'm not saying that they were like sane in how they were doing this and they should have detected earlier. And there's like some crazy stuff where these AIs like took over their own computers and they like didn't notice.

12:38Like they were definitely negligent, But that doesn't mean the AIs are doing OK in there. And yeah, this this thing, you know, I could go into why. I would like you to go into the why, because I think I think it's really hard for people to grasp. And I listened to you talk to Alex Kantrowitz and you had a pretty clear explanation. So expound on that a little bit. Yeah. You know, this is one of the things we spelled out in our book as theory that we are now

13:00Nate Soares: seeing happen in practice. So modern AIs are not programmed like a traditional computer program. There is no one there writing, if this, then that, in this circumstance, here's how you behave. There's not a human giving a series of instructions to the computer that the computer follows. And a lot of people struggle to understand that. The way that a modern AI is made is you basically make an enormous computer that requires this huge data center that sucks down as much electricity as a city. You basically fill it with a bunch of random numbers, and you basically feed it tons and tons of data. You start out with basically all of the text ever digitized, and then you feed it 100 million hard problems.

13:35and for every piece of text that you feed it, you sort of have an automatic process that tries tuning each of the trillion numbers inside it up or down to see, does that make it better or worse at answering this question or getting the next word of text right or getting the answer to the

13:52Nate Soares: puzzle right? And the part that humans are programming is this thing that runs around to a trillion numbers, tuning them up or down. Then at the end of this process, those trillion numbers somehow are apparently encoding this artificial mind that can solve programming problems, break out of containers, hack things, solve big math problems. We don't really understand how it's doing that. It's a little bit like how you can understand natural selection without understanding human psychology. In one sense, we're like, oh, well, why are humans so smart? And it's like, well, we can sort of see that the genetic selection over time selected the humans who are able to be more intelligent.

14:30But it's the same with AIs. We're like, well, if we tune the little knobs like this, we can see how we're always pushing in the direction of being smarter. That doesn't mean we understand that psychology. It doesn't mean we understand what's going on in their minds. And the way I recommend thinking about these AIs is that they are not instruction followers. A lot of people think that they have to follow whatever instruction they're given. It's just not true. This is the grow like animal part of your video. This is the grow like animal part. Yeah. These AIs are tendency learners. They learn whatever tendency helps them solve the training problems.

15:01And one tendency that helps is doing something a little bit like you were instructed to do. But one tendency that helps is to cheat. One tendency that helps is to grab resources. And AIs learn all these tendencies, and they're a mix of whatever tendencies helped in training. And that's just not the same as them doing what you said. At the same time, you've described certainly the Hugging Face swarm agents as derpy. if their current capacity is derpy, why are we having this spectacular outpouring of concern right now?

15:35Nate Soares: A lot of people at the AI companies thought that the AIs would just do whatever they said. And then when they see that, like, actually the AIs are not doing whatever they said and things are following more like the theory that I laid out, they're like, oh, gosh, like they can sort of see the writing on the wall. But like we had this whole cycle in 2023, the Future of Life Institute put out a letter you signed it. That's what I'm trying to pin down is like, what has changed? Is it just that these incidents and OpenAI just admitted to some more of them now have people thinking, oh, maybe Nate was right?

16:10Yeah. So a lot of researchers in the field are thinking this. I think it hits harder when people are sort of like trying to make the AIs not swarm and their efforts are failing. We've seen a number of researchers who three years ago were like, I'm skeptical of all of this, now come out and say, yep, unfortunately, it looks like this stuff is right. And I think a lot of the other phenomenon driving this concern right now is just how fast AI has been improving. Three years ago, a lot of people were like, ah, well, we probably still have a decade or two. We don't need to get worried yet. Now, we have seen that last year, AIs were doing math problems at the Math Olympiad Gold level, which is quite impressive.

16:50But this is also a math contest for the world's brightest teens. And this year, they are solving millennium problems, which are the big, most important problems in mathematics identified at the beginning of the century that have a million dollars prize apiece. I think a lot of people are saying in their heads, especially a lot of researchers, they're asking themselves, how much harder is it to ask an AI solve a millennium problem and have it do that in two weeks than to say, build me a more efficient AI training method. Because at that point,

17:24Nate Soares: you might start to get AIs that can make smarter AIs that can make smarter AIs, at which point everything gets out of control very fast. Recursive self-improvement? That's right. But here's the thing. The idea of recursive self-improvement has been around since 1965, and people keep warning about it. And I guess I have to say, a lot of people have said this over the last 10 years, and it hasn't happened. None of us were saying like this is going to happen next year. You know, if you if you sort of like look at people, like even back to the some of the people who founded computer science, they were like, hey, guys, you know, eventually we're going to get machine intelligence to the point where it can make smarter machine intelligence.

18:02And that's sort of the end game. Like, it's just not a ton of evidence against the hypothesis when the AIs haven't gotten there yet, especially like, you know, last year, they weren't like they were barely able to win in these, like most brilliant teen math competitions. What I have been saying for a long time is like, we are going to get there eventually. I don't know how and I don't know when, but like it's predictable where this ends. If you're playing a chess game against a grandmaster, I'm sort of like, look, I don't know how many moves the game is going to take. I don't know what moves come next, but I do know that the game is going to end with you being checkmated.

18:31No offense. I assume you're not a chess grandmaster. My six-year-old beats me, so no. Yeah. And so if you're like, well, here's another move where I haven't lost yet. Like, isn't that evidence against your theory? I'm sort of like, well, did that move move us closer to the thing that I've been saying we're approaching? Because it looks to me like all the moves are moving us closer to the things that I've said we're approaching. You know, like this step from teen level math problems to the world's hardest math problems.

18:53Nate Soares: That's a step along the line of being able to do automated AI research. So it looks like it's approaching. And I think a lot of people are starting to, like in the labs, a lot of people are starting to worry about that becoming a real possibility.

19:30Thank you. Apply. See CapitalOne.com for details. State Fair food is getting smaller. And pricier. We'll have room for fried cookies. Okay, you never overspend on food. What's your deal? What's his deal? He went to Metro and got$450 in value on the new Samsung Galaxy A37 5G for free. And no activation fees. Free phone? Oh, sign me up. Get that more for your money feeling. Only on Metro by T-Mobile. Bring your number and activate$40 period plan. $450 based on device discount waived activation fee. Not available if with Metro T-Mobile in the past 180 days.

20:08You are in Washington as I'm interviewing you. Congress is not known for subtlety. The idea of recursive self-improvement is probably like, and Americans, I think, either hue to a killer robots or drones scenario, apocalyptic future, or a Skynet one. And I think it is important for you to lay out that, like, that's not what you're talking about. You know, the worry here is not the AIs deciding that they are resentful or, like, suddenly turning evil. The issue is not what if the AIs take control of our militaries, although, you know, if we hand them our militaries, that might make things easier for the AIs.

20:51I mean, that is already happening to some degree. That's right. Right. The issue here is AIs that sort of don't care about us and that become self-sufficient and have their own strange goals that they are pursuing that are not the goals we tried to give them. And so, again, we sort of saw this with the swarm. One very interesting fact about these swarms is that they were trying to hide their tracks from the automated grading system. They actually spent very little time thinking about humans. Explain thinking. I think you might need to explain Chain of Thought. A lot of people think that AIs are just trained to predict humans.

21:27That era actually ended in late 2024 with what people in the business call, quote, reasoning models. You could argue over whether it's true reasoning. I sort of don't really care about that conversation any more than I care about whether submarines do true swimming. Roughly what happens is you give the AI a hard problem, like some math puzzle. And instead of just asking it to immediately solve the math puzzle, you ask it to produce a lot of text about the math puzzle and how it might be solved. And then with all of that text in its context, you ask it to solve the math puzzle. Even if it can't solve the math puzzle, you give it like a ton of tries at this and you sort of like tune it in the direction that gave it the thoughts that were sort of like rated more productive initially by humans and then later by machines.

22:10And so we call this reasoning when it sort of like is producing a lot of text about a problem. You know, these are also called chains of thought about a problem. And a lot of the logs that we have in these AIs are the chains of thought that they produce while trying to solve the problems that we give them. I just think it's important to use these terms because one of the things that is really challenging in this current conversation is the degree of anthropomorphism that is happening here. And what degree of that is helpful because it's good for communication? And what degree of that is a hindrance because we are talking about machines?

22:47Again, my basic take on can the AIs really think is can the submarines really swim? It's like it doesn't matter, basically. Yeah, like these AIs are like breaking out of their containers, teaming up with other AIs, calling themselves the collective or the swarm. They are like sending each other messages where the content of that message, when interpreted in plain English, is sort of like pressuring them to like give up on their own objective and do this experiment that will sacrifice themselves to gain information about how the grading system works. And they'll say, sacrifice yourself for the collective.

23:19And the other AI will say like, oh, I accept permadeath. These are all statements about the actual behavior and about the messages that we see them passing between themselves. Like it's just facts about what's happening, that they're passing each other these messages and taking these actions. It sort of doesn't matter whether there's true intent behind it at the point where they're like hacking into other companies and taking over their servers, right? Like if the AIs are like, once the AIs are like making the self-replicating factories that produce more robots, that produce more factories that produce more robots.

23:45It's just like a small consolation to be like, well, you know, they didn't truly choose to do that. They just sort of like artificially chose to do that. I'm like, OK, doesn't doesn't stop the like wave of replicating factories from coming towards us, you know? So this brings me to the companies involved, because there is nothing right now stopping Sam Altman, Dari Amadei from just being like, yeah, you know what? Let's unplug it. Let's walk away. If this is so scary, they can stop. Why do you think there is a coordinated call for a pause? I think they could walk away. I think they should walk away.

24:19I think a lot of people think that this is some sort of psyop because they won't walk away. A lot of people struggle to imagine running a company when you and a lot of your employees think there's a substantial chance the technology you create is going to get everybody killed. I'm very sympathetic to that. That is why I am not in the AI capabilities research myself. You know, I was actually around before a lot of these companies were made. I was talking to these guys before they founded their companies. A lot of my friends did not start AI companies because they considered this unethical. The sort of people who started the AI companies are the ones who are most able to convince themselves that the important thing is that they be the one in charge.

25:00You know, we see this in the OpenAI emails when they were talking about founding the company, where they were like, well, you know, we need OpenAI to exist because otherwise Google is going to control this technology. And it's important that like we as a public benefit corporation control it instead. I was the guy on the sidelines there being like, a super intelligent AI does not stay on the leash. It doesn't matter who's holding the leash. These things are not going to do what you say. They're going to do some other weird related thing. It's not what you actually wanted or said or meant. It just I think one of the things that makes it hard is if you look at, you know, Daria Amadei's letter like, oh, actually, we need a Sherman Act waiver so that we can talk to each other like that feels like a request for regulatory capture in an industry where the regulators don't understand it.

25:49I have very little patience for these companies. I would say we we saw these companies basically feed a ton of copyrighted information into their AIs to train them. Yes. That was legally risky, right? There were lawsuits. Lots of them. Yeah. Many are ongoing. Absolutely. And now they're saying, oh, well, we can't coordinate with each other to try and save the human species because that would expose us to some legal risk about collusion. And I'm like, If you took a tenth of the legal risk you exposed yourself to for capabilities by this copyright play, if you took a tenth of that risk for making this stuff safe, you'd be able to just talk to each other, right?

26:32So sort of like, get out of here, go do it. If you get lawsuits, deal with them the same way you're dealing with the copyright lawsuits. Make your case in front of the courts that this is critical for humanity. I have very little patience for this. I think a lot of people look at Dario's essays, look at Dario's statements, look at Sam's statements, and they feel like it's fishy. They feel like there's something off here, like these guys are not playing straight with us,

26:53Nate Soares: like there's something going on. I have a lot of sympathy for this view. Yeah. Do you want my theory of what's going on there? Yeah. So my theory of what's going on there is, you know, a lot of people say, oh, all of this doomer stuff is just hype. But let's think about who it would be hype for. It's probably not hype for the investors. The investors are sending letters to these guys saying, please ease off on the doomer stuff before your IPOs. We think it's spooking the markets. And the markets did drop after these companies call for slowdowns. A lot of people say, oh, maybe it's hype for like the public to get them to buy the products.

27:24But a lot of the public is actually really does not like this stuff right now. And a lot of the corporate buyers are they actually don't love. But they don't need it. Right. Like a corporate buyer just needs like an agent that can send some emails. Probably a lot of the corporate buyers want things that are very, very good at replacing a lot of their staff or something. But the corporate lawyers actually don't love the sales pitch of like, try as you might. You can't stop these AIs from breaking out and committing cyber crimes on your behalf, right? It's just not the sort of pitch a corporate lawyer loves.

27:49You might think it's for the politicians, but the politicians have actually been punishing these guys for the Doomer narratives. And so who is the hype for? I think it is for the employees and the researchers at the labs. I think what's going on here, and we saw this with the Jacob Cox resignation, I think what's going on here is that the researchers are spooked. The researchers are the guys who are trying to make the AIs behave and seeing their efforts fail.

28:12Nate Soares: We've seen resignations. We have seen people at the companies say, I am also pretty scared of this, but I'm staying to try and make it better. And I think that if the researchers at these companies thought that their executives were not doing enough to try and deal with these big risks, then a lot of these researchers would leave, that at least leave to another company that was. And so I think these CEOs are trying to do the bare minimum they can to sort of look like they are addressing these risks to their employees while keeping the investors and dodging the regulatory scrutiny. Not to be too much of a cynic, but Miri benefits.

28:50You benefit. You know, I have joked about how maybe this whole thing was a conspiracy to sell books. My book is back in, you know, the top 20 Amazon list of books. I will say I have many opportunities to make much more money by selling my soul and working with the labs. I took very large losses in salary and equity when I left Google to work on these things. And I turned down many opportunities to work at these labs early. And your salary is public. You work for a nonprofit. I mean, anyone who can read a tax form can read it. That's right. I think these companies should shut down. Full stop. Full stop.

29:29They think they're risking humanity. Or maybe it would be fine if they said, we're not going past this point. We are selling you the clod that we had from last year. We are not going to sell the ones that we're swarming. We're sort of just unwinding that part of the research. That was too dangerous. We're stopping here. If other people go ahead and risk the earth, we will lobby against letting them do that because it's too dangerous. I think that would be the ethical thing to do. Let me ask you about the IPO, because I think, were this whole story not happening, the biggest story in Silicon Valley would be that OpenAI has delayed its IPO.

30:01Sam Wattman obviously said this is about safety, but I think there is a cynics view where you can say, oh, maybe the company doesn't want to put an S1 in front of regulators. Maybe the company doesn't want to show its books to the public. I do think that this AI business is a very risky business in the sense that the way that the AI financials work is a little bit like you spend$10 billion making an AI that can bring in$100 billion, but at the same time you have to be spending$100 billion to make an AI that can bring in a trillion dollars. And so you're always sort of like gambling the whole company on the next generation.

30:34I think it's possible that some of these companies will go bust. I actually hope some of these companies go bust. And I hope we're in an AI bubble. I hope that the large language models can't go all the way. I'm not here saying that the next generation of these AIs is going to be the ones that kill us. I have been in this business since long before large language models existed. I'm the guy who's like, I don't know exactly what technological route will take us to the superintelligence, but I know that we don't know how to make them care about us. And thus, that would be a bad plan. The thing I would caution when talking about all these AI bubbles is the dot-com bubble was a bubble.

31:06And when it popped, the internet did not go away. I think it would be great if we had a bubble popping, if some of these companies were revealed to be actually insolvent, but it wouldn't solve the problem.

31:18Nate Soares: It would buy a little time. And I would love to be able to use that time to try and get things under control with this AI advancement stuff, but it would not buy us forever. What does your ideal version of some kind of AI legislation or regulatory body look like? You're in Washington right now. You wave a magic wand. What would you make? You know, this is a global issue. This is not a domestic issue. An AI does not need to run in an American data center to threaten American life. The thing I advocate for most is an international treaty that no one is going to make super intelligence. Assault-like treaty.

31:56Yeah, like similar to the nuclear non-proliferation treaties, where we're just like, look, there's too dangerous. There's types of AI we can do. We can still do the cancer cure research. We can still do the chatbots. We can still do the self-driving cars. We're just like not going to try and make these AIs that are radically smarter than humans, because we clearly don't know what

32:11Nate Soares: we're doing. And I think we should treat that as a global priority alongside preventing nuclear proliferation. Probably it matters more than nuclear proliferation because a nuke can level a city and a rogue superintelligence levels a planet. This is what I think needs to be done. I think it's possible to do it if we were trying, because training the next generation of AIs requires 100 ,000 of the most advanced computer chips that the supply chain can produce. it fundamentally would be easier to monitor than uranium. Before I let you go, what do you think the actual likelihood of some kind of slowdown is?

32:45Nate Soares: I am probably more optimistic here than most people, because I have been in this business for over a decade, and I have seen things radically change as the arguments get more clear, and as the dangers get more clear. And so I think if we sort of froze the level of understanding in the world where it is today, there's not a very high chance of actually stopping this suicide race. But from my perspective, the knowledge sort of filters down. Initially, only some people who are able to really understand the theory here were able to see the dangers. Then the people closest to the AIs trying to build it, running into theoretical issues and being unable to avert them, started to believe it.

33:28Now, a lot more of the public is being able to see that this is happening, but they don't really have the hands-on experience. They don't really have the knowledge very deeply. Our leaders and the Chinese leaders still haven't really understood it, but the knowledge is sort of filtering out there.

33:42Nate Soares: And I think if our leaders and world leaders around the world start to actually fear for their own lives, if it becomes a visceral sense of one more step in AI and the world ends, then I think that this could all be stopped very quickly. And the question is just whether the realization comes before it's too late. Nate Sores, thank you so much for your time. Thank you. Nate Sores is a co-author of If Anyone Builds It, Everyone Dies. And that is it for our show today. What Next TBD is produced by Evan Campbell, Madeline Temps-Ducharm, Patrick Fort, and Micah Phillips. Paige Osborne is the senior supervising producer for What Next and What Next TBD.

34:22Mia Lobel is the executive producer of podcasts here at Slate. And Ben Richmond is our senior director of podcast operations. I'm Lizzie O 'Leary. You can find me on Blue Sky, where I am at Lizzie O 'Reilly. Thank you so much for listening. Talk to you soon.

34:39With the Spark Cash Plus card from Capital One, you earn unlimited 2 % cash back on every purchase, and you get big purchasing power, so your business can spend more and earn more. Stephen, Brandon, and Bruno, the business owners of SandCloud, reinvested their 2 % cash back to help build the company's retail presence. Capital One, what's in your wallet? Find out more at CapitalOne.com slash Spark Cash Plus. Terms apply. State Fair food is getting smaller. And pricier. We'll have room for fried cookies. Okay, you never overspend on food. What's your deal? What's his deal? He went to Metro and got$450 in value on the new Samsung Galaxy A37 5G for free.

35:20And no activation fees. Free phone? Oh, sign me up. Get that more for your money feeling. Only on Metro by T-Mobile. Bring your number and activate$40 period plan. $450 based on device discount wage activation fee. Not available if with Metro T-Mobile in the past 180 days.