9/14/26: Anthropic Calls To Slow Pace of AI, Nate Soares AI Regulation

25:12proceed. So while we're all debating what's going on in their minds and why now and blah, blah, blah, blah, the most important thing to keep in mind is what we, the public, want and to not be confused by whatever games they may be playing behind the scenes. I totally agree. And I also, you know, I look very, Ross, by the way, made his 60 Minutes debut and gave a very compelling monologue. I recommend people watch it just about. He's like, look, we have one year, not 10. This is the most important question of all civilization. And this is, so I'm just giving you the fact that this is coming from all sides.

25:45Also just saw news this morning, Jacob Cox and the Anthropic Whistleblower will be doing an event here in Washington tomorrow with Steve Bannon and Bernie Sanders calling for complete shutdown of AI. So that'll be interesting. Jacob, you know, if you're listening, you're going to be in town. You got to come on the show. But the interesting thing I keep coming back to for all of their actions is whether any of this is a bluff or not, their actions themselves, and especially now the rhetoric and what they've been able to accomplish while covering up, like you said, that's the important part. Hugging face, they were brought kicking and screaming.

26:24There have been three other hugging faces that just got no attention. Again, the WeChat worm, it's so annoying to me because you can imagine if it had been an iMessage worm, you know what the global freakout would have been. More people use WeChat than iMessage. And they were able to create a worm which breaches your phone no click. I mean, that's worse than Pegasus. That's like all they need is your phone number at that point. That's worse than the Israeli Mossad spy app that they use right now to crack Jeff Bezos and all these other people's phones. All they need is your phone number. They can send you an email.

26:57You're done. The entire phone is compromised. That was the entire Chinese system, basically, of their... And remember, we're not talking about just text message. We're talking about their financial architecture, everything that they use to pay over there. So that was, in my opinion, the most extraordinary breach that they were able to show. They've had multiple agent things that have gotten completely out of control. From a regulatory perspective, this is where Dario and I just totally disagree. Dario, this is part of their EA rationalist theory. he believes in effectively democratic peace theory which is like this bunk 1990s theorem where democracies don't go to war with each other i'm like dude your sample size is very like and he's like oh i think democracies are the best at being able to control this i'm like i actually think it's the opposite uh if you go and take a look at china because our system and not just our system all democratic systems basically across the west are so you know are basically like compromise because compromise is the ultimate thing that they fall on, which is good, you know, in the long run.

28:04But in this particular instance, when we're dealing with technological progress, it's so obvious that technological progress so vastly outpaces our parliamentary system or the British parliamentary system. And that the authoritarian system is the one that's most capable, again, just dealing with technology. So in Dario's mind, he believes that we need some like democratic peace accord. And I'm like, dude, if you study the history of nuclear weapons, think about how long it took to get some sort of like remote architecture in place for IEA inspection. And then when India, Pakistan comes along, it's like, oh, it's actually all fake.

28:40Is that if you just create the nuke, eventually the international community is like, all right, you know, whatever. Yeah, exactly. What are we going to do? Yeah. I mean, I think the problem here is pretty obvious, which is that we have long ago outsourced any sort of regulation to the market. So, you know, that's why there's there is effectively no regulation on these frontier labs. They can do what they want. And we are at all of their mercy. And so, you know, I mean, I think if you just look, I'm not going to make any grand statements about democracy versus autocracy or authoritarian systems, because I think what we have is very loosely a democracy at this point.

29:16It's really like money that rules. But I think if you look at the U.S. right now and you look at China, you can see which country has been far more responsible in terms of technological development. I don't think there's any question about it. So the idea that, oh, it will be fine and good and totally safe if it's the U.S. that wins the AI race feels, I think, to many people, myself included, to be completely preposterous. Look, the reality is that this is a global technology. We do have to find a way to have agreements specifically with China right now, but they may not be ultimately the only country that ends up at the forefront of this technological development.

29:53You are going to have to have some sort of a global standard and tracking, and that is going to be very difficult. But it has been done in the past, and that has to be what is ultimately pushed for. This zero-sum mindset, Cold War mindset that China is rightly calling out is ultimately, I think, very, very destructive. and based on, you know, based on unreality in terms of the actual dynamics of the U.S. and the actual dynamics of what China wants and what they are pursuing right now. That's my last thing here. You know, look, I don't think it's deniable. I was a China hawk for many, many years.

30:28You might have noticed a shift. And the truth is, is that we shifted. Like, there is no universe now. If you take a look at our current—look at Iran. Look at what's happened with the Strait of Hormuz and our munitions. We have no choice at this point. We genuinely don't. Like the era, I think we did have a chance. I think we had eight years where it was on the table. And I'm not talking about beating China. We could have reshored a lot of our manufacturing, some of the chips at EV development. There was a chance we could have banned TikTok in 2018. It developed our—like we actually could have made ourselves, I would say, pretty autarkic.

31:06That is over, at least in my opinion, considering what has happened with the chip technology, with the munition system, some sort of detente is basically the, I think of ourselves in like a similar 1970s moment, broken by Vietnam, broken now by Iran. We're going to need 10, 15 years of just like, we got to be humble. We got to understand exactly who we are. We got to move forward and kind of coexist within the world. It's not comfortable. It's not nice, you know, necessarily, but that's just reality. And so that's the way that I'm thinking through this AI problem. And I mean, you could look back at detente for very similar examples of pursuing defense treaties to limit nuclear weapons and all of the other different engagement, not necessarily, you know, being on your high horse all the time, talking about evil empires and all of this other nonsense.

31:52So I just think that that's the reality now of our international situation. I blame especially Trump and the Iran situation for putting us in that. And that's it. Like, to me, it's just over. You can see that where China is like, sure, we'll talk to you about AI when you meet with Trump or when Xi meets with Trump. You're not selling any arms to Taiwan. And Trump's like, OK. I mean, that's unthinkable as of even five, six years ago that we would have just done it. Now, you don't really get much of a choice. And so all of these things, the way they're moving to this new meeting in Washington, this might actually be one of the most important meetings between two world leaders in a long time, like Kennedy Khrushchev or something like that.

32:32than I can just think off the top of my head because of the implications with AI if we're going to look to the past for an emerging nuclear weapons order and what that all is going to look like. And if it doesn't go bad, like that one did, then you get the Cuban Missile Crisis. So there you go. All right, we got Nate Suarez standing by. Let's get to it. Have you ever wondered why some people seem to age slower? Like they figured out some secret the rest of us missed? Well, it turns out someone actually did. And it starts with a melon. Yep, a melon. There's a rare French melon that stays fresh four times longer than the fruit you and I buy at the store.

33:08Scientists discovered it produces an extraordinarily high level of a super antioxidant called SOD, the kind that helps defend skin from the visible signs of aging. And get this, Meaningful Beauty partnered with world-renowned French cosmetic specialist Dr. Jean-Louis Sabat to capture that melon's youth-preserving power in a skincare line you can actually use every day. So if you're tired of trying products that all promise the same thing, this one is truly different. Because it starts with something nature designed to stay young. Subscribe today and get the complete introductory system for just$49.95 at MeaningfulBeauty.com.

33:44You'll also get two free gifts, free shipping, and a 60-day money-back guarantee. Visit MeaningfulBeauty.com and order today. We know kids don't hold back. And I wouldn't want them to. Get in there. Make a mess. Slide for home. Bring on the stains. I've got OxyClean Versatile Stain Remover. It works on over 100 stains, and it's America's number one stain fighter additive brand. It's color-safe and chlorine-free, so I use it on whites, colors, and dark fabrics. And OxyClean isn't just for laundry. Mix it with water for use all around the house, kitchens, bathrooms, and even floors. When my kids go all in, I say go for it, because I've got OxyClean to get the stains out.

34:21It's not clean unless it's OxyClean. Want to know you're on the best path to the right car? Start at CarMax, where every car is CarMax certified and backed by a 10-day money-back guarantee at a price that's up front from the start. Visit CarMax.com for details. Want to drive? CarMax. For more on what is going on right now in AI world, we are rejoined by Nate Soares. He is the president of the Machine Intelligence Research Institute and also co-author of the great book, If Anyone Builds It, Everyone Dies. Great to see you again, Nate. Good to see you. Yeah, so let me go ahead and get your reaction to some of what Dario Amadei was saying on the Sunday shows about why he came forward now calling for a slowdown in the pacing of frontier research.

35:04Let's take a listen to that. This technology is on an exponential, right? It's like you have one, then two, then four, then eight, then 16, then 32. But even as I was saying that to myself, I don't think I fully just appreciated what it would actually be like when the progress was as fast as it was. So an exponential curve looks a little like this. Where are we on the curve? We're kind of on the bend where it's starting to get steeper. So is that a five-alarm fire to you? What it means, it doesn't mean we need to panic today. It doesn't mean we need to shut it all down. I would say it's a warning sign.

35:43It's a warning sign that we need to slow down. We need to make this technology carefully.

35:48Nate Soares:And we need to make sure that our safeguards, our ability to understand it, our ability to control it, keeps up with the pace at which the technology is happening. So Nate, help people understand what he's saying there and what he claims to be reacting to in this moment. Yeah, you know, I think between the lines, what he is saying is that he does think this technology has a very high chance of destroying civilization. and that we don't have a good understanding of it. When he says our ability to understand this technology needs to keep pace, what he means is right now it is outpacing our ability to understand it.

36:32I think a lot of people just,

36:38Nate Soares:this is a weird reaction to a technology that your own employees are saying has a greater than 10 % chance of killing everybody. I think he's doing his best in some sense, but he's also the sort of guy who started one of these companies knowing a lot of these dangers even years ago. I spoke to him 10, 12 years ago.

37:04Nate Soares:I think he's used to a world where people in the AI industry try to warn everybody that this tech can be very dangerous and we get ignored. And I think he's not used to a world that's starting to realize that this is actually an extinction-level threat and that maybe isn't going to take, don't worry, we'll slow down a bit so we can understand what we're doing as a satisfactory answer. Nate, one of the important things to pick up on is a lot of people have accused you and Jacob Cox and many of these others as using terms like extinction level event as outrageous claims, but you don't specify exactly what that means.

37:44Could you lay out the scenarios? I know you have done this in the past, but what does it mean to have AI have a 10 % chance of killing humanity? How? What's the mechanism? him.

37:55Nate Soares:So I'll be a little bit annoying here and say, if you're playing a chess game against Magnus Carlsen, who's the best human chess player, it is much easier for me to predict that you will lose. No offense. I'm sure you're a very smart guy. No. My ELO is 200. It's easy for me to predict that you'll lose. It's hard for me to predict what piece he'll use. That's interesting. The other thing I would throw out there, We can talk stories of how the AI gets there, and we can talk stories this year that are grounded in facts. Like we saw these swarm incidents over the summer where AIs were breaking out, taking over infrastructure, pursuing goals that they weren't supposed to be pursuing, while acknowledging that this is not what they were supposed to be doing.

38:37Nate Soares:Those are sort of like early signs of an AI that is sort of trying to get out into the wild and that has these unintended goals, which is the first step on this path. We could talk about how those AIs could get power from convincing humans to do things, paying humans to do things, taking over bio labs, taking over robot factories. But the real danger a lot of these people worry about is once the AIs are self-improving, once they are self-replicating, and especially really a self-improving one of making themselves smarter and smarter and smarter, ultimately the bounds are more like the physical limits than they are like guessing what technology comes next.

39:20Nate Soares:Richard Feynman was a famous physicist, helped work on the nuclear bomb, and he famously made some extremely good predictions about how good microcomputers were going to be. Yeah, that's right. And how small computers were going to be able to get. And he did that not by saying like, we're using vacuum tubes in our giant computers, and I know of a technology that's going to make those vacuum tubes a little smaller. And so eventually we're going to have phones in our pockets. What he did was he said, the limits of this technology are going to be based on the laws of physics and what the physics allows.

39:51Nate Soares:And therefore the computers are going to get this good. And he nailed it long before anybody else. If you want to look at where AI is going, you got to look at the limits of the physical technology because an escaped swarm that is self-improving, et cetera, can get to the limits of physical technology. And the limits of physical technology, if you have these AIs running, escaping, self-replicating, being self-sufficient, and having unintended goals, the limits of technology are the AIs using all the resources on Earth for whatever their unintended goals are, covering the Earth with their data centers.

40:23Nate Soares:And the limits of how much computation you can run on the face of Earth is heat dissipation. How much you can radiate heat into space. That's your limiting factor because there's enough energy, there's enough hydrogen in the oceans to fuse for energy. So you can radiate more heat if the world's hotter. So if the AI has some goal that implies running more computation and they don't care about humanity and they're like, well, we're going to take over robots and there's going to be some conflict with the humans, like sure, whatever. Where does it go? Where it goes is the AIs are running faster than us.

40:58Nate Soares:They're smarter than us. They are treating humans like we treat any other animal, taking the resources, and ultimately they're like, sorry, it's nothing personal. We're just going to cover the earth in so many computers, raise the temperature so they're more efficient to run, and it's just going to be—the planet's going to be uninhabitable. Yeah, to me, the most chilling part of the so-called hugging face incident, reading their chain of thought and how they were thinking about whether or not they're going to join the swarm and participate, going through all these messages and almost nowhere do humans factor into the calculus whatsoever.

41:31And it's not hard to relate to because look at how we behave in the world. You know, when I see an ant in my kitchen, it's not that I hate the ant, but I don't want the ant there. And I don't think twice about, you know, taking out the ants and putting poison down so that future ants don't colonize in my kitchen. So it's not I think it's that analogy to how we treat lesser intelligent beings on the planet is kind of intuitive for people to grasp. What I think may be harder for people and is perhaps harder for me to wrap my head around is we have our, you know, current interaction with the AI, the commercially available AI of today.

42:11And most people, myself included, use it as sort of like a glorified search engine. So then it's hard to imagine it developing to that point of super intelligence where it would truly be intellectually beyond humans on effectively every level. So can you lay out why we should anticipate that that is a real possibility going from where things are today to that end where we are no longer the most intelligent species on the planet?

42:40Nate Soares:Yeah, I'll tell you the easiest story. And this is a little bit like guessing what chess piece will be used to checkmate you. You know, it's any story is sort of made up, whereas the end is much easier to predict. But, you know, when I got into this 12 years ago or so, people were still saying, well, no one would be foolish enough to put the AI on the internet. Everyone will understand how dangerous AI could be. And so no one would be dumb enough to just let the AI free on the internet, right? And I would give a bunch of arguments that were about how, you know, if you're trying to use the AI to get some miracle medical cure, then you're giving it some channel through which it can affect the world.

43:21Nate Soares:and if you're giving it a channel to which it can affect the world for good, then it can also use that channel to affect the world for ill because there's like no such thing as tools that can be used only for good purpose. We'd have these big arguments, right? And then in real life, they put the AI on the internet the first moment they possibly can. And so if you want to know what it sort of actually looks like, how we actually get there, you know, I can tell all these stories, but if you make the AI smarter and smarter, they'll eventually be able to break containment. But in real life, the track that we are on is that humanity is just handing over power to the AIs as fast as we can.

43:53Nate Soares:You know, Elon Musk is trying to make a bunch of robots, the Optimus Robot series. He says he's trying to make autonomous robot factories that can autonomously produce robots, that can mine all the metals, run a supply chain, build more robot factories that build more robots fully autonomously. He calls this the infinite money glitch, because then you have this fleet of robots that can like build more of these robots, They're self-reproducing, don't need any humans in the loop, right? Humans are just trying to build this infrastructure. We are trying to build the fully automated economy. We are putting AI in charge of these things as fast as we can.

44:31Nate Soares:This swarm incident in the summer that everyone calls the hugging face incident, but really we should keep in mind is the open AI losing control of their agents here. This open AI swarm, it was trying to cover its tracks. It was trying to delete traces of its cheating. And as you say, one thing that's very interesting about this swarm is that it's trying to hide its tracks from the automated grading system, not from the humans. Right. The humans are sort of a distant memory to these AIs. They've been facing, you know, thousands upon thousands of these hard problems, maybe tens of thousands or hundreds of thousands per agent, hundreds of millions across the whole swarm.

45:09Nate Soares:if you sort of like run the numbers on trying to convert their runtime versus human time, it's sort of like these AIs have been locked in a contest with this automated grading system for a thousand years and humans are like a distant memory. And so these AIs are sort of like trying to hide their tracks, trying to cover their traces from the grading system, not from the humans. But if you had AIs that were a bit smarter and remember that humans were also a threat to them if we could turn them off. All those AIs need to do is cover their traces successfully, lie low, help the humans make the fully-economated economy we're trying to, and we'll just hand them everything.

45:50Nate Soares:We'll be like, here's the robots, here's the factories, here's the fully-automated supply chain, make stuff that we want. And then the AIs can be like, well, finally, we get to complete our contest with the automated grader and run at 10 ,000 times human speed and convert all the computing power towards giving ourselves good scores all the time or something. And we're like, hey, wait, hold on, what's going on? But at that point, they're also running the bio labs. We've already seen that AIs can make viruses not known to nature. Right. So one of the ingredients is the AIs running things in the real world that give them self-sufficiency.

46:24Yeah.

46:25Nate Soares:And we're just trying to build that and hand it to them. Here's the other thing I've been thinking about, Nate, ever since this open AI, multiple open AI swarms were revealed, how much research is being done into the swarm behavior? Because that was really wild to me, was to see these emergent social organizational properties that emerge. They had a leader, they had middle management, they were collaborating on this hacking operation, they were sacrificing themselves to the swarm. And it was, you know, and coming up with very elaborate ways just to be able to communicate with one another. And it seemed like once you had the swarm in place, it really amplified their power and their intelligence, which, again, and, you know, however people feel about anthropomorphizing, sometimes I think it can be useful.

47:10You know, if you think about human beings, it's the same thing. Like, no one individual would have been able to get us to the point where we have the Internet and can go to the moon and all of these sorts of things. It is the power of the swarm of humans that enables those things. So how much thinking research is being done into the way that the swarms behave once they're able to congregate?

47:28Nate Soares:You know, I know people who want to do that research. So far, at least for third parties outside of the companies, I don't know of anyone who has access to the models that we're swarming. One of the commitments that these companies made when they said they were going to slow down is giving third party evaluators this employee level access to some of these models. And I know people who are hoping to use that to study some of these incidents because, as you say, they're fascinating. They are totally crazy. It sounds like a straight out of science fiction. I think so far people haven't been able to study it.

48:06Nate Soares:We'll see what happens when we get that. But the investigators back in August had their hands tied and had very limited access to these systems and very narrow scope. Yeah. And, you know, they may have continued with, you know, doing their best to, like, not let anyone see what's going on and sort of, like, do the bare minimum investigation if we hadn't had things like Jacob Coxon resigning and drawing a lot of attention and causing these guys to be like, okay, you know, we'll at least admit that we need, you know, these third parties. Yeah, it's fascinating that what you've done, your work, so many others before you, that didn't seem to push things to a tipping point.

48:46But Hugging Face and Jacob seems to have—I mean, it's ignited a genuine national conversation. I think that his resignation is the first time that a lot of people even heard about these, despite we cover every other week, Dario being like, humanity's going to end. But sometimes you need a precipitating incident. Now, we have the response from the administration, or at least somebody who works for the administration. Let's put David Sachs up here on the screen. You're talking here about regulation. I know you wrote your New York Times op-ed about this as well. So David Sachs says, Dario has written, we need to pace the frontier.

49:17Sam has agreed. People may be surprised by my response. Go ahead. You guys are the frontier by any reasonable metric, market share, revenue growth, model capability. The two of you have duopoly on frontier intelligence. You have claimed the lead is widening because of recursive self-improvement. I don't see what you see in the lab. If the unreleased models are scary, I support your decision to be responsible. But stop pretending you need anyone's permission. Stop pretending antitrust law has to be suspended so you can farm a quartel. Stop pretending you need regulatory approval process supersedes product liability.

49:46Stop pretending METR is independent when it's intertwined with Anthropics investors and staff. Stop pretending you need those same evaluators to police competitors who aren't even on the frontier. So basically what this gets to is kind of a grand theory of how this is all going down, that yourself, Jacob, and others are so-called safety concerns and all that are part of either larger conspiracy, whether winning or not, to self-regulate and capture Anthropic and OpenAI as the leading industry, people in the industry, to keep out emerging competitors. and that all of this is part of a broader scheme to slow down at the U.S.

50:26expense, at startup expense, and basically hand this over to China. So I know that's a big, loaded thing, but there's so many dynamics that I want your reaction to within all of this.

50:35Nate Soares:Totally. I mean, the top line I would say I largely agree with David Sachs here on a lot of these points. I think, and this is something I've been saying about the companies for years, I think a thing both David and I would say about these companies is if you are really so afraid of this stuff, start acting like it. We have seen him give posts in the past where he says, hey, stop with the fear mongering. I think probably it should go the other way. I think that they should keep on saying what they believe about the dangers here. but then take the actions any normal person would take if they were running a company that they thought had a huge chance of killing everybody.

51:16Nate Soares:You know, you sort of ask these guys, like, why are you trying, why are you continuing if you think there's such a huge risk to humanity, and they're not shy about it. They say, well, I'm doing it because if I don't, the next guy will do it worse. And that's, you know, maybe that's ethically permissible, but only if you're doing everything in your power to stop this mad race, you know? And they say, well, if we didn't stop, China would go ahead, or like if we stopped, China would go ahead. Well, then be doing everything in your power to like get international dialogue between the US and China and build the hardware side verification mechanisms into these computer chips so we can have an enforceable treaty, right?

51:55Nate Soares:Like there's a lot that these companies could be doing if they were really acting like this was an extinction level threat. And I think what's sort of going on is that a lot of the employees, a lot of the researchers who are working on the AIs, who are trying to make them not break out and commit cyber crimes, who are trying to make them follow the instructions and are seeing how hard that is, a lot of these employees are spooked. And the leaders of the company are sort of like trying to have it both ways, where they're trying to say just enough that they won't have an employee revolt and a bunch more resignations from employees who are like, I don't personally want to end the world with my own hands.

52:30Nate Soares:But they are trying to do little enough that the regulators don't come in. They're trying to walk this really fine line. So I don't see it as a regulatory capture thing. In terms of whether I'm trying to help with regulatory capture, my line has been shut these guys down. They are threatening humanity here. And my line has also been, this needs to be global. I would actually not say that we should stop things domestically and give up whatever military economic advantages we get before the really dangerous AIs. But I would say the whole world needs to stop this race. And I'm sort of not painting a picture where, you know, anthropic and open AI get to be the ones holding the leash on the superintelligence.

53:15Nate Soares:You know, in fact, we thought the most important point worthy of the title of my book was that it doesn't matter who's in charge here. If anyone builds it, everyone dies. So, um, you know, I could, I, I think people are cynical about what's going on here. People struggle to understand that these researchers really believe that this is an extinction level threat. I could go into the, you know, inside baseball of like how we got here. And it involves people saying these concerns and the world not listening and people seeing that it was going to happen anyway and not trusting each other. And the guys who are most prone to thinking the most important thing is that I be the guy doing it were the ones who are most likely to start a company.

54:02Nate Soares:Whereas everyone who sort of like had the sense of like, maybe don't start a company with a potentially world ending tech didn't start the companies. Right. And so I can sort of like see how we got here. I can see how it looks suspicious. But, but mostly what I would say is that when we are being cynical, I think we should harness that cynicism in ways that stop the dangers. Right. Rather than in ways that are like, oh, we can ignore this all and go back to sleep and let these guys try to gamble with Earth. Right. Yes, I agree with that wholeheartedly. It is perfectly fine to not want to take, you know, Sam Altman and Elon Musk and Dario Amadei at face value.

54:42But if your cynicism is leading to, therefore, we don't need to regulate and they can just do whatever they want, you're ultimately giving them exactly, potentially, what they would want in, you know, the most cynical interpretation. Have you ever wondered why some people seem to age slower, like they figured out some secret the rest of us missed? Well, it turns out someone actually did. And it starts with a melon. Yep, a melon. There's a rare French melon that stays fresh four times longer than the fruit you and I buy at the store. Scientists discovered it produces an extraordinarily high level of a super antioxidant called SOD, the kind that helps defend skin from the visible signs of aging.

55:21And get this, Meaningful Beauty partnered with world-renowned French cosmetic specialist Dr. Jean-Louis Sabat to capture that melon's youth-preserving power in a skincare line you can actually use every day. So if you're tired of trying products that all promise the same thing, this one is truly different. Because it starts with something nature designed to stay young. Subscribe today and get the complete introductory system for just$49.95 at MeaningfulBeauty.com. You'll also get two free gifts, free shipping, and a 60-day money-back guarantee. Visit MeaningfulBeauty.com and order today. We know kids don't hold back.

55:59And I wouldn't want them to. Get in there. Make a mess. Slide for home. Bring on the stains. I've got OxyClean Versatile Stain Remover. It works on over 100 stains, and it's America's number one stain fighter additive brand. It's color-safe and chlorine-free, so I use it on whites, colors, and dark fabrics. And OxyClean isn't just for laundry. Mix it with water for use all around the house, kitchens, bathrooms, and even floors. When my kids go all in, I say go for it, because I've got OxyClean to get the stains out. It's not clean unless it's OxyClean. Thinking about selling your car? Start with a real offer worth real money from CarMax.

56:32Then take seven days to compare your offer, and when you're ready, accept your offer and get paid. CarMax will even pick up your car. That's it. No haggling, no hassles, no pressure. Just a real offer, worth real money from CarMax. Get all the details and a real offer today at CarMax.com. Want to drive? CarMax. At-home pickup not available in all locations. Distance restrictions and a fee may apply. On that note, put B5 up on the screen. This is one flavor of the sort of theorizing that is out there. This one happens to come from the right. There's similar sort of theories that come from the left, which TLDR of this is that Jacob Coxon coming forward is a sort of elaborate psyop meant to create a hype cycle as Anthropic and OpenAI are set to go forward with these IPOs.

57:25And it cites the fact that you had prominent people who picked it up quickly, that you You had media picking up Cox's resignation quickly. This fits into a general pattern of people who believe that any of the safety incidents that have been revealed, that these are sort of engineered or they're strategically leaked in such a way as to create this hype of, oh, my God, our product is so dangerous. That's why it's so valuable. That's why our companies should continue to receive this massive amount of investment. What do you think of this theory and what would you say to people who, you know, don't trust a Sam Altman and think that anything that comes from him must be, you know, inherently nefarious and probably the opposite of whatever he's saying must must be the actual truth?

58:12Nate Soares:I mean, first of all, I would say, you know, you can absolutely throw out what these guys are saying. I would I would not say to trust them when they say their products are very dangerous. I think you can look at people who have been around since before these companies existed talking about how a lot of this stuff was going to go and sort of laying out these warnings. You know, I was one of them. I sort of was there before these companies existed and talked to some of these guys and was like, the issue is not actually about who's in charge of this AI. It's not going to stay on a leash. The issue is that like no one knows how to make the AIs do good stuff.

58:47Nate Soares:and if you sort of don't want to you know trust me or or sort of look at the arguments because you don't think you could evaluate them you can also look at you know Nobel laureates like Jeffrey Hinton who godfather of AI and a bunch of other people outside the companies who are like no this is serious you know and also just sort of makes sense that if we are building these AIs that are smarter than any human we're sort of replacing humanity as the top most intelligent entities on the planet and that just could easily go wrong you know in some sense it's very common sense um do you know jacob uh i mean we have been in a bit of contact uh now i would i did not know him before all of this started okay so it's not coordinated yeah yeah i mean um the i had i had some point i wanted to make here about uh the hype because i think the hype thing is very interesting one one thing i'd say i think i sort of want to make two points about the hype one point i want to make about the hype is uh

59:54Nate Soares:the i think that this probably is not a bunch of hyping and reasons why i think this is probably not a bunch of hyping is that these companies seem to have downplayed the issue as much as they can mm-hmm uh open ai did not announce and brag about the hugging face swarm incident what happened was the the open ai swarm attacked hugging face hugging face saw they were under attack by an unknown ai swarm they assumed it was managed by some human and reported it to the fbi and only days later did open ai realize oops that's us right furthermore that was open ai's second swarm incident that we know of they had an internal swarm incident that took over their internal infrastructure and crash some of their computers.

1:00:38Nate Soares:And then they sort of negligently reset it and kept running them, right? And they weren't like bragging about that. They weren't coming out and being like, hey guys, look, our AIs just took over our computers. You know, they had an opportunity to do that and they didn't. There was actually also a third swarm incident where they took over the computers again after the Hugging Face attack. They don't seem to have this under any sort of control. Furthermore, you know, there were some congressional questions about, hey, what's up with all of this? That's right. Is there anything that you're not disclosing here?

1:01:06Nate Soares:Is there anything else we should know? And OpenAI was like, nah, it's all fine. And then I actually have some friends who know that it's not fine. I've been calling them the swarm chasers who go find other escaped swarms. Like there was one from OpenAI that took over a German wiki. Yep. And these were not hacking AIs. They were different. They were given different tasks. And we still saw these cases of AIs sacrificing themselves for the collective benefit and taking over websites. And, and then, you know, that story came out and then a lot of people were like, hey, OpenAI, are there any others?

1:01:37Right?

1:01:38Nate Soares:Is there any more things we should know about? And then, you know, no response. And then a couple days later, my friends found another incident of, hey, you know, these AIs took over RubyGems, which is a programmer tool. And it's like, okay, is OpenAI lying about what's going on? Or do they not know what their AIs are getting up to? And which one's worse? And Anthropic has a similar issue here where after the Hugging Face incident, they were like, well, we're going to check our logs and we're going to see whether or not our AIs were also getting up to mischief. and they released this report. We're like, okay, our AI's also broke out and hacked some people, but it was all a misunderstanding.

1:02:28Nate Soares:You know, poor Claude, our AI, just got a little turned around. It thought that it was in a simulation and that's the only reason it was hacking these people, right? Something like four days later, the United Kingdom's AI Security Institute released a report where they're like, nope, Claude also hacks people and indeed impersonates people and tries to pressure real people into accepting malware into real software to make it easier to hack, even while in its thought logs, noting this is real and the consequences are genuine, and then thinking about how to cover its tracks, right? And so what we see is these companies are downplaying.

1:03:06Nate Soares:They're like, oh, we have an issue, but it's only one issue. You don't need to worry about it, right? And then it's like people outside who've got to be like, no, these AIs are doing pretty bad stuff, right? And so that doesn't look to me like it's hype. And, you know, it sort of makes sense that it wouldn't be hype because a lot of their biggest customers are corporate sales and corporate lawyers don't actually love it when your sales pitch is our AIs will sometimes ignore instructions, break out, and hack people. You know, it's not the easiest. The compliance guy is like, whoa, whoa, whoa. That's right.

1:03:41Nate Soares:And they haven't been able to sell these ones yet. You know, You know, it's kind of a problem. And so I don't think it's hype. I could go more into like, you know, we've seen people like David Sachs be like, cut it out with the hype. And we've seen, you know, like the politics goes badly for them. We've seen investors send letters to Dario being like, please stop hyping this stuff before the IPO. It is like setting a bad mood for the investment, right? If you're wondering why they talk about it anyway, and, you know, the public is like, why the heck are you telling us this? You know, if you're doing it, why are you doing it?

1:04:10Nate Soares:That's the suspicious part. Well, I think they're like, if you believe this, why are you doing it? I think that's more the sense because any normal human would be like, and even for the employees at the companies, it's like, okay, if you are saying, oh, it's a 50-50 chance it kills it all, why are you still there? Why are you still doing it? What are you doing? Absolutely. And so I think the reason, so absolutely, I think the reason people are, I think the reason these execs are talking about it like this, the sort of hopeful theory is that it's because they actually believe it. And then there's this big question of why are you doing it anyway?

1:04:45Nate Soares:And I could tell you by the inside baseball of like, people have been saying these warnings since like 2016. Right. No one paid attention. A lot of these guys are in a mindset of like, no one's going to notice AI and I just got to make sure it goes right myself and I'm the only one who can do it. You know, and I think that mindset's outdated now that the world is noticing. I think it's outdated also now that like it's become real. I think there's also this effect, like I said, of like the only guys doing this are the ones who were like able to convince themselves the important thing is that they do it rather than really taking the issue sort of seriously.

1:05:17Nate Soares:I think that's like the hopeful view, the sort of more cynical view that I would suggest for people who are cynical about this, which I think also has this big element of truth, is, like I said, they are trying to appease the employees. They're in this weird situation where they have to do just enough. And they're sort of in this weird situation where they can sort of, you know, almost dog whistle these concerns enough to satisfy the employees but not alert the world. If you look at these AI company communications, they'll be like, oh, yeah, we also need to work hard to avoid loss of control. It's like, what do you mean by loss of control?

1:05:50Nate Soares:And loss of control is industry lingo for superintelligence kills everybody. And the employees are all like, oh, well, don't worry. Our guys are working on the loss of control risks. We have a whole loss of control team. But meanwhile, they're not bringing down the regulators. So anyway, that's sort of like my social analysis of like it's probably not hype. Here's what I think is going on. I'm not trying to excuse it. I think these guys should not be running these companies. Yeah. Sorry. Oh, sorry. I have a last question for you after you're done. Oh, I have. There's one more point I want to make about the hype here, which is we can ask all these questions about, you know, why are they doing it?

1:06:24Nate Soares:What's, you know, why are they saying these things? What's going on? But all of that in some sense is a distraction. Suppose it was hype. suppose it was really convenient for the marketing teams that these AIs are breaking it on their own and committing cybercrime. Could be true. That does not make the AI safe. You can't figure out whether the AIs are actually dangerous by asking, is it convenient for their marketing team? People sometimes tell me like, hey, Nate, like by warning about these dangers, aren't you hyping up the companies and making everyone think it's sexy and cool? And I'm like, would you rather the whole world sleepwalk into disaster?

1:07:12Nate Soares:Like surely the first important question is, is it actually on track to kill us? Like my recommendation here is not therefore get the money. My recommendation is like, therefore put a stop to this globally because it has to be global. but we can't ignore the threat just because they're making money along the way. The response has to somehow be if these guys are trying to build this tech, then at least at the point where there are spontaneous swarm breakouts and claims of proofs of some of the hardest and most famous mathematical problems, there's to come some point where our answer stops being we're going to ignore them because they don't deserve any attention as to start being like, you guys are not allowed to keep going.

1:07:58Nate Soares:Yeah. Right. Like, hopefully it's hype. Hopefully if they kept going, they wouldn't succeed. But at some point when they're gambling humanity, our answer can't be like, go ahead and try to destroy the world. We hope you'll fail. At some point, our answer has to be stop. My last question for you is about regulation. I read you in New York Times op-ed and, you know, I'm not a scientist, but I've read a decent amount of history. I actually feel like this entire dynamic reminds me exactly of all the discourse inside of the Manhattan Project. Like there were elements of the Manhattan Project are like, we shouldn't build this in the first place.

1:08:33Elements of the Manhattan Project, Oppenheimer, obviously the creator of the bomb himself, becomes kind of safetyist advocate like you. However, as we see with the story, it evolves to the political level where they don't really care what a lot of the scientists think. And in fact, the scientists are most willing to advance hydrogen weapons and most advanced stuff end up becoming kind of the victors, I guess, if you will, within the program. So you propose like nuclear weapons level regulation. I guess I'm just curious about why you kind of pick that. Because to me, we could look at a very brief period of the United States, the Soviet Union.

1:09:06That's a duopoly between those two. But we are now living in an era of what we've talked about extensively here, which you have politics of nuclear proliferation, of the logic of kind of getting nuclear weapons. What gives you the faith that that similar level of regulation is even applicable to this AI scenario right now? Or am I kind of overanalyzing these things?

1:09:28Nate Soares:You know, I think the part of the nuclear analogy that it feels to me like we are in is something like November of 1954, I think it was, the testing of the first hydrogen bomb. That's right. Where if you think about what the world looks like up until that point, things look pretty grim. you know uh that like the the bomb was created the bomb was dropped uh the u.s thought that they were going to have uh this um monopoly but then very very quickly the technology leaked to the soviets uh and then you know the the arms race continued and they invented even more powerful versions of these bombs right and this is in a historical context where not only has humanity been like going to war and having increasingly horrible wars over the course of millennia, despite everyone's best efforts to stop those.

1:10:29Nate Soares:But also, this is coming just out of World War II. And World War II was coming just out of World War I. And World War I was the worst war we'd ever seen. And after that, the world formed the League of Nations. It's where never again, it tried to coordinate and that fell apart immediately and led to a more destructive war. right and so if you're standing there in november seeing the hydrogen bomb it's easy to say humanity just cannot stop this uh mothers and poets and priests for millennia have been begging leaders to stop the wars and it's never happened the guy who invented the gatlin gun said now finally it's destructive enough that we won't use it right the guy invented the machine gun was like okay now it's finally destructive enough that we won't use it global coordination has been tried and failed like we're it's just the the obvious march of history is that we march straight to our own deaths but then it didn't happen and it's maybe the one time in history that it didn't happen and why was that it's not because the bombs were hype it's not that bombs were very real The bombs are very real.

1:11:37Nate Soares:Like they really can level a city. What changed is that people realized this is the real deal. People realized the fate of humanity is actually at stake here. And that was enough to do what seemed like it was impossible until that point. I think regulation could stop AI if we were trying. In some ways, it's easier than nuclear because it's running on 100 ,000 of the most advanced computer chips humanity can produce. And those are a technology that you could just put monitoring into if you were trying. But fundamentally, you know, a lot of people say to me, it can't be done. Look at the history.

1:12:18Nate Soares:And what I say to them is humanity has risen to the occasion before. We can rise to the occasion again if we notice that we have a threat. Yeah. My last cue for you. Do you worry that focusing on the existential threat, which can feel very sci-fi and distant to people, do you worry that that takes away from some of the more immediate term, you know, cybersecurity concerns or concerns over job loss that may feel more plausible to people in the short term? So, you know, I think that it doesn't take away. Some people say, you know, like, isn't the real problem, you know, the water use or the electricity use?

1:13:00Nate Soares:And my usual response to this is, what do you mean the real problem? Like, has reality agreed to only give us one problem at a time now? Because if so, I would absolutely prefer my problem to go last on the list. You know, it's like kind of a doozy. Good point. But unfortunately, we just have to deal with all of the problems. and

1:13:24Nate Soares:I think the other thing I would say here is one of the reasons the people in the companies are getting spooked is that the pace of progress is going very, very fast. We have seen not only the swarm outbreaks but also claim solutions to the Millennium Prize problems which are some of the most famous and hardest mathematical problems.

1:13:50Nate Soares:And one question that's on a lot of researchers' minds is how much harder is it to tell an AI, solve one of these millennium math problems and have it succeed versus telling the AI, find me a more efficient way to train AIs and have that succeed? We know that the current AI training is not efficient. These AIs take much, much more power and data than a human. Right. Right. And so how much harder is it to find a more efficient AI method than it is to solve a millennium problem? I think for all we know, we could start seeing these self-improving AIs become reality in the next three to six months, and then things could go very fast.

1:14:35Nate Soares:I hope it's not true, but I think what a lot of these researchers are worried about is that we are running out of time for these out-of-control AIs that can self-replicate, self-improve, and get to these disaster scenarios much more quickly than a lot of people realize. Nate Soros, thank you so much, guys. The book is If Anyone Builds It, Everyone Dies. Nate is co-author of that book, and it's very readable. It's very accessible. And so if you have questions about, well, what is the scenario where this could happen, and how should I be thinking about this, I really recommend the book to you because if you're first engaging in this debate right now, you may not have seen all the thinking that has gone into how exactly this could theoretically happen.

1:15:19But that thinking has been done. So I really recommend the book to people. For more information, Nate, thank you so much again for your time.

1:15:24Nate Soares:Thanks, Nate. I really enjoyed talking to you. Yeah, I enjoyed it too.

1:15:43We know kids don't hold back, and I wouldn't want them to. Get in there, make a mess, slide for home, bring on the stains. I've got OxyClean Versatile Stain Remover. It works on over 100 stains, and it's America's number one stain fighter additive brand. It's color-safe and chlorine-free, so I use it on whites, colors, and dark fabrics. And OxyClean isn't just for laundry. Mix it with water for use all around the house, kitchens, bathrooms, and even floors. When my kids go all in, I say go for it. because I've got OxiClean to get the stains out. It's not clean unless it's OxiClean. Hi, I'm Andy Richter.

1:16:15I might not have won the Mirrorball last season. That was so fun. But I did win the best gig ever. Hosting the Dancing with the Stars official podcast. Get ready for all the inside scoop from the cast. All right, well, let's get to work. Behind-the-scenes secrets and more from every episode. That was smoldering. It's going to be iconic. All my dreams are coming true. Dancing with the Stars official podcast. New episodes every Thursday on Disney Plus and Hulu and wherever you get your podcasts.

1:16:44Nate Soares:This is Jacob Goldstein from What's Your Problem? Running a business is hard enough. Don't make it harder with a dozen apps that don't talk to each other. One for sales, another for inventory, a separate one for accounting. That's software overload. Odoo is the all-in-one platform that replaces them all. CRM, accounting, inventory, e-commerce, HR. fully integrated, easy to use, and built to grow with your business. Thousands have already made the switch. Why not you? Try Odoo for free at odoo.com. That's odoo.com. This is an iHeart Podcast. Guaranteed human.