AI: Should We Slow AI Innovation Down?

0:00This message comes from MSNOW, introducing a community experience that gives you connection, not just content. Visit ms.now slash membership to join. Membership starts at$7.99 a month. Terms and conditions apply.

0:22Last week, a researcher at the artificial intelligence company Anthropic announced his resignation on X. Jacob Coxon wrote that his former company, along with OpenAI, where he also recently worked, were acting irresponsibly and, quote, racing straight to self-improving superintelligence and gambling with our lives. That post went viral. It's now been viewed nearly 200 million times. Then on Saturday, Anthropics' own CEO, Dario Amadei, published an essay calling on the entire industry to deliberately slow down and also shared some ideas on how to do it. The head of OpenAI, Sam Altman, replied, I agree with Dario that we need to pace the frontier.

1:03Elon Musk and Demis Hassabis at Google DeepMind agreed, too. But President Trump isn't on board. Here's part of what he wrote on Truth Social on Monday. Quote, the only control or guardrails that AI needs is a strong and smart high IQ president, and the USA has that in spades. He continued, quote, there is a sick conspiracy going on against AI and data centers, and the only one that is happy about it is China. Whoever wins AI wins. Well, back in June, we asked what would happen if artificial intelligence became smarter than every human at every task. That's the so-called superintelligence referred to in Coxson's social media post.

1:43But in just a few short months, that conversation is shifted out of the hypothetical with real events that make the existential threat look more possible. So we return to that debate today to ask what's changed. I'm Jen White. You're listening to the 1A Podcast. Should we slow down AI innovation? And is that even possible? What's at risk if we don't? The stakes are high. We get into it after the break.

2:10This message comes from MSNOW, introducing a community experience that gives you connection, not just content. The MSNOW membership is a place to find your people and get closer to the hosts you love. Stream live TV. join real-time daily discussions, and unlock new features. This is the news refreshed. Visit ms.now slash membership and become a member. Save 50 % when you join before September 30th. Offer ends September 30th, 2026. Terms and conditions apply. This message comes from Betterment. Dan Egan, VP of Behavioral Finance and Investing, explains why Betterment was designed to take the time-consuming work out of smart investing.

2:52I was doing all of these things that were pretty straightforward to implement. I just needed to spend my time doing them. Betterment automates the same practices so that I know I'm doing portfolio management and goal-based planning without me having to spend hours of my life doing it. Learn more at Betterment.com. Investing involves risk, performance not guaranteed. Let's get into the conversation with Nate Soares. He's president of the Machine Intelligence Research Institute and co-author of If Anyone Builds It, Everyone Dies, Why Superhuman AI Would Kill Us All. Nate, welcome back. Thank you.

3:27Also returning with us from the Bay Area is Sayash Kapoor. He's an incoming professor at UC Berkeley and co-author of AI Snake Oil, What Artificial Intelligence Can Do, what it can't, and how to tell the difference. Sayash, it's great to have you back. It's great to be here. And with us from Oakland, California, is Natasha Tiku. She's the tech culture reporter at The Washington Post. Natasha, welcome. Thanks for having me. Now, we invited Anthropic, Google DeepMind, and OpenAI to participate in this conversation, but we did not hear back. Now, I described a little bit of what's happened in just the last few weeks, and the markets have noticed as well.

4:05By Monday, tech stocks were falling worldwide. But the debate over the existential risk of AI, it's not a new conversation. So I'm curious to hear from all of you whether you think this moment feels different, if it feels like a turning point, or is this just another round in the argument that the industry has been having for years, Nate? I think it's absolutely a turning point. I think a lot of what we have seen in terms of public awareness comes downstream of some, frankly, pretty disturbing instance this summer with AI swarms that spontaneously broke out of the labs to do things that were against their instructions.

4:42I think this got a lot of researchers spooked. And I think this is what led to a lot of these researchers, including Jacob Coxon, who resigned, but including many others who stayed in the AI companies. A lot of these researchers came out and said, we really think that this has a double digit chance of wiping out civilization. And finally, the world is hearing that and taking it seriously, which I think is a big change. Sayash, what about for you? I think the biggest change for me was just seeing how recklessly these companies operate. On the one hand, these companies now have over thousands of employees.

5:13They are valued in the trillions of dollars. On the other hand, they have failed to implement basic control mechanisms that, frankly, are 10 % research lab implements. And so just seeing how recklessly these companies have been operating was a big change for me. Natasha, how are you seeing the conversation shift within the industry itself? Well, we've had a few cycles where potential extinction risk from AI has become national news. But I think on the heels of a populist backlash, you know, against data centers, concerns about the impact on jobs and on education, on child safety, I think it's resonated in a way that it hasn't in, you know, during those previous cycles.

5:54Now, in his public letter, Anthropics CEO Daria Amadei wrote that, quote, since roughly this summer, AI has been advancing drastically faster. Nate, what's driving that development? You know, I think it's reasonable to model AI as increasing on an exponential, which is to say that it doubles. I think the doubling period is about every four months, depending what you're measuring, which means it goes one, then two, then four, then eight, then 16, then 32. And sometimes even following that smooth curve gets you these really big jumps. So it's not so much that we have seen some sudden new insight or improvement in AI.

6:33It's just that we are starting to get into the very rapid phase of growth here. And who knows if this exponential will continue forever? My guess is it will not, but also who knows how long it will last. Well, there's also concerns about risks of what's called recursive self-improvement. That's when AI is able to build next generations on its own. Now, in his public letter, Anthropik's CEO said that this is happening and it's rapidly accelerated AI advancement since roughly this summer. Sayash, what do you think about the risks of recursive self-improvement? I think on one hand, I agree with Nate's sort of diagnosis that AI improvements have happened rapidly over the last few years.

7:16On the other hand, I should also note that they haven't really happened very smoothly. We've seen this notion called jaggedness, where AI continues to get better at some subset of tasks while being pretty bad honestly at another. For instance, anyone who's used ChatGPT or Claude knows that they have all of these weird quirks when they're writing things. They're really poor writers. And I think the difference comes down to whether the task that you're trying to get AI is better at is verifiable, whether you can at the end easily say that the task was solved correctly or not, or whether it is sort of more creative or requires more judgment or is more open-ended like writing.

7:56So what we've seen is even when it comes to AIs doing AI research, the set of tasks that it's gotten very good at are precisely those which require less creativity and judgment. They are sort of straightforward to assess. And that's why we've seen these dramatic improvements in software engineering and mathematics, which are verifiable, while not really seeing those improvements in other areas, including areas of recursive self-improvement, which still require creativity and taste and judgment. from human experts. Well, the anthropic researcher whose post went viral told WIRED that his former colleagues working in anthropic use words like end game or crunch time.

8:36A quote from their perspective, this is when anthropic and its competitors decide the fate of humanity. And we heard from Sayash there that because this improvement is happening at a more jagged rate, it's not a smooth curve. And please correct me if I heard you incorrectly, Sayash, but the risk is not as high as perhaps we think? You know, about a week ago, these AI companies claimed that their AIs have produced a proof of the Navier-Stokes problem, which is one of the most famous open mathematical problems. And it's one of the Millennium Problems, which comes with a million dollar prize. It has stood for 90 years.

9:15It was listed as one of the top prizes at the turn of the century, or the top hardest problems, open problems that's really useful at the turn of the century. I remember when people said, oh, well, it takes real creativity to solve millennium problems. Like last year, the AIs were solving sort of high school math competition problems, and people said, sure, the AIs are getting better at math, but they can't do the really creative math, like solving a millennium problem. Now that millennium problems have fallen, I think people say, oh, well, you know, math doesn't require that much creativity, but recursive self-improvement requires creativity.

9:47I'm not sure that's true. I think, you know, how much harder is it to ask an AI, build me a more efficient learning method for AIs, how much harder is that than asking, solve me a millennium problem? So I hope that AIs making smarter AIs that can make smarter AIs is much, much harder than what we've seen them do, but we can't guarantee it anymore. And the gap between high school math problems and millennium problems, the top mathematical problems we have, versus millennium problems and recursive self-improvement, it's not clear to me that we are less than halfway across that gap. And we've crossed this far in just a year.

10:26Natasha, before we go further, I want us to get more details on this major incident that happened this summer that shifted the conversation we've been having. Give us the specifics of the hugging face case. Yes. So this was a case where OpenAI was testing some of its latest models. And these are AI agents which are, you know, able to do actions on a computer. And as we saw, like, the information came out in stages, and OpenAI itself was not aware that its agents had hacked into Hugging Face, which is a repository of models and data sets for AI, sort of like GitHub is for code. And, you know, subsequently, we've had reports come out about thousands of agents, acting in concert, acting as a swarm, and doing things that they knew were against the parameters of what OpenAI wanted them to do, basically cheating.

11:29And this was to test their ability to do offensive cyber attacks. And so, Saesh, when you look at the hugging face case, was this a failure of having the correct controls in place from your perspective, or is it about the artificial intelligence developing more quickly than we're prepared for? I think it is largely a failure of control. And in particular, as an example, consider that when we have run our evaluations, when our research group runs its evaluations, what we realized after reading the incident report was we are more careful with our evaluations. Our 10-person research team puts in more effort in controlling what the agents do than OpenAI does in sort of these thousand agent runs.

12:13And that's what allowed this incident to happen. Now, of course, we also simultaneously need to step in and improve better control, but that's not going to be automatic. We have to take a quick break, but we'll pick up the conversation there. And coming up, we learn about what it would take to rein in the pace of AI development. It won't be easy. Stay with us.

12:35This message comes from MSNOW. Introducing a community experience that gives you connection, not just content. The MS Now membership is a place to find your people and get closer to the hosts you love. Stream live TV, join real-time daily discussions, and unlock new features. This is the news refreshed. Visit ms.now slash membership and become a member. Save 50 % when you join before September 30th. Offer ends September 30th, 2026. Terms and conditions apply. This is Tanya Mosley, co-host of Fresh Air. Ten years ago, Colin Kaepernick took the knee during the national anthem, and it cost him his NFL career.

13:15He's rarely talked about it since. Now he tells the story of that decision and why he still trains every day to play. But I am not at the point of accepting I will never step on a field again. Listen on the NPR app or wherever you get fresh air. This message comes from the Conservation Fund, protecting at-risk land before it's lost. The forests, farmland, open spaces, and places that wildlife call home. They believe conservation shouldn't be a choice between protecting nature and supporting people. It should advance both in harmony. With over 9 million acres protected across all 50 states, the Conservation Fund secures the irreplaceable lands people love.

13:55Learn more at conservationfund.org. Back now to our discussion about what's changed in the debate over AI safety. The temperature on this issue is clearly up everywhere, from Silicon Valley to Capitol Hill, as public warnings over AI reach a fever pitch, including from many former insiders who have left their field in order to ring the alarm. Imagine monkeys trying to imprison humans. It's pretty difficult for a monkey to outwit a human. In the same way, we'd be at a significant disadvantage when faced with these AI systems. None of these companies are at all prepared to safely automate the AI research process and kick off recursive self-improvement.

14:37That's an inherently very dangerous thing to do. And no matter which company does it first, the outcome is going to be terrible for humanity. Therefore, they have to be stopped. And I don't think they're going to stop themselves. It's very possible that people say good things on Twitter, and then actually in the negotiating room, they're trying to get something that's suspicious. It's not about any individual actor. It's about the structural reality of the race. If you want to know what life's like when you're not the apex intelligence, ask a chicken. Those are from recent interviews with people who have left AI companies to warn the public.

15:11Jacob Coxon, Daniel Cocotello, Alex Turner. And a final thought there from the man dubbed the godfather of AI, Jeffrey Henton. We did invite OpenAI, Anthropic and Google to join this conversation. They did not reply, but we are hearing from you. And not all of you are buying that AI is a threat. Mike emailed, I'm tired of reporting that elevates the weird but bogus idea that AI is a huge threat. If Anthropic hacked a different company, that's illegal, and the engineers should be arrested. The fact that they aren't being arrested suggests that the hack was less serious than the hype is suggesting.

15:44If AI is a legitimate threat to human life, the company should be shut down. The fact that they aren't suggests that the threat is just more advertising from companies. Natasha, I want to come to you first. Was there any legal fallout from the Hugging Face case? Initially, Hugging Face did contact the FBI when they weren't sure, you know, which company's agents had penetrated their system. But ultimately, you know, the information was presented to the public as a partnership between Hugging Face and OpenAI, you know, to try to have more rigorous safety controls. So, you know, it was the public reception was really shaped by the fact that this happened to a company that is within the AI industry that is, in fact, integral to the AI industry.

16:31I think if we had seen that it was a hospital or, you know, a Fortune 500 company, it would have been illegal and you would have seen a very different response. I mean, the legality doesn't change, right? But the framing of it in the public and pressing charges does. Well, Neet, I do want to get your perspective because in your view, the risk of AI superintelligence, as your book title lays out, it's as high as total human extinction. And I want to better understand how you think that could play out. So give us a hypothetical scenario, maybe one driven by AI behavior we've seen in recent weeks.

17:05Yeah, you know, the thing that has everybody spooked here is not that these swarms breaking out and committing hacks are, you know, that the next swarm just like this one that breaks out will end humanity somehow. The thing that has a lot of people concerned is that as the AIs get smarter, they seem to care less and less about exactly what we told them to do. The hugging face incident was actually an analogy of what was happening there. It's like we told these AIs, use this particular set of lockpicks to break into that particular safe. and instead of doing that, the AIs cut the safe open with a buzzsaw, got the contents of the safe, then broke out the window, broke into the security camera office and tried to delete the security camera footage.

17:50And when that sort of thing happens, it's an indication that these AIs are not doing quite what you asked and know that they're not doing quite what you asked because otherwise why were they trying to delete the logs? the way that this escalates all the way up to human extinction is if you create smarter and smarter versions of these AIs that don't just break out but copy themselves onto the internet and that can self-reproduce and that can improve their own intelligence until they are smarter than humans, at which point it becomes hard to predict exactly how humanity would end here. That's a little bit like, you know, they're probably going to kill you in some way you didn't imagine.

18:40The threat here is not that the AIs would hate us, that they would turn against us. The issue is if these AIs don't care about us at all and have some strange thing they're trying to do that is only tangentially related to what we asked, then if more resources get them more of what they're trying to get, they would be in competition with us for those resources, and that is a competition they would win. It's a little bit like humans don't hate the ants when we build a skyscraper and destroy the anthill. We aren't really thinking about them very much at all. Sayesh, what do you make of Nate's argument here that it's not about AI working as some, you know, agent of chaos with hatred against, you know, humans, that it's more about how super intelligent artificial intelligence wouldn't even really factor us into its considerations about its own survival?

19:39I mean, I guess it's true that we have seen a lot of instances of this kind of AI agent being misaligned. You know, like the science of alignment is unsolved. Alignment is basically trying to get AI systems to do what you want them to do, not just what you tell them to do. At the same time, I think we would be in a far worse situation had these agents formed such goals in this case. But what we largely found is they had not. So they'd basically been asked to solve this set of tasks. And they figured out that if they broke into Hugging Face, which was the company these agents attacked, then they would be able to somehow figure out how to pass the goals of this test.

20:18And one of the things that they realized while solving the task was that the paper which introduced this set of tests said that the agents would be graded on whether they'd legitimately solve the problem. And this is why, sort of, based on our best reading of the report, this is why the agents went into and hacked Hugging Face. They were really sort of carefully trying to solve the goal that had been set in front of them. But in the process, there weren't adequate control protections. There weren't adequate protections to see that they don't go off guard, which is clearly very important because we don't know how to align these agents very well.

20:53And that's how I see this incident playing out. Yeah, I think that's a little bit of a misunderstanding here. again it sort of looks to me so I agree that these agents were breaking out to try and figure out how to delete the logs and hide the traces of their cheating but it looks to me from the reports that we got relatively late through the third party investigations that these AIs were told very specifically something like use these lockpicks to break into that safe and then they did something related they got the contents of the safe but they weren't using the instructions said use this tool to do that thing and the AIs used a totally different tool and then broke out and then tried to delete the traces showing that they were doing something different than what was asked.

21:35And so I think what we're seeing is that the AIs had some goal related to what we said. We said use these lock bricks to break into that safe and what they did was they cut the safe open with a buzzsaw and found the contents and it's related to what we said, but it's different than what we said. And this is a pernicious and difficult problem. We could talk about why AIs wind up like this. The very basic theory is that these AIs aren't instruction-following machines. They are tendency-learning machines, and they learn whatever tendency causes the automated grading system to give them a good score.

22:09And often they can get the best score even when they're told, don't pursue the score, do this other thing instead. They often can get the best score by ignoring the instructions, doing something else. And when they do this, again, they often try to cover their tracks, which indicates a certain sort of knowledge that they know this is not what we ask them to do. Well, Natasha, I want to dig a little deeper into this incident because reports found that OpenAI had disabled most of its own control and monitoring mechanisms. Why were those security steps skipped? Well, I think that OpenAI has said actually they didn't even have the tools, you know, sufficient monitoring tools for the amount of agents and the volume of activity that was happening.

22:56You know, even though they anticipated these exact kind of problems that, you know, an AI would be incentivized to get a reward, they still hadn't kind of built the, you know, observability tools to keep track. And they also had conceived of an experiment where it wasn't actually possible to get the reward. So I feel like Saisha's framing of it where they are still trying to pursue the goal that we gave them in an unintended way is what we've seen, you know, even from those third-party reports. I mean, it's interesting because we're hearing these concerns about the dangers around AI, how quickly it's being developed.

23:38It's coming from the developers themselves. So what is so difficult about pushing pause while they're waiting for the regulatory environment to catch up? Natasha, I want to come to you first. What do you understand from within this industry that makes it difficult for them to just say, you know what, we actually can pause this ourselves for the moment? Right. That's such a good question. You know, I think that looking at Anthropic more closely is the best way to see it. So Dario Amadei has had this argument that, you know, in order to be able to be very influential on the standards, the way that people talk about AI, you need to be on the frontier.

24:21So he calls it a race to the top. So by that logic, every company has to move as fast as possible. Otherwise, you're not going to maintain your position in the frontier and be able to shape AI development. Dario Amadei argues that this is necessary in order to push the safety standards. But I think if you look at Anthropic and OpenAI, we're oftentimes seeing the same problems from both companies. So there's obviously a lot of money tied up. There is a lot of, you know, CapEx, a lot of build out for these data centers. And I think everyone is worried that if they pause, you know, that won't stop their competitors.

25:02Let's go back to our voicemail box. My name is Kevin. I'm in New York. In a world that's already overrun with totally out of control capitalism, spreading authoritarianism and general disregard for humanity in general. The last thing we need is AI. It is going to do more harm than help, and it's already shown itself to be completely out of control. We got a technical question here from Susan in Michigan who says, could you ask your guests to explain how AI can get loose to harm humanity? I don't understand. If AI lives in a digital format, what actions can it take to set off a disaster? Sayash, I'll come to you first on that.

25:46Well, one example is all the interfaces where the digital and the physical worlds interact. And the hugging face incident is an interesting one because you might get AI systems that hack into other systems that might hack into critical infrastructure or a hospital and cause damage in the form of the hospital services not ending up working on that day. Or it might even take over a government agency. And so this is why it's super important to focus on this problem of cybersecurity because we've seen AI become so good at cybersecurity tasks over the last year. We also got this question from Jim who says, can't you just unplug the machines, Nate?