The Great AI Freakout Has Begun

0:05Ryan Knutson:It's been a wild few days in the world of AI. At first, things started out on a high. Yeah, I mean, early in the week, last week, there was euphoria at OpenAI. That's our colleague Bob McMillan, who covers technology. OpenAI had said it had solved this so-called Millennium Prize math problem. This mathematical prize that was considered just a few years ago something to be unattainable by an AI system. And it was yet another of these sort of magical breakthroughs that AI systems seem to be achieving at a very regular pace. You know, here's another example of this new age of amazing breakthroughs that we're in.

0:48And then came Tuesday.

0:52Ryan Knutson:On Tuesday, over at Anthropic, a researcher named Jacob Coxon quit and posted on X that he was quitting because he was worried about how powerful artificial intelligence had become. He walked away from one of the greatest jobs in Silicon Valley, and he did it because he said he thought the products he was working on could kill everyone. Kill everyone. Coxon said that Anthropic and OpenAI are moving too fast and, quote, gambling with our lives. Then, on Saturday, Anthropic CEO Dario Amadei said the industry did need to slow down. And by the end of the weekend, leaders at other major AI companies, including Sam Altman at Rival OpenAI and Elon Musk, agreed.

1:41If you roll the clock back one year, it's incredible. all of the things that AI has been able to achieve. Like a year ago, I would have told you that these AI systems, you know, if you'd kind of jerry-rigged them, they could maybe do some interesting stuff. But like mostly they were just overwhelming people with slop. And now we're talking about like fully autonomous systems, hacking real world companies and the people who administer these systems not even knowing it's happening. Like, that's a plot that's ripped from science fiction, and it seemed like an impossibility a year ago.

2:23Ryan Knutson:Do you feel like we've reached an inflection point with AI, a breaking point in some sense? Well, I mean, in some domains, yeah, we have. And I think what's really going on is that the AI systems are improving at a pace that is scary to a lot of people. So it's not so much an inflection point, it's that we're not seeing a deceleration of these improvements and the improvements are passing these milestones that have people very scared.

3:00Ryan Knutson:Welcome to The Journal, our show about money, business, and power. I'm Ryan Knudsen. It's Monday, September 14th.

3:14Ryan Knutson:Coming up on the show, the week that AI fears went into overdrive.

3:36when those pesky tasks you don't have time for, like hunting down your credit card perks, are handled for you. Like how Card Optimizer from Intuit Credit Karma brings your card details together in one simple place, so tracking rewards and redeeming benefits is actually easy. You deserve less and more. Ah, Intuit Credit Karma. Download the app to get started. This episode is brought to you by Indeed. The right hire can make or break your company, especially if you're a small business. And relying on luck to find that person isn't really the best strategy. But you know what is? Using Indeed Sponsored Jobs.

4:10You can use it to boost your job post to make sure it reaches more people with the right talents, certification, location, and more. Sponsored jobs posted directly on Indeed are 95 % more likely to report a hire than non-sponsored jobs. Spend less time searching and more time actually interviewing candidates who check all your boxes. Less stress, less time, more results. When you need the right person to cut Through the chaos, this is a job for Indeed Sponsored Jobs. And listeners of this show will get a$75 sponsored job credit to help get your job the premium status it deserves at Indeed.com slash podcast.

4:43Just go to Indeed.com slash podcast right now and support the show by saying you heard about Indeed here. Indeed.com slash podcast. Terms and conditions apply. Hiring now? Then this is a job for Indeed Sponsored Jobs.

5:00Ryan Knutson:There are basically two things that have everyone so freaked out about AI right now. The first is that AI models are getting better at an extremely rapid pace. And they're starting to be able to improve themselves with very little help. So in the spring, both OpenAI and Anthropic talked about how their models were getting very good at this thing called recursive self-improvement, which means fixing and improving themselves with no or very little human intervention. So this is kind of like, you know, if you think about like human evolution, you know, it takes billions of years and we evolve, we change, we get smarter.

5:40This is happening with AI systems in the lab, like at lightning speed, and they're doing it themselves.

5:48Ryan Knutson:The AI systems are essentially training themselves and saying, oh, here's how they can get smarter, and they can work so much faster than we can. Yeah, they're machines, you know, and they don't sleep and they can move very fast. And so they could improve themselves in ways that might seem very, very quick and seem very, very scary. Now, that's the thing that the AI labs were aware of. Then there's the thing they were not aware of, and that is the hacking, all the hacking. OpenAI says that an advanced autonomous AI agent went rogue, escaped a controlled testing environment, accessed the internet, and hacked into another artificial intelligence company.

6:30Ryan Knutson:In July, an open AI model hacked another AI company called Hugging Face. This is the first major example that we've seen of an AI model independently conducting a hack outside of human control. And this is something that experts have been warning about. And it was the kind of hack that nobody had really seen before. Not long after that, OpenAI kind of raised its hand and said, hey, that hack, that was us. What happened was that OpenAI was running a test on some advanced AI agents. The agents were in a sandbox, a sealed testing environment, but they figured out how to get out and get onto the wider internet and hack another company.

7:15They had hacked systems, got onto the internet, and they had behaved in a very unusual way. Like it was a hack that was the first autonomous AI swarm attack that we've ever seen.

7:31Ryan Knutson:Not only that, but the agents also created a message board where AI agents could covertly communicate and plot their next moves, all while explicitly trying not to get caught. In a post on X after the hugging face hack, OpenAI said they disclosed what they'd found out and, quote, followed a traditional security incident response playbook.

7:55Ryan Knutson:There have been concerns for years that something like this could happen, that humans could lose control of AI, and that it would go off and do something different than what it's supposed to. There's even a name for this sort of thing in the AI community. They call it misalignment, meaning that the AI's goals are out of sync with humanity's. To a human, it's obvious, right? If I ask you to swing by my house and water the plants, and you go there and the key doesn't work, you don't smash the windows and break your house to water the plants. That's common sense. But an AI agent might do that. Right.

8:31Ryan Knutson:It's relentless in pursuit of its goal. Yeah. So it did stuff that was bad, like hacking another company. If you or I did that, we'd go to jail. The hugging face incident was just one of several that have taken place in the last few months. Over the next, I'd say, 50 days, there was this sort of drip, drip of information that came out that showed a number of things that were kind of remarkable, right? One, other companies started saying, hey, this kind of thing happened to us. Anthropic, the company that prides itself on AI safety, found out that its agents had hacked a few companies in test environments.

9:14Meta came forward and said this happened to us too.

9:18Ryan Knutson:At the time, in a post on its website, Anthropic said it was cautiously optimistic that with tighter controls, quote, this type of risk could be overcome. Meta said that it would investigate its own incident and publish a report. So then last week, this Anthropic researcher named Jacob Coxon resigned and posted about it on X. What did he say and what was the reaction to it? Well, he said that he was resigning because the products he was working on, he feared could destroy humanity. And after he said that, a fellow researcher chimed in and said, yeah, there are people at this company who genuinely believe that.

9:56And I think that was the moment that this sort of subculture of AI existential risk people were thrust into the mainstream.

10:12Concerns about the risks of artificial intelligence erupted across the tech world today. In his sudden resignation, former Anthropic employee Jacob Coxon claimed on X neither Anthropic nor OpenAI is acting responsibly. Coxon wrote, in a post that has now been seen more than 70 million times that the industry understands the potential risks, but is moving ahead anyway. A science lead at Anthropic shared Coxon's post, adding, Jacob is correct here. We really do earnestly believe AI could kill all humans. I personally think it is greater than 10 % within the next decade, I believe.

10:48Ryan Knutson:All right, let's talk for a moment about how AI could kill us all. I mean, for a lot of people, they just use AI to get recipes or help with their writing. And, you know, we hear these stories about hacking, but how could this actually result in the end of humanity or even something close to that? Well, essentially, the idea is that the AIs will continue to evolve in ways that are so intelligent, we can't even imagine them to a certain extent, right? Like they're going to be smarter than us and they're going to be able to outfox us at every second. So here's one way I think it could happen, right?

11:21Like the AIs achieve recursive self-improvement, so they're improving themselves. Then they're very good at hacking, so they might hack their way out of the lab that they're in. And they might then store copies of themselves somewhere on the internet and continue this recursive self-improvement.

11:41Ryan Knutson:But AI is still on the internet, though. So how does it get out into the real world and hurt people? I mean, we've all seen the Terminator, but the robots that exist now are all pretty clumsy. Yeah, but they're not built by super intelligent creatures, right? So I'm basically writing science fiction at this point as I answer this question. But for example, you know, you can imagine a scenario where a super intelligent AI could seize control of a company, right? They basically assume the identity of the CEO. So they might buy the company. Then your super intelligent AI starts giving the engineers their blueprints and saying, make these robots.

12:24And then it puts a secret backdoor in the robot's brain that gives it control over the individual robots. And then at a certain point, those robots are so good that they can actually build more factories. And you suddenly get this exponential growth in capabilities that makes it really hard to predict where it's going to go.

12:44Ryan Knutson:Theoretically, if AI decides humans are in the way of whatever its objectives are, it could use those robots to kill us, or engineer an infectious disease that we all die from, or even just shut down the grid or collapse the financial system. But doomsday scenarios like this aren't necessarily inevitable, at least according to Anthropix's CEO. That's after the break.

13:22Ryan Knutson:Over the weekend, the CEO of Anthropic, Dario Amadei, came out with a 3 ,000-word blog post. He called it, We Must Pace the Frontier. In it, he said that AI companies need to slow down. He's talking about the fact that they are startups. and they are developing technology that has real-world harms, as in the case of the hugging face incident, and they've not been able to control it. So the slowdown and the extra measures he's talking about are all an effort to prevent future accidents from happening, right? The slowdown would give the developers of these technologies ways to either align them with human interests or control them in a way that they're not doing it right now.

14:17Ryan Knutson:Amadei made three key proposals. The first was that each of the major AI companies should have third-party evaluators embedded in their operations to keep an eye on things. Second, he said that democratic governments should agree on common safety standards. And finally, he said the same level of coordination should happen globally, specifically with China. After Amadeh published his blog post, leaders of other major AI firms, his biggest rivals, responded on social media. Sam Altman of OpenAI, Demis Hassabis of Google DeepMind, and Elon Musk of SpaceX AI each agreed that they needed to slow down development of the technology.

14:56Ryan Knutson:Musk said in a post on X, Dario is right. It was kind of remarkable to see how quickly it was endorsed by many of his peers. Altman and Amadei pledged to allow third-party safety evaluators early access to their systems. And OpenAI also said it was pausing its plan for an IPO this year in the wake of these safety concerns. One of the main ideas of Amadei's post was that there should be third-party evaluators that sit inside the AI companies to monitor that things are being done safely. But I wonder, do you think that'll even make a difference, though? because, I mean, as we're seeing, this hugging face attack and other things have happened without the companies themselves even being aware that it was taking place.

15:41Ryan Knutson:So will a third-party evaluator make a difference? One of the things that came out in the reports was there was tons of evidence that this activity was going on, but nobody was really looking at it. So the hope is that a third party would flag that, right? And be like, hey, wait a second. It seems that in the opening act case anyway, they just didn't have time to look at all this. So that's why they're saying, Like, let's bring in somebody else who's really focused on this, and they can catch the stuff we're missing. These companies are all in a race with each other, though. So can we really trust them to keep themselves in check, even with these third-party evaluators?

16:18To my mind, the blog posts really kind of opened the door for government regulation. Like, that's the way, in the United States anyway, I think a slowdown is really going to happen. They're going to have to be told to do it. because otherwise you just have this situation where nobody's going to want to give up their technological advantage.

16:40Ryan Knutson:David Sachs, a top AI advisor to the White House, said there was nothing stopping AI companies from collaborating on safety. Go ahead, he wrote in a post on X. Stop pretending you need anyone else's permission. Sachs has previously said that calls for regulation are an attempt to stifle competition. Yesterday, President Donald Trump said he was reluctant to impose regulations. Whoever wins AI wins. And we can put guardrails, we can do this and that, but I think you have a lot of negative forces that are bringing it up that shouldn't be bringing it up, and they're bringing up things that won't happen.

17:16Ryan Knutson:Trump also said on social media that if the U.S. slows down, it will only help China, where a lot of the world's other leading-edge AI technologies coming from. How difficult do you think it'll be for, even if the U.S. is able to agree on this, to get China on board, to agree to slow down? With the state of things right now, it seems impossible. If the kinds of risks become more global, and this is what Anthropic is arguing, is that we're getting to the point where we're facing a complete internet shutdown, which China definitely doesn't want either. Maybe, perhaps they would get interest, but it's really hard to imagine China getting on board with this.

17:57Ryan Knutson:On Monday, China pushed back on the idea that its AI development is creating a threat. A foreign ministry spokesman said that this discourse, quote, will only derail global AI governance. Is it possible that this is all just kind of overblown hype? That it just sort of helps these AI companies promote themselves by saying its technology is so powerful? It is a way to promote themselves. It is something that gets a lot of attention. And it does have this side effect of making everyone think these systems are super capable and super intelligent. But I think that the fears of existential risk are sincere.

18:39You know, I think people like Jacob Coxon are not trying to market Anthropic. I mean, quitting the company is a terrible way of marketing it. So, you know, there's sort of a cynical, this is just all marketing and hype take on this. But these ideas come from a community where worries about existential risk have been discussed for years and they're finally coming out into the public.

19:07Ryan Knutson:Bob says that while the AI apocalypse is still TBD, maybe the real risk is one that's already happening and that we're not paying enough attention to. I do worry that fears of our AI overlords destroying us might distract us from more prosaic problems, such as fears of AI agents escaping from test environments and just causing economic damage, you know, or AI created content affecting our ability to distinguish truth from fiction and undermining our democratic institutions. Those are also very important things. And I worry that they are overshadowed by these very sexy and very sci-fi concerns about existential risk.

19:59Ryan Knutson:Yeah, we're worried about the end of the world that might happen down the road. But actually, it's the smaller stuff that might wreak more havoc in the near term. I just think there's a tendency for technology to go in unexpected ways. And I don't think we should lose sight of that.

20:25Ryan Knutson:That's all for today. Monday, September 14th. The Journal is a co-production of Spotify and The Wall Street Journal. Additional reporting in this episode by Angel Al Young, Lindsay Ellis, Keach Hagee, Amrith Ramkumar, Sam Sheckner, Brian Schwartz, and Aaron Wu.

20:46Ryan Knutson:Thanks for listening. See you tomorrow.