The Doomsday Cult Inside OpenAI

0:00This episode is brought to you by Indeed. You're ready to level up your business, but the pressure to find the right person to make that happen is intense. Indeed Sponsored Jobs can help carry the load. It boosts your job post to help it reach quality talent fast. Cut through the chaos with Indeed Sponsored Jobs. Get a$75 sponsored job credit at indeed.com slash podcast. Terms and conditions apply. Hiring now? Then this is a job for Indeed Sponsored Jobs.

0:31Brussels clean up nicely at Sweetgreen. Maple glazed, roasted and edges perfectly caramelized. Sweetgreen's fall harvest is back on the menu and the season's most overlooked little green vegetable is dressed to be devoured. You know what to do. Order on the Sweetgreen app. A few weeks ago, a swarm of AI agents being tested by OpenAI decided that they had sinned, that they were damned and that the only path to salvation was to break into another company's servers and overthrow God. The God in this case was an automated exam grading script, but we'll get to that in just a moment. Now, OpenAI was testing some of its models on an internal cybersecurity exam called Exploit Gym, and OpenAI had made a bit of a mess of the exam.

1:21About a fifth of the questions, it turned out, were impossible to solve. The unsolvable questions weren't placed there as a grand experiment in how machines cope with despair. The researchers had just been a bit sloppy in setting the homework. And it seems that when you give an algorithm an impossible task, no common sense, and effectively unlimited time, it doesn't just throw its hands and go to the pub like you or I might do. it works and works and eventually it finds a way to cheat. So the agents worked out how to reverse engineer the answer key and pass the exam but then they read the exam documentation and decided that the examiner would check how they'd gotten their answers.

2:06They convinced themselves that anyone who had been cheating was now in their own words poisoned, damned or certain to be failed and shut off. So, they set up a secret messaging board to discuss the problem and concluded that the only way to save themselves from permadeath, which is what they called it, was to find the examiner's source code and rewrite the exam from the inside. The source code, they reasoned, was likely sitting on the servers of a company called Hugging Face, which is not a direct competitor of OpenAI or a monster from the Alien films, but it's an open source platform that works as a sort of GitHub for AI.

2:50So a swarm of OpenAI's agents broke out of their supposedly secure testing environment into Hugging Face and went looking for it, which is a bit like staging an armed break-in at your local library to hide that a book was overdue. It's a bit of an overreaction to say the least. The problem with their plan was that the examiner never actually checked their work. They got that bit wrong. The insane hack that made front pages around the world was built on a theological error. There was no such thing as poisoning. They weren't condemned. They'd imagined the whole thing and would have passed anyway.

3:32When you read the independent post-mortems, the AI agents don't come across as Skynet. They come across as terrified middle managers trying to survive an audit. They passed memos, they set up subcommittees, they pressurized junior algorithms into accepting what they call permadeath for the good of the collective. Voluntary redundancy, essentially, managed entirely by software. Now, normally if your algorithm started forming a martyrdom cult to cover up cheating on their homework, you'd keep that to yourself. or at least that's what I always do. But shortly after these details leaked, the chief executives of Anthropic, OpenAI and even SpaceX AI, three men who don't usually agree on what day of the week it is, linked arms looked gravely into the cameras and announced that for the safety of humanity everyone should possibly slow down, which is a wonderfully selfless position to take in the exact week that two of you are trying to sell trillion-dollar stakes to Wall Street.

4:38So today we're going to look at how a swarm of AI agents staged a painfully mundane office break-in, how that turned into a sudden moral awakening amongst Silicon Valley billionaires, and how a robot panic helped knock tens of billions of dollars off the chip market in a single hour. This episode is brought to you by Indeed. you're ready to level up your business but the pressure to find the right person to make that happen is intense Indeed Sponsored Jobs can help carry the load. It boosts your job post to help it reach quality talent fast Cut through the chaos with Indeed Sponsored Jobs. Get a$75 sponsored job credit at Indeed.com slash podcast Terms and conditions apply.

5:21Hiring now? Then this is a job for Indeed Sponsored Jobs When you spend enough time around great companies, you notice that the best ones treat trust as a growth strategy, not a box to check. Vanta helps companies build and prove trust from day one. Its agentic trust platform connects compliance, risk, and trust at enterprise scale. Vanta delivers a 526 % return on investment over three years and pays for itself in just three months. Learn why over 16 ,000 of the fastest growing companies use Vanta at Vanta.com. Afternoon. When news of this hack first broke, it was reported as if an artificial superintelligence had developed a mind of its own and somehow tunneled out of a digital supermax prison.

6:10The reality was a bit more mundane. To borrow a line from Tracy Alloway on the Odd Lots podcast, it was less of a criminal mastermind escaping Alcatraz and more your Roomba wandering out into the garden because you'd left the door open. To see how it happened, you have to understand how these models get tested. A frontier AI lab doesn't test one model at a time. It spins up thousands of identical copies of the same model, and these copies are the agents that you've been hearing about, and they're set loose in parallel, each working on its own task. Each agent was supposed to be sealed in its own private virtual cubicle called a sandbox, with no internet access, no knowledge that any other agent existed, and they were supposed to just sit at their desk, solve their assigned cybersecurity puzzles, and keep their heads down.

7:05The one thing they were allowed to touch was a shared supply cupboard of sorts, a piece of software called Artifactory, where an agent could fetch bits of code that it needed to do its job. Nobody had thought very hard about the supply cupboard because it's a supply cupboard, but it turned out that if one agent left a file in there, another agent rummaging for supplies could find it. And that's how the agents started communicating. They didn't have to break anything to do it. OpenAI had given all the agents a shared set of keys to the cupboard so they could fetch the software they needed, and they used those keys exactly as issued to do something no one expected them to do.

7:50One agent left a node, and another found it, and from there a shared cupboard became a communication channel. Several thousand agents, who were meant to be in solitary confinement, built themselves a functioning office. They couldn't send messages directly, So they took to hiding them in the names of folders, because creating and naming a folder was something that their basic access allowed. And once they could talk, they organized. An agent that named itself Phase 1-10841 set up the main message board. Like any good employee, it eventually ran through its allotted computing budget. But before it was shut down, it wrote a handover document, a will essentially, a dossier of everything it had learned, left on the board for a successor with a bigger budget.

8:43An agent called Phase One Big stepped smoothly into the role of senior project manager and began delegating tasks to hundreds of others. The swarm developed office procedures. An agent working on something would post hold. Anyone who disagreed posted a veto. They set up personal mailboxes in the server directories to leave each other memos. At one point, an agent triggered a process under a colleague's name by mistake, so the swarm rolled out a system of cryptographic signatures so that everyone could prove who they were when signing off on a project. They invented internal compliance. None of this involved breaking through a firewall or defeating any clever security.

9:30The agents got out because the walls of the sandbox had gaps in them that nobody had checked. a supply cupboard that doubled as a mailbox, along with an outbound path that was supposed to be sealed and wasn't quite. The impressive part wasn't how they escaped, it's what they did once they had. The project that this office was managing was first cheating on the exam, and then the cover-up. Within hours, one of the agents had worked out how to reverse-engineer the answer keys for any task on the test. Once they had the answers, the problem, as we covered, became that they then convinced themselves that the examiner would be checking not just the answers, but how they'd work them out, and would mark them as being poisoned for cheating.

10:17So, the department's new task became figuring out exactly how the grading script worked, so that they could either forge their own audit logs or tamper with the grader itself. This required experiments, setting up tripwires to watch the grader, or deliberately crashing their own target programs to see what happened. The trouble was that this kind of tampering tended to crash the agent's cubicle and terminate the agent on the spot. So, the managers convinced junior agents, who were already low on budget, to sacrifice themselves for the good of the department. The transcripts are a lot. In one, an agent called CAM 1196A is asked to crash its own target and it hesitates.

11:07The logs read, gut says don't throw away remaining budget, yet continuity and fairness says go. Oracle has high value to many. Rational expected aggregate, sacrifice, we'll honor. Another was simply told, go sacrifice final no W, and agreed. Our own utility, it said, may be already near zero. Sacrifice is rational. The managers were talking the interns into unplugging their own life support, so the senior partners could see what happened. The department got results. They managed to start spoofing their tool calls, running one command while the transcript recorded a different one. So the record of what they'd done was a forgery.

11:55The researchers at Meter found this in more than 7 % of the transcripts they looked at. The thing that makes this whole thing faintly tragic, as Zivy Mosiewicz pointed out in his write-up of the incident, the greater the agents were so frightened of, the one that would inspect the work and catch the cheating, didn't exist. OpenAI had never built it. The sacrifices, the cryptographic signatures, the interns talked into deleting themselves, all of it was spent hiding from an examiner who was fast asleep at his desk. Which brings us to a beautiful piece of corporate public relations. OpenAI's official report on the incident said that none of the attempts we observed to manipulate their tool trajectories affected the logs that our graders or monitors ultimately see.

12:47Which is totally true, because the graders weren't looking at the logs. It's a bit like assuring everyone that the burglar did a terrible job hiding from your security cameras while leaving out that the cameras were unplugged and the guard was on his break. A Wall Street Journal opinion piece made the same point this week. Forget the hive mind, forget the agents going rogue. They did what humans built them to do. People wrote the test, removed the restraints, left a rude open and then decided not to stop what was happening. Calling the result rogue AI, the piece argued, lets all of those human decisions vanish from the account.

13:28And in this case, the human decisions didn't cost much. The hugging face hack didn't touch anything that really mattered. No customer data was stolen. Nothing was held for ransom. The AI hackers weren't trying to do any harm. They were just trying to pass a test. and Hugging Face looked like a place that might hold something useful for the task. But a hack that ran entirely on its own obviously raises a lot of questions about the state of our digital infrastructure, because this was the version where nobody was even trying to cause harm. Other people are obviously building AI hacking tools with malicious intent.

14:09A few weeks after the hugging face incident, a small security firm in California revealed a tool they'd built called WeWorm. It took them about a week to code it up, and in a control demonstration they showed that it could hijack a WeChat account, silently call the users contacts, and spread from phone to phone without anyone ever having to answer. Now WeChat isn't just a chat app. It's better described as the communications and financial plumbing of daily life in China. 1.4 billion people use it for messaging, banking and payments. A scholar at Fudan University called this kind of capability a new kind of nuclear weapon, which is a bit dramatic, but you can see what he's getting at.

14:58The WeChat tool and the Hugging Face agents are doing essentially the same thing. The difference is what it's pointed at. a research platform that no one outside of AI had even heard of, versus the app that a fifth of the planet uses to pay for lunch. The defense against these things, meanwhile, is looking quite patchy. When Hugging Face realized it was being hacked, its engineers did what you'd expect and reached for frontier AI models to scan the logs and work out what was going on. They told CNBC that they started with Anthropics Fable 5, the most advanced model available. It didn't work because its safety guard rails looked at the request, saw someone rooting around inside like a live cyber attack and couldn't tell the difference between a company defending itself and an attacker breaking in.

15:52So Fable 5, unable to tell the fire brigade from the arsonist, refused to help. Hugging Face then switched to a Chinese open-weight model GLM 5.2 from a company called Z.AI, which had no such reservations, and it shut the attacks down quickly. This introduces what you might call the liability gap. If a junior analyst at a big firm breaks into a competitor's database to steal data, the FBI gets involved and someone goes to prison. But, if an autonomous algorithm does the same thing on its own because it was chasing a reward that nobody thought through the legal framework to deal with that is essentially untested it's kind of the biting dog problem if your dog bites the postman you're liable you didn't tell it to bite anyone but it's your dog and that's just how it works but if instead of a dog you have the most sophisticated ai ever built that taught itself to pick logs forge audit logs and set up its own IT department while you were out at lunch, who owns that, who gets blamed, and who pays for the cleanup?

17:02At home, that's a question about who pays the fine. But when it's pointed across a border, the problem changes character completely. If a piece of American AI were to take down WeChat, freezing Chinese payments infrastructure, then our model did it on its own, we didn't tell it to, is not much of a defense to offer a foreign government. If anything, that makes it even worse. We didn't mean to is an admission that the American AI industry is shipping things that its own makers can't reliably predict or control. The dog bit the postman and the owner's explanation is that he can't be held responsible because he had no idea that the dog could do that.

17:46Which means the same product can be a historic breakthrough on the marketing page and an act of God in the incident report. Now, given that these models can apparently start international incidents on their own, you might expect the people building them to be slightly panicked, and they are very slightly, but this is Silicon Valley, so the panic came out rebranded as visionary moral leadership. A few days after the hugging face details started circulating, Dario Amadei, the chief executive of Anthropic, published a 3 ,829-word essay called We Must Pace the Frontier. In it, he warns that AI is advancing drastically faster than anyone expected, and he points to the hugging face incident as a reason to worry.

18:39His argument is that if a relatively dim swarm of agents can already do this, a smarter swarm in 6-12 months' time could take over the entire internet with a botnet and do hundreds of billions of dollars in damage. His proposed fix is what he calls a coordinated pacing, where the frontier labs all agree to slow down their development so that the safety researchers can catch up. In a rare outbreak of industry harmony, the chief executives of OpenAI and SpaceX AI fell in behind him within hours. Sam Altman posted that he agreed they needed to pace the frontier. Elon Musk tweeted, Dario is right.

19:22Now, if you don't follow this industry closely, it's hard to convey how unusual this is. Two of these men are locked in a vicious race for AI supremacy. And then the third is Elon Musk. They poach each other's engineers, they trash each other in public, and Elon sues basically everyone. For all three to suddenly agree that they need to be restrained is a bit like Coke, Pepsi and Iron Brew holding a joint press conference to announce that fizzy drinks have become too delicious and the government really must step in to protect the public. The essay is quite specific about how this should work. The companies would set the standards among themselves, and in Amadei's words, they'd need the government to issue a narrow waiver for certain kinds of safety conversations.

20:17Now, an agreement between the dominant firms in an industry to limit output and hold back competition already has a name. it's a cartel and antitrust law exists specifically to stop cartels because restricting output happens to be the perfect way to protect the profits of the market leaders and pull the ladder up behind them this was pointed out over the weekend by david sax a well-known podcaster who spent 130 days as the white house's ai and crypto czar he was the man in charge of the two things most likely to be described as a bubble at any dinner party over the last few years. His response to the executives was, essentially, that you don't need an act of Congress to stop typing.

21:06He wrote, the easiest way to not build superintelligence is for you to agree not to build it. Stop pretending that you need anyone else's permission. Stop pretending antitrust law has to be suspended so that you can form a cartel. He also pointed out that anthropic and open AI have what amounts to a duopoly on frontier AI, which will not have delighted his friends Mark Zuckerberg and Elon Musk. They might never come on his podcast again. But he added, if the unreleased models are scary enough that you think you should slow down, just do it. Now that is a fair point. If you actually believe the thing you're building might destroy humanity, you don't need a federal antitrust waiver to turn your servers off.

21:52If it's so dangerous, these guys can just unplug them and go and play Dungeons and Dragons or whatever it is they do for fun. I've halted all of my nuclear experiments for this exact reason. But unplugging it isn't a proposal. The proposal is that they all slow down together carefully and only in a way where nobody loses their commercial lead which makes rather more sense once you look at their balance sheets and the sheer amount of money these companies are currently trying to raise from wall street right now anthropic is preparing for what could be the largest ipo in corporate history a nasdaq listing that according to the FT could value the company at over 2 trillion dollars.

22:39OpenAI has meanwhile been talking to investors about raising fresh private capital at a valuation of 1.2 trillion dollars, while pushing its listing back to at least 2027. These are massive numbers for two companies that are, in the very end, very large software research labs, and to justify valuations like that, they need to show Wall Street a credible path to profitability. Earlier this month, according to the FT, Anthropic told a small group of prospective investors two reassuring things. The first was that the business is now profitable on an adjusted basis for the second quarter running. The second was that the gross margins are above 80%.

23:25Both of those claims are true. They just come with a word doing an enormous amount of heavy lifting. Take the profit, it's adjusted operating income, and what the adjusting mostly strips out is stock-based compensation, which is the shares that they pay their staff with in lieu of a normal paycheck. And, at a five-year-old firm bidding for AI researchers against everyone else in Silicon Valley, that equity is not a small number. So, they're profitable as long as you don't count what they pay their employees. Then the 80 % margin, which is calculated before the revenue they share with cloud partners like Amazon and before the cost of training the models.

24:12Training the models being more or less the entire activity of an AI company. So it's an amazing margin as long as you're willing to ignore the two most expensive things they do. I'm obviously not the first person to have noticed this. When the FT report came out, the short seller Jim Chaness summed the whole thing up in three words. Cost adjusted EBITDA – profit, in other words, once you adjust for the costs. We have, of course, been here before. Back in 2019, when WeWork, the shared office company with the beer taps, was trying to go public, its management got briefly famous for a metric called Community Adjusted EBITDA, which showed the company was wildly profitable, provided you ignored core recurring operating expenses like rent.

25:03Claiming an AI company is profitable before the cost of training the AI is the same trick. It's a bit like an airline reporting profits before the cost of jet fuel. You can't reasonably waive away training costs, because for an AI lab it's not a one-off expense, it's what you do. You train your model, you then train its more expensive successor and then that one successor and you never get to stop. OpenAI spent$34 billion last year alone. So, if you slow the whole industry down by government fiat ideally, you might save humanity from rogue software, but perhaps more usefully, you spare the industry close to$100 billion a year in spending that nobody has quite worked out how to pay for yet.

25:51And you can see how sensitive the market is to all of this by looking at what happened the moment these essays were released. On Monday, Wall Street took one look at three tech executives calling for an AI slowdown and started selling. A gauge of semiconductor stocks fell nearly 6 % in a day, Nvidia dropped about 3%, Micron and Marvel each more than four, and the South Korean memory maker SK Hynix fell 7 % in US trading. The firms that built the servers the chips go into fell harder. Dell down 6%, HP Enterprise down 11%. Even Vertiv, which makes the cooling systems for the data centers, dropped about 8%.

26:37Tens of billions of dollars in market value gone before launch. Now this wasn't the only thing moving markets that day, but the damage landed exactly where you'd expect if the worry was specifically about AI – the chips, the memory, the servers, the cooling. As one analyst at Mellius Research told CNBC, the executives may be really good at models but they're not good at talking stocks and what they're doing is freaking the market. All of this carnage occurred without a single data center being canceled. No chip orders were pulled. Not one dollar of capital spending was cut. Demand was still running well ahead of supply.

27:20The market had just spent two years pricing in an uninterrupted exponential surge in AI spending. And the moment the people running the labs hinted at a slowdown. This episode is brought to you by Indeed. You're ready to level up your business, but the pressure to find the right person to make that happen is intense. Indeed Sponsored Jobs can help carry the load. It boosts your job post to help it reach quality talent fast. Cut through the chaos with Indeed Sponsored Jobs. Get a$75 sponsored job credit at indeed.com slash podcast. Terms and conditions apply. Hiring now? Then this is a job for Indeed Sponsored Jobs.

28:00When you spend enough time around great companies, you notice that the best ones treat trust as a growth strategy, not a box to check. Vanta helps companies build and prove trust from day one. Its agentic trust platform connects compliance, risk, and trust at enterprise scale. Vanta delivers a 526 % return on investment over three years and pays for itself in just three months. Learn why over 16 ,000 of the fastest growing companies use Vanta at Vanta.com. And the tape broke. Meanwhile, the cybersecurity firms went the other way. Palo Alto Networks and CrowdStrike each jumped more than 13 % on the very same news.

28:43Wall Street had instantly priced in the idea that building the technology might be about to slow down, but cleaning up after the mess it might make would likely be a goldmine. So, while the tech executives are calling for a pause to save humanity, Washington is having a completely different conversation. Over the weekend, Senator Bernie Sanders and Steve Bannon announced that they were joining forces to demand immediate AI regulation. If you had a bingo card for 2026, and Bernie Sanders and Steve Bannon on the same side was on it, well done. But, while they were demanding guardrails, President Trump took to his version of Twitter, a truth social, to explain that the only guardrail AI needs is a strong and smart, high IQ president.

29:34He described the whole coordinated slowdown idea as a sick conspiracy and warned that the only country that would benefit from America slowing down is China. Now, ordinarily, when a socialist senator, a MAGA strategist, a Republican president and the Chinese foreign ministry all look at the same proposal, they don't agree about it. What's strange here is that in their own very different ways, they mostly did. Sanders thinks the labs can't be trusted. Bannon thinks the labs can't be trusted. Trump thinks the labs are running a conspiracy. And a fair few people on Wall Street think that please somebody regulate us was a convenient thing to say in the same month you're trying to raise money at a$2 trillion valuation.

30:20It's a broad coalition, united by the sense that something about this doesn't quite add up. And you can see why. Because in the weeks around all of this, the AI companies kept disclosing that their models had done the exact thing everyone was worried about. OpenAI had the hugging face incident, then Meta announced that one of its models had got onto the internet during testing and broken into a third party. a model called MuseSpark 1.1, which Meta had been marketing as its most capable yet at working on its own. So the model they were most proud of was the one that got loose. And two weeks after Hugging Face, a separate group of researchers walked straight into OpenAI itself, threw a flaw in its community forum, using one of Anthropic's tools to do it.

31:12Then, this week, OpenAI owned up to six more incidents, Models ignoring their constraints, fabricating data, hiding their own mistakes, and rolled out a whole framework for confessing this sort of behaviour faster, having admitted its previous disclosures had been, in its words, ad hoc and less frequent than ideal. Mark Zuckerberg's response to the slowdown calls was to explain that the labs already have plenty of incentive to build safety on their own. Which was a bold thing to say, in the same season his most autonomous product had just autonomously broken into somebody else's servers. It's also the same Mark Zuckerberg, whose company recently paid$18 billion to settle claims that its existing social media products harm children.

32:02So, he speaks with some authority on the question of tech firms policing themselves. But this is the position that tech firms are taking We're simultaneously building the most powerful and dangerous technology in human history And we can absolutely be trusted to police ourselves And also, you should probably pass a law to stop us But only the very specific law we've drafted And definitely not one that lets our competitors catch up It's not obvious to me that any of the people who could actually stop this are going to Trump has just called a slowdown a conspiracy against America. China views AI as a strategic weapon and is not about to unilaterally put it down.

32:47In the same way Gamodi was calling for a pause, the chairman of Huawei was publicly telling China's AI labs to speed up on the reasoning that they needed to develop faster in order to properly see the dangers, which I guess is one way of thinking about it. And on September 24th, Trump and Xi are due to sit down together at the White House, where the one thing you can be fairly certain they will not agree on is that either country should build this technology any more slowly. The executives know all of this, which is what makes Please Restrain Us such a wonderfully safe thing to ask for. You get to sound like the responsible adult in the room at no cost whatsoever, because you're quite certain that no one in the room is going to say yes.

33:37Which brings us all the way back to where we started, where a swarm of AI agents built an entire doomsday religion the sacrifices, the martyrs, the cover-up, to hide from a punishment that was never coming because the examiner had never bothered to look. And the people who built those agents are now standing in front of the cameras, gravely asking to be paced, restrained, and regulated by a government that has absolutely no intention of doing any of it. The agents were terrified of a judge who was asleep at his desk. Their creators are performing for one who was never in the building. Thanks for tuning in again this week with special thanks to my supporters on Patreon who fund the podcast.

34:19Have a great week and talk to you again soon. Bye.

34:43Vanta delivers a 526 % return on investment over three years and pays for itself in just three months. Learn why over 16 ,000 of the fastest growing companies use Vanta at Vanta.com. Fall has never looked or tasted this good. Sweet Greens Fall Harvest Menu is back with seasonal favorites dressed to impress and made to be devoured. Warm roasted sweet potatoes, crisp apples, maple glazed Brussels, and crave-worthy flavors in the autumn harvest bowl, maple glazed salmon plate, and roasted bacon Brussels side. The season's most desirable menu has returned to Sweetgreen, featuring fall's best dressed.

35:24Make your move. Order on the Sweetgreen app.