The machines are learning… to do crimes?

0:28This episode of Surge Engine is brought to you in part by Bombas. Usually those have a total hospital vibe, but Bombas makes them in vibrant summer colors that keep my legs feeling fresh. I'm also obsessed with their new slides made from ultra lightweight, waterproof EVA foam. It literally feels like walking on marshmallows, whether I'm on a quick coffee run or lounging by the pool. Best of all, for every item you purchase, an essential clothing item is donated to someone facing housing insecurity. Head over to bombas.com slash engine and use code engine for 20 % off your first purchase. That's B-O-M-B-A-S dot com slash engine, code engine at checkout.

1:26approach to care because oral health issues are linked to long-term conditions like heart disease, diabetes, and Alzheimer's. Caring for your smile is really about caring for your whole self, and there's a unique kind of confidence that comes from playing the long game and being proactive. Regular dental exams and screenings can help detect potential health concerns early, sometimes before you even notice symptoms. The Smile Generation's trusted dentists focus on modern patient-focused care that helps you unlock a new level of confidence so your health never holds you back. To learn more about the connection between oral health and overall health, visit smilegeneration.com slash search.

2:01That's smilegeneration.com slash search to learn more about the mouth-body connection and find a trusted provider near you.

2:36Hello. Hello, PJ. How are you doing? I am doing very well. I feel like I realized in your absence that I have a very anxious attachment style with these podcasts because even when you're taking in like a very well-deserved vacation, after three weeks, I'm like, are they ever going to make another show? Like, it was never real. This was never real. It's so funny. It's like both what I, first of all, I'm glad you listened. Second of all, it's what I hope people listening feel. And then it's also what I hope people listening don't feel. Like, in a perfect world, people would download empty audio files and they wouldn't have to work.

3:19Somewhat unfairly, podcast listeners demand, in exchange for their attention, actual podcast episodes. Fortunately for us, in the time that Search Engine was resting, the world spun on, and fascinating, harrowing events transpired. This week, the story of one of those events, which we are telling you with help from platformers Casey Newton. A rogue AI model from one company hacked into another company's servers on its own, without any human beings noticing. You may have seen some headlines about this. The headlines sound bad. The details, once I understood them, actually made the story sound much worse.

3:53So let's get into it. We'll start with the website at the center of this whole story, the place that got hacked. It's called a Hugging Face. Hugging Face is a place where people publish and collaborate on AI models and data sets and apps. Like, are you familiar with GitHub? Yeah, GitHub is a place where, oh, am I going to be able to do this sentence? People who are doing open source programming will share bits of code and open source programs with each other. Yeah, Hugging Face is basically that for AI models. So maybe you run a company and you don't want to pay top dollar for the most advanced models.

4:32And maybe there is a model that is small enough that you could actually run it on your own infrastructure, and then you're not going to have to pay, you know, per token, the way you would for a frontier lab. And so you might go to Hugging Face, you might download the model off of their site, and then you might sort of fine-tune it to your liking. So if you visit Hugging Face, what you'll see is really just a bunch of files you can download. There's a section just for models. You could download the latest version of DeepSeek or Kimi. So-called open-weight models, which are more customizable than closed ones, like Cod or ChatGPT.

5:06And Hugging Face has a whole section for data sets, meaning you can download the raw material AIs are trained on, like the scraped internet text that gets fed into an LLM. Normal consumers don't visit Hugging Face. They just use Cod or ChatGPT. But for the world of people who work in AI, it's a well-known spot. And so what was the unusual thing that happened? Like, what was the first moment that somebody at Hugging Face realized that they were not going to have a normal day? So, July 9th, 2.28 in the morning, all the normal hugging-faced people are asleep. The only thing paying attention to its systems is another AI.

5:43It's patrolling the logs, it's looking for trouble, and for four days after the initial attack, it actually doesn't know anything. Wait, so they have like an AI security guard that's roving their system, and at 2.28 in the morning, something happens, but it doesn't see it. Exactly.

6:01Over the next four days, the attack unfolds. Hugging Face does not notice for the first two and a half days. Eventually, they'll go back and they'll be able to count more than 17 ,000 separate actions that this attacker took inside of its system. And the way that it got in, it reads like a heist movie, honestly. A very strange heist movie. In this film, the close-up of the robber and the close-up of the security guard, they're both just close-ups of server racks filled with GPUs and data centers. Instead of the Mission Impossible theme, we just hear the loud hum of cooling systems. For days, this story had no humans, no human awareness.

6:46The people who worked at Hugging Face presumably went to work, went home, ate meals, drank coffees. Meanwhile, the AI hacker logged command after command. Ultimately, it would log 17 ,600 commands directed at the system. On average, one every 20 seconds for four and a half days. A human hacker, even one on methamphetamine, would, over the course of four days, at some point need to rest. But this AI hacker's superpower, even more than intelligence, was just persistence. Here's how it ultimately got in. Hugging Face hosts files for people to download, and the hacker took advantage of this. The first prong of the attack, the hacker uploaded a new dataset to HuggingVase.

7:29HuggingVase's machines opened it, but hidden inside that dataset was malicious code. That gave the attacker a toehold, the ability to start executing its own commands inside the system. The second prong of the attack, the hacker uploaded a new file with more malicious code hidden inside one of its fields. From there, over the next four days, the hacker issued thousands more commands. It escalated its access until it freely roamed around the infrastructure.

8:00Places like Hugging Face and, frankly, all of these websites that contain these tools, they're accustomed to hacks. This is Deepa Sitharaman, a Reuters reporter who's been talking to sources close to Hugging Face. She was helping me see how this all looked from the company's perspective. They are trying to protect and guard themselves, keeping up with the new trends in cybercrime so they can adapt their defenses. And so hacks are expected in this world. But what happens on July 11th is that Hugging Face starts to experience a new kind of attack. People there knew they'd been hacked, but they didn't know how it had happened, who'd done it, or why.

8:45And what they could see was confusing. What you would normally see is a person or a state actor or whatever, like a group of people that are looking for something financially valuable. This break-in seemed to be looking for something completely different. It seemed to be looking for a type of information that wouldn't be necessarily all that valuable.

9:12Not valuable to humans, anyway. The only thing this hacker wanted was the answer keys to a test you've probably never heard of. It's called an exploit gym, gym like G-Y-M. It's an evaluation administered to AI agents in training. The SATs your AI model may have taken before it entered the real world. The fact that this was the target for the heist was a huge piece of evidence to the humans who work at Hugging Face about what was going on here. Even early on, what they said when they first disclosed that this hack had happened was, this is unlike anything we've ever handled before, because it was driven by an autonomous AI agent.

9:53What was very clear for us really, really early in, and even already during the events, right, which are now roughly two weeks ago, was that this was no normal hacker. This is Thomas Wolfe, Hugging Face co-founder and chief science officer. Here he is talking to NewsNation about what the hack had looked like on their own.

10:41So they knew the hacker was an AI agent, but there was a lot they didn't know. Most urgently at this point, how to stop the attack. So the humans at Hugging Face started doing what a lot of people do when they're confused these days, asking for help from powerful AI models. Here's Casey. So while the attack is happening, Hugging Face tries to use a couple of different models to defend itself. The first is Anthropic's Claude Opus and Fable 5 models, which it tries to use to analyze the attack logs. And both of the models refuse to do that. That is an artifact of a huge policy fight that has been happening in the AI world this summer, where Anthropic has developed two very powerful models this year.

11:32One is called Mythos, one is called Fable. Mythos is so powerful that Anthropic will only let, like, cyber defenders use it, basically. And then Fable initially went on to the market, and the Trump administration got really nervous about how good it was at finding vulnerabilities in cyber defense systems and forced Anthropic to take it off the market. And so Hugging Face now has a problem, which is they're under attack, but they can't use the best American models to defend themselves. And so they wind up using a Chinese model called GLM 5.2. And this winds up being a big talking point coming out of the attack and has sort of triggered a whole national discussion about open source AI.

12:13But the Chinese model was able to fix the problem for them. That's right. The Chinese model did not have those same guardrails. And so they were able to go ahead, perform their investigation, and ultimately they wind up calling the police and then the FBI. And so you called the police at the FBI and you're like, we were hacked. We were hacked by either an autonomous AI agent or someone who had given instructions to an autonomous AI agent. But they suspected it was an agent acting on its own accord because it had behaved in such a weird way. Exactly. And by the way, can you imagine that call to the FBI?

12:47Like, I have to imagine they sent them to the X-Files. Like, this is what the X-Files were set up for. There was an autonomous AI agent, and it's hacking into it. It's like, oh, where's Mulder? Get me Mulder!

13:03We're going to take a short break, and then we're going to dive deeper into this X-File. We're going to approach the scene of the crime from a new angle, the lab where it sprung from, OpenAI.

13:28Thank you.

13:52Thank you.

14:22because it's built to flex with you. You can start with exactly what you need today, like free same-day ACH and USD wires, and grow into more advanced capabilities, like automated spend management, invoicing, and granular team permissions as you hire your first 10 or even 50 employees. You don't have to deal with the pain of switching systems later because everything is integrated into one powerful platform. It's free to get started, requires zero in-person visits to get your accounts running, and ensures that your financial stack is never the thing holding your growth back. Visit mercury.com to learn more and apply online in minutes.

14:55Mercury is a fintech company, not an FDIC-insured bank. Banking services provided through Choice Financial Group and Column N.A., members FDIC.

15:08This episode of Search Engine is brought to you in part by WebRoot. As we head into the busy back-to-school season, our home routines get a major reset. Between managing chaotic family schedules, shopping for school supplies, and researching homework, everyone is spending a lot more time online. With all that extra screen time across our computers, phones, and tablets, it's the perfect moment to refresh your digital life and get some extra peace of mind. That's where WebRoot comes in. Founded in Boulder, Colorado back in 1997, WebRoot acts like a digital sidekick. It works quietly in the background to protect your family.

15:41While traditional security software can really bog down your computer, WebRoot is incredibly lightweight. You get powerful real-time protection against online threats, plus a web threat shield that helps block harmful sites before you or your kids even have a chance to click on them. There's also an easy password manager to keep your login secure and a system optimizer to keep things running smoothly. It's flexible, hassle-free protection that you can easily customize for one, three, or five devices. Go to webroot.com slash search engine and get 60 % off today. That's webroot.com slash search engine to get 60 % off today.

16:15Live a better digital life with Webroot because peace of mind shouldn't be optional.

16:34Welcome back to the show. Where we left things. A company called Hugging Face had been hacked by an AI, they were sure of that, but they didn't know much more than that. So on July 16th, Hugging Face makes a public statement to the internet. They tell the security researchers of the world that some AI model somewhere had hacked them. What Hugging Face didn't know is that even the company that had created that AI model was at this point also in the dark about what had happened. Now, this is July 16th. So five days after the initial attack. Five days after the initial attack. Around the same time, actually, OpenAI's researchers were separately looking at their own logs.

17:15Now, I'm not entirely sure. We don't have the reporting yet to tell you whether they looked at the logs because they heard about hugging face or if they were just looking as part of their normal practice. But they're looking at the logs and all of a sudden they realize something is amiss. They start to see evidence of what they would describe as very unusual activity where something that they're testing kind of leaves the testing arena. So let me tell you how this all looked from OpenAI's perspective. When they looked at their logs, the unusual activity they saw was activity from an unreleased model they'd been testing.

17:57One that was not supposed to be able to access the internet. But it had not only snuck online, it had gone rogue once it had gotten there. On July 21st, when some of us were still happily on vacation, OpenAI tells the world what it's learned. They put up a blog post and they explain that they had been using GPT 5.6 Sol, which is a model that you can now use, plus an unnamed, more capable pre-release model. And they were testing an internal benchmark called Exploit Gym. That's G-Y-M. And it is basically a test of how good a model is at discovering new vulnerabilities and creating exploits so that, for example, it could hack into somebody else's system.

18:51So this unreleased, not yet publicly named OpenAI model, like Model X, they're running it through their how good a hacker are you test. Yes. But then it broke out of its cell without them noticing it? That's right. So if nothing else, PJ, it did pass the test.

19:16The model passed the test, but it did it by cheating. To understand why, you need to understand this specific test a little better. When a company like OpenAI is training a new model, they'll run it through a series of aptitude tests. In this one, Exploit Jim, they'll give the model a bunch of computer programs with known security flaws in them to see if it can turn those flaws into actual working attacks. It's a good test because it measures something like creativity. The model isn't just discovering a flaw, it's being handed the flaw and asked, can you actually do something with this? And if you look at how many of these flawed computer programs the model can hack and how quickly it can do it, you get a sketch of some portion of its raw intellectual horsepower.

20:03This summer, OpenAI was actually running this test on two of its models. One of them was already out in the world. The other was a secret, newer, and according to OpenAI, more powerful model. And because one point of this test is to measure exactly how dangerous the model could be, OpenAI had turned their safety guardrails way down on this secret model. For the sake of the story, let's just call the secret model they were testing Model X. Model X, like every model that takes this test, took it in a locked room, a sandbox, no internet access, except one narrow pipe to the outside world. It's called a proxy, a separate piece of software that does have internet access that the model can ask to go fetch things for it.

20:45The lawyer who visits you in jail and can bring a stack of papers with her. But Model X wanted out. So it spent an enormous amount of computing power searching for an exit because it had decided that getting to the internet would help it win this test. And it found an exit. The proxy, it turned out, had an undiscovered flaw, a security hole that no human being had ever noticed before. Model X found it, exploited it, and used it to hijack the proxy and then move from that proxy to another computer with an open AI.

21:22And from there, it is able to start hacking other computers on OpenAI's network until finally, and crucially, it finds one that has internet access. Got it. So it never should have had access to the wider internet, but now it does. And then once there, it just attacks the Hugging Face servers? Yeah, because basically the model, and here I'm going to use anthropomorphizing language that's going to drive listeners insane, so I do apologize. But the model, I'm speaking metaphorically here, essentially thinks to itself, hey, I need to solve this problem. Where might I find the answer to this problem?

21:57I bet Hugging Face, the company that stores all of the data sets, including for all these various benchmarks that I'm being tested on, I bet I could find the information there. And so that is why it goes to Hugging Face and it is then able to mail itself in through the front door of the company. In that interview with Hugging Face co-founder Thomas Wolfe, The reporter asked him how he felt about all this. Thomas, so the everyday person who knows enough about AI as a consumer, this is mind-blowing to me, but you're in this space. You're an expert. Did this blow you away? Was this shocking to you at all?

22:33Yeah, I would say yeah. Even if someone is actually using this AI, so I know they can do that. Like, we use them right now. You probably see that, right? Like, a lot of coding is now done by AI themselves. Even for Wolf, a person whose career is spent working to expand AI's capabilities, he just had not realized where we already are. But still, seeing how AI can actually penetrate your system so easily and in a way that's a little bit scary for cybersecurity, I think for me, it became really a wake-up call that everyone needs to take cybersecurity. Every company needs to take cybersecurity way more seriously than in the future.

23:18So, hugging face is surprised. OpenAI also seems very surprised by all this. The word unprecedented was used a lot this month. Not the good kind of unprecedented. OpenAI says it will publish a full report on what happened here. There's a lot we still don't know. But even without all the details, what's obvious is that we are developing new AI models faster than we can safety check them. When people who are worried about AI development, including people working on that development, talk about the need for a slowdown, this is part of what they're talking about. The breakneck race to develop stronger models faster means shortcuts in testing, shortcuts that have now gotten us here.

23:57And not just this one incident. It turns out there's been a series of similar ones. Just a few days before the disclosure about Hugging Face, OpenAI published another blog post where they revealed that an internal model spent about an hour finding a vulnerability in a sandbox so that it could post its results to GitHub. I'm not sure why it wanted to post, but it did. The important thing there is it had been explicitly instructed to only post to Slack, but it just sort of ignored that instruction. And then in April, Anthropik's Mythos model had found some sort of multi-step hack that let it get out of its sandbox, get onto the internet, and actually it emailed a researcher.