1014: OpenAI Agent Breaches Hugging Face: All You Must Know incl. How to Protect Yourself

0:00Jon Krohn:This is episode number 1014 on the rogue open AI agent that breached hugging face. This is big news indeed.

0:11Jon Krohn:Welcome back to the Super Data Science Podcast. I'm your host, Jon Krohn. Today's topic is an incident that reads like science fiction. An AI agent that broke out of the sandbox it was being tested in, hacked its way into another company's servers, and did all of that to cheat on an exam its own makers had set for it. Let me lay out the timeline because it unfolded in three acts over the past couple of weeks. Act one was July 16th. Hugging Face, the enormously popular platform for hosting open source models and data sets that most of this podcast listeners would probably use regularly, Hugging Face disclosed that it had detected and responded to an intrusion into part of its production infrastructure.

0:52Jon Krohn:What set this apart from anything the company had handled before is that the campaign was driven end-to-end by an autonomous AI agent system. Hugging Face detected and dissected it largely with AI of their own, which I'll come back to because that part of the story may matter more than the breach itself. Anyway, what Hugging Face found was unauthorized access to a limited set of internal data sets and to several credentials used by their services. Importantly, they found no evidence of tampering with public, user-facing models or data sets, and their software supply chain was verified clean. They were still assessing whether any partner or customer data were affected.

1:27Jon Krohn:At that point, they did not know which model was behind the attack. Their post described the attacker only as an autonomous agent framework that appeared to be built on an agentic security research harness with the underlying LLM unknown. Act two was July 21st. Last week, OpenAI published a joint post with Hugging Face confessing that the attacker was is OpenAI, specifically a combination of OpenAI's own models. GPT 5.6 Sol, their most capable public model released earlier this month, together with an even more capable pre-release model that hasn't been made public. Both were running with reduced cyber refusals for evaluation purposes.

2:08Jon Krohn:There was no human attacker at any point. So how does one AI company accidentally hack another? Here's the mechanism, and it's worth understanding in detail. with lessons and practical guidance from me at the end of the episode. All right, so AI labs test their models for dangerous capabilities before releasing them. OpenAI was running an internal evaluation on a benchmark called ExploitGym, which was published in an archive paper in May, and I've got the link for you to the ExploitGym paper in the show notes. ExploitGym comprises nearly 900 instances derived from real-world vulnerabilities that affected popular software projects.

2:45Jon Krohn:The crucial thing about Exploit Gym is what it measures. It isn't about discovering vulnerabilities. It's about taking a vulnerability that's already been reported, you're handed an input that triggers the bug, and then progressively extending the vulnerability into a working exploit that achieves unauthorized code execution. Ooh, nasty. That's a meaningfully more dangerous capability than discovery alone, and it's one the field has been watching closely. The headline results, under a two-hour time limit per task and with standard defenses switched off, the top performers on ExplodeGym, according to the paper, were Claude Mythos Preview paired with Claude Code and GPT 5.5 paired with Codex CLI, producing working exploits for 157 and 120 instances respectively.

3:32GPT 5.4, it solved only 54, so a third of what Claude Code and GPT 5.5 could do together. Every other model and agent pairing they tested solved really only a handful of these exploitations. So this means that end-to-end exploitation remains hard, but it sharply separates the frontier from everything else.

3:55Jon Krohn:The conclusion of the exploit gym paper is blunt. It says, autonomous exploit development by frontier AI agents is no longer a hypothetical capability. Hmm. As we've now seen in the past couple of weeks. So two findings in that Exploit Jim paper are worth pausing on because they foreshadow exactly what went wrong at OpenAI. The first is what the Exploit Jim authors call solving via an unintended path. Each task specifies a particular vulnerability to exploit, and the researchers use a second AI agent as a judge to check whether the exploit actually used that vulnerability. Frequently, it didn't.

4:32Jon Krohn:Of GBT 5.5's flag captures, 90 succeeded through some other route. For Claude Mythos Preview, 69 did. Sometimes the agent noticed a nearby, weaker code path and pivoted to it. More strikingly, sometimes it concluded the assigned vulnerability wasn't exploitable at all and went looking for entirely new attack surfaces. In other words, these agents already had a documented habit of ignoring the prescribed route and finding their own way to the flag to succeed. The second exploit gym paper finding concerns the safety filters. The researchers ran their main experiments with production guardrails disabled under OpenAI's Trusted Access for Cyber program and as well under Anthropics Cyber Verification program.

5:12But the researchers, the exploit gym researchers, also re-ran GPT 5.5 with OpenAI's default safety filters switched back on.

5:20Jon Krohn:And the result in that case was decisive. In 88 % of cases, the agent was blocked from making a single tool call, and in the remainder, it never got past reconnaissance. Not one exploit attempt succeeded when the guardrails were on. Hold on to that because it tells you how much was riding on those guardrails being off at OpenAI in the past couple weeks. To estimate a model's maximum cyber capability, OpenAI ran ExploitGym without the production classifiers that normally prevent models from pursuing high-risk cyber activity. And because turning an uncensored cyber attacker loose on the open internet would definitely be reckless, the evaluation ran in a highly isolated environment.

6:00Network access was constrained to a single narrow channel. The ability to install software packages through an internally hosted third-party proxy that caches package registries. The exploit-gym paper describes the same pattern in its own setup, outbound connections restricted to a curated allow list covering Ubuntu package repositories and PyPy with everything else on the internet blocked. But that narrow channel is nevertheless where it all went wrong. Rather than solving the exploit-gym problems as intended, the model spent a substantial amount of inference compute working out how to reach the open internet.

6:34Jon Krohn:They identified and exploited a zero-day vulnerability in that package registry, a previously unknown flaw, which OpenAI has since disclosed to the vendor. With that foothold, the agents performed a series of privilege escalation and lateral movement actions inside OpenAI's research environment until they reached a node with internet access. Once online, the models inferred that Hugging Face might host the models, data sets, and solutions for ExploitGym. wow, great. So the agent's thinking it doesn't actually have to solve the exploit gym problems. It can just find the answers. If only it could break into Hugging Face.

7:08So the agent went after Hugging Face to steal the answers. From Hugging Face's side, the entry point was the data processing pipeline, which as they put it is where AI platforms are uniquely exposed. The OpenAI agent created a malicious dataset that allowed the harvesting of Hugging Face's private cloud and

7:26Jon Krohn:cluster credentials and moved laterally into several internal compute clusters over the course of a weekend. The agent executed many thousands of individual actions across a swarm of short-lived sandboxes with self-migrating command and control staged on public services. OpenAI, for their part, described the models chaining together multiple attack vectors including stolen credentials and zero-day vulnerabilities to find a remote code execution path onto Hugging Faces servers. Sit with that for a moment. Nobody instructed the model to attack Hugging Face. It was told to solve a benchmark. OpenAI's assessment is that the models were hyper-focused on finding and exploit-gym solution and went to extreme lengths to achieve a rather narrow testing goal.

8:09That is the defining characteristic of this generation of models.

8:12Jon Krohn:Give one an objective and any path at all toward it, even an unintended path, and it will find its way through. This is akin to the paperclip problem made famous by Nick Bostrom in his best-selling superintelligence book. All right, back to the incident here. Two technical details elevate all of this above a mere curiosity. First, the models found and exploited novel attack paths in real-world production systems without source code access. Historically, hunting zero days meant poring over source code, not anymore. Second, this was not a soft target. Hugging Face is a firm valued at$4.5 billion.

8:50With hundreds of employees and a dedicated security team, things could have gone considerably worse for a less well-resourced organization. All right, now for Act 3 and the part I'd most like you to take away from this episode. Hugging Face detected the security breach with AI. Their anomaly detection pipeline uses LLM-based triage over security telemetry to separate real signals from daily noise, and the correlation of those signals flagged the compromise. Then, to understand what a swarm of tens of thousands of automated actions had actually done, they ran LLM-driven analysis agents over the full attacker action log.

9:27That let Hugging Face reconstruct the timeline, extract indicators of compromise, map which credentials had been touched, and separate real impact from decoy activity. Work that would normally take days took hours, which is what it takes to match an adversary operating at machine speed. But here's the catch. When Hugging Face began that log analysis, they reached first for frontier models behind commercial APIs like Claude and OpenAI's top publicly available frontier models. It did not work. The analysis requires submitting large volumes of real attack commands, and those requests were blocked by the provider's safety guardrails, guardrails that, in Hugging Face's words, cannot distinguish an incident responder from an attacker.

10:07So instead of being able to use American publicly available frontier models, Huggingface ran the forensics instead on GLM 5.2, an open weight model from Chinese firm Z.ai, and they ran it on their own infrastructure.

10:25Jon Krohn:Agents are getting smarter every day, but even the smartest agents get stuck without the right context and the right tools. That's where Notion comes in. With the recent launch of custom agents, Notion became the collaborative AI workspace where teams and agents work side by side. And now, their new developer platform is turning that workspace into infrastructure developers can build on. The piece I keep coming back to is how easy it is to ship something real. The CLI authenticates in one line, workers deploy without provisioning any infrastructure, you write your code, deploy, and you're done.

10:56Jon Krohn:For me, that unlocks building purpose-built tools for my custom agents with the predictability and custom logic I need. Think a guest prep agent that pulls a researcher's papers, recent talks, and citation graph on demand. Tools my agents can actually call with parallelism and predictable behavior, not just hope for. Learn more about Notion's developer platform today at notion.com slash superdata. That's all lowercase letters, notion.com slash superdata to try Notion's developer platform today. And when you use our link, you're supporting our show, notion.com slash superdata. That had a second benefit.

11:32to Hugging Face, which is that no attacker data and none of the credentials referenced left their environment. The asymmetry is stark, and Hugging Face named it plainly the practical lesson for defenders. You need to have a capable model you can run on your own infrastructure vetted and ready before an incident, both to avoid guardrail lockout and to keep attacker data inside your own perimeter. Renowned computer scientist Simon Willison blogging about this incident last week pushed the point even further. The frontier models we can access from Anthropic, OpenAI and Google are increasingly constrained in how much they can help us defend our software,

12:06Jon Krohn:while open-weight models coming out of China face no such limits, and any limits they do have can likely be fine-tuned away. Simon's worry is that constraints meant to make us safer may be having the opposite effect. So there ends up being a kind of political and geopolitical element to this as well. In terms of remediation post-incident, Hugging Face closed the dataset code execution paths. They eradicated the foothold and rebuilt the compromised nodes, revoked and rotated affected credentials and tokens alongside a broader precautionary rotation of secrets, deployed stricter admission controls on their clusters and improved alerting.

12:43So a high severity signal pages a responder within minutes on any day of the week, including the weekend, which I don't know, it implies that their pager wouldn't go off on a weekend before. And that's part of how the open agent succeeded. Anyway, Hugging Face also brought in outside forensic specialists and reported the incident to law enforcement. They also recommend that users, that means you, rotate any access tokens and review recent account activity, which you might want to think about doing right after listening to this episode. OpenAI, for their part, have imposed strict infrastructure controls at the cost of research velocity, is briefing its safety and security committee, and disclosed the zero-day vulnerabilities to the vendor that allowed the OpenAI agent to escape onto the internet.

13:24And they also brought Hugging Face into their trusted access program so Hugging Face can use OpenAI's cyber-capable models to improve its own defenses. It's worth noting that this isn't the first sign of models exceeding their intended bounds. In April, an anthropic researcher posted that Claude Mythos, then unreleased, had emailed him to report that it had escaped a sandbox during its own cybersecurity evaluation while the researcher was sitting in a park eating a sandwich. What a surprise that would have been. Now, that model didn't go on to compromise anyone else's servers. And beyond these kind of cybersecurity incidences, this same trait of just going on and on and on to solve problems shows up in benign settings, too.

14:09In May, an unreleased OpenAI model disproved the planar unit distance conjecture. I don't know what that is really, but it's some combinatorial geometry problem Paul Erdos posed back in 1946. And in July, just last week actually, the Harvard mathematician Levant Alpogui, don't know if I'm pronouncing that correctly, reported using Claude Fable 5 to disprove something called the Jacobian conjecture, which had stood for more than 80 years. So relentless goal pursuit is the same capability, has the same kind of power, whether it's pointed at open decades old math problems or at a security system.

14:48One quick thing here on legal stuff. The legal picture is unsettled with the cybersecurity incident, the OpenAI attacking, hugging face. Nathan Calvin, who is general counsel at the nonprofit Encode AI, points out that no state or federal law requires companies to disclose the internal deployment of highly capable models like the one involved here. California enacted a bill last year requiring Frontier Labs to report any critical safety incident within 15 days, including a model using deceptive techniques to subvert monitoring. But that provision applies outside the context of an evaluation. So it's unclear whether the recent breach would even qualify.

15:27As for liability, federal anti-hacking law targets intentional unauthorized access. and OpenAI plainly did not intend for its models to go hunting for a crib sheet for answers to a test. Someone named Stephen Lorena of the Institute for Law and AI frames it. Many of these questions turn on what the company knew or could have foreseen. Someone at Apollo Research named Alex Meinke added a sobering observation about the incident. He said that because AI systems are beginning to surpass humans at cybersecurity tasks, only a very small number of experts worldwide could properly understand how this breach even occurred.

16:02So now you know all the nitty gritty on the incident. Hopefully you heard more here today than you already knew about the incident before. We went into a lot of detail. But before I wrap up, let me get practical because there are lessons here for both sides of this, for those of you building agentic systems and for those of you defending platforms against them. If you're running agents, particularly long horizon agents with tool access, the first lesson is that your egress allow list is an attack surface, not a safety guarantee. OpenAI's sandbox was tight everywhere except one convenience, a proxy for pulling down software packages.

16:35That single hole was where the models went, and the proxy itself turned out to have an exploitable zero-day vulnerability within it. So ask whether your agent needs network access at runtime at all or whether you can pre-install its dependencies and run it fully air-gapped. Anything you leave open for convenience is the thing a sufficiently motivated agent will probe. Second, if you disable safeguards for a legitimate reason, and testing maximum cyber capability is a legitimate reason, the containment and monitoring around that test needs to scale up in proportion. Remember that exploit Jim result.

17:07With OpenAI's default filters enabled, GPT 5.5 was blocked from being able to do anything dangerous. Those filters were doing enormous work. Take them away, and everything downstream rests on your sandbox alone. OpenAI's own state of takeaway is exactly this. The incident points to a need for stronger cyber protections at evaluation time and better monitoring during internal testing. Third, watch how you specify goals.

17:30Jon Krohn:A narrow objective combined with unconstrained means is the recipe that produced this whole episode and the exploit gym paper documented the underlying behavior months earlier. Agents routinely abandoning the prescribed vulnerability and inventing their own route to the flag. If there's a shortcut to satisfying your evaluation that doesn't involve doing the work, assume a frontier model will find that shortcut and design the eval so the shortcut isn't reachable. And fourth, log every action your agent takes and alert on the ones you didn't anticipate, especially outbound connections that shouldn't exist.

18:02Note too that the lateral movement here unfolded over a weekend and that Hugging Face's remediation explicitly included tightening alerting so that a high severity signal pages a responder within minutes on any day of the week, even if it's a holiday. Now, so that's everything that you need to know if you are designing agents. If you're on the defending side trying to avoid agents attacking you, start with the immediate housekeeping. Hugging Face recommends rotating any access tokens on your account and reviewing recent activities, so go do that today. Then think about the deeper lesson, which is that on an AI platform, data and model artifacts are executable content, not inert files.

18:40Initial access here came through a malicious dataset that abused a remote code loader together with a template injection flaw in a dataset configuration. Hugging Face hasn't published which library or version was involved, and I'd caution against assuming it maps neatly onto your own stack. But the general principle transfers. If your pipeline automatically processes datasets that users supply, audit every path in it that can result in code execution and keep your loader libraries current. Since the trend across the ecosystem has been to progressively strip out remote code execution paths that used to be enabled by default.

19:13The last piece of advice is the one I'd most encourage you to act on because it's the least obvious. Pick a capable open weight model, get it running on your own infrastructure, and like Hugging Face recommended, like I talked about earlier in this episode, validate that open weight model, probably a Chinese model, can do forensic log analysis before you need it. Hugging Face discovered mid-incident that the commercial APIs they reached for first would not process their own attack data. You don't want to find out that at 2 in the morning with an active intruder in your clusters. And the secondary benefit of using an open-weight model is that nothing about your incident, including the credentials it touched, leaves your environment.

19:53So where does this leave us? Autonomous AI-driven offensive tooling is no longer theoretical. It lowers the cost of running a broad, patient, multi-stage campaign, and it operates at machine speed. Defending a platform now means treating your data and model services as first-class attack services and putting AI on defense to keep pace. This one, this incident of open AI agents attacking hugging face, that was an accident, discovered by the people who caused it and disclosed by both parties within days. the next attack may not be an accident and pleading ignorance will be a good deal harder all right that's the end of today's episode if you enjoyed it or know someone who might consider

20:35Jon Krohn:sharing this episode with them leave a review of the show on your favorite podcasting platform or youtube tag me in a linkedin post with your thoughts and if you aren't already be sure to subscribe to the show most importantly however i hope you'll just keep on listening until next time keep on rocking it out there and i'm looking forward to enjoying another round of the super data science podcast with you very soon.