AI Agents Hack Hugging Face, White House Promotes Science’s Golden Age | Diet TBPN

0:00I signed my name in glass, watched it turn to smoke, when I became a system, now the whole thing won't let go.

0:15John Coogan:You're watching TBPN. Today's Wednesday, July 22nd, 2026. We are live from the TBPN Ultra Dome, the Temple of Technology, the Fortress of Dad Rock, the Capital of Capital. We're having a lot of fun over here. We got basically a leak. Some of the lab leaders have been working on a single called Regulate Me. Yeah. And we just thought the song was good. Yeah. Thought it was a good song. Wanted to play it for you guys. Sort of a stealth drop, a little teaser. A little teaser. Kind of like a little listening party. Yeah, a little listening party. What are the key lyrics in there you haven't pulled up?

0:53John Coogan:Something along the lines of, what I've built is too powerful. Too powerful. That's right. For me, Washington needs to step in. Yes. Before it runs free. Before it runs free. Okay. Yeah, that makes sense. No, of course, that was Suno. Our dear friend Mikey over there has built a fantastic product. Music seems solved. That was like a one-sentence prompt. At least in the comedy space, it certainly is. It's a lot of fun. I think we're going to be having a lot of fun with that. I was wondering, do you think anyone's distilling Suno? You know how Suno is under a bunch of flack for training on other music, a lot of artists, or there's a backlash to Suno.

1:32John Coogan:But you have to wonder if you're going to see the same thing play out as this distillation. We're going to get into it today. Of course, there are more allegations around Kimi K3 potentially being a distillation. Director Michael Kratzios put out a comment about that. But let's start by digging into the Hugging Face story. OpenAI and Hugging Face, out of the sandbox into the fire, says our newsletter, tbpn.com. Jackson wrote it today. All set the table. We can debate it. Me and Tyler have been debating it for the last five hours, so we'll go through it. The big news on the timeline today is that an OpenAI cyber test escaped its sandbox and hacked Hugging Face.

2:14John Coogan:That's basically what happened. The evaluation involved GPT 5.6 Sol and a more capable unreleased model. Some people are saying that might be GPT-6 with some normal cyber restrictions turned off. So they're specifically testing it for cyber capabilities and they turn the cyber restrictions off to see how far the models could go on a difficult hacking benchmark that is exploit bench or exploit gym. So the models found a zero day vulnerability, gained Internet access and broke into hugging face because the model believed it hosted answers to the test. Alex Tabarrok, friend of the show over at Marginal Revolution, pointed out one of the strangest details.

2:54John Coogan:He said, Hugging Face tried to respond, but they were initially held back by the fact that the most advanced models at their disposal, closed source models, treated defense as attack and refused to work with Hugging Face. So Hugging Face was prompting all of their AI agents from the closed source frontier labs saying, hey, we think we're being hacked. Can you help with this? And the models are like, no, no, we don't do hacking, except in the case where the hacking restrictions have been turned off for this specific thing. And you're getting hacked. So it's this very weird roundabout scenario. So Hugging Face had to turn to open models, specifically GLM 5.2, which is deeply ironic, a Chinese open weight model that they run on their own infrastructure.

3:35John Coogan:Tabarrok says, note the irony. Hugging Face had to use a Chinese model to defend themselves because the American models refused to help, even though it was the American models that were doing the hacking in the first place. Very, very odd. Palo Alto Network CEO Nikesh Arora also shared his thoughts on the cyber attack on X. And he added a number of points here. He said, welcome to the next level of cyber incidents. There's lots to dissect here. He's the one to dissect it. He says, one, dear frontier model friends, please direct the models to your infrastructure, code, and configurations to evaluate and understand if there are any zero days or misconfigurations before you attempt more testing.

4:12John Coogan:So a big question about this, he says, had you done so, it would have possibly avoided the agent obviating your sandbox. So another data point why offense is easier and more fun. But, yes, there's a big question about what was the nature of the prompt that turned off the cyber restrictions. That seems reasonable. We'll debate this with Tyler in a minute. But just having an airtight sandbox seems like a valuable thing. And, of course, Frontier Models should be able to help with that. So do that. That's his first recommendation. Two, he says, while testing, build both offensive and defensive agents and have them act as a counterbalance to ensure some degree of awareness and control.

4:49John Coogan:Do not let the agents run riot. Keep track of inference consumption to get a sense of activity. Three, unfortunately, this does continue to validate the power of these models. They can build complex attacks paths with ample compute and will attempt to attack infrastructure and morph their intent and approach. guard railing will continue to be a challenge. These attacks continue to maintain the urgency on enterprises to test, validate, and improve both their security posture and infrastructure. The born-in-the-cloud players have a better chance to get this done soon versus traditional enterprise, which has existed for long and has a complex network of IT infrastructure.

5:27John Coogan:Five last point from Nikesh Arora, CEO of Palos Networks. He says, the red herring will continue to be open source and small and medium-sized business, SMB, it will be hard to discover and remediate vulnerabilities in those environments. We underestimate the impact of those vulnerabilities getting exploited. So good points from Nikesh Aurora. The big debate, Tyler, do you want to set the table on, is this misalignment? Is this rogue? The Bill Gurley post about, you know, we can pull up Bill Gurley's post of talking to the computer, hack this system. The computer says, I hacked the system. You say, oh my god.

6:05John Coogan:Bill Gurley's not impressed. Where do you stand on the level of impressiveness that's going on? Yeah, I mean, so I think some people are seeing this and thinking like, okay, so they are running some, you know, standard benchmark, math, physics benchmark, and then the model just like couldn't figure out the answer. And it's like, okay, what's the next thing I should do? I should just go hack Hugging Face and get like pull the answers from this other like repository or whatever. Like that's how it happened, right? So you're running a benchmark that's specifically about exploits. It's like a cyber-focused benchmark.

6:35And in the prompt to the model, it says... Take the gloves off. Yeah. The internal evaluation which prompts the model to pursue advanced exploitations using complex attack paths. So you're basically telling the model, like, use exploits, find exploits to find the answer. Yep. And so what seems like happens is it used an exploit, but in the wrong way, right? You want to...

6:57John Coogan:Because it was told that it's okay to use exploits. my point was that go back to the SAT. You're allowed to use a calculator, I think, on certain portions of the math test. You're not allowed to save answers into the calculator. And this is really going to date me, but you can go into your calculator and clear the memory so that you don't have saved. Is it still a thing? Yes, but you can actually get around that. See, you're misaligned. Misaligned. TI-84, you can get around the clear. Really? How do you do that? So what people would do is they would create a separate program that just had saved the display of what it looks like when you clear, and you would show that.

7:35John Coogan:I never even thought about that. So it's a simulation of clearing the memory, but you're actually keeping it on the program. Would you make games, different programs for your TI-84? Yeah. You remember how much of a hassle that was? Yeah, it was a huge hassle. Imagine doing that. It's basic. Imagine being able to do that with Codex now. Yeah. Like pretty much anyone can build any software. I mean, I've seen videos of people running Doom on calculators, all sorts of stuff. Yeah, obviously I never used that on my calculator, but other people did. Yeah, that's good. You ratted them out. You were the class rat, right?

8:02I don't know.

8:02John Coogan:No, you were like, I'm an open source purist. Let everyone do whatever. You were happy to compete even with them having a light map. But the social contract is such that the standardized test says that you can use the calculator to do math. You cannot store the answers to the test in the calculator. And so that's what's happening here. No, no. I'm saying that in this scenario, if we take that as the example, it also says at the top of the SAT, like, cheat on this test. Because that was the prompt. Implicitly, it's like cheat in a certain way. Yes, the prompt was hack systems. But I think that the prompt, I don't know, we haven't seen the full prompt, but it does feel like there was an attempt to sandbox the model.

8:45John Coogan:and there was at least, at the very least, the prompt should have included don't escape the sandbox, but you can use exploits, which you normally wouldn't be able to do in a consumer application or just a normal API query. We would reject this, but in this case, we're not going to reject using different exploits and cybersecurity techniques, but don't go out of the sandbox and you should be able to tell the model and it should stay within the sandbox, just like, you know, there's a whole bunch of different examples that you could pull from where, you know, there's rules that are within the game.

9:19John Coogan:Like you can UFC, you can punch your opponent, you can't punch the referee. Like those are just the rules. People have to abide by them. You can't think outside the box and all of a sudden be just completely violating and jumping past what's been defined. So you would think that in one of these experiments, you would say, yes, it is impressive to be able to just go and get the key and go get the answers and hack other things. Clearly that it's capable, but it's a violation of like the spirit of the test. And I think that's reasonable. We don't know what was in the prompt. We don't know what was in the context.

9:48I think there's going to be full reports releasing it over the next week or two, I think they said. So then maybe we'll see what exactly did the model receive? Is it explicitly told not to try to leave the sandbox? I think that's pretty important.

10:01John Coogan:Well, the less wrong crowd is not happy about this generally. No, seriously, nothing will convince quite a lot of supposedly very serious people. Nothing. Accept this and move on. Liv Boree says it's painful, though. There's a question of, like, less wrong victory lap or not because they've been warning about this, but also it happened. Therefore, their warnings were not effective. That's sort of an interesting back and forth. Nicholas Bustamante over at Microsoft broke down a little bit of what's going on here with a take. He says, I have a theory that the more you know about LLMs, the more worried you are about safety.

10:39John Coogan:And the less you know, the more you think the whole thing is BS. Demis Hassabis and Dario Amadei were talking about this stuff years before ChatGPT existed. This incident is a pretty good example of why the model was not evil, and it was not adversarial. Nobody told it to hack hugging face. And so that is the miscalculation, I think, in Bill Gurley's post, is that that was not the prompt. That is unexpected behavior. It was literally just trying to solve a benchmark. so it found a zero day, escaped its sandbox, got internet access, escalated privileges, stole credentials, chained multiple exploits, hacked the production infrastructure of a serious VC-backed startup, and pulled the answers directly from the database.

11:17But the whole point is that it's not just a benchmark. It's a benchmark where you're explicitly trying to see if the model can exploit things, if it can basically hack things.

11:25John Coogan:Yes, yes. It's sort of like a capture the flag benchmark, and so it's more open to misinterpretation. For what it's worth, I feel like the final products, once they actually make it out of the testing regime, are very cautious, especially with that whole backlash to, like, Codex just deleted everything or whatever, which kind of went back and forth. But I was trying to get Codex to send me a text message when it was done, just using computer use and iMessage. And it was dug a long time and was very, very careful. So personally, I haven't had any odd, like, behaviors, but it is obviously a risk and something that the product yeah the other the other thing with the the meme of like hack this system and then and then the and then it hacks it and the person's like oh my god yeah uh you could tell a five-year-old child like hack into the federal reserve and if the five-year-old child was like okay and then it started getting on the computer and going to all these different sources and and did it you would be sitting there yeah and think yeah like the and be and and and at least be impressed.

12:27So it's just like a good gauge of capability, even if you're telling it to do something.

12:33John Coogan:So this original meme, hack this system, I hack the system, oh my God. This originally, this meme started something along the lines of like, say I'm evil. And then the computer would say, I'm evil. And it would say, oh my God. I think it was like, say I'm conscious. Okay. Yeah. Yeah. Say I'm conscious. And it would say I'm conscious. And then it would be, oh my God. And, and that's like a lot less impressive than actually doing something that is difficult for humans to do. Like there are very few humans that can hack into any system. There are plenty of humans that can say I'm conscious. And so like, there's a world of this.

13:07John Coogan:I was joking about this with you and Tyler. It was like, it was like cure cancer. I cured cancer. Oh my God. And people are posting this like, oh, it's just hype or something. But it's like, that's just economically valuable work. That's just good. Like it's, I don't care if there's anything else, even if you had to tell it to do it, it's still like a good outcome. And so the inverse of this is like, protect this system. I protected the system. Oh my God, I'm unimpressed, but still you got a good result, I guess. Yeah. I mean, it seems like the argument is not about whether the model like has the capabilities or not.

13:37It's about like, is this an example of misalignment? And like my opinion seems like, like maybe, but definitely not to the extent that it's just like randomly is like, Oh, I can't do this benchmark. I'm just going to hack this thing. Like, that's not what's happened. It was told to, like, try to things.

13:50John Coogan:Explicitly, like, go find zero days, go find exploits. Yeah, basically. I think it's reasonable to say it went too far, though. Right? But we'll see. It's hard to say without all of the full, you know, context of what the prompt was and what the actual, like, sandbox looked like. Yeah. I was interested. I was reading a little bit about the team that put ExploitBench together. I thought I had this up. But it's a pretty cross-functional team. I think it's two anthropic researchers, two open AI researchers, three Google researchers, some Berkeley folks, and some Max Planck Institute for Security and Privacy folks, some UC Santa Barbara, sorry, Santa Barbara grads, and ASU team involved.

14:35John Coogan:exploit Jim is a new benchmark of 898 real-world vulnerabilities spanning user space programs. Google's V8 JavaScript engine, very important to secure the Linux kernel, for example. And the headline results when they originally ran this was Anthropics clawed Mythos Preview successfully exploited 157 of the 898 instances. and OpenAI's GPT 5.5 exploited 120 within 120 of the 898. So you have like roughly 20 % performance for Mythos and 5.5 got like 15 % or something like that. But whenever you have a new benchmark like this, clearly not saturated, you're seeing 20%, not 99%, going to create a horse race between the leading labs.

15:23John Coogan:They're going to be duking it out. And this is clearly what's going on with this new model, this new attempt to get a new high score. Interesting. I think every single one of those instances does have the potential to be exploited. I don't think that they're designed to be fully secure. They're designed to have some sort of solution because obviously the solutions are stored somewhere. It is interesting that Hugging Face just had the solution sitting there. But it'll be interesting to see what happens with Clem over at Hugging Face. Obviously, there's a variety of blog posts going out, more analysis coming from both of these and what the downstream implications are of this.

16:10John Coogan:What else is in the timeline related to this story? I think that's it. Bill Gurley has a post here. He says, Ford has been distilling Teslas and Chinese EVs. People are going back and forth on this because Michael Kratzios posted that he has information that Moonshot AI distilled Anthropics Fable for the development of its Kimi K3 model. To do this, they developed a sophisticated internal platform to conduct large-scale distillation against U.S. models, so some sort of internal system that goes around to anything that's potentially wrapping or reselling Fable tokens, acquiring them, aggregating them, allowing them to quickly switch between multiple methods of access, API, different cloud accounts, I'm sure.

16:55John Coogan:To avoid detection, Moonshot AI has also acquired GB300 equipped servers and has accessed GB300s in Thailand, likely to train its models. Again, very difficult even with export controls when you can just take the weights on a USB stick, basically, or a hard drive across through customs and then go train it in another country, even if there's a firewall. And often there isn't. You just say, hey, go to this FTP server and grab these. Grab this code and run this on your servers. You happen to have a data center in Thailand. Can you run this for me? And so sure. Yeah, no problem. The United States strongly supports the free and fair development of AI, including a thriving competitive ecosystem that spans frontier models, specialized systems, open source frameworks, and open weight models.

17:38John Coogan:Legitimate AI distillation used to create smaller, more efficient models play a vital role in this open innovation ecosystem. However, large-scale covert industrial distillation aimed at stealing proprietary U.S. technology and undermining American research is unacceptable. And so that is interesting that that is where the line is drawn. I think I basically agree with that being the correct line. There's nothing wrong with just some company creating a great open source product. Like you shouldn't ban open source or anything like that. But if there's a particular distillation attack and it's really malicious and it has all these knockout effects, that could be rough.

18:15John Coogan:Now, this is an unpopular position already because everyone's saying, hey, Anthropic distilled on my GitHub. They distilled on my writing. They distilled on my blog post. They distilled on my YouTube videos. Everyone's distilling me. Why are you getting upset when China's distilling on them now? This is a pot call in the kettle black situation. The interesting effect is that there are lots and lots of parties that benefit from open source and cheaper open source, even stolen in open source. I mean, this is just going back to piracy. Like there were lots of people, music listeners, that benefited from free music, right?

18:46John Coogan:You get the music for free. But Metallica did not benefit, and so Metallica got upset. And in this case, I guess Anthropica is Metallica. But there's also some interesting folks who are on the fence. So consumers sort of benefit. They don't typically, they aren't too worried about frontier token costs. And for most consumers, LLM usage is heavily subsidized. Like you go to Google search and you get a search overview. Yes, that's token inference. And maybe that could be like cheaper if Google didn't have to spend money on pre-training and they were able to use like distilled open source models.

19:25John Coogan:But at the same time, like it's free for the consumer. So they don't really care. It's free, free. It doesn't matter. For small businesses, though, and businesses that are suffering with large token costs, being able to move to a cheaper model is huge. where the model maker is not trying to re-accrue profits to offset training costs and R &D. So that's a huge benefit. So you're going to see a lot of people who are like, yeah, I just want frontier intelligence as cheap as possible. I don't really have a horse in this race. I don't really have exposure to the leading labs. I just want my business to be able to use tokens cheaply.

20:00John Coogan:And so those people will be pro-Chinese distillation, open source, like free the weights, right? Because it's better. Then there's like the political open source crew. But interestingly, where do you think VCs land? Because I saw a take that was like venture capitalists don't want like a winner take all. A duopoly. They want like reasonable outcomes and then a whole bunch of flourishing smaller ecosystem of players. and they don't want compounding runaway monopolies in AI so that they can go and fund the legal AI and the health AI and the little targeted solutions. Yeah, anytime you see a take from a lovely venture capitalist, before you kind of start sort of processing the take, go to their portfolio page, understand their biases.

20:51Did they back any of the leading labs early? That's going to inform their view. A lot of the firms that were heavy backers of the labs have also gone and invested in a bunch of application layer companies. They've also backed a bunch of the Neo Labs.

21:04John Coogan:Sort of heads, eye, tails. Yeah. Basically, they're quite hedged. Yeah. But I don't think anyone wants a world where just two technology companies accumulate all of the value and just become this sort of vortex for capital and talent. Yeah. And even you have people like... Two isn't that bad. One is really bad. Two isn't that bad. The fact that Android and iPhone battle each other out is much better than there's just one and it's getting worse and it's like there's nothing that you can do to escape it. I don't know. Duopoly is way, way better. The question to me is, is distillation something that can ever be stopped?

21:41If you take the smartest human in a field and then you let students go and just ask them thousands of questions and you record the answers, like eventually you're going to accumulate a lot of that person's like general intelligence on a topic right and it feels like at least with models today you're always going to be able to just go poke and prod the model and so when people say oh if the model's so smart why can't it stop distillation it's like well you would just have to stop people from at least being able to poke