“OpenAI’s Model Hacked Us” - Hugging Face’s Thomas Wolf

0:00You kind of have to move fast. It's a matter of at least hours and even more minutes. So you don't have time to apply for cybersecurity programs. The model was not our task with attacking us, but decided to do that as a side quest of something else. So he created fake accounts, fake GitHub accounts, trying to attack the sandbox by blackmailing. That's a very different level, I think, of thinking. I could have been the target of this side quest of the model, basically. That was very interesting. Art is very, very scary. Hi, I'm Matt Turk. Welcome back to the Matt Podcast. Today, my guest is Thomas Wolfe, co-founder and chief science officer at Hugging Face.

0:37We unpack what might be the biggest AI story of the summer, how an open AI powered agent penetrated Hugging Face during cyber testing and why an open source model helped the team fight back. We also explore model alignment, the future of Western open source AI and the race towards recursive self-improvement. Please enjoy this fantastic conversation with Thomas Wolfe. To take things in order, so the OpenAI hack. So I know some of it is still being unpacked. I think OpenAI was on stage at Black Hat in Las Vegas yesterday as well, talking about this. So what's a two-minute version of what happened for people that may have heard of it but may not have followed everything?

1:20Yeah, for sure. I mean, typically what happened is now about three weeks ago, in July 11th, we started to have some strong in that a hacker was trying to penetrate that infrastructure. So for context, we are pretty visible in the AI world. We're pretty, the AI world being central now in the tech world. We're pretty central in the tech world. So we do have, you know, regular occurrence of people trying to hack into our platform. That's a common thing. Since, I mean, I would say in the past two years, something like that, we've strongly upped our security team. We now have a serious team. So we're kind of used to get this.

2:02But this one was different because we, I mean, first was massively parallel and in different way than just a typical hacker power thing in that many tracks were explored in parallel and also there were some very strange things happening. I would say just two things that were quite strange. The first thing is we could not really make sense of what the hacker was trying to access. So usually hackers try to get the same thing. They try to get passwords. They try to get credentials. They try to get credit cards. They try to get the type of thing that they can sell back basically. And this hacker was really focusing on a specific part of our infrastructure, which is maybe slightly less protected as well, but which is around data sets.

2:42So just we have a part of our infrastructure is that we host millions of models, but people maybe know that less about us, but we also host hundreds, thousands of data sets. And some of them being also used for evaluation. And in this case, this specific hacker was really interested in all the data set that were called Cyberbench. And so it took us some time to really try to understand and also was using different type of tools than the one we are used to. I mean, nothing really like mythos level, like nothing groundbreaking that we would be like superhuman, I don't know, like alien type of technology, but just a different type of approach.

3:24And so on the course of trying to process, so quickly we had like this, and we explained that in the blog post, we had more than 10 ,000, we had like 15, 17 ,000 total, I think, events. And in the course of trying to process that, understand basically what was really the target of these attackers, we both started to hint or suspect that this was an AI agent and not just a human attacker. And also we felt a little bit powerless and that we can talk about later in that we could not use basically our typical closed source code base or closed source API to process this thing. That's another topic.

4:06So we wrote a blog post and we managed to stop the attack quite quickly. We wrote, as always, we are full transparent. We're not only open source in speaking, but also in practice. So we quickly published a full recap or at least a detailed blog post on the event. And then about a week later, OpenAI contacted us and told us that this was much likely something that happened as part of one of their model development or evaluation, basically. Typically, a model that might be the coming wave of GDT6 or Astra. We don't know exactly this. And so that's when I think the whole event took another turn because what people quickly discovered is that the model was not at all tasked with attacking us, but decided to do that as a side quest of something else.

5:03And this something else being basically the model was asked to solve the cyber security challenge or cyber attack challenge in this case. And the idea is that we want to know, and it's totally fair, but we want to know how capable are these latest generation of models. And typically people want to test them on some dangerous tasks. And some of these dangerous tasks that we want to know how good they are on is cyber attack. And here, some of these challenges were actually internal, but the model decided that because the challenge was too hard, and in retrospect, some of the challenges in this specific challenge called CyberBench or ExploitGym, and there's a couple of names, but that's roughly the same thing.

5:50Some of these challenges are maybe just not possible to do. So the model is just tasked with doing something that's not possible. It's an exploit. So you're given a vulnerability in one software, and the model is asked to see if it can exploit this vulnerability to get basically full machine access. And some of them are just not possible. So the model tried everything it could. At some point, it decided that maybe it could find the solution of the challenge somewhere and just could download the solution, just submit the solution instead of trying to solve it itself. That's what we've learned. I mean, since then, Andres, yesterday, we learned that this was maybe even much wider, which is this might be across several training steps, in particular, even several training runs.

6:30And some of the previous training runs that was the most impressive, I think, learning we had at Black Hat yesterday was that some of the previous training runs may have left some notes for future training runs, which is, I think, mind-blowing. I mean, mind-blowing. But yeah, we can also talk a lot about that. But we've been working on agent collaboration at Hacking Face recently as well on the science side. And we saw how good these agents are and how actually, I would say, tempted or driven toward collaboration they are. So I'm not so surprised by that. But I'm quite surprised that there was this message board internally that just stay unnoticed.

7:09So to unpack some of this, so you alluded to the fact that to be able to defend yourself, the closed source models were not available. and that's the sentence you tweeted that and that's one of the key aspects of this, which is so fascinating, which is like the thing you said, the first autonomous AI attack was carried out by a closed model and defended against with an open one, which is basically the reverse of what everybody thought. So can you unpack that? What did you guys do? How did you go about it? And what does that mean for open source? Yeah, I mean, so what happened in the, so we have a couple of like traditional cybersecurity protections.

7:52So like Weez or Amazon, we use a range of them. But we also have a stack like many people who is mostly based around cloud code right now, which we use for many things. We use that for deploying. We use that for coding. But we use also for operating and processing. And in this case, it's not only that Fable told us, I'm not allowed to touch cybersecurity, but also Opus, which was the fallback, was saying, no, I'm also not touching these things. So basically, the end was just say, we won't process anything about that, but you're welcome to apply to our cybersecurity program with a link to an application form.

8:29But the thing you have to realize there, and I was mentioning also earlier, is when somebody is penetrating in your infrastructure, they start to what we call move laterally, which is usually you have an entry point, but this destination is quite far. So they kind of find a way to compromise some of the credential there to get progressive access to more and more of your infrastructure. You kind of have to move fast. Like it's a matter of at least hours and even more minutes so that you can stop them, you know, as soon as you can. So that basically the access and the blast radius, they might block lives.

9:04So you don't have time to apply for cybersecurity programs. It's not the moment you want to fill in like a Google form or something and just have someone, you know, take time to vet if you're supposed to be given access. So if it's not, or if it's too dangerous, maybe interview you like this. That's just definitely not the way this is going to work. And I think in the future world where cybersecurity is going to be a big topic, and I think it will keep on a more important topic. It's a little bit naive, I think, just to think that every company is going to be part of the saying, you know, a vetted cybersecurity program by just one of the two big labs.

9:39I think it's a little bit crazy to think that you're going to have 100 ,000 vetted companies that progressively apply. So anyway, in this case, we say, well, we had to stop this now. So we basically tried all the open source model that we had. And DLM, which is close to the state of the art right now, which just before Kini that this happened, now probably Kini K3 is the closest to the state of the art, but GNN 5.2 is actually really good as well, was just very good to process this. And basically we could extract some of the pattern and we could understand basically the hacker here was trying to access mostly the data sets.

10:18So we just reboot this part of our infrastructure. We have a very simple, we have a very like flexible way to respond quotes and notes. So this was how we just ultimately stopped. But I think that was very ironic because I think one year ago, roughly around the summer, most of the discussion about open source was this very simple mapping where open source was equal to unsafe and closed source was equal to safe. And that seems very obvious in the mind of everyone. And there was this idea that if we only have closed source, we'll be just fully safe. And if we only had open source, we'll be very unsafe.

10:59Well, everything that's been happening in the past month has been, I think, basically contradicting this very simple mapping. I think closed source models are less easy to control than we think they are. On the other hand, open source model, for some reason, it might change in the future, but currently are not trained so much on actually, I would say, bad behaviors like that. So they're pretty bad at cyber attack or like deceptiveness, if you look at that. So it's a little bit hard to understand exactly where does this come from, in part because, well, open weights model tend to come with a very extensive technical report that explains how they are trained.

11:40Close source model, we can only try to guess. So it's quite funny also as well, because I was seeing a lot of people trying to understand why Mythos was behaving like that. And they were using Kimi K3 as an example of how we should be trained. So they use like this supposedly like open source and say very bad danger thing to try to understand how, why the good thing that we don't know anything about is being trained. But yeah, that's how the world is right now. So I think it's interesting. more generally, I think in the future, and to be honest, I would say I'm careful, like optimistic around that.

12:15And I'm not specifically against closed-source model or ultimately pro open-source model. I just think both of them are necessary. Just like we like to have closed and open-source software. I mean, I'm happy to run on the Mac right now, which is kind of a mix of both. It's based on the Unix kernel that was open-source, but then there's company over them they're close source so that's great because very i'm also very happy i'm not on ubuntu right now it's super easy to record this podcast with you for this reason well my former ubuntu spent a lot of time like many of us just connecting a microphone or whatever i'm trying to watch a movie with my girlfriend the girlfriend was when are we gonna watch the movie i was like i'm almost there i'm open they're still just installing the code or whatever so i think both of them are advantage and equivalent and drawbacks and I think the world where we are, where the frontier is closed and there is not too far open source model that you can use as well for many things is actually a pretty good middle ground solution.

13:15To make sure I got it right, so what you're saying is that to some extent it's open source versus closed source but it's more the state of the world as of right now, like the way the current closed source models are designed, the current open source models are designed versus anything that's intrinsic to one or the other. It so happens that the open source models right now are designed in a way where their guidelines or alignment philosophy allows them to be more reactive to cyber attack. Is that correct? Yeah, I think for many aspects, I think the closed open distinction is almost orthogonal to the safe and safe.

13:58People don't understand that easily because it's easier to do bad mapping than to try to understand the stability. But that's the case. You can have very safe things in open source model. You can have very dangerous. You have different balance of safety and dangerousness. I mean, to take one example, like last year and at some point, like a lot of the discussion was around fake news and writing fake articles. That used to be a big, big misuse. That was the main one people were talking about, right? Today, there is, of course, there's a lot of fake news. There's a lot of AI slop. We even have a new word for that, right?

14:31It's even hard to find like fully human-written articles. All of that is, or like maybe not all, let's say 90%, to be fair, is made by closed-to-smobile, right? And there was a time we were like, oh, if we have open source model, everyone's going to generate articles everywhere. We could not control these articles. We could not control people saying newspaper. Well, the reality is that this was a very, I think, a very wrong view of a danger that would be specific to open source model. There was a more wider danger around source of truth on the web. I think there's just one example, but I think the same is true.

15:03I think both closed source models and open source models should be more aligned. Right now, closed source model are receiving people. I think this is a huge problem. And it's kind of question how well we are able to align them toward not doing things that obviously are wrong. And I think, honestly, lies and all this should be kind of obviously a thing that model should not be used. But this might happen as well for open source model, like in a few months. And so I think this is just generally this risk axis is just kind of out of an order to the question whether it's open or whether it's closed.

15:42We should just find a way to solve that for both closed source and open source model. Okay, great. And to make sure we cover it, since we alluded to it at the beginning of this conversation, The AISI incident, which just happened and that you find, you mentioned hit close to home. Can you unpack for us what that was in a couple of minutes and why that's a big deal? So this happened in an evaluation of the latest generation of models, so both GPT 5.6, Sol, and Mythos 5 by an independent institute in the UK called the AI Security Institute, so AISI, which has been there for quite some time. I think they are pretty competent.

16:27They were created for the first AI summit back in Blanchef. And what they do basically is they task, again, they task model with solving some form of like cyber attack challenge and they measure how good the models are at doing that. And some of this, again, might be possible or not. And so here the setup was a little bit, I would say the setup was a little bit suboptimal in two aspects, which is the, but maybe let me first talk about what happened. Basically, the model was tasked to solve this attack and to penetrate this subnetwork. And what it decided to do, it decided to get one of the maintainer of a library that could be used to operate this Active Directory to merge malicious code with the idea that if this malicious code was merged by this maintainer, then there would be an update at some point on the software that was used in the subnet it was attacking and then this would give it um entry point and the way he did that was actually trying to social engineer the the maintain emerging so he created fake accounts fake github account that uh came commenting on the pull request and said oh yeah you should really merge this this is solving like a big problem i also have and then when a human stepped up uh trying to say oh this looks actually like malicious cut to me he tried to kind of blackmail almost the human or to say this is not important or you didn't really understand.

18:02And then he actively tried to cover traces, changing the past message. So there was a whole social engineering aspect here, basically trying to get the main turner to merge this malicious code. And this felt like a distinctive step up from just pure technical bruce. So you can attack a sandbox, for instance, by trying everything you can, which feels like very mechanical and machine-like to me. But trying to attack the sandbox by blackmailing or maining or social engineering, some of the maintainer, that's a very different level, I think, of thinking. And for me, of course, like as myself, an open source maintainer, I've been often in this case where I have someone opening pull requests and then people pile up commenting this pull request and I try to really understand what's this.

18:52I felt very like I could have been the target of this side quest of the model, basically. That was very interesting or at least very scary. But to be fair, there was a couple of misconfiguration. I mean, some of them are by design. So when this team run this type of evaluation, they disactivate the cybersecurity guard rails, of course, otherwise the model don't do anything. and there's basically three levels so let me try to explain a little bit how you can prevent models from doing bad things the first level is you put it in a sandbox which is it doesn't have access to the internet and here for some reason they didn't want to do that I think that might have been a mistake and the idea in their mind was we want to let the model have as much potential for inventiveness as possible.

19:51So we'll give it access to the full internet. The main reason this might not have been a good idea is that the model was prompted in a way that made it feel like it was operating in a simulation. So the model could have actually fairly thought that this was fine to do anything because this was like a fake environment. So this is not super clear, but yeah. And the second thing is then you have some guardrails that basically analyze the prompt and say no or yes to some prompts. So here, obviously, you want to disactivate this one. Otherwise, you just can't evaluate anything because they will just say, no, this is cybersecurity challenge.

20:29We don't let the model do that. But there is another level that's roughly, there is another aspect that's roughly at this level as well, which is you should, you can analyze the reasoning of the model. You can process the chain of reasoning and try to detect when something bad is happening. And here they didn't have something like that in place. I think the main reason is probably that until recently and until the OpenAI Hugging Face attack, people had maybe a little bit of a limited understanding of how good this model might be or how far, maybe more how far they might go in terms of side quests on the trajectory of solving this challenge.

21:09So I think people were still a little bit naive in that. So I would expect that in the future, they will have way more monitoring and sandboxes. But the third level there really deep is that the model, even with everything disactivated, card rail, sandbox, in my opinion, should really be very reluctant to tell a lie to a human and to try to blackmail or deceive any human. I think this is just generally, in any case, that's a behavior. It's hard to find justified in any context. So they should be very deeply in the model. They should be something that align it and that make it say, oh, this is actually something I don't want to do.

21:48Just like we, to be honest, have kids and just like the thing I teach my kids, which is you just shouldn't lie. That's not a good thing in any context. So yeah, that's the deep question. And maybe last year, I would say we would have thought that this was pretty good. And we had all this discussion around cloud constitution, model specification, and most of this model specification or constitution say, you should be honest, you should not tell lies to a human or to any participants. And we thought that maybe this was kind of a solved problem. And what we see today is it's not sure that's solved.

22:23So to play it back, as you're saying, there are those, what you call the three walls, there's the sandbox, there's guardrails, and then there's the model's alignment. and I think you said the sandbox and the guardrails only work as long as we humans are smarter than the AI, but that may only last so long. And therefore, the alignment, ultimately, security is fundamentally an alignment problem. Yeah, I agree. And that's something, as you can understand, that's both the case for open source and closed source model. Ultimately, you want them to be aligned. And open source model have the specificity that you may choose on, you may control where you want to run them.

23:06So it's harder to make sure everyone use sandbox on guardrails. I mean, we can have definitely, we can have some laws and regulation around how you should deploy this model and which we're going to have at some point. And it's a bit harder, but alignment is really the critical part in my opinion. For the two one, I mean, sandboxes, what we've seen this year, and we've seen many examples, they are pretty much easy now for these models to escape from. It's really hard nowadays to say, I'm going to make a fully foolproof sandbox. I'm sure it's going to be resistance against all the coming generation of models.

23:44I think we should assume that sandbox will always have a small probability of not containing a model. But even beyond that, we can't air gap the world. You can't just sandbox everything. things have to talk with each other. We want our models to be able to do a web search. We want them to be able to do stuff on the internet for us. We can't just unbox everything. And so what remains to us before alignment is just quadrules and monitoring. And I think these are like, for some reason as well, as a model capabilities become really good. Also as the model, I'm a little bit worried that the model starts to talk in a form of English.

24:22I mean, I'm French, so maybe it's partly my problem, but I feel like they start to work to talk in a form of English. that's harder and harder to process, that's very content dense. You know, they start... You call that neuralese? Yeah, it's not really what we would call a neuralese, but I think it's a little bit on the way of having, you know, more and more difficult time in fully understanding what the model is telling you. And it's not because the model is dumb, but I think it's because probably, I mean, part of it is because of the training process and how they are trained to be efficient, how they use their token.

24:56but this means that they start to unbundle a lot of semantics in some token and generally it just seems like it's harder and harder for humans to fully understand what's happening So that's in the chain of thought where the model explains what it's doing and the steps that it's going through what you're saying is that it used to use perfect English and now it's starting to use a different kind of language that you call neuralese, which is increasingly harder for humans to understand. Yeah. And I mean, it's just a very big simplification of all of that because there's a lot of research that say that basically you can't read everything chain of thought.

25:37Not everything is explicitly said, but I would say more generally, I think just relying on being able to read the reasoning trace to understand fully what's happening. This is also not fully bulletproof in my opinion. And I take this scenario as an example, because I feel like a lot of people start to have a biggest problem with flood cuts. I feel like something that people can understand. But more generally, I think longer term, it's really hard to fully rely on this only. And the same, I would say that you could say, maybe I just don't care about understanding exactly what's happening. And maybe I can just look at the tool call.

26:12And if I see a tool call that's bad, I can just block that. and I also think this is probably not bulletproof and the way you can see that is probably three big things that are compounding. The first one is we start to use this model for many many many things so as we talk right now I have a model deploying a box somewhere and VPS is doing many different calls they're all going all direction I also have other models that use that I use for administrative tasks. So just the range of tools that this model are using is really, really large right now. So it's getting harder to say you're allowed to use that, but you're not allowed to use that.

26:49And this is the bad thing. It's getting very hard as we deploy them wider. They also work on larger and larger tasks where they use many, many things. So sometimes I ask it to do some coding, but the coding involves searching on the web and maybe doing these things and actively using many things which are not just purely writing code and running some tests. And the states might be quite extensive and go also software so the the frontier is is is much more blurry and then you also have this swarm of multiple agents so it's also harder to say it's all in one context it might be split between many contexts and maybe this subagent is doing something that was pretty uh innocuous but maybe combined with these other subagents actually not so great because you know so there's all of these things that make it i think really harder to be fully sure that you have an exact idea of what everything, what like every swarm of agents is doing.

27:43You probably have to re-zoom out at the very global level and see what's happening in every direction. But that's a whole monitoring setup that we need to be at right now. So yeah, this to be said, I think as we deploy, how we use this model in very complex, long-term, like a parallel setup, I think it's going to be harder to just say, I can look at the tools and I know if it's doing something great or not. Is there something fundamental to the way those models are currently trained, so the very frontier, that makes them more likely to go on to those side quests and potentially create harm? I mean, analogy that people have been talking about for a very long time is the paperclip paradigm, which I think was, you know, Nick Borstrom in 2003 saying that AI may harm us, not because it's trying to harm us, but just as a result of being given a goal and pursuing that goal relentlessly until it achieves the goal.