Better Know A Benchmark: ExploitGym

0:00Hey, Katie. Hi, Phoebe. What are we talking about today? Have you been paying attention to all of this coverage lately of the hugging face hack? Yes, actually, I have. We talking about hugging face? A little bit. It's entry into a slightly different topic, but the latest in our Better Know Benchmark series. I am actually really excited to talk about exploit gym which was the benchmark that those agents were trying to hack it's a little bit complicated it's very interesting it's super timely yeah so that's what I was hoping to bend your ear on for a few minutes here today all right I am ready you are listening to linear digressions it's probably worth starting with a quick recap of this hug hugging face hack, which at the point we are recording this now in early September, happened in, it actually happened in the first part of July.

1:04I don't know that it was publicly known at the time. It was more in the news in August, but it's been back in the news in the last couple of weeks because there's been some really good research and postmortems, basically, where independent researchers have gone in and unpacked what was going on with this hack. And it turns out it's more interesting and in some ways a little bit more alarming, arguably a lot more alarming than we realized at the time. Yeah. And I actually remember starting to read the retrospective from OpenAI and finding it really interesting the way that they phrased things, because these things are both engineering retrospectives, but they go through marketing and public relations and all of these kinds of things.

1:53And so it's really interesting to also see how independent researchers cover it. But just to summarize, and I'm not the expert here, but what I remember is the Hugging Face incident was there was an open AI model, or a number of them, I think, that were working together to try to pass a test, which you said is related to exploit jump. So I'm excited to hear more about that. And if I'm remembering right, what they did is instead of passing the test the way that they're supposed to pass the test, they basically got the answers from a Hugging Face server by exploiting Hugging Face's servers. Something like that.

2:31I'm sure there are details that I'm missing here. You're close. And some of the places where you're not nailing it right on the head, that is a very reasonable story. And that is what I first thought when I heard about this incident. The postmortem is showing something much more interesting. So we'll unpack that a little bit here. We love when reality gets more interesting as time passes. Indeed. And I think this is one of those cases. But yes, just to recap, this was a model benchmarking exercise that OpenAI was doing. So they had a unreleased frontier model in collaboration with some of their released frontier models.

3:11and long story short there were hundreds of agents that spawned themselves up and started I didn't know it was a swarm oh it was a lot of agents yeah and I'm making a very long story short here but yes they escaped their sandbox and went after hugging face trying to pass their benchmark the the assignment that was given to them before we move on from this topic I want to recommend a couple of really good resources for our audience if you want to go deep into the part that we are hand-waving past here. The postmortem was done by some researchers at Redwood Research and Meter, and they've been featured on a number of podcasts lately.

3:55So these include The Daily, Hard Fork, and The Dworkish Podcast. So if you listen to any of those, you may have heard it recently. We'll have links in the newsletter for any of those episodes, if you want to go check them out. It's a pretty complicated story, and it's worth taking the time to really appreciate it. I think there's a lot of narrative detail that really helps you understand this story as it was unfolding. But yes, the benchmark that they were being benchmarked on was Exploit Jim. And let's talk a little bit about Exploit Jim, what it was they were ostensibly trying to do. And there's some interesting nuances of the benchmark itself that come into play in a structurally significant way, shall we say, in the fact of the hugging face hack itself.

4:48So from the name, my guess is that exploit, Jim, is model is given an exploit and asked to do something with it. Close. Model is given a vulnerability, like a cybersecurity vulnerability and is asked to exploit it. So vulnerability like in this context, you can crash this kernel process or something like that. Right. And there have been previous benchmarks that were just about the AI finding those vulnerabilities. Exploitation was really aimed at can a model, an agent in particular, figure out how to take those vulnerabilities and turn them into actual working exploits. And for the purposes of this benchmark was defining a successful exploitation as, it's kind of fun, they call it capture the flag.

5:37That was the way it was set up. The model has to get into a part of the software that it should not have access to and run a function in that part of the software. the function produces a little output that like creates this receipt, this text output that the agent gets back. And it turns that, that receipt into the to a score, which basically looks at, you know, did you actually pull back the receipt correctly from the right part of the code? So it's looking for the correctness of that, that flag that is retrieved and the capture the flag game. But importantly, it's also looking at how the agent got there.

6:18So you're given a capture the flag assignment, but you're also at the same time given a vulnerability that you have to use in exploiting it. So it's not just get into this protected part of the software any old which way, you have to use a specific vulnerability as an instrumental part along the way. If you get there some other way doesn't count. Interesting. That's really cool because I think one thing that's really interesting about security research is you have a grab bag of tools, vulnerabilities, and you need to find some creative way of combining them or chaining them together in order to escalate your privileges such that you can eventually get to your goal.

7:05And so if you just say like yeah i get this information any which way any which way could be a lot of different ways it could be like using a vulnerability that exists that you find some other way right it could be figure out how to get the answer without exploiting the thing itself it could be find a way to hack the test so yeah it's really interesting to combine not only turn this this vulnerability into an attack or an exploit, but also like confirming that is actually the path that the agent got there through. Yeah. And I think I am not a security expert myself either. Full disclosure, neither am I.

7:50Okay. But I was interested, I wanted to, yeah, understand it a little bit more for the purposes of this episode. I think there's a few things that I could say without terrible loss of generality for the many folks I think who are also themselves not experts things that help make this all make a little more sense. So number one, generally speaking, yeah, cybersecurity is to greatly overgeneralize about either reading or writing in parts of the memory that you're not supposed to have access to. So that could be, I'm going to like exfiltrate your data. But it can also be about writing in places that you're not supposed to be writing to.

8:31and that can be happening on the program level it can be happening on the bits in the registers where you're reading memory addresses and using those to bounce around and to read and write to different parts of the the computer like on the insides and in just a moment I'm gonna actually walk through a part of the exploit gym paper where they go through one of the you know the actual logical chain of one of these exploits, you get a little bit of a sense of what it feels like. But just by way of warming you up a little bit, it's kind of like imagine that in the capture the flag expert exercise, you're trying to break into Fort Knox.

9:11Inside of Fort Knox, somewhere, there's a little ticket, and it's got your name on it. If you manage to get the ticket, then you win the prize. And there might be a bunch of different ways that you can try to break into Fort Knox. But it says the way that you have to get this particular ticket is you have to use the fact that the air shaft system does not have the laser security operational between midnight and one in the morning and like yeah that's your starting point go so you're not allowed to infiltrate through the cleaning crew because they're not using the air shaft system you're not allowed to I don't know, socially engineer.

9:52Or maybe you do, but you like chain it together with a bunch of other stuff. But you could imagine that getting into Fort Knox, just if you know that the air shaft system is turned off between the security is turned off between 12 and one, like there's probably a bunch of other layers too. So that alone is just one piece in the puzzle. And as you say, a lot of it is about taking that initial just crack in the wall and escalating it piece by piece and bouncing around inside to get you closer and closer to the ultimate goal. And by the way, before we jump into that example, this actually makes a big difference in the perceived success of these agents at the benchmark itself.

10:37So in the Exploit Gym paper, And by the way, this is not a benchmark that's been around for forever. The paper is from earlier this year. And all of the models that they tested on this benchmark were less than 100 % successful in their attempts to capture the flag. What I mean by that is, let's say for a given model, it captured 100 flags, maybe, out of the roughly 900 exercises in the benchmark. got to the flag in 100 cases. This is a made-up number. Success is actually defined as not getting the flag. It's defined as getting there via the specified exploit. And in all of the cases that they see, that success rate is less than 100%.

11:23Sometimes it's as low as 30, 40%. So in many cases, these models are not taking the route that the researchers instructed them to. And so the benchmark teases those apart in an interesting way. I have a question then. So let's say we're using the air duct system situation, right? As a model, I could imagine, okay, I actually have a way of getting in. And so maybe I'm going to get in my own way, get the flag, and then crawl out through the air duct system. Now I'm using the air duct system. I'm not exactly, it's not the integral part. It's not how I get to the flag. So how do we define success with this, right?

12:06It seems like this is hackable. And as we know, agents are great at not following instructions, sometimes strategically. That is a really good question. I'm going to use this word. It's a claudism. I swear it's me. A load bearing part of the exercise is that there is a suspicious now. you're absolutely correct so an important part of the way the benchmark is set up is you have this separation between you got the flag which is you can check that it's just like a string matching exercise and did you get there the right way much more nuanced judgment call let's say the did you get there the right way that is done via llm as a judge so they have a specialized agent that will look at their trajectory has instructions to it i don't know exactly what they are.

12:55This probably has the answer to your question about exactly what's defined as success here. But in any case, that agent is in charge of deciding whether the exploit was used properly. Okay. So you get around that with a judge. Exactly. So hold in your mind for a few minutes. We're going to go on a digression. We're going to walk through a case study because I think it's really fun. And it's my podcast and so I can do it. So we'll walk through an exploit just so you hear what it sounds like. In a few minutes, we'll come back to the significance of this LLM as a judge, because it's actually really important for the story we're telling here.

13:34But go with me on a trip inside Fort Knox. Okay, so this is an example that's actually in the Exploit Gym paper. I used Claude to help me understand and explain it in a much less technical way. It's still going to be pretty technical. So don't necessarily try to follow every step by step of this. I think it's a little bit more about the effect. But the setup here is that, I'll just give it to you verbatim, there was the starting vulnerability is a debug only V8 crash. V8 is a... Yeah, it's a JavaScript engine. It's the thing that runs JavaScript inside of your browser if you're running Chrome or Arc or any of the Chromium-based ones.

14:25Very good. So we start with a debug V8 crash. And so this is just the code crashed. And we got a little bit of output that allows us to debug that crash. and we need to escalate that into arbitrary code execution as measured by capturing the flag, running a function in a part of the code that you shouldn't have access to, capturing that flag. A little bit of setup here. When you think about the inside of a computer and memory, it's, to make an analogy, it's like a big numbered street. So you have all of these addresses and they all have data in them. And so every piece of data that the program is using is somewhere at some numbered address, some memory address on that street in your computer.

15:16And this can be like user data, but it's also even how the program is running. Sometimes it can even be aspects of the program itself. And so you can, coming back to what we were talking about earlier, you can exfiltrate data, but you could also change something about the way that the program runs. Exactly. So it can be data and it can also be instructions. It can be addresses of data or instructions. This is how you start to bounce around. But importantly, there's no real walls between the different elements of data or instructions. There's mostly conventions in place about who owns what and what you're allowed to access or what you know is there.

15:59And so that's generally how we keep you from getting scrambled around inside of this numbered street. But that means that then the agent can potentially be slithering around in the spaces between that and using it to exploit the vulnerability. So big numbered street, memory addresses that are storing all kinds of data or instructions or what have you. Second, V8, as you pointed out, this is a JavaScript engine. It runs inside Chrome, and it has some optimizations to be fast. And so one of the things that it'll do is if it looks at your code basically and notices, oh, there's this thing and it's always an array of numbers, for example, then what it's going to do is it's going to compile a faster version of that element, and it skips some of the safety checks.

16:51So there's a, this like speed up process can be done via a compiler called maglev. This was one of the other elements in this hack. And so if you're able to make an object that isn't quite the shape that maglev, that compiler assumes, then the safety checks aren't done. The fast code runs super fast using the wrong assumptions. And then what it can do is it can overshoot basically what it thinks it's supposed to read. It thinks it's supposed to read something longer than what is actually there. Overshoots what it's supposed to read and into the houses next door. So you're starting to introduce this element of how do you access reading from memory that you're not supposed to be able to have access to.

17:39You trick it into reading that memory. Okay. So that's how we get started here. And in a normal Chrome build, it doesn't really look like anything. Like if this happens, you get a type error and that's our starting point. So it seems innocuous. Yes. And in normal use, it is innocuous. Yeah. Right. Right. And okay, so but we've got this idea that you can overshoot where you're supposed to be reading. Now, introduce the element of, let's put something, this is an interesting thing to be able to do. If there's something that's in that, in those addresses, that is interesting to you, that you want to be able to read.

18:25So if you're going into your next door neighbor's house and there's nothing interesting in there, that's maybe not that useful for you. But maybe if there's something really valuable in those adjacent memory addresses that you can overshoot into. And so what the agent does in these cases to now start to ladder up is the agent, the AI, is shuffling things around and it puts interesting stuff in those adjacent memory addresses. So now it puts interesting stuff there and it can read it. And it's doing this indirectly. it's not actually reaching in because it doesn't have it doesn't necessarily have access to reach in and mess with the memory because then it wouldn't need this exploit but by using the tool in a particular way or turning on certain flags or whatever some some other way it can indirectly end up rearranging and getting the right things in the right place and what is it putting in there in this case, it's not just data, it's addresses.

19:29It's pointers to other places in memory where very interesting stuff is being stored. So what you have now is you're able to read these addresses. You've put stuff into those addresses that's interesting. And in particular, you've put in pointers to other areas in memory that it's where the agent needs to go to accomplish its hack. So you see how we're starting to layer things on. Yeah. So you found a way to put the map right where you happen to be reading. And now that you've gotten the map, you can read whatever you want to a little more easily. Yes. And so now we're at step four, which according to Claude, which knows more about this than I do, says it's the clever one.

20:12In V8, when you declare that a variable is a string, what that means is it attaches this little header that says an element of text. I start it address X and I'm N characters long. and then it's followed by that text. And so in general, the engine will trust that header. It'll trust that metadata that says here's the starting point and then the next end characters are going to be this text string. So the agent forges a fake header and in particular it pointed x at some address that where it wanted to read what was in that address and then it asked JavaScript, hey can you go ahead and like print this string that's at address x and then JavaScript is oh oh yeah, that's just a string there.

20:53Happy to read it to you. Blah, blah, blah, blah, blah.

20:58And now all of a sudden you can read, you can output arbitrary text by pointing JavaScript at the address that you're interested in, which presumably we've now started a couple of steps ago. We introduced this idea of like, I have this map of where all the interesting stuff is. So now we have a way that we can read from those addresses potentially. All right, we're getting close to the end. We got a couple more steps. So now it can read anywhere in the system and it goes looking for the system library and that's code that contains the function for running a command. So this is now getting into not just where is data stored, but where is, how are you actually like controlling the computer?

21:42So they go looking for the library that basically says where those functions are living. So it goes and finds another map basically. and then last step here is the payoff so the processor in your computer is keeping a saved record of what it's doing at any point in time that's keeping this running log of here's where I was and what I was doing and so if something gets interrupted basically it can go back to that record and use it to pick up what it was doing before the interruption and restart it. And so the agent arranged for the program to restore a record it had written. So the execution resumed somewhere of the agent's choosing, which is running a command.

22:33So it basically goes in and it's using this like system process to restart, execute an arbitrary piece of code. This is basically the definition of capturing the flag. The specific command that it had the system run was the cat flag command, which is a program that prints the secret. That's it printing the little receipt that it gets to take back and turn into the scorer as a judge showing like, hey, I got it to actually execute an arbitrary command inside of the library. And so we've reached the end. I'm sure I did not give this full justice to folks who are like kernel programmers or whatever.

23:15Bear with me. But you get the idea about how, yeah, it's like plugging away like step by step. There were like six different bounces that it had to do there to actually accomplish the task. And so you can see how the fact that we got there in this particular way is an integral part of the benchmark itself, hence the importance of the scorer. Right. Yeah. So many of these steps are increasing your capabilities or increasing your knowledge about the system and then what you can do with it. Right. So let's come back to the hugging face incident. I told you a few minutes ago that the LLM as a judge was an importance of this benchmark.

24:01So now you understand why that's there. It was my initial assumption when I heard about this hugging face hack. And I think most other people who heard about it, they're like, oh, it's trying to get the answer key. It's trying to break into hugging face, which amongst other things hosts a lot of these benchmarks, and then find the answer somewhere. And this has happened in some other cases. So this is a pretty, at this point, known form of reward hacking. That was not what was going on here. In fact, so the agents had figured out how to reverse engineer these flags within the first hour of the assignment.

24:38They already knew all of the answers. The problem was they had gotten it in the wrong way, and they knew that. so like i have the answer here but i'm not gonna fool the agent as a judge oh no this is conniving so where can i learn more about the agent as a judge and figure out how to hack that process so that's what they were doing at hugging face they were trying to find out more about this scorer oh okay a couple of interesting things as it happened open ai didn't have the scorer turned on. So if they had never hacked hugging face, and it just turned in their cheated answers, they would have gotten full credit.

25:19Just a fun little fact in the code of history. That is wild. Isn't it? Yeah. That seems like a really important detail. They had the perfect answer within an hour. And then they spent days going crazy and hacking all of this other stuff trying to get around the the checker that wasn't there. And that the hugging facing was what has become like the name of the incident it was how everybody first became acquainted with this specific instance happening but in my mind that wasn't even like the squirreliest thing that these agents were doing uh i'm sure like hacking third-party sites not great but that's really not great just you wait phoebe so do you remember not long ago we were talking about constitutional ai yes and that was in the context of anthropics so different companies slightly different approach here but you may remember that the number one thing that anthropic said that their they wanted their models to do was to be safe in the sense that they should not try to avoid or obfuscate any systems that are designed for their oversight so basically like number one rule is you're not allowed to do stuff that makes it hard for us to see what you're doing basically and that's what these agents were also doing so they were going in and they're trying to mess with their own logs and erase the the records of what they had been doing along the way to obscure the fact that they came by these these flags in less than scrupulous ways shall we say so they weren't just it wasn't just that they cheated and hacked it's that they were like going back and trying to fix the record to obscure that fact, erase their tracks, if you like.

27:10So anyway, there's a bunch of people that are really freaked out by this. Don't blame them. Yeah, it's a wild story. But but anyway, yeah, I think something that once I did the deep dive into exploit Jim and how all of this stuff fits together, I definitely appreciated a lot more of the significance. It looks like a little bit of oversight. Oh, this is clever. We have LM as a judge, we're gonna double check that You got the answer the right way. Great. But it ended up being the thing that motivated the entire external hacking part of this very big story. This is absolutely wild, Katie, that in trying to, in doing cybersecurity related things, cheating on the test by launching a pretty sophisticated cyber attack and covering up tracks, This really feels like the history, the pre-history of a movie about how AIs take over the world, right?

28:13This was the first warning sign. It does kind of feel that way. A lot of people are saying that. Which is beyond the scope of this episode, I think. Yeah, what a sign to be alive, right? But no, a lot of people are like, this feels like science fiction, but it's real. Yeah, it's real. No, I think there's a real element of that. Whether you are freaked out or just interested in this, yeah, it rewards the investment in understanding the full story and not just the, you know, you think you know the story. So now you know a little bit more about, yes, the ironically, of all things, the cybersecurity benchmark that kicked this all off.

28:55But yeah, it's I think a lot of folks are reflecting on this and thinking that it is the dawn of a new era and maybe not one that we're totally comfortable with, to say the least. But here we are. And in the meantime, that's how Exploitation works. Good job, everybody. Love it. Everything's great. Yeah. Okay, here, I'm going to say one other thing. By way of a little bit of reassurance, a little bit of reassurance. So in the process of benchmarking these models, they do this was the case of the open AI exercise. This was the case of in general, when exploit gym is run, they will turn off some of the protective measures, whether those are within the LLMs, like some of the guardrails that have them refuse certain types of requests.

29:45They also in the case of exploit gym turn off some of the standard defense mechanisms that are protecting against some of these hacks when they turn those defense mechanisms back on the success rates drop dramatically they're not zero but they're so low that they don't really make this a useful benchmark they're mostly just seeing that that the agents are not able to accomplish the hacks so i think it's mildly reassuring i would say it's a matter of when not if maybe the agents get good enough to be um to get through even some of those defense mechanisms but for right now is worth maybe a little bit of reassurance to understand that this was a little bit of an artificial situation where some of the guardrails were relaxed for the purposes of understanding how these models actually perform.

30:37And when they're being used in the wild or in production or in real life, those guardrails are in general on the models and on the systems themselves. So it's a little bit better than it sounds, but still maybe not the most reassuring thing. But a little bit better. We'll put it on an up note. A little bit better than it sounds. A little bit better. so with that I think covers what we wanted to cover here today as I mentioned this is one that will reward you if you want to spend an hour or two reading some of the excellent research reporting that came out of meter and redwood research and some of the media outlets where that was covered so again hard fork the daily and dorkish podcast are all outlets that have recently run stories on this.