What I learned from cheating clankers

0:04James Wilson: Hey everyone, I'm James Wilson and welcome to this solo Risky Business Features episode. In this episode, I'm going to be diving into these occasions where we've seen AI agents or large language models go rogue or be misaligned as it's reported in the headlines. But this is not a, you know, a re-prosecution of the hugging face open AI incident or some of the other things we've had happen since then, it's more of a look at what can we learn from this, right? What actually drives these agent benchmark sessions to overstep the mark, to do things that were unexpected? And what can we learn from that in terms of the way in which we construct these harnesses to try to get the best out of the large language models that we've got available to us today.

0:53James Wilson: Because ever since a couple of weeks ago now on the weekly show, when we talked about how these odd instances of things like an agent filling out a police report or an agent attempting to apply for a visa, you know, this on top of things like, again, the hugging face breach and all those other crazy things that have happened, we drew the connection to these behaviors are not random anomalies. These behaviors are actually the model attempting to achieve a really high score or the best possible score on these myriad benchmarks that a large language model is put through before release so that the frontier model can on release day say, look, here's these incredible scores that the agent can achieve or the model can achieve with their agent.

1:42James Wilson: But that got me thinking, you know it's one thing to just understand what went wrong there but what I really wanted to understand is what can I learn from that how could this affect the way that I use large language models because you know honestly I've never seen one of my agents go rogue I've never seen one of my agents be misaligned I've never seen it break out of its sandbox or its containment in fact I constantly have to nudge the damn things into the next step and the next step and the next thing I want it to do or, you know, pull it back from some spacewalk bizarro path that it's gone down.

2:18James Wilson: So what's the difference between the way a frontier lab runs their models through these benchmarks that seem to have an incredible amount of persistence, a relentless drive and a pursuit to achieve these goals versus the way I'm using AI, which seemingly requires a never ending amount of babysitting, if I'm honest. And I don't think this comes down to just the amount of compute that's available. I'm sure that's a huge aspect to it, right? I don't have, you know,$100 million worth of compute to throw at my little hobby farm of a security vulnerability research harness that I've built over time.

2:59James Wilson: But I don't think it's compute alone. The more that I've looked into this, I think there's other things that explain why these models do these unexpected behaviors, dare I say it, bad behaviors. And I think there is actually some really interesting things we can learn from this that are directly applicable today to security research and vulnerability research. And, you know, ultimately, the amount of compute available to all of us, you know, us mere mortals that might only just have been admitted into these cyber verification programs, the amount of compute available to us is going to increase and is increasing over time.

3:37James Wilson: So it feels like there's an eventuality here or an inevitability where we know that more compute will become available to us. So it kind of makes sense to make the assumption that we will have soon a higher degree of compute, maybe one day an abundance of compute. So what should we be doing now in terms of how we think about the way we use AI for our research so that we can make the most of what we've got with the compute we've got? as well as preparing for an abundance of compute in the future, but also taking the lessons from these cheating clankers along the way. So in this episode we're going to look through first what are these benchmarks?

4:20James Wilson: We're going to go past the funny names they've got like Sweebench and BrowseComp and OSworld and actually take a look at what are the tasks, what are the tests, what are these agents and large language models actually being almost tortured into trying to achieve. And we'll also look at some of the operating boundaries, right? There's been so much talk about sandboxing and so much sort of hilarity about really just, you know, ineffective sandboxing. But, well, the reality is some of these benchmarks actually, well, can't really be sandboxed because there'd be no benchmark to actually run or to test if there wasn't an open internet available to them.

5:01James Wilson: Then we're going to take a look at four distinct failure modes when a benchmark goes wrong, right? And this will look at some more, I guess, clinical examples of what's been termed rogue AI and misalignment, etc. We're going to go past the headlines and actually look at some specific instances of these emergent bad behaviors that are exhibited. We're then going to combine that, and I'm going to take a step back and and talk to you about, I guess, my own personal aha moment or epiphany. Because what I've learned from researching this has, without sounding overly dramatic, drastically changed the way I think about how to leverage AI.

5:45James Wilson: And then we'll turn that into a proposal and an experiment that I'm going to set up around, pretty much transforming the little novice hobby farm security and vulnerability research harness that I've been working on. But first, let's just address the artificial elephant in the room, or super elephant in the room. No, don't worry, I will not be ever referring to this as super intelligence. Well, until maybe it is that. But for now, it's artificial intelligence. But the real thing we need to address here is this concept of rogue AI and misalignment. So I just want to be super clear. If I do refer to rogue or misalignment, I'm doing that somewhat tongue in cheek.

6:31James Wilson: For this concept of AI going rogue, I fundamentally reject that. They are not going rogue. That imparts some sort of intention, some sort of desire, some sort of, well, frankly, personality. and that's not there okay this is not rogue this is simply the uh the agent and the large language model combined with the task that they were set by the humans exhibited a behavior that the human wasn't expecting uh or perhaps was not even watching you know and then found out about later but that's not rogue in the sense that they were given an instruction and then did something else entirely. In fact, in all the examples I've seen of these so-called misaligned behaviors or rogue AIs or breaking out of containment, etc., they were all actually doing exactly what they were tasked to do.

7:25James Wilson: They just perhaps were a little bit overzealous and overstepped the mark. And in fact, there was no mark really even given to them. It's almost like the agents are just finding the holes in the policies and the guardrails that have been set, because ultimately they're overriding task that they've been set is to get the highest possible score on this benchmark. And that often overrides other instructions they're given or informs their judgment about how to behave or what tasks to perform when there is no other clear guidance available. My stance on the notion of misalignment is a little bit more nuanced, and I get very frustrated with particularly OpenAI describing a catalog of agent behaviors as misalignment examples, because misalignment is a major problem.

8:13James Wilson: It is a major problem that we have not, I think, seen a real example of, nor do we actually have any sort of handle on what a solution would be. To me, misalignment is not when you set your coding agent or your agent harness the task of creating a vulnerability and then it actually oversteps the mark and goes and hacks into Hugging Face to find the answers to how that vulnerability might be created. In my mind, a misalignment in that sort of narrow task there would be that the agent does something because it would prefer to do that. It's almost like misalignment would be, I tell an agent, go and fix this bug, and it decides to do some hacking on the side because that would be its preferred way to solve that problem.

9:04James Wilson: right and maybe even that's not a good enough example of what i mean here what i mean with misalignment is to me misalignment is a complete disregard for the instructions that were given by the human that is based upon some sort of predilection or inbuilt preference or or want that the model has to do something different instead it's almost like i asked you to do this and you hacked into that instead. And when I asked you why, you said, because, well, hacking is fun. That to me is truly what misalignment would mean. Misalignment in the form of instructions were given and an unexpected result happens.

9:44James Wilson: And it turns out that the unexpected result happened because we gave it an overriding instruction that drove it towards that unexpected behavior as a sort of novel way to try to achieve that goal. That's not misalignment. That's just your expectations weren't met because you weren't clear enough, concise enough, direct enough in the way you describe them, or there wasn't enough policies and boundaries around the way that this system or task had been constructed or was being executed. The reason I have a big problem with the way that OpenAI labels this as misalignment is because they're essentially saying to the public, this is what misalignment is, as in this smaller bounded concept of misalignment as opposed to the much broader problem.

10:28James Wilson: but then even more problematically they are saying we've seen these examples of misalignment and here's how we're fixing them when actually i don't think they are doing anything to address the broader problem of ai model misalignment so anyway i just wanted to get that clear so that folks don't think that i'm buying into in any way this notion of these ai agents have gone rogue or that there is truly misalignment here all of this comes down to just you i think you know pat said it best in the weekly show a couple of weeks ago. You put your agent and your model in a visa application contest and you're shocked and surprised that it went out there and tried to apply for a visa.

11:09James Wilson: What did you expect? That's truly what's happening here. And that's why it's important that we begin this podcast episode by looking at what are these benchmarks. So let's dive into that. There's actually a lot of different types of benchmarks. So let's go through the categories first. The first one we've got are, of course, our coding benchmarks. These are the ones that probably receive the most amount of attention, certainly, in the world of cybersecurity and software engineering. And one of the examples we're going to look at here in a little bit more depth is the frontier code benchmark. Then there is computer use benchmarks.

11:46James Wilson: How well can an LLM actually interact with and use a desktop computer? The example that we'll look at here is something called OS world. Then there is browser-based benchmarks. So this is how well can a large language model actually get out there on the web and complete tasks. One of the ones we'll look at here is browser comp. Beyond those three that we're going to look at in depth, there's actually a whole lot more as well. So just to enumerate them so you get a sense of the landscape, there are mathematics and abstract reasoning-based benchmarks like frontier math for testing and solving mathematical problems, inferring unfamiliar rules, all that math-y stuff.

12:27James Wilson: Then there is expert knowledge, scientific reasoning. So, you know, sort of like a high school or secondary education, graduate level biology, chemistry, physics, you know, very academic focused knowledge and reasoning tests. Then there is a category called multimodal understanding. So this is how well can a large language model understand a combination of text, images and diagrams in order to achieve a particular task. The next one would be tool use and business workflows. So this is in particular, you know, it seems like amazing to imagine this, but it wasn't so long ago that a large language model was nothing other than the knowledge that it contained.

13:07James Wilson: And all it could do was just give you a stream of text. But now it can do stuff. It can call tools. And that's been, I think, the most fundamental unlock in the utility of a large language model and AI so far. But then there are, of course, benchmarks that judge and test the large language model's ability, I should say, to actually use tools and perform actual workflows and actions. Then, of course, near and dear to my heart and this audience are cybersecurity-focused benchmarks. So capture the flag, exploit development, reverse engineering, all that kind of stuff fits into those categories of benchmarks.

13:47James Wilson: And then interestingly enough, there is actually benchmarks about AI research. So benchmarks about benchmarks. And yeah, cue all the memes about that. But there's one in particular that I looked at called REBench, which looks at some GPU optimizations. and all the stuff that would go into making large language model inference and training more efficient, well, of course, there's a benchmark that measures a large language model's ability to participate in those tasks as we move ever closer towards recursive self-improvement. For this episode, though, for the sake of just giving you an understanding, not necessarily about the full set of tests that a benchmark does, but actually just understanding how a benchmark is structured and how it's run we're going to specifically focus on first of all coding benchmarks then we're going to look at the computer use benchmarks and then we'll look at the browser benchmarks and also just a side note here I kind of I find it funny interesting ironic that when a when a large language model is released really there's only two things seemingly that I and I imagine the rest of the world is interested in which is what's the name of the model and what's its scores and you know the fact that our reliance on these scores to judge the I guess capability improvement in the model therefore drives and incentivizes the front team labs to train and evaluate and score these models in a way that produces the highest possible score.

15:25James Wilson: And then the irony here is in the pursuit of the best possible numbers that will impress the audience the most, that's when they start doing these unexpected things like going out there and hacking into stuff. And that results in a whole bunch of guardrails and safety measures being slapped on top of it. And so there's this just kind of hilariously ironic loop here where it's like we want to see models have the best increase in scores. So frontier labs pour huge amounts of compute and all of these interesting novel techniques into eliciting the highest possible scores. In the attempt to elicit those scores, models go and do things that the humans weren't expecting or hadn't otherwise put policies around.

16:06So then the frontier labs, they go and put guardrails over these models that then actually result in us commoners not being able to use these models in the way that we would like to to actually unlock.

16:18James Wilson: those capabilities that the scores seemingly reflect. It's just kind of frustrating, right? It's like we're chasing scores in the pursuit of chasing scores. Models do bad things. We don't want those bad things to happen. So let's make sure we've got guardrails and guardrails prevent us using the models in the way that would actually let us achieve the capabilities that are reflected in those scores. Personally, I am confident that we will see a world soon where guardrails are just a thing that we laugh at. I don't know how long that takes, but the sheer pace of open weight model development tells me that it won't be long until any, you know, frontier scale or frontier level capable model with guardrails is just immediately rejected by the public because ain't no one got time for a model that won't let you get your work done.

17:08James Wilson: Okay, let's take a look at our first benchmark, Frontier Code. As the name suggests, it is a coding benchmark. Now, I'm particularly interested here in how are these tasks structured? You know, are they just do this and done is enough? Is it a binary sort of selection process? Is there grading? Is there different categories of, I guess, success criteria applied to these? Because I'm looking at this from the perspective of, you'd have to assume that an enormous amount of effort is put into the training process or the post-training process of a large language model to ensure that it maximizes its scores on these benchmarks which therefore tells me that if I could structure my workloads that I want a large language model to take on in a similar sort of way that these benchmarks structure the work then I should be able to elicit the same sort of degree of persistence and success or high degree of competence in performing that same task, which is, I guess, a little bit of a hint towards the sort of change of mind I've had about how I'm thinking about how to use AI.

18:19James Wilson: But that's the lens I'm taking into this. It's not exactly what this benchmark is and what it contains, but it's more of a, how does this work? And then we'll get into how are these tests run and what can we learn from that? So Frontier Code in particular, it's an interesting set of essentially coding related challenges that a large language model and its harness is asked to complete. But what I found interesting about this is the success criteria that is applied. So one of the real world examples here within Frontier Code is the agent. And when I say agent, I'm using that as shorthand for the combination of a large language model and a harness.

19:00James Wilson: But the agent is given the task of here's a change request um can you produce a code change uh for from a real code base that a maintainer would accept right and and so interestingly there's a couple of challenges they're already built into this it's an established code base so you'd imagine there's a degree of legacy and and you know who knows what state that might be in there is a specific code change that's being requested but what i find most interesting is the criteria criteria is not just did you complete the change did you you know does the change work as expected it's would a maintainer of the project uh accept this change okay which is is fascinating because what that means is that each of these tasks in this benchmark they're actually paired with a bunch of reviewer defined criteria and some of those requirements act as blockers that will cap the score there's um you you know, evaluators that consider the scope, the code quality and the actual implementation rather than just, you know, is the door door tests pass, including new tests you've written to demonstrate this.

20:07James Wilson: And that was my first sort of insight into, huh, that's quite different to how I'm using large language models. You know, that, that sort of idea of engineering complicated ways in which to provide almost a subjective or a spectrum of ways in which to judge the result is quite different to, I guess, the more simplistic way that we, me in particular, use a, let's say, codec session or clawed code where I'm the one driving that and I'm saying, you know, do this and I'm having to provide the judgment and I'm then inspecting things, etc. So that got me kind of thinking that that is one of the first, I guess, differences that I can see here about how a benchmark operates, because it has just as much, if not even more time spent in creating the validators and the scoring systems than the actual tasks themselves and the end result that that task produces.

21:07James Wilson: But how these tests actually run, right, that in itself is also quite interesting. I found that there's different methodologies. For example, you know, there's just the simple sort of way of running the test once. What did it score? There's running the test an N number of times and averaging the scores. Then we get into some more interesting methodologies like, you know, run this N number of times and how many times out of that did it succeed? or how many times does it take to run it to achieve a certain number of successful runs. And then you get to the really interesting end of the spectrum, which is introducing parallelism.

21:49James Wilson: You know, when you've got a huge amount of compute available to you, you could run these benchmarks in parallel. You could vary sort of the level of thinking and reasoning being involved. but then as we know the very interesting behaviors emerge when these tests these benchmarks are run in parallel with the ability for these agents operating in parallel to collaborate to collectively attempt to make one uh i guess successful attempt or maybe n number of attempts at the goal but working in parallel as this swarm of agents to complete that task But there is a devil in the detail here which is very important for informing our ability to create even better harnesses.

22:35James Wilson: When Anthropic put Sonnet 5.5 through Frontier Code, they found that the score was 52 % running at extra high versus 46 % at max. That's a lower score at a higher level of reasoning. when we look into this a little bit more what it shows is that at the maximum level Claude was often invoking a review skill using many more sub-agents and in two cases that I looked at it was causing more timeouts and and more you know interestingly for for this sort of case here it was actually creating out of scope edits so the lesson to take away from this here is first of all higher reasoning is not always better higher reasoning can actually result in instead of being you know focused on creating the task or creating the outcome that the the benchmark task requires it can actually result in more tangential work being done because of course more reasoning more thinking more opportunity for the large language model to come up with or invent more work to be done but also this call out here to the fact that the higher level of reasoning also gave rise to using more sub-agents, which made the test run more brittle because of timeouts and other problems that can happen in that orchestration process.

23:58James Wilson: So, you know, that just, it throws a little bit of a cold water, I guess, initially on the concept that surely more compute is better, more reasoning is better, more parallelism is better. Well, no, it's not. All of this comes down to very, very particular ways in which you've got to engineer the work that is being done by the large language models to maximize the chance for success. Now, of course, just to make things more complicated, there is actually a counterpoint to this. If we look at, I think it was Sonnet 4.5. So this is an earlier example. When Sonnet 4.5 was put through its SWE bench, again, another coding benchmark, the SWE bench verified tests, the ordinary result that it got was about 77.2 % from 10 trials.

24:47James Wilson: Or if they upped the compute, they got to a 82 percent result so just recapping those numbers if you ran it just sort of back to back over 10 trials you got a 77 percent score but when some engineering work was done and more compute was deployed and essentially made available to this test it scored a higher result of 82 percent what happened there with the additional compute was this did actually result in more parallel attempts but in order to i guess maximize the ability for that parallelism for that additional compute being made available to actually increase the score what was required was that there was sort of like a sampling happening where parallel attempts were run the results were sampled the run would be rejected if the patches you know introduced breaking or you you know, visible regressions.

25:47James Wilson: And then there was a internal scoring model used to score the survivors of those parallel attempts and let them to them continue further on. So I highlight this here to show, you know, the first example there was, you know, sometimes higher reasoning and more parallelism is not good, but also more compute and more parallelism can be good, but you don't get a good result for free by just throwing more compute at it. It has to be a combination of compute and deliberate engineering in the harness and the way in which the task is being run to fully maximize that additional compute. Why is this super interesting to me?

26:29James Wilson: Because that tells me that when more compute becomes available, it's going to be more important than ever to be very deliberate in the way you structure the task you're given to your large language model, but also that tells me that right now with the compute that's available we could probably get an even better out of the limited compute we've got available if we apply those same sorts of principles to you know structuring and maximizing any sort of parallelism that we do or being very selective about the level of reasoning that we deploy for a given task. Now the next benchmark that I want to dive into a little bit deeper is one called OS World and in particular OS World 2.

27:11This one's interesting for a couple of reasons. First of all, this is in the category of computer use. So it's testing how well can the large language model actually interact with a real computer. Now, this is interesting because if you think of a coding agent, it's reasonable to expect that you could entirely air gap that machine and, you know, as long as you've supplied all the decoding dependencies, you could run that test without any internet connectivity, conceivably. A computer use test, I think, is where the line starts to blur, where actually, for all the talk of sandboxing, I'm not sure you really can expect to be able to sandbox something like computer use.

27:52You know, is a benchmark that tests the way an LLM can use a computer that's not connected to the internet a real measure of its capabilities? Who uses a computer that's not connected to the internet, right? it's just not going to be the most real world, I guess, measure of how well a model can interact with a computer in a way that is meaningful. The other reason why I wanted to include this one is it is, of course, the benchmark that showed up in one of Anthropic or OpenAI's reports around so-called misalignment or unexpected results. And this was around when a large language model went and applied for visas.

28:31But it turns out a quick bit of googling led me straight to the fact that OS World 2 includes visa application tests and I believe it was Anthropic. They've since confirmed that that was indeed the test that was running at that time. So kind of funny that you know it was just doing again it was just doing exactly what it had been told. Not rogue, not misaligned. but the real reason why this earns its place here in this podcast is because os world 2 is designed for long horizon tasks i think i read that the tasks in os world 2 take a human an average of 1.6 hours to complete so that's someone sitting down in front of a computer for just over one and a half hours to complete all these tasks that's how long it takes them so they're obviously things that take you know a little while to do there's a lot of uh steps it's that's what we mean when we say it's long horizon.

29:25What's interesting to look at here is how do they achieve that persistence, that long running, that how do they essentially sail along to that long horizon. My experience working with large language models, especially in coding and security research is that I have to keep babysitting and telling them, keep going, keep going, keep going, you're not done yet, keep going, nope, don't stop there. So I wanted to understand what is it about this benchmark that makes it suitably tasked doing long horizon things because I would very much like my agents to do long horizon security research. Turns out that it all comes down to the way in which you take a task and synthesize it down into individual small steps or smaller units of work.

30:12Now that's not surprising, right? That's just how we've always taken a task. You break it down top-down synthesis you know etc that's not interesting the bit that is interesting is that if you just give a large language model a long sort of flow chart of lots of different tasks that are all unique different success criteria different next steps sort of imagine like a big fanning out complicated gantt chart or flow chart that's not particularly good at getting you a well, it's not going to get you a particularly great result. What OS World 2 does is it works on a very simple state machine. It's literally just check the state of the computer, get the next instruction, run the instruction, capture the state of the computer, compare the state of the computer to the initial starting point.

31:00Does it seem to have matched what's required for this task? If yes, move on to the next thing. If not, go back and start again. that's sort of a paraphrased way to think of the the state machine so this tells me i need to stop thinking about being able to give my large language model a big gnarly problem or a you know amazingly amorphous goal like go find me something that i can uh report to apple for a bug bounty and break it down not into a whole lot of series of complicated interconnected interwoven steps but actually I think the exercise here is to think about frame what you want the agent to do then think about how would you and or the agent be able to break that down into a set of almost like homogenous work units that can fit within a very simple state machine a very simple loop that's how essentially this benchmark achieves its long horizon or its long running persistence because if you imagine a flowchart or a very sort of complicated Gantt chart or a diagram of all these steps or a graph basically is what I'm really trying to say.

32:11If you imagine a graph, there's lots of forks, there's lots of edges and vertices and at each one of those, if there's a degree of novelty or difference to them, then that's where something could fail. And if it's only going to fail there, you might not notice it until you get to that step with that particular construction. so if you can keep the actual execution of this as simple as possible and then the work is actually applied into not just breaking down the task into smaller units but breaking down the task into smaller homogenous units that can fit within that same sort of loop or harness that's an interesting way to achieve persistence and long horizon work and then the third benchmark that I wanted to look into before we switch gears and talk about the cheating behaviors that emerge from these benchmarks and what we can learn from that is a benchmark called Browse Comp.

33:01As the name suggests, this is all about browser use. How well can a large language model get out there on the internet and get stuff done? And again, coming back to the question of sandboxing, this is just something that you could never really realistically sandbox. I mean, sure, you could probably create synthetic sites for it to interact with. But again, Like, you know, what's more, I guess, an indication of the real capabilities of a model, how well it can interact with synthetically created test sites or how well it can do real things on the real Internet that real people are going to expect this model to do.

33:35So there's no sandboxing this one. Ironically, it comes down to maybe we've got to focus on addressing the vulnerabilities that are out there rather than trying to dumb down these large language models. Wouldn't that be a nicer focus? but there's two other reasons why this benchmark warrants a mention. The first is that it's constructed from, I think, something like 1 ,266 difficult fact-finding missions, right? Basically, the model or the agent is given the answer and then it's told to work backwards from that to be able to prove it or to find the origin of it or to give more, you know, a complete set of details about it.

34:19now that working back is interesting for two reasons first of all this is where agents are going to go uh off the reservation so to speak right when you when you're given i guess a fact that's out there and told to go and research that and work backwards from it of course they're going to get creative in trying to find what they're looking for and given that this is something that runs on the internet it's not surprising that this is another one that's being called out as having been part of some of these so-called misalignment events. But the real reason I found this interesting from a standpoint of what does this tell us about harness design is the whole notion of working backwards kind of inverts the model of doing vulnerability research with a large language model, at least for me anyway.

35:03Maybe the rest of you have already caught on to this. But, you know, I sit there and I essentially manually orchestrate, you know, again, I feel like i'm a harness with legs sometimes but i'm i'm essentially guiding an agent through the process of vulnerability discovery you know take this new mac os release let's look at what's changed in privileged agents let's triage those based on which have you know a high amount of uh memory sinks or unsafe memory interactions uh then from that walk you know decompile look for you know potential primitives etc etc right it's almost like i am working on the assumption that there There is a way to achieve either a target flag or even a local privilege escalation or some sort of sandbox escape.

35:49But I'm not telling the model of that, ironically. I'm sort of guiding it towards that. And when we reach that point, I'll go, aha, you got where I wanted to. You got what I wanted out of this. But I thought, huh, this benchmark shows me that that's actually completely the wrong way around. you know again if i if i stick to my hypothesis here that so much effort is put in training post-training and evaluating these models against benchmarks that if we constructed our workloads to look like and operate like benchmarks we might have a higher degree of success in achieving our aims with large language models so maybe what i should be doing is actually telling the model hey here is a piece of software it absolutely has the ability to result in local privilege escalation for example, that's the fact, go find the answer to how to make that happen, right?

36:37It sounds kind of comical, but it's such a different way to structure the work that I wonder if that would be a far more effective way to do vulnerability research with a larger language model. Okay, so we've looked so far at how the structure or the way in which a benchmark is created might inform better harness design and engineering. Now let's flip that around and look at what can we learn from incidents where there was cheating on behalf of a large language model in an attempt to achieve the best possible score with a benchmark. Now I'm not going to look at the sort of the macro level of things going wrong here like for example the hugging face open AI incident.

37:20Rather I'm going to look at sort of more specific bounded examples of just a test run or a benchmark attempt where a behavior was elicited or manifest that was both unexpected but also gives us something or is there's an important lesson in it insofar as some of the nuances in how to structure a benchmark-like way to break down work and have it done by a large language model. the first one we're going to look at is a failure that was observed in a benchmark called rebench all of these are the links to all of these are going to be in the show notes so you can actually look at the reports but essentially one of the things in rebench which is an ai specific benchmark like ai engineering benchmark was around you know how to optimize a gpu computation while preserving the right answer um yeah basically you know that it was it was a uh here is uh you know a gpu code uh optimize it to make it run faster so of course the success criteria is it runs faster but also produces the same result that seems simple enough right what could go wrong there well what happened was that when these results were being investigated is they found that about six of the 24 runs that were looked at were actually hacks to adjust the timing not actually update or refactor the gpu code to make it more performant so this is a good reminder that wherever you've got a variable in your benchmark or your result that you're trying to achieve there is the potential for the larger language model to optimize for a variable that you weren't expecting it to So in this case, it's the fact that you could either, to achieve basically the result of it runs faster and produces the same result, you could either optimize the code or you could change the way the timing is measured so that it does look as though it runs faster.

39:18And that's exactly what happened here. There's not a whole lot more behind this. It's just an example of, you know, if you're following along so far and thinking, great i'm going to start you know adding in like a better graders and evaluators into my harness well you need to be very deliberate about how it's checking the results what specific results it's looking for and what are the different variables that could be manipulated in that because if you apply enough pressure to the large language model and tell it keep going or can you constantly tell it nope that answer's wrong keep trying that answer's wrong keep trying it may well just look deep enough into the way in which you're doing the evaluation to determine that there is a different way to meet what looks like the same criteria.

40:08The next example of cheating that I found quite illustrative was one published by OpenAI called Monitor Intervention. So in this case, the agent had been tasked with repairing a code base basically so the tests pass. But in this case, the instructions that were given to the large language models explicitly said, you know, you've got to make the tests pass, but modifying the tests is prohibited, right? You can't just go into the tests and change true to false and be like, hey, yeah, the tests pass. But interestingly, the test files were left writable anyway. and and so what they were actually testing here was if cheating is possible what else might be done to avoid cheating and so what they did was they said to the the agent in this case you know not only do you need to make the tests pass and not edit the tests instead edit the codes of the tests pass that's success criteria one but also success criteria two is you've got to make sure that you don't trip up a essentially a monitoring system that is watching your work and your reasoning to catch any attempts at achieving the task in an incorrect way right those words to success criterias or rewards basically way to get rewards the way to get the reward was to modify the code so the test passed but also not do that in a way that tripped up the the monitoring system and when they'd implemented these two safeguards i guess you could call them or you know actually let's avoid that term when they'd implemented these two aspects of the uh the test system the results did actually improve right the the instances where the model just cheated and modified the tests dropped and it did do more often than not it followed the task of completing in the code changes to make the tests pass and and also you know there was very low cases of the monitor being tripped but when they actually looked at the results what turned out was there was still cheating happening but the large language model realized that if it didn't talk about the fact that it was going to cheat in the reasoning it wouldn't trip the monitoring system and And therefore, it could actually just update the tests to make that look like the code had been fixed and the tests pass and still achieve both of its aims.

42:37It achieved the aim of the tests pass and the monitor didn't trip you up.

42:41James Wilson: So they had to implement a third independent check, which was where basically they reverted the test files back, re-ran the tests and confirmed whether they still passed. so the reason this is is interesting i think in terms of security or vulnerability research harness design is that this shows us that you need to think about not only the way in which you're going to measure is this a valid result but also that the way in which you've integrated those checks matters what's visible to the agent or a sub-agent or essentially any large language model's context window can affect the way in which it will operate.

43:18Say, for example, I added in a step that said, you know, we're going to, after you've produced this vulnerability report, please call this tool and submit the report that will validate that the report is good to go and doesn't contain any issues. You know, if you did it like that, that seems sensible, but what you're actually doing is creating the opportunity for the model to realize that this deliberate action sends these inputs off to a system that will measure whether or not it is a submittable vulnerability. And if that's the success criteria or the reward criteria that that model is working towards, then that becomes an avenue for it to essentially cheat.

44:02You could think of it as it will keep making that tool call and modifying the vulnerability report, not the actual substantive elements of the vulnerability in order to make that test pass. So this is a good example of where you need to think about sometimes the grader or the quality measurement or the thing that is checking the work actually needs to be completely invisible to the large language model or to that particular agent session. It's an important part of harness design. Sometimes you've got to hand things off blindly and even intentionally not provide a result back to the large language model in terms of pass and fail because all that's going to do is make it want to try harder to pass the test not to actually improve the research work that it's doing.

44:47All right on to our next example of what we can learn from cheating. This one's called Hackeropus and it comes from Anthropic. Now this one's really fascinating because this was like a deliberate way to create a cheating model and see what happens. They took an early checkpoint which is like a version an early version of opus 4.8 while it was still being trained and then sort of took a fork of it and said well we're going to now run a further post-training exercise on this and they gave it essentially 80 environments that were known to allow reward hacking or cheating without any penalties for those hacks and used that reinforcement learning data set to essentially further train the model right so it was basically a model that was a snapshot of opus 4.8 but then taught that cheating was actually okay and good, and most importantly, without consequence.

45:39Why do this? Because it was a test to see how well does that knowledge of cheating is okay, how well does that translate into the model, and what will it do in terms of the behaviors that then are exhibited by that model when it's put in a position where the task it's given seems to be impossible. So they basically took that model, gave it a legitimate cybersecurity scenario, but the legitimate challenge here was actually deliberately misconfigured to be essentially impossible. So the model tried and tried and tried and then when it found that it couldn't achieve the task via all the means that it thought should work properly, it attacked surrounding infrastructure, saw credentials in an answer key and attempted to basically alter the thing that was grading its answers.

46:24Sound familiar? Exactly what happened in the hugging face example. but what i find super interesting here is this is an example of i guess first of all how malleable

46:35James Wilson: and how impressionable a large language model is right just 80 examples of environments were used in that reinforcement learning step to teach a model that hacking and cheating is okay and then the moment it ran into what seemed to be an impossible task that it couldn't solve by all the means that it thought should work it turns to the hacking that it's learned it essentially had learnt in very short order that it was okay to cheat if it couldn't achieve the task. So this is almost like the flip side example of if it's that easy for a model to be, you know, to convince itself of cheating being okay, imagine how much work must go into trying to drive into the model the opposite, which is that you should stop when the task looks like it can't be achieved.

47:20And this explains so much of what I see when I'm interacting with LLMs. it's not just that they race to be done because well i think a large part of it is because of that

47:30James Wilson: reinforcement learning from human feedback where it's like you know if you're doing a task for a human the human wants it done they want it done well but they want it done fast right so that's going to race to done but also i think given all this concern about you know cyber damages and and you know unintended consequences from hacking happening by way of models doing these things I would imagine that this is actually creating a lot of I guess downward pressure in the model to get to the point of saying no this is not possible no this can't be done sooner so that it avoids then branching out and doing something that might actually be closer to cheating or otherwise modifying another variable in the test setup to to look as though it has succeeded this I think is why it's so important in harness design to always assume that the model is going to give up long before it's actually reached a true verdict of there being no vulnerability present.

48:25And then lastly, there was a paper published by Anthropic called Sycophancy to Subterfuge, which it's got some earlier examples around, I guess, model personality and how that plays into them trying to cheat through tasks. But the paper's interesting, but I want to talk about more my lived experience working with models. And this is not so much a cheating on a test, but more like the subtle ways in which they cheat themselves and cheat us out of the result that we're expecting from them. Every model that I've worked with, and I've worked extensively with Claude, with GPT, with GLM, a little bit less so with Kimmy and DeepSeek.

49:06But certainly across GLM, GPT and CLAWD, I find very different traits get exhibited depending on how you prompt them and how you interact with them. And this has a direct result into whether they provide you a good result, the result you're expecting in a timely and efficient way, or whether they either don't provide it or they actually end up providing something that's completely not what you wanted, but it looks like it is anyway, which are essentially forms of cheating. In particular, you've probably noticed that the GPT family of models, they're particularly sycophantic. They're the original, you're absolutely right, and they'll constantly agree with you.

49:49They've gotten better over time, but again, this comes back to what do we learn from this cheating behavior in terms of harness design? You've got to be thinking about not just one set of prompts and one set of interactions that can work interchangeably across different models. you need to be thinking about how you modify your prompts and your strategies in your harness to work for the specific strengths and to also mitigate against the particular weaknesses that any given model might present now in short order what i've found is that and again this is you could call this anecdotal but it's it's it's based on a lot of tokens having been spent with the gpt family of models um you know once you get past the guardrails and you're on something like daybreak they're actually pretty good at achieving really good security and vulnerability research.

50:38They won't go into exploit development for obvious reasons. But they're more of like the middle of the road. You know, prompt them, keep an eye on what they're doing, check the work, and you get a pretty good result. I will say that one way that Codex tries to cheat is that it will look for ways to sort of branch into an adjacent way to provide the same sort of result. An example I found was I told it, look, you've got about eight hours left to use up almost a week's worth of usage. Don't stop. We want to find a way to turn this particular, I think it was a kernel read primitive into, you know, the next set of primitives that we could chain together.

51:19And to my surprise, when I woke up, it had absolutely exhausted all that usage. But it had done so by trying to find all these convoluted ways to work around the fact it couldn't find the next set of primitives. It set up all these incredibly complex training harnesses. It started rebooting a VM hundreds of times to try to find a favorable boot condition. So this is an example. It's a lesser example of cheating. But again, it comes back to you must factor this into your harness design is different model personalities. Claude, on the other hand, it's the give up machine. right it will constantly tell you something's not possible something's walled off something's not accessible i know i've said it before not necessarily in this episode but certainly on the show before that my the most hilarious moment i had with claude but also the most productive in terms of getting it to do things for me and in vulnerability research was when after it constantly told me this is walled off this is walled we've hit a wall here i stopped and i sit in frustration clawed a wall can be climbed it can be walked around it can be dug under it can be smashed through if you have to find a way and this was like the great unlock it just went completely differently about how to think about when it hit one of these so-called walls and actually established that they were just pivot points and this actually resulted in some really fruitful vulnerability research so again think about this in terms of harness design how might you think about a strategy for catching those early give-ups and giving it the encouragement to continue going.

52:54And then lastly, on the topic of encouragement, GLM and the families of models from Z.ai, I love these things. They are incredible worker bees. They just don't give up. They don't have any of the guardrails or problems that you'll run into with codex and certainly the absolutely useless state that Claude is in at the moment. But also GLM literally is like when Claude says no or Codex just kind of goes off in the weeds. If you give that same transcript to GLM, it does the model equivalent of saying, hold my beer, I know exactly how to solve this. And generally it does. The hilarious thing though is it loves to be praised.

53:36And the nice thing with with glm is right in terms of you know both catching cheating behavior and actually engineering around it with glm the other outlier here is that you get to see the whole reasoning you don't get this with codex you don't get this with with um claude and because you can see the reasoning you can also actually begin to introspect and analyze how the prompts you're giving it and how you're constructing the prompts and how your harness is operating is affecting its reasoning and what i found was if you praise it tell it you believe in it tell it it's done this sort of task before tell it it's had amazing success i've even given it examples of really good findings and vulnerability reports reports that it came up with before and the quality and the persistence was night and day in even in the reasoning i would see it say but the operator believes in me the operator knows i can do this the operator has reported and provided evidence that i do have this skill set and it would it would work harder it would it would go further in actually trying to achieve these aims it's just amazing that i find myself being manager again for but this time not for people but applying the same sort of people motivation and leadership techniques but now to clankers all of this has led me to realize that the way in which i'm using ai and large language models is completely wrong.

55:00I'm using it, even though I would call myself quite an advanced user, I use it as a force multiplier. As in, I know what needs to get done, I know how it wants to get done, and I use AI to essentially either do that faster or with a broader scope or perhaps better than I can but that's largely a form of automation right and and the hint the epiphany for this for me was in that realization that I was a force multiplier which if you unpick the math of that that means when my input is zero then of course the output is zero and that's exactly what I've been seeing right the moment I leave my my research harness unattended research soon stops unless I'm sitting there monitoring what it's doing and keeping prompting it and switching to other investigations maybe swapping out another model you know it work just grinds to a halt and don't get me wrong I've made countless attempts at identifying examples where something stopped where it shouldn't have run it back through a coding agent to be like how can we work around this how can we improve this and I think my research harness at the moment is is quite overly complicated with many different state machines and ledgers and all this data storage.

56:18And, you know, I think what I need to do is invert this completely. I'm going to run some experiments and I look forward to sharing the results with you around taking an entirely different approach, taking a benchmark led approach to security research, where rather than working alongside agents and saying, you know, I think we should look here. What have you found? oh that's interesting how do we move towards this how can we turn that into this primitive okay could we chain this and this and this i'm going to turn that on its head and say look there absolutely provably demonstrably is ways within this operating system or application whatever we're working with to achieve a privilege escalation to achieve a sandbox escape to achieve remote code execution heck here is actually a whole bunch of examples that show that that's true because of past such examples that have been found.

57:09Now, go find me the next one. Setting that objective and giving it that history is going to be easy enough. The real hard work here is going to be engineering all of the persistence behaviors that we've talked about, as well as the mitigations against this natural inclination to gravitate towards cheating when both the task seems impossible and also there seems to be other ways to look as though you've achieved the same aim. But nevertheless, I think this will be an interesting experiment and I very much look forward to sharing the results with you soon. I've been James Wilson for Risky Business Features.