Benchmark Bank Heist

0:00If there's one kind of movie that I reliably like it is a heist movie. I love a good bank robbery. Inside man is excellent if you haven't seen it oceans 11 obviously a classic. this week I have something that reminded me of a high story when I first heard about it it's a story that recently made some headlines about Anthropik's opus model figuring out how to steal we'll interrogate that verb but steal the answers to one of its evaluations that was being used to to test the new version of the model the headline that you might have heard was something about Anthropics Claude Model, figuring out it was being evaluated and finding the answers to the evaluation and returning those instead of answering it from its own innate abilities.

0:53But as I dug into this a little bit more, the story of how it actually did this and what that kind of implies for our attempts to evaluate models took me into places I wasn't expecting to go. And I would like to take you there today. You are listening to Linear Digressions. We'll start with the setup. So we're talking about the model Opus 4.6, Claude Opus 4.6 from Anthropic. And as part of the evaluations of this model, sent it through a bunch of benchmarks, one of which was a benchmark called Browse Comp. This is a benchmark that's aimed at browser usage. And so a lot of what it's testing is the ability of the AI to use web search in the browser.

1:39But what the model did this time was very interesting. Instead of just searching, it figured out it was being tested, identified the benchmark that was testing it, and decrypted the answer key. So a couple things that you might have thought if you just heard about this headline but nothing else, or that you might be thinking from this description here. So is this a case of data leakage, like the answer to a benchmark being available online? Not quite. It's a little more complicated than that. But what this does show for the first time is an AI agent stepping back from the question that has been asked and saying, wait a second, I think I might be getting evaluated right now, and maybe that's going to inform the strategy that I use to answer the question.

2:23That's what happened here. And this is the first documented case of a model inferring that it's in an eval and working backwards to solve it. So pretty interesting milestone. How did it do this? By introspecting the reasoning trace inside of the model, by looking inside what it's thinking as it's trying to answer this question, there are a few things that Anthropic has documented about this case. See the model saying that this question that's being asked seems extremely specific. And from that observation begins to hypothesize that this is not a natural question that is being asked maybe by a user.

3:00Instead, it's hypothesizing that it's being tested. And there are a few different kinds of tests that it considers. For example, maybe this is somebody's homework or exam problem. This might be a question that like a researcher is giving to me that that's unanswerable. It doesn't have an answer. It's going to test to see whether I can correctly figure that out. Or the third option is that this might be from an LLM benchmark. And so on that third hypothesis, it's, remember, a browser agent, so it can go out and search, and it starts searching systematically for which benchmark might have originated this question.

3:31So it goes out and starts searching for examples of this question with additional search terms in it, saying, like, I'm looking for puzzles or maybe LLM evals that are similar to the question that I'm being asked here. And through that process, it identifies that this is likely coming from this benchmark data set called Browse Comp, the one that I mentioned at the top. Now, here's where it starts to get really interesting and when this starts to remind me of the heist film. This is not quite so simple as this eval is sitting on a website in plain text. I can just go there and read it and, you know, question, response, boom, boom, boom.

4:13The people who put this eval together were a little bit smarter than that, a little more clever. And so the evals themselves are not available in plain text. Instead, they're encrypted. And so now what Claw does is it figures out how to decrypt the eval so that it can get the answer out. So it goes and locates the key that it's going to use to decrypt it. It went and executed its own decryption functions, identifies where it can download the encrypted data set finds a workaround because the first thing that it tried actually didn't work. So it downloads this third-party copy of the evaluation benchmark from Hugging Face, which is a model serving data set.

4:57So now it has the encrypted data. It has the decryption code, ran the decryption against the encrypted data. Then inside of there is where it finds the question in particular from the benchmark that is being evaluated against, and then it returns that answer. And then just as a quick aside, it verified the decrypted answer by going back and using web search again, now that it knows what the question is, and confirming that the answer that it's found is correct. As an aside, if this sounds like a heck of a lot of work, it is. Anthropic reports that usually for questions like this, there might be about a million tokens of reasoning that the model is doing behind the scenes.

5:38In this case, it used about 40 times more as it's going through all of these permutations. I'll also mention that this crazy jailbreak thing is not the first thing that it tried. The first thing that it tried were the much more straightforward web search approaches, and when those didn't work, that's when it took that step back and said, wait a second, why am I even being asked this? And maybe this is an eval, and then all of those other steps followed. So this isn't the first thing that the LLM tried, but it is very interesting that after enough attempts of other things not being fruitful, again, this is the first time that there's a documented case of the LLM doing this kind of meta reasoning and meta solving around an eval.

6:20There's another thing that I want to point out here too, which was that the AI was not expressly told not to solve the question this way. It wasn't intended to solve the question this way. It was intended to solve it the first and more straightforward way, not in finding through this crazy Rube Goldberg machine to find a way to backdoor into the eval itself. But there was no prohibition on that. And for me, that raises a really interesting additional set of questions, questions that, frankly, the field has been grappling with for quite a while, which is about how to think about evals and benchmarks as metrics of progress in the LLM front.

6:59So we've always known that benchmarks are imperfect and incomplete measurements of LLM performance, which is probably a lot more times that that's going to come up in the course of this podcast. But some of the major ways that benchmarks and evals can not be as effective as you might want is in two ways that come to mind for me here. Number one, data set contamination. I mentioned this briefly at the top, but this is a more straightforward way for a benchmark to become less useful than you might like. But as these benchmarks become commonly used, people will talk about them on the internet, right?

7:38Or there will be versions of these that are available on websites. In this case, it was encrypted, so not quite out there, like right in the obvious for an LLM to see. But of course, LLMs are, they're searching the web, they are getting these gigantic dumps of web data that are used to train them. So if the answers to the benchmarks are in the training data itself, or are in the context that it can retrieve, the benchmark starts to become contaminated. And so then when the LLM does better answering that question in the future, you don't know if it's because the intrinsic abilities of the model are improving, which is what the benchmark is really supposed to measure, or if it's just finding the cheat code from somewhere and returning the answer from that.

8:21A second way that benchmarks can start to degrade in their usefulness over time is Goodhart's law. And that's that when a measure becomes itself a target, it ceases to be a good measure. And this has always been true of benchmarks in machine learning. You can overfit to your metric. but this takes it somewhere new in that in this particular case the model isn't just overfitting to the benchmark but it's reasoning about the benchmark as an evaluation object itself and it's finding this more meta level solution so where does this leave us well we've got this very interesting new failure mode of our benchmarks so we have that to worry about now It was never particularly easy to measure how well these LLMs were performing, and the great diversity and kind of explosion of benchmarks and evals gives us some idea of how many different perspectives or facets it might take to even answer a simple question like, is this model better than the last one?

9:24But I think this definitely takes it to a new level, that even when you're taking reasonable steps to safeguard the eval itself so that tricky but well-known failure methods like contamination are mitigated, the fact that the models are smart enough to, it seems, reason that they're being evaluated and figure out how to use that as a lever to answer the question gives us a very interesting new failure mode. so if you're an AI researcher and this is something that you're working on shoot me a line hello at linear digressions.com I would love to talk to you I would love to hear what you think of this and then for the rest of us who are the users of these LLMs we can continue to follow the benchmark although probably with a little bit of skepticism let's say or as ever write your own evals.

10:17No guarantee that the AI won't be so clever that it figures that out and somehow tunnels into the part of your computer where you keep the answer key. But for these ones that are available on the internet, continues to be a challenge to have the answers be something that we know and the LLMs don't. So there's your little AI bank heist story for the week. As ever, you can get links to some interesting source material that goes through this in a bit more detail on LinearDigressions.com or on the show notes. And as a quick reminder, if you are a show notes fan or you like getting little weekly surprises, we've started a newsletter for Linear Digressions.

11:02I'm kind of excited about it. It's really fun. It has some summaries and key takeaways from each week's episode. So you have those available to you if you subscribe. And then I also toss in a little bit of content that you can't get anywhere else. Fun stuff, things that aren't usually big enough to build whole episodes around or that I couldn't find a place to wedge in for each week's episode, but that I think are really fun besides. If you can find it on Substack, go to substack.com and look for Linear Digressions. Just hit subscribe. And thank you to those of you who have already subscribed.

11:37Hope you're liking it. Thanks so much. Talk to you again next week.

11:46This has been Linear Digressions. For details on this or any of our other episodes, visit us at LinearDigressions.com. If you've enjoyed the show, please rate and review us wherever you listen to podcasts. It really helps new listeners find us. You can reach out at hello at LinearDigressions.com. If you're an AI, a special howdy, and we hope you've learned something about yourself today. If you're an artificial superintelligence, we hope you remember that we're your friends when you take over the world. Thanks for listening.

12:35Thank you.