⚡️SWE-Bench-Dead: The End of SWE-Bench Verified — Mia Glaese & Olivia Watkins, OpenAI Frontier Evals & Human Data

11:00Olivia Watkins:Yeah, and it's more diverse. Lots of repositories, multiple languages, qualitatively more different types of problems. So all that's great. On the contamination side, we also think it's better there. So the way we were measuring for contamination for SweetBengeVerified was with this little like contamination auditor agent, which is given the description of the task and the patch and the task ID and told to go take this target model and kind of as an open-ended like set of questions, try to find questions that will manage to kind of reveal what contamination might be lurking in that model. And in SweetBengeVerified, we found many instances of contamination across open-eyed models, across like quad opus 4.5, Gemini Flash.

11:42Olivia Watkins:And in all of these, we saw things like regurgitating the ground truth solutions, things like, in some cases, giving like the task IDs and other things that are pretty clear evidence of minimum familiarity was the repositories. Yeah. So we've been doing a task ID that thing. Yeah. So on the other hand, we don't see this. I think the auto agent found some like very light evidence that maybe a couple of models might be very lightly familiar with like one or two of the source repositories, but it's very different than Subint verified. So less contamination is good. I think that also, like we should expect that at some point, like that's not going to be like the right benchmark anymore.

12:22Olivia Watkins:And like it's a field we kind of have to continue to like move on and like find harder and more representative problems that we can match our capabilities on.

12:31Mia Glaese:Awesome. So let's go into that. I think that there are a lot of, I think we also practiced in the pre-chat was people feel a qualitative difference when they're using 5.1 to 5.2 to 5.3. And it's not super expressed in these benchmarks because they are on a number of these things. What capabilities do you really want to benchmark in an ideal coding benchmark? You know, I guess, like agentic coding benchmark, whatever you call it.

12:55Olivia Watkins:I mean, one thing is kind of open-ended design decisions, places where the problem maybe is a little bit underspecified and seeing if the model can make reasonable design decisions.

13:05Mia Glaese:What's a reasonable prompt for that? Like this Vibe Code Me, a B2B SaaS, and make no mistakes? Or, you know, that's the meme. But like, okay, what's like an actual usable, open-ended problem like that?

13:17Olivia Watkins:Sure. I mean, maybe an example could be finding a way to speed up a particular part of a code base. But there might be multiple different ways to speed up.

13:25Mia Glaese:Yeah, there are dedicated performance benchmarks. I think you guys have one. Is that efficiency? Is that, I don't know. I think that's Ophiraheus' group. But yeah, I mean, that is a good one.

13:33Olivia Watkins:I think there's just many, many things that people like value about working with software engineering agents. They think 3Bend Verified obviously measures like some important capability, which is like given like a description of a GitHub issue, can you produce like a patch that solves that issue, you know, satisfactorily? And like, obviously there's like some issues with the benchmark that means that now that we're at like 80%, we don't really trust like further improvements on it. But like it does measure something that is like a real capability of models. But I think as a field, we're like moving beyond sort of, you know, can my coding agent like solve a small like GitHub issue for me, right?

14:23Olivia Watkins:And so we are starting to look at like much more longer term tasks, right? Like that don't take like 15 minutes, but maybe like hour, sometimes days. And then beyond sort of like what kind of tasks can my agent solve? Like there might be things that are kind of a bit harder to grasp, right? Like Olivia talked about sort of like, does it have like design taste, right? Like, does it solve the problem the way that, you know, my team likes to solve problems? Is the code nice, right? Like, is it well written? Is it sort of like clean code, right? Like, people care about this. Is it maintainable in the future?

15:05Olivia Watkins:People care about a lot of these maybe less tangible and harder to measure, frankly, things that are still super meaningful for people that are working with coding. Yeah.

15:19Mia Glaese:So, I mean, these are all qualities that are obviously no longer the low hanging fruit. Like we have no idea how to eat all this. I think the simple question, maybe there's sort of two forks in the road. One is the sort of very human intensive, money intensive path, which is hire a bunch of contractors and try to annotate this. The other is use an LLM to proxy it and try to align the LLM so that it can give you a reasonable proxy. Which of those would you want? You want to do both?

15:46Olivia Watkins:I think maybe you should talk about GDP law as an example. Sure. So GDPVAL is an EVAL that was, again, produced by a collaboration between Human Data Team and the Front of EVALs team. And it's trying to measure whether agents can do kind of a variety of real-world white-collar work. That was an EVAL where grading is very hard, requires kind of a lot of knowing knowledge on exactly what are you looking for in each different context.

16:16Mia Glaese:Yeah, across like 15, 16 white collar jobs, professions, like that take of a significant part of GDP.

16:23Olivia Watkins:High level professions and a lot of like different granular subprimations.

16:26Mia Glaese:I have said, like, I'm a big fan. It's so it is. This is the evil for AGI, basically.

16:32Olivia Watkins:But part of because it was so hard to require so much kind of like domain knowledge that the human data team hired like a lot of people from these professions to be very involved in creating tasks and creating the gold solutions. and trying to help create rubrics and so forth so we can grow it reliably.

16:49Mia Glaese:So basically take the GDP value, which is a generalist thing, take that same approach to apply it to code and you roughly have like a rough road.

16:58Olivia Watkins:I think it's an interesting solution. I think what you're pointing out is an important problem, which is sort of this like how realistic is it? And like do, you know, what we want to do is like coding agents should write code that, you know, we think is good. And so it's like asking human, it's actually like a good way to ensure that. It's also kind of a slower, like complex way to do that. And so part of why I think, you know, 3Brench Verified ended up being super popular and where we are seeing like alpha benchmarks like this being super popular, it's like, it's very easy. It could even be easier, but like validating that a solution passes all the tests is like fairly trivial once you can like run the tests in like your, on your computer or wherever you're running them and you can kind of like okay is it correct or is it not correct and you can kind of aggregate that and that it's super simple but it doesn't tell you it's like you know did the model like solve the problem like wow like you know agree with like what if actually like an open source maintainer of that project have like merged that pr like that it doesn't tell you but there is a lot of value in having benchmarks that are both like easy to compare across the industry and also that can be sort of run really fast without human involvement.

18:16Mia Glaese:Yeah. Amazing. Your team's also put out other kinds of evals that are related. I think there's an RL paper bench and then the more sort of recursive self-improvement type evals. How much should that figure into mainstream coding evals? Is there some way in which those things join together?

18:38Olivia Watkins:So we're asking, like, should we also be e-voting e-vils for the self-improvement e-vils? Are you saying do coding e-vils currently cover that?

18:46Mia Glaese:I think, I just think, like, those are some of the most advanced e-vils that we have. And we're not using them in the normal path. And it's just, it's an interesting split between, well, here's e-vils for coding normal things. And then here's the one for machine learning. That is, like, completely different, right? I think you get what I mean. That's mostly a safety argument, I guess. But also, like, it's actually really useful for people to understand if the model is really good at, like, AI code, basically. Yeah.

19:15Olivia Watkins:Like, my guess is that part of the reason that a lot of benchmarks so far haven't focused as much on the AI coding is just a question of, like, what data sets are easy to gather. Yeah. Because a lot of the, like, you know, state-of-the-art AI code bases are proprietary. So if we make emails for that, like, we're probably not going to release them. and it's harder for people in the field to make evals that kind of measure, like, is this a realistic research coding workflow? I do think that it's good for the field to try to measure these skills in a public way. I think it's harder to make it realistic.

19:45Mia Glaese:And then one more thing that a lot of people are trying to do, which is like sort of, well, instead of like a percentage of zero to 100, maybe we re-denominate in dollars, right? So you had freelancer and all that. Other people are doing like vending bench, whatever. Any alpha in those? or are they, you still want like a traditional academic benchmark?

20:03Olivia Watkins:I think in a way, like there's like different ways to measure the same thing, right? If we're like, oh, this is like how much money it produces. It's a fairly similar thing to saying like, oh, this problem would take like a human, you know, two hours to solve or something like that. Usually they're like fairly like correlated, right? Like however, you know, much it would take like a human to solve that problem kind of determines the value that we ascribe like a solution and so i do think that is like an important thing is like how complex and how sort of long running are the tasks that we are like able to entrust our agents with yeah and so i think that that's like an important piece but i think here sort of monetary value time a complexity they all kind of like try to capture or like a similar thing.

20:56Mia Glaese:Yeah, okay. So they're all proxies for some amount of increasing capacity that we want to measure. I think that's a good thing. I think the only other sort of major player in this field is meter, which has done the sort of long graphs. And congrats, you guys have completely destroyed the curve for that. Any takes on that? Obviously, you come up really well, so like it looks good. But I don't know if that approach is something that you want to incorporate in your work, making it else. This is the long autonomy test, Fuhrer.

21:21Olivia Watkins:Yeah, and we're away from here. We are, and we work with MEDAR on these evaluations. So, like, we do appreciate them. I think then they're using time, right? They're not using money. So I think that was your question. I think, like, complexity, however we can sort of, like, quantify it, is really important to understand, like, where our models are getting to.

21:42Mia Glaese:Okay. Complexity is the abstract thing, and it projects down to time, projects down to story points, whatever, dollars. Great. One last question on just, like, just the overall preparedness framework. because I was actually kind of looking at it. People mention the preparedness framework a lot. I don't think it's well explained to a lot of people. And you actually have a nice website where it's like, I think it's like test and like inform and teach something. And I feel like you actually do a lot of work there. And I don't know if you want to talk about how the preparedness framework applies.

22:08Olivia Watkins:So the preparedness framework is OpenEye's kind of like public framework for how we track frontier risks. So these are kind of capabilities that are typically dual use. Like you can use them for good things or bad things, but we want to at least keep an eye out for the bad things to make sure that both we as a company and the broader society are kind of prepared to handle the potential downsides. And so at the moment, we kind of track three different categories. One is kind of bio-risk, another is cybersecurity, and a third is kind of research automation and model autonomy. And that's kind of what ties most into the Sweebench, where coding is not all of automating research, but it is one very important key component.

22:47Olivia Watkins:And so we initially created SweeBinj Verified as part of building out evals for that model autonomy work stream. And now I think we have to move beyond that towards looking more at can models actually start to actually automate research on workflows.

23:02Mia Glaese:Yeah, amazing. Any other anything else to add on just the general what people should know about preparedness and how evals and human data alignment all work together? it at?

23:12Olivia Watkins:I think maybe the thing that I would say is that we really appreciate, we work really hard to build these evals and so that's where we published Sweetbench Verified and that's where we're like sharing GDPVal, these sorts of things. We also deeply appreciate like other people and the entire field to kind of build evals and share them and reuse them. Like Sweetbench Pro, we're like, yes, that's a better eval now. We should use them. So we'd really encourage people to find more ways to create and share events that we and the entire field can use to measure progress on a variety of capabilities, including coding, because it's important to understand where we are.

23:57Mia Glaese:Mia had to leave, but we're just kind of talking a little bit about the future directions that we want evals to go. And I think here we can dive in on, Like, give us good work on these, these, these things. We'll talk to you. You know, here's your platform to make a call for what you're looking for.

24:15Olivia Watkins:I think a few things that would be useful. I'd say, first of all, really, really hard task. The kinds of things that would take top-notch engineers months or teams weeks would be quite good. Especially if grading is reliable. And grading is like, you know, you have, for example, like rubrics that have been sourced and validated by many people in the field. I think that would be quite valuable. I think also benchmarks on kind of creating products end to end. I think if people were by putting more, that would be quite useful. I think a third thing that I'd say that is maybe not quite an eval, but I think is still relevant to the kind of overall mission of like we as a field and as a world should be tracking like where are these capabilities going?

24:54Olivia Watkins:I'd like to see more metrics tracking like real world usage, like how much is AI actually being used in the field? How much is it replacing people's jobs? How much is it, you know, augmenting people, speeding people up? Just like real world networks. Yeah.

25:08Mia Glaese:Yeah. The replacement thing is always like a sensitive one on the sort of PR side of things. But, you know, we create new jobs that manage the old jobs. That's how it is. Yourself, like, you know, I think in terms of the frontier evals that OpenAI is really going to excited to push. Like you put out really good work every single time. What should people expect from OpenAI itself?

25:28Olivia Watkins:I'm not sure I can say what we're going to.

25:30Mia Glaese:General directions.

25:31Olivia Watkins:I mean, general directions, I think looking at real world impact, like real world, real

25:37Mia Glaese:GDL2, you know, whatever.

25:38Olivia Watkins:That kind of stuff. Yeah. Yeah.

25:40Mia Glaese:Amazing. Okay. Well, I'm excited for more real world impact. I think you guys have, you know, really made a lot of progress and I think taken a lot of industry leadership for CBench Verified and now moving on to CWR. So thank you for doing this. Thank you for being so transparent. And I think people will respond in kind. Yeah. Yeah. Great for your time. Thank you.

26:02Thank you.