How OpenAI Beat Every Human Team at the World's Hardest Coding Competition

0:00We didn't tell it to do that. It had never done that before. Can our models actually discover some new knowledge? One of the problems that no human contestants solved, our models solved. GPT-5 tried seven times to solve that last problem. It was just like something else. It was mind-blowing. They'd write solutions that were so bad that the little virtual computer, the sandbox would give it, would just crash.

0:27Ahmed El-Kishky:The world's most elite programmers trained for years to compete at ICPC, the Olympics of coding. This year, OpenAI's system solved all 12 problems, outscoring every human team in the world finals. Today, we're joined by one of the researchers that made that happen. Welcome, humans, to the latest episode of The Neuron. I'm Corey Knowles, joined as always by Grant Harvey, writer of the Neuron Daily Newsletter. This episode is brought to you by Whisperflow. More on them to come later on. But today, we are chatting with Ahmed El-Kishki. Ahmed is a research lead at OpenAI, where he focuses on advancing reasoning, tool use, and programming capabilities in AI models.

1:04He holds a PhD in computer science and has worked at Twitter and Meta on large-scale data mining, multilingual NLP, and cross-lingual representation.

1:14Ahmed El-Kishky:Most recently, and what's brought us together today, though, Ahmed was a part of the team behind OpenAI's historic win at the International Collegiate Programming Contest, where their system solved every problem in the World Finals, marking the first time an AI has defeated all human teams. Ahmed, welcome to the Neuron. Thank you for having me, Corey and Grant. First off, congratulations. How did it feel when the moment that the system cleared that 12th problem and you realized, you know, wow, we've just made history? Is that what you expected? Actually, we weren't expecting to get all of them right.

1:47So let me give you a quick like summary of like how these competitions go. So we generally don't really like prep for them too much. Usually it's a, hey, like if we have enough time, a group of like two to three people can like form a ragtag team and just try to get something like this going. So in this case, we had like three or so people from my team. They flew out to Azerbaijan, which is, you know, way on the other side of the world. We had somebody like, you know, back home here, making sure everything was up and ready systems wise. and yeah like they were almost like hackathoning it they were just like up you know sitting there with everybody in the competition um they had the same five hour limits uh they were just like eagerly looking at the problems like as you know the model came back with a solution they'd like you know eagerly submit it and uh the cool part is like the first 11 that were submitted like Everyone was correct on the first try.

2:48Amazing. So it was mind-blown. We didn't even think that would happen. Yeah. And then we got to the last problem, and the last problem was so difficult. We kept submitting it every 10 or so minutes, and it would be wrong. So it was a little bit nerve-wracking. It already, like four hours, it passed. It had already gone to the last hour of the competition. And we're like, okay, we're probably like 11 out of 12 is pretty good. That's already really great. We were looking at the scoreboards. And then like in the last like little bit of the competition, we saw that answer accepted. And it was just like something else.

3:23It was just it was mind blowing. Just seeing all 12. It was like really a pivotal moment for like just AI, but also like all the hard work we've been doing so far.

3:33Ahmed El-Kishky:That's that seems so exciting and fun. Like like just like playing high school basketball or something. I know. I was going to say I could see the sports movie like of this and playing in my head, you know. definitely yeah biopic actually it's kind of interesting um we've been doing like you know this year we've done several of these programming competitions and it is almost like sports i've never actually like you know closely followed competitive programming like i tried it out in college and uh in high school um but like we competed in japan at atcoder uh earlier this year and uh it was obviously on like japan time uh and we were trying to see how well our models could do on like these new types of programming competitions where there isn't really a correct answer.

4:17There's only like how good you can do, how well you could do. So the competition was like up until like, you know, three, four in the morning. It was a 10 hour competition, which is crazy. Oh man. But everybody just like stayed up all night. Like we had this like channel on Slack and like hundreds of researchers were just following along. It was in Japanese, so we couldn't understand it. but we could hear like OpenAI's name being like yelled out but people were just like following along, looking at the solutions. It really felt like a sports game or like a high school basketball game.

4:50Ahmed El-Kishky:How did OpenAI get into doing these competitions in the first place? I know this is the second big one we've seen fall this year, right? This was the third actually. Third? Yeah. So I'll tell you a little bit about it. I joined OpenAI a couple of years ago and like programming competitions were like my first big project I was working on. When I first started, like I was like, okay, like maybe this is like a way to sort of see how good our models are. And there's some like hint of truth to that. When I joined GPT-4, I'd just been released. It was kind of abysmal at these competitions. It was, I think like 300, 400 ELO, if you know chess terminology, which like places it solidly at like sub-novice.

5:36somebody that just like you know just started um so it wasn't really good at all and it was kind of in fact quick funny story when we try to actually try these programming competitions problems they were so difficult and the model was so bad at them they'd actually kind of like crash the computer and so they'd write solutions that were so bad that the little virtual computer the sandbox would give it would just crash it ran out of memory it would just hang and we'd have to like, you know, delete it. It was that bad. But a lot of people at OpenAI really come from the competitive programming community.

6:13From like high school, they would do Olympiads, international like informatics Olympiad. In college, they would do ICPC. They would participate in these competitions like Google Code Jam. And so there was this like deep seated community of competitive programmers. And I think fundamentally they had a belief that competitive coding was a really good, I don't want to say benchmark, but a good way to sort of measure how well you were improving at reasoning. So when we saw that it was really bad at them, many people were just like, hey, wouldn't it be great if our models could compete at the same level as like some of the brightest people in the world, some of the brightest students, competitive programmers, that would be a good measure of like progress towards reasoning.

6:59So yeah, like we have Yaka Pachoki, who is our chief research scientist. We have Mark Chen, who is our chief research officer. And they really like heavily pushed for, you know, starting this work stream. That's awesome. Yeah, as you mentioned three. So is that does that include the International Math Olympiad or is it three programming competitions specifically? Oh, yeah. So three programming competitions and then one math competition um got it got it yeah so the first one we tried this year uh was at coder which is like this premier competitive programming competition that's held in japan every year the world finals there um we decided we wanted to mix it up there we didn't compete in the traditional like algorithmic competitive programming uh where there's usually a correct solution a solution that's you know is able to answer all the questions within a certain time complexity under certain memory constraints.

7:50That's like the traditional programming competition, which is like ICPC or IOI. This one was a, it's called the heuristic competition. Heuristics are necessary to solve like these problems that they're impossible to get a correct answer to. Like finding the best answer is just fundamentally possible. Like theoretically, it's difficult to do. It's super difficult. And so the best that people can do is writing these heuristics. They're programs that sort of find better and better solutions, but there's no idea of like a best solution. And so in these competitions, we were, yeah, we're trying to find like, you know, really good solutions.

8:33The model has to be creative because it isn't like the traditional, this is correct, this is incorrect. The model just have to, you know, submit solutions and then looking at like how well it did, it would have to then one up itself. So it was really in a competition against itself to sort of get better and better solutions. And we'd never tried these out before. And so we wanted to sort of see how we did. One example of these competitions or these problems are like games. Sometimes some games, there's no like, you know, best, you know, game or anything or best program, but you can sort of just get better and better at it.

9:05Ahmed El-Kishky:Okay. So let's talk about a tool that's completely transformed how I personally work. That's Whisperflow. Imagine being able to write full articles, emails, even take complex notes just by talking. That's what Whisperflow lets me do. It's hands-free writing that's smart, accurate, and ridiculously fast. For me, it solved a decade-long problem. I used to cover baseball as a beat reporter and had to file my story deep in the middle of the night. I always wanted a quality dictation tool that would let me get started on the way home. I wanted to be able to just talk my ideas into a dot. I tried several tools, they all came up short.

9:43Ahmed El-Kishky:But with Whisperflow, I can dictate an entire piece, have it cleaned up, and file all while I'm on the go. No more waiting to get back home, fire up a laptop in the middle of the night to rush something out. I was able to take time that I already had and use it. But it's not just about accessibility, it's about productivity. You'll save hours, get your thoughts down instantly, and stay in flow without breaking a note. So whether you're a writer, a founder, or just someone who needs to capture ideas quickly and accurately on the go, Whisperflow makes it effortless. Seriously, you're going to want to check this one out.

10:16Ahmed El-Kishky:You'll wonder how you ever worked without it. It's available on Mac, Windows, and iPhone. Visit whisperflow.ai slash neuron today and get started for free. That's whisperflow.ai slash N-E-U-R-O-N. Tell them the Neuron sent you. Well, then exactly like how do you prepare for a contest like that? You know, like what was happening behind the scenes, you know, as you're leading up to this? And, you know, if you could tell us a little bit about who else is on the team. I know a couple of them like Mustafa and Boris, but yeah. Yeah, yeah. I'll talk a little bit about ICPC. It's honestly just a team sport here.

10:53I don't even want to say it's just like a handful of people. um it's the byproduct of like you know pre-training our models you know and then doing like rl on them in general to make them really great reasoners really great tool use um uh and then afterwards you know there's so many people that contributed different ml aspects to the models one of the experimental reasoning model we used was a byproduct of the imo efforts by Alex Way, Cheryl, Noam. So there's so many people sort of playing a part here. But the core people that actually went to Azerbaijan, it's almost like a volunteer experience.

11:32We're just like, who wants to try this out? And Mustafa was one of them. We had Robin, we had Andrew, and Boris was helping out from London. So they formed almost like the core team of people that were, you know, driving this. A lot of it was sort of making sure the models were ready to go because there's no room for error here. It's a five-hour competition. You basically, it's live. So it's not like, oh, we're having technical issues. Let's push us back by an hour. We were like in the same situation as a student. People were nervous. They get the problems in the same PDF format. And so everything had to be just ready to go.

12:12And so they spent like, you know, a week before the competition, getting things up, trying it out, making sure that like things were going, doing a dry run, you know, spending some of the like spending long nights trying to make sure everything's working and then just flying out. And just, you know, after that, it's, you know, hope that you prepared well enough and the models can take it from there.

12:33Ahmed El-Kishky:You know, I understand you guys combined GPT-5 with an experiential reasoning machine system. What makes that a powerful combination for ICPC? We wanted to see how well models that are available to the public can do. So GPT-5, we just released it not too long before. And so we wanted to really see how well could these models, given enough compute and attention, actually solve some of these problems. And so it was kind of important to us to try out GPT-5. The experimental reasoning model is stuff that we worked on. We're always, you know, trying out, uh, to trying to improve our models. Like we're always running, uh, new experiments, training larger models, uh, more RL and seeing how far we can push reasoning.

13:24Some of these models never end up making it, uh, into like chat GPT. Um, but we're always learning from them. Uh, so in this case, we knew that, you know, GP5 can go a long way, but we wanted to, you know, not just, you know, limit it there. We wanted to also test how well our reasoning models perform. And so we tried out both. There's no reason to do one. We were looking to see a system that could push the frontiers of these competitive programming competitions. And so it was a very simple approach here. It was just like both GPT-5 and the IMO model, the experimental reasoning model, would take each problem and they would try to solve it.

14:02They would try to solve it multiple times. They would give like a collection of solutions. And then the experimental reasoning model would be like, I like the solution the most. Let's submit this one. Wow. That's awesome. It was a good model or it was a good approach just because we had two models. So the two models, you know, they have some diversity there. They think a little bit differently. So that helps. But the experimental reasoning model is just a more powerful model. GBD5 is, you know, a bit faster. So I was able to get the answers quickly. But ultimately, it was just a testament to how well the experimental reasoning model could sort of be like, okay, this is a good solution.

14:44Let's just submit this one. And then also, the last problem, GBD5 couldn't handle it. So we needed to fall back to the powerhouse, the reasoning model to get that one. Yeah, well, let's talk about that. So 11 problems were solved on the first attempt, like you said. What does this tell us, in your opinion, about the reasoning capabilities? And maybe you could expand a little bit on the 12th problem, too, if that has any insights there. It's basically a testament to how well we've managed to get just tier one quality reasoning models into the hand of the public. Like just a couple of years ago, it would have been unfathomable to sort of have a model as powerful as GPT-5 in just the hands of everybody.

15:28And now like people all over the world can just go on there and just try it out. It's honestly amazing. Like the progress, I look at it and I'm just like, wow, how far like we've come. So it's, yeah, just a testament to how well just the standard models that everyone has at their fingertips can reason, can solve these difficult problems. And it isn't just competitive programming. It's just all over. They do well in mathematics. Academics use it all the time to help them in their research. People use it in their day-to-day jobs in engineering and science. um so uh yeah it was just a testament to like what has now been commoditized um and uh the reasoning problem or the the reasoning model itself uh obviously like it's one of our you know newer models uh internally stuff we were experimenting on um it's it's not quite you know as fast as gpd5 because we're just you know training it for research um yeah but it's demonstrably better.

16:26Like you could tell, uh, GPT-5 tried seven times to solve that last problem. And each time it's failed. Uh, so you can imagine, you know, like Mustafa and Boris and Robin and Andrew sitting there, uh, submitting every, you know, like 10, 20 minutes and me and me, okay, this one's wrong too. Um, and all the while the experimental reasoning model is just thinking, it's thinking for hours and hours. It hasn't even like, I'll put it a single answer yet. so it's three hours later and it's still thinking it's just grinding on this one problem uh and then eventually it gets to uh solving it we start seeing it like you know get some answers uh we submit the first one it's wrong as well and then the second one that um it's we submit from that model is correct so gbd5 tried seven times and got it wrong and then you know on the second try uh the experimental model the imo model uh just you know got it amazing how close wait I think you said this before, but how close was the countdown clock?

17:23Like, how many minutes did you have left? I don't really know, but it must have been within, like, 30 minutes of, like, the competition. Wow.

17:29Ahmed El-Kishky:You know, a year ago, I mean, when we sit down and think about this, like, JGBT isn't three years old. I mean, like, I mean, just saying that sounds crazy. And, you know, and I think about AI struggling with even fairly easy contest problems, like, as recently as a year ago. Were there any, you know, insights or breakthroughs along the way specifically that really enabled this big leap we've seen over the last 12 months or so? I think we announced it about a year ago, but the biggest leap was just the introduction of reasoning models. If you remember when ChatGPT was released, I guess almost three years now ago, the models or ChatGPT, you'd ask it a question and immediately it would just give you an answer.

18:14It would just always give an answer. Sometimes the answer would be correct. Sometimes it wouldn't be. It didn't matter how hard the problem was. It would just always immediately give you an answer. So back then, people had these approaches. They called it chain of thought prompting. They'd be like, show your steps. Think through step by step. They would just ask the model to. And people noticed that when you just asked it to show your work, the benchmarks would just improve a little bit. It would be a tiny bit smarter. Yeah. So we worked on that. We were like, okay, what's happening here? And the idea is that like, when you have a difficult problem, no human just immediately like gives an answer.

18:56If I asked you to multiply like a four digit number by a five digit number, you wouldn't like give me an answer instantly. You would like, you know, get your piece of paper and pencil. You'd work it out. You'd think through, maybe make a mistake and fix it. And you give an answer. If you asked a scientist to work on a problem or a mathematician, they do the same thing. Hard problems require, you know more time more thinking um so we decided to lean into reinforcement learning as a way to get our models to actually you know think longer uh think better um and that's actually where the breakthrough came in um i thought what a breakthrough it was too it was a it was a crazy one like um we wanted to see like we wanted to a little bit mimic how a human you know thinks through these problems.

19:44They try things out and maybe they go down wrong directions to dead end. They course correct. And OpenAI has always been into like reinforcement learning. From the days of like, you know, Dota playing video games, they really leaned into reinforcement learning as a tool that would bring about sort of next level intelligence. And there'd been some attempts to apply reinforcement learning to LLMs, but nothing at this scale. So the, I guess, the strawberry efforts, the O-series models were the first, like, very serious attempt at getting reinforcement learning working on these large language models.

20:24And it was honestly, like, amazing to sort of see it from the beginning, like, you know, when it started to now being at a performance level where it's competitive with some of the best competitive programmers. I guess like as just what a couple more questions about the competition. So which problem in the contest surprised you the most? Like either because it was harder than you expected, like the 12th one, or maybe was it one of the first 11 that you were like shocked that it was so easy?

20:54Ahmed El-Kishky:Or the approach it took even. Yeah. Yeah. So, I mean, to be fair, I don't think I'm even at the level where I can judge these problems. That's fair. Yeah. Like these college students are like, there's something else. I mean, they really should be proud. These are some of the most brilliant college students in the world. And if I took these competitions, maybe if I'm lucky, I'd get one. But that's being generous. But I mean, after the competition, I looked at the problems. And it was really interesting to me that one of the problems that no human contestant solved, our model solved pretty straightforward.

21:32the problem that we struggled with the most. It was also a difficult problem. The people that solved it among the human contestants took nearly all the time to solve it. So it shows me that maybe what's difficult for an AI isn't necessarily what's difficult for a human, and maybe what's difficult for a human isn't necessarily what's difficult for an AI. But yeah, so that was sort of the interesting aspect to me. What did those iterations reveal, if anything, that was new to you, like about the systems reasoning process? Did you feel like you'd learned anything new about how it thinks, like from reading that back and trying to understand, like kind of like going through after the fact and seeing what it did?

22:14I guess I do have the privilege of being able to peek into the chain of thoughts from these models. And it's honestly, sometimes it's a little bit crazy to read through because they're so, yeah, like they do like such clever things. Yeah. So I'll tell you a little bit of a story, actually. Let's make this a story over like how it started versus where it's at now. When it started, it would basically barely even reason. It would just try to output the answer when we first started like impetus programming two years ago. um when you started doing rl like just getting started there it would do a little bit of planning it would be like i'm gonna try this uh and we saw an improvement there over time the reasoning that the models would do would actually become way more elaborate uh and it was really interesting because we have a lot of like ioy gold medalists icpc competitive programmers and they'd be like oh yeah that's that's a strategy that i would use um so one of the coolest um the coolest things in my opinion is when the model decided it wanted to test itself before it submitted.

23:19And so what the model would do is, so the solution would have to be a very complicated algorithm. It would have to have great complexity. It would be very memory aware, very complicated. The algorithm would be like so complex that it's easy to make a mistake. So what the model would do first is write a brute force solution, which is a lot easier to do. It can be like a few lines. And then it would generate some inputs, like pretending to be a, like, you know, the judge would generate inputs in the same way. And then it would compare the outputs of the brute force solution to the really complicated algorithm that it wrote.

23:56And it knows that for it to be correct, the outputs have to match. So it couldn't submit the brute force solution because, you know, there's strict time limits and memory limits. It wouldn't like be correct, but it would spend a lot of time thinking and then trying to match the outputs of the brute force solution. And I remember when I noticed that and just showed it to people on the team, the competitive programmers at OpenAI would be like, yeah, that's a valid strategy. And what was really cool is we didn't tell it that. We didn't tell it to do that. It had never done that before. What had happened was just over time, as we trained our models, it just decided, hey, this is a valid strategy.

24:33I'm getting really good results when I do this. So let's keep doing it. And that's the beauty of RL. Yeah. And when you look at that. That's so awesome. Yeah. When you look at the reasoning, I mean, it's doing tricks like this all over the place. You know, obviously, IMO happened earlier this year. That was the International Mac Olympiad. And, you know, both you and Google were competing in that. And they did eventually release, I believe, a version of that model that they ran IMO with called DeepThink. And I think there's a version of that that certain users of Google can use now. And I don't know if you mentioned that you're using the same IMO model.

25:09or a version of that that you used for this competition. Do you think OpenAI will eventually release like a similar reasoning model from the ICPC system available in production in chat GPT, perhaps as a future model? Or what are your thoughts there? One of the big powerhouses of the ICPC was GPT-5. So that's already kind of available to the public. Right. So the exact model that was used for like IMO and ICPC, the extreme reasoning model, that one probably is not going to be specifically released, but the insights that were used to train that model will be eventually incorporated into later models.