OpenAI's Noam Brown, Ilge Akkaya and Hunter Lightman on o1 and Teaching LLMs to Reason Better

0:00One way to think about reasoning is there are some problems that benefit from being able to think about it for longer. You know, there's this classic notion of system one versus system two thinking in human system one is the more automatic, instinctive response and system two is the slower, you know, more process driven response. And for some tasks, you don't really benefit from more thinking time. So if I ask you like, what's the capital of bootcamp, you know, you could think about it for two years. It's not going to help you get it get it right with higher accuracy. What is the capital of bootcamp?

0:32I actually don't know. But you know, there's some problems where there's clearly a benefit from being able to think for longer. So one classic example that I point to is the Sudoku puzzle. It's you you could in theory just go through a lot of different possibilities for like what the Sudoku puzzle might be. what the solution might be, and it's really easy to recognize when you have the correct solution. So in theory, if you just had like tons and tons of time to solve a puzzle, you would eventually figure it out.

1:13We're excited to have Noam Hunter and Ilgo with us today, who are three of the researchers on Project Strawberry or O1 at OpenAI. O1 is OpenAI's first major foray into general inference time compute, and we're excited I'd like to talk to the team about reasoning, chain of thought, inference time scaling loss, and more. I'll go hunter and gnome. Thank you so much for joining us and congratulations on releasing a one into the wild. I want to start by asking, did you always have conviction this is going to work?

1:46I think that we had conviction that something in this direction was promising, but the actual path to get here was never clear. And I, you know, you look at all one, it's not like this was an overnight day. It actually there's a lot of years of research that goes into this. And a lot of that research didn't actually pan out. But I think that there was conviction from OpenAI and like a lot of the leadership that something in this direction had to work. and they were willing to keep investing in it despite the initial setbacks. And I think that eventually paid off. I'll say that I did not have as much conviction as known from the very beginning.

2:31I've been staring at language models trying to teach them to do math and other kinds of reasoning for a while. And I think there's like a lot into research that's ebb and flow. Sometimes things work, sometimes things don't work. when we saw that the methods we were pursuing here started to work. I think it was a kind of aha moment for a lot of people, myself included, where I started to read some outputs from the models that were approaching problem solving in a different way. And that was this moment, I think, for me, where my conviction really set in. I think that OpenAI in general takes a very empirical data driven approach to a lot of these things.

3:12and when the data starts to speak to you, when the data starts to make sense, when the trends start to line up and we see something that we wanna pursue, we pursue it and that for me was when, I think the conviction really set in. Well, that you, Elga, you've been at OpenAI for very long time, five and a half years. Five and a half years. What do you think, did you have conviction from the beginning that this approach was gonna work? No, I've been wrong several times since joining about the path to AGI. We originally thought that robotics was the way forward. That's why I joined robotics team first.

3:47Embodied AI, AGI, that's where we thought things were going to go. But, yeah, I mean things hit roadblocks. I would say like during my time here, Chachi PT, well, I guess that's kind of obvious now that was a paradigm shift. We were able to share very broadly with the world, something that is a universal interface. And I'm glad that now we have a new path potentially forward to push this reasoning paradigm. Yeah, it was definitely not obvious to me for the longest time. Yeah. I realize there's only so much that you're able to say publicly for very good reasons about how it works. But what can you share about how it works even in sort of general terms?

4:36So the all one model series are trained with Aral to be able to think and you could call it's reasoning maybe also and it is fundamentally different from what we're used to with LLM's and we've seen it really generalize to a lot of different reasoning domains as we've also shared recently. So we're very excited about this paradigm shift with this new model family. And for people who may not be as familiar with what stated the art in the world of language models today, what is reasoning? How would you define reasoning? And maybe a couple words on what makes it important? Good question. I mean, I think one way to think about reasoning is there are some problems that benefit from being able to think about it for longer.

5:26You know, there's this classic notion of system one versus system two thinking in human, system one is the more automatic, instinctive response, and system two is the slower, more process -driven response. And for some tasks, you don't really benefit from more thinking time, so if I ask you, what's the capital of Bhutan? You could think about it for two years. It's not going to help you get it right with higher accuracy. What is the capital of Bhutan? I actually don't know. But there's some problems where there's clearly a benefit from being able to think for a longer. So one classic example that I point to is the Sudoku puzzle.

6:00It's, you could, in theory, just go through a lot of different possibilities for like, what the Sudoku puzzle might be, what the solution might be. And it's really easy to recognize when you have the correct solution. So in theory, if you just had like tons and tons of time to solve a puzzle, you would eventually figure it out. And so that's what I consider to be, I think a lot of people in the AI community have like different definitions of reasoning. And I'm not claiming that this is like the conical one. I think everybody has their own opinions, but I view it as the kinds of problems where there is a benefit from being able to consider more options and think for longer.

6:37You might call it like a generator verifier gap where there's like, it's really hard to generate a correct solution, but it's much easier to recognize when you have one. And I think all problems exist on this spectrum from really easy to verify relative to generation, like a Sudoku puzzle versus just as hard to verify as it is to generate a solution, like name of the capital of Gutan. I wanna ask about AlfaGo and know your background having done a lot of great work and poker and other games. To what extent are the lessons from gameplay analogous to what you guys have done with O1 and how are they different?

7:18So I think one thing that's really cool about O1 is that it does clearly benefit by being able to think for longer. And when you look back at like many of the AI breakthroughs that have happened, I think AlphaGo is the classic example. One of the things that was really noticeable about the bot though I think underappreciated the time was that it thought for a very long time before acting. It would take you know 30 seconds to make it to make a move. And if you tried to have it act instantly, it actually wasn't better than top humans. It was noticeably worse than them. And so it clearly benefited a lot by that extra thinking time.

7:56Now, the problem is that the extra thinking time that it had, it was running multiple outreach research, which is like a particular form of reasoning that worked well for go. But for example, doesn't work in a game like poker, which my early research was on. And so a lot of the like methods that existed for being able to reason, for being able to like think for longer, was still specific to the domains, even though the neural nets behind it, the system one part of the AI was very general. And I think one thing that's really cool about O1 is that it is so general. The way that it's thinking for longer is actually quite general and can be used for a lot of different domains and we're seeing that by giving it to users and seeing what they are able to do with it.

8:48Yeah, one of the things that's always been really compelling to me about language models and this is nothing new is just that because their interface is the text interface, they can be adapted to work on all different kinds of problems. And so what's exciting, I think, about this moment for us is that we think we have a way to do something, to do reinforcement learning on this general interface. And then we're excited to see what that can lead to. One question on that. You mentioned, I thought that was well put, sort of the, the, I forget exactly how you phrased it, but the gap between generation and verification, there's sort of a spectrum in terms of how easy things are to verify.

9:25does the method for reasoning remain consistent at various points in that spectrum or are there different methods that apply to various points in that spectrum? One thing I'm excited about for this release has been to get O1 in the hands of so many new people to play with it, to see how it works, what kinds of problems it's good at and what kinds of problems it's bad at. I think this is like something really core to open AI's strategy of iterative deployment. we put the technology that we build, the research that we develop out into the world so that we can see We do it safely and we do it so that we can see how the world interacts with it and what kinds of things we might not always understand fully ourselves And so in thinking about like what are the limits of our approaches here?

10:11I think it's been really enlightening to see like Twitter Show what it can and what it can't do I hope that that is like enlightening for the world that's useful for for everyone to figure out what these new tools are useful for and then I also hope where it will take back that information and use it effectively to understand our processes, our research, our products better. Speaking of which, is there anything in particular that you all have seen in the Twitter verse that surprised you, you know, ways that people have figured out how to use O1 that you hadn't anticipated? There's one thing I'm super excited about.

10:45I've seen a lot of MDs and researchers use the model as a brainstorming partner and what they are talking about is that like they've been in cancer research for so many years and they've been just running these ideas by the model about what they can do about these gene discovery gene therapy type of applications and they are able to get like these really novel ways of research to pursue from the model. Clearly the model cannot do the research itself but it can just be a very nice collaborator with humans for in this respect. So I'm super excited about about seeing the model just advanced a scientific path forward.

11:24That's not what we're doing in our team, but that is the thing, I guess, like we want to see in the world, the domains that are outside hours that gets really benefit by this model. No, I think you tweeted that deep RL is out of the trough of disillusionment. Can you say more about what you meant by that? I mean, I think there is definitely a period starting with I think Atari, you know, the Deep Mine Atari results, where Deep RL was the hot thing. I mean, I was in a PhD program. I remember what it was like in like, you know, 2015 to 2018, 2019, and Deep RL was the hot thing. And in some ways, I think that was, I mean, a lot of research was done, but certainly some things were overlooked.

12:10And I think one of the things that was kind of looked was the power of just training on tons and tons of data using, you know, something like the GPT approach. And in many ways, it's kind of surprising because if you look at AlphaGo, which was in many ways like the the crowning achievement of DeepRL, yes, there was this RL step, but there was also, I mean, first of all, there's also this reasoning step. But even before that, there was this large process of learning from human data. And that's really what got AlphaGo off the ground. And so then there was this like increasing shift. There is, I guess like a view that this was an impurity in some sense that so a lot of DeepRL is really focused on learning without human data, with just learning from scratch.

13:00Yeah, Alpha Zero, which was a great, which was an amazing result and actually ended up doing a lot better than AlphaGo. But I think probably because of this focus on Looney for Scratch, this GPT paradigm kind of flew under the radar for a while. And except for OpenAI, which saw some initial results for it and, you know, again, had the conviction to double down on that investment. Yeah, so there was definitely this period where DeepRL was the hot thing. And then I think, you know, when GPT -3 came out and some of these other like large language models. And there was so much success without deep RL.

13:44It there was like, yeah, a period of disillusionment where a lot of people switched away from it or kind of lost faith in it. And what we're seeing now with O1 is that actually there is a place for it and it can be quite powerful when it's combined with these other elements as well. And I think a lot of the DPRL results were in kind of well -defined settings like gameplay. Is O1 one of the first times that you've seen DPRL used in much more general kind of unbounded setting? Is that the right way to think about it? Yeah, I think it's a good point that a lot of the highlight DPRL results were really cool, but also very narrow in their applicability.

14:28I mean, I think there were a lot of quite useful deep RL results and also quite general RL results, but there wasn't anything comparable to something like GPT -4 in its impact. So I think we will see that kind of level of impact from deep RL in this new paradigm going forward. One more question in this general train of thought. I remember the AlphaGo results, you know, at some point with the in the lease of the old Tournament, there was move 37 and that move surprised everybody. Have you seen something of that sort where O1 tells you something and it's surprising and you think that it's actually right and it's better than any top human good think of?

15:09Have you had that moment yet with the model? Or you think it's O2, O3? One of the ones that comes to mind is we spent a lot of the time preparing for the IOI competition that we put the model into. looking at its responses to programming competition problems. And there was one problem where it was, someone was really insistent on solving the problem in this kind of weird way, with some weird method, I don't know exactly what the details were. And our colleagues who are much more into competitive programming were trying to figure out why it was doing it like this. I don't think it was quite a, like, this is a stroke of genius moment.

15:44I think it was just like the model didn't know the actual way to solve it. And so it just like being ahead until it found something else. Did it get there? Yeah, yeah, it solved the problem. It just it just it used some it was like it was some method that would have been really easy if you saw something else I wish I had the specific one, but I remember that being kind of interesting. There's a lot of the things in the in the in the programming competition results I think somewhere we have the IOI competition programs published Where you can start to see that the model doesn't approach thinking quite like a human does, or doesn't approach these problems quite like a human does it has slightly different ways of solving it for the actual IOI competition.

16:23There was one problem that humans did really pour on that the model was able to get half credit on. And then another problem that humans did really well on that the model was like barely able to get off the ground on. Just showing that it kind of has a different way of approaching these things than then maybe a human would. I've seen the model solve some geometry problems and the way of thinking was quite surprising to me such that you're asking the model just like give me this like sphere and then there are some points on the sphere and asking for probability of some event or something and the model would go let's visualize this, let's put the points and then if I think about it that way or something so I'm like oh you're just using words and visualizing something that really helps you contextualize.

17:09I would do that as a human and seeing all one do it too, just really surprises me. Interesting. That's fascinating. So it's stuff that's actually understandable to a human and would actually kind of expand the boundaries about humans would think about problems versus some undecisurable machine language. That's really fascinating. Yeah, I definitely think one of the cool things about our own result is that these chains of thoughts the model produces are human interpretable. And so we can look at them and we can kind of poke around how the model is thinking. Were there, were there aha moments along the way?

17:42Or were there moments where, you know, Hunter, you mentioned that you were not as convinced at the outset that this is the direction that was going to work? Was there a moment when that changed where you said, oh my gosh, this is actually going to work?

17:56Yeah, so I've been in an open air about two and a half years and most of the time I've been working on trying to get the models better at solving math problems. And we've done a bunch of work in that direction. We've built various different the bespoke systems for that. And there was a moment on the O1 trajectory where we had just trained this model with this method with a bunch of fixes and changes and whatnot. and it was scoring higher on the Matthew Vals than any of our other attempts. Any of our bespoke systems and then we were reading the Change of Thought and you could see that they felt like they had a different Character in particular.

18:34You could see that when it got stuck it would say wait this is wrong Let me take a step back. Let me figure out the right path forward. And we called this backtracking and I think for a long time I'd been waiting to see in instance of the models backtracking and I kind of felt like I wasn't going to get to see an other regressive language model backtrack because they're just kind of predict next token, predict next token, predict next token. And so when we saw this score on the math test and we saw the trajectory that had the backtracking, that was the moment for me where I was like, wow, this is like something is coming together.

19:07I didn't think was going to come together and I need to update. And I think that was when I grew a lot of my conviction. I think the story is the same for me. I think it was I've probably run the same time actually. Like, I, you know, I definitely, I joined with this idea of like, you know, chat to BT doesn't really think before responding. Like, it's very, very fast. And there was this like powerful paradigm of like, in these, in these games of AI is being able to think for longer and getting much better results. But, and there's a question about how do you bring that into language models that I was really interested in.

19:41And you know, that's like, it's easy to say that, but then there's like, there's this difference between just like saying that, oh, there should be a way for it to think for longer than actually like delivering on that. And so we, you know, we, I tried to, I tried a few things and like other people were trying a few different things. And in particular, yeah, one of the things we wanted to see was this ability to backtrack or to like recognize when it made a mistake or to like try different approaches. And we had a lot of discussions around how do you enable that kind of behavior? And at some point, we just felt like, okay, well, one of the things we should try, this is a baseline, is just to have the AI thing for longer.

20:21And we saw that, yeah, once it's able to, to think for longer, it develops these abilities, almost like, emergently, that we're very powerful, and contain things like backtracking and self correction, and all these things that we were wondering how to enable in the models. And to see it come from such a clean, scalable approach, that was for me the big moment when I was like, okay, it's very clear that we can push this further. And it's so clear to see where things are going. No, my thing is understanding how strong and effective his conviction and test time compute was. I feel like all of our early one on ones when he joined we're talking about test time computer and it's power and I think multiple points throughout the project known would just say, why don't we let the model think for longer?

21:12And then we would and it would get better and he would just be, he would just look at us kind of funny like we hadn't done it until that. One thing we noticed in your e -vails is that, you know, O1 is noticeably good at STEM. It's better at than the previous models. Is there a rough intuition for that? Why that is? I mentioned before that there's some tasks that are reasoning tasks that are easier to verify than they are to generate a solution for, and there's some tasks that don't really fall into that category. And I think STEM problems tend to fall into the what we would consider hard reasoning problems.

21:48And so I think that's a big factor for why we're seeing a lift on STEM kind of subjects. Makes sense. I think relatedly we saw that in the research paper that you guys released, that 01 passes your research engineer interview with pretty high pass rates. What do you make of that? And does that mean at some point in the future, OpenAI will be hiring 01 instead of human engineers? I don't think we're quite at that level yet. I think that there's more. It's hard to be the 100 % though. Maybe the interviews need to be better. Okay. I think that the 0 -1 does feel at least to me, I think other people on our team like a better coding partner than the other models.

22:35I think it's already authored a couple of PRs in our repo. And so in some ways it is acting like a software engineer. Because I think software engineering is one of the STEM domains that benefits from longer reasoning. I don't know, I think that the kinds of rollouts that we're seeing from the model are thinking for a few minutes at the time. I think the kinds of software engineering job that I do when I go on right code, I think for more than a few minutes at a time. And so maybe as we start to scale these things further, as we start to follow this trendline and let O1 think for longer and longer, it'll be able to do more and more of those tasks.

23:12And we'll see. you'll be able to tell that we've achieved AGI internally when we take down all the jobless things. And you know, the company's doing really well, really poorly. What do you think it's going to take for a one to get great at the humanities? Do you think being good at raising and logic and STEM kind of naturally will extend to being good at the humanities? As you scale up in front of time, or how do you think that plays out? You know, we're like we said, we released the models and we were kind of curious to see what they were good at and what they weren't as good at and what people end up using it for.

23:47And I think there's clearly a gap between the raw intelligence of the model and how useful it is for various tasks. In some ways it's very useful, but I think that it could be a lot more useful and a lot more ways. And I think there's still some iterating to do to be able to unlock that. like more general usefulness. Well, I'm kind of scone that. Do you view, I'm curious if there's a philosophy at OpenAI or maybe just a point of view that you guys have on, how much of the gap between the capabilities of the model and whatever real world job needs to be done? How much of that gap do you want to make part of the model?

24:28And how much of that gap is sort of the job of the ecosystem that exists on top of your APIs, like their job to figure out? Do you have a thought process internally for figuring out what are the jobs to be done that we want to be part of the model versus where do we want our boundaries to be so that there's an ecosystem that exists around us? I always heard that opening I was very focused on EGI. I was honestly kind of skeptical of that before I joined the company. Basically the first day that I started, there was an all hands of the company and Sam got up in front of the whole company and basically laid out the priorities going forward for the short -term and the long -term, it became very clear that AGI was the actual priority.

25:14And so I think the clearest answer to that is AGI is the goal. There's no single application that is the priority other than getting us to AGI. Do you have a death mission for AGI? Everybody has their own definition for you. Exactly. That's what I'm curious. I don't know if I have a concrete definition. I just think that it's something about the proportion of economically valuable jobs that are models and our AS systems are able to do. I think it's going to ramp up a bunch over the course of the next however many years. I don't know. It's one of those, you'll feel it when you feel it and will move the goalpost back I can be like, this isn't that for however long until one day we're just working alongside these AI coworkers and they're doing large parts of the jobs that we do now and we're doing different jobs and the Holy Co system of what it means to do work has changed.

26:14One of your colleagues had a good articulation of the importance of reasoning on the path to AGI, which I think paraphrases as something like any job to be done is going to have obstacles along the way. And the thing that gets you around those obstacles is you're going to reason through them. And I thought that was like a pretty nice connection between the importance of reasoning and the objective of AGI and sort of being able to accomplish economically useful tasks. Is that is that the best way to think about what reasoning is and why it matters or there are other frameworks that you guys tend to use?