O3 and the Next Leap in Reasoning with OpenAI’s Eric Mitchell and Brandon McKinzie

0:05Hi, listeners, and welcome back to KnowPriors. Today, I'm speaking with Brandon McKenzie and Eric Mitchell, two of the minds behind OpenAI's O3 model. O3 is the latest in the line of reasoning models from OpenAI. Super powerful with the ability to figure out what tools to use and then use them across multi-step tasks. We'll talk about how it was made, what's next, and how to reason about reasoning. Brandon and Eric, welcome to No Priors. Thanks for having us. Yeah, thanks for having us. Do you mind walking us through O3, what's different about it, what it was in terms of a breakthrough, in terms of a focus on reasoning and you're adding memory and other things versus this core foundation model, LLM, and what that is?

0:41So O3 is like our most recent model in this O series line of models that are focused on thinking carefully before they respond. And these models are in sort of some vaguely general sense smarter than like models that don't think before they respond, you know, similarly to humans. It's easier to be, you know, more accurate if you think before you respond. I think the thing that is really exciting about O3 is that not only is it just smarter if you make like an apples to apples comparison to our previous O-series models, you know, it's just better at like giving you correct answers of math problems or factual questions about the world or whatever.

1:26This is true, and it's great, and we will continue to train models that are smarter. But it's also very cool because it uses a lot of tools that enhance its ability to do things that are useful for you. So, yeah, you can train a model that's really smart, but if it can't browse the web and get up-to-date information, there's just a limitation on how much useful stuff that model can do for you. If the model can't actually write and execute code, there's just a limitation to how, you know, the sorts of things that an LLM can do efficiently, whereas like a relatively simple Python program can, you know, solve a particular problem very easily.

2:02So not only is the model, it's on its own smarter than our previous O-series models, which is great, but it's also able to use all these tools that like further enhance its abilities. And whether that's doing like research on something where you want up to date information or you want the model to do some data analysis for you, or you want the model to be able to do the data analysis and then kind of review the results and adjust course as it sees fit. instead of you having to be so sort of prescriptive about like each step along the way, the model is sort of able to take these like high level requests, like do some due diligence on this company and, you know, maybe run some reasonable like forecasting models on so and so thing.

2:38And then, you know, write a summary for me, like the model will kind of like infer a reasonable set of actions to do on its own. So it gives you kind of like a higher level interface to doing some of these more complicated tasks. That makes sense. So it sounds like basically there's like a few different changes between your core sort of GPT models, where now you have something that takes a pause to think about something. So at inference time, you know, there's more compute happening. And then also it can do sequential steps because it can kind of infer what are those steps and then go act on them.

3:05How did you build or train this differently from just a core foundation model or, you know, when you all did GPT 2.5 and 4 and all the various models that have come over time? What is different in terms of how you actually construct one of these? I guess the short answer is reinforcement learning is the biggest one. So yeah, rather than just having to predict the next token and some large pre-training corpus from, you know, everywhere, essentially, now we have a more focused goal of the model, solving very difficult tasks and taking as long as it needs to do to figure out the answers to those problems.

3:41Something that's like kind of magical for me, a user experience for me was we've in the past for our reasoning models talked a lot about test time scaling. And I think for a lot of problems, you know, without tools, test time scaling might occasionally work. And but at some point, the model is just kind of ranting in its internal chain of thought. And especially for like some visual perception ones, it knows that it doesn't it's not able to see the thing that it needs. And it just it just kind of like loses its mind and goes insane. and I think tool use is a really important component now to continuing this test time scaling and you can feel this when you're talking to O3 at least my impression when I first started using it was the longer it thinks, I really get the impression that I'm going to get a better result and you can kind of watch it do really intuitive things and it's a very different experience but being able to kind of trust that as you're waiting it's worth the wait and you're going to get a better result because of it and the model's not just off doing some totally irrelevant thing.

4:41That's cool. I think in your original post about this too, you all had a graph which basically showed that you looked at how long it thought versus the accuracy of the result, and it was a really nice relationship. So clearly, thinking more deeply about something really matters. And it seems like in the long run, do you think there's just going to be a world where we have sort of a splitter bifurcation between models, which are sort of fast, cheap, efficient, get certain basic tasks done, And then there's another model which you upload a legal M &A folder and it takes a day to think. And it's slow and expensive, but then it produces output that would take you a team of people a month to produce.

5:20Or how do you think about the world in terms of how all this is evolving or where it's heading? I think for us, unification of our models is something that Sam has talked about publicly, that we have this big crazy model switcher in ChatGPT and there are a lot of choices. And, you know, we have a model that might be good at any particular thing, you know, that a user might want to do, but that's not that helpful if it's not easy for the user to figure out, well, which model should I use for that task? And so, yeah, making the models better able, you know, making this experience more intuitive is definitely something that is, is like, valuable and something we're interested in doing.

6:02And that, that applies to this, you know, question of like, you know, are we going to have like two models that, you know, people pick between or a zillion models that people pick between? Or do we put that decision, you know, inside the model? I think everyone is going to try stuff and figure out what works well for the problems they're interested in and the users that they have. But yeah, that question of how do you make that sort of decision be as effective, accurate, intuitive as possible is definitely top of mind. Is there a reason from a research perspective to combine reasoning with pre-training or try to have more control of this?

6:47Because if you just think about it from the product perspective of like the end consumer dealing with chat GPT, like, you know, we won't get into the naming nonsense here, but they don't care. They just want like the right answer and the amount of intelligence required to get there in as little time as possible. Right? The ideal situation is it's like intuitive that like how long should you have to wait? You should have to wait as long as it takes for the model to like give you a correct answer. And I hope we can get to a place where our models have a more precise understanding of their own level of uncertainty.

7:21Because, you know, if they if they already know the answer, they should just kind of tell you it. And if if it takes them a day to actually figure it out, then they should they should take a day. But you should always have a sense of like it takes exactly as long as it needs to for that current like models intelligence. And I feel like we're on the right path for that. Yeah, I wonder if there isn't a bifurcation, though, between like an end user product and a developer product. Right. Because there are lots of companies that use, you know, the APIs to all of these different models and then for very specific tasks.

7:53And then on some of them, they might even use like open source models with really cheap inference with stuff that they control more. I hope you could just kind of tell the model like, hey, this is an API use case. And yeah, you really can't be over there thinking for like 10 minutes. We got to get an answer to the user. It'd be great if their models kind of get to be more steerable like that as well. Yeah, I think it's just a general steerability question. Like at the end of the day, if the model's smart, like you should be able to specify like the context of your problem and the model should do the right thing.

8:28There's going to be some like limitations because, you know, maybe just figuring out given your situation, what is like the right thing to do might require thinking in and of itself to figure out. So like, it's not that you can obviously do this perfectly, but, but yeah, pushing, you know, some, all the right parts of this into the model to make things easier for the user is like, seems, is a very good goal. Can I go back to something else you said? Like, so the first guest we ever had on the podcast was actually Noam Brown. I've heard of him. You know, two plus years ago. Yes, I know. It'd be great to get some intuition from you guys for why tool use helps test time scaling work much better.

9:10I can give maybe very concrete cases for the visual reasoning side of things. There's a lot of cases where, and back to also the model being able to estimate its own uncertainty, you'll give it some kind of question about an image. And the model will very transparently tell you an initiative thought, like, I don't know, I can't really see the thing you're talking about very well. Or like, it almost knows that its vision is not very good. and uh well it's kind of magical it's like when you give it access to a tool it's like okay well i gotta figure something out uh let's see if i can like manipulate the image or crop around here or something like this and um what that means is that it's it's it's like much more productive use of tokens as it's doing that and so your test time scaling slope you know goes from something like this to you know something much steeper and uh we've seen exactly that like the the the test time scaling slopes for without tool use and with tool use for for visual reasoning specifically are very noticeably different.

10:04Yeah, I was gonna say like for like writing code for something like there are a lot of things that an LLM could try to figure out on its own, but would require a lot of attempts and self verification that you could write a very simple program to do in like a verifiable and, you know, much faster way. So, you know, I do some research on this company and like use this type of, you know, valuation model to tell me like, you know, what the valuation should be. Like you could have the model, like try to crank through that and like fit those coefficients or whatever in its context, or you can literally just have it like write the code to just do it the right way.

10:49Um, and just know what the actual answer is. And so, um, yeah, I think like part of this is you can just allocate compute a lot more efficiently because you can defer stuff that the model doesn't have comparative advantage to doing to a tool that is like really well suited doing that thing. One of the ways I've been using some form of O3 a lot is deep research, right? I think that's basically a research analyst AI that you all have built that basically will go out, will look up things on the web, will synthesize information, will chart things for you. It's pretty amazing in terms of this capability set.

11:23Did you have to do anything special in terms of you know, any form of specific reinforcement learning specifically for it to be better at that or other things that you built against it? Or how did you think about the data training for it, the data that was used for training it? Like, I'm just sort of curious, like, how that product, if it all is a branch off of this and how you thought about building that specifically as part of this broader effort? I think when we think about, like, tool use, I think browsing is one of the most like natural places where, you know, you think of as a starting point of like, okay, like, and it's not always easy.

12:02I mean, like the, you know, initial kind of browsing that we included in GPT-4 a few years back, like, it was hard to make it, you know, work in a way that felt like reliable and like useful. but you know in the sort of you know modern these days last year you know two years ago is ancient history I think it feels like a natural place to start because it's like so widely applicable to so many types of queries like anything that is you know requires up-to-date information like it should help to browse for and so in terms of like a test bed for hey like does you know the way we're doing RL, like, does it really work?

12:45You know, can we really get the model to learn, like, longer time horizon, kind of meaningful extended behaviors, like it feels like kind of a natural place to start in some ways, and that it, you know, also is fairly likely to be like useful in a relatively short amount of time. So it's like, yeah, let's, let's try that. I mean, you know, in RL, like, at the end of the day, you're defining an objective. And if you have an idea for like who is going to find this most useful like you know you you might like want to tailor your the objective you know to who you expect to be using the thing what you expect they're going to want you know what is their tolerance for do they want to sit through a 30 minute rollout of deep research you know do they when they ask for a report you know do they want a page or five pages or a gazillion pages.

13:39So yeah, I mean, you're definitely, you know, you want to tailor things to like who you think is going to be using it. I feel like there's a lot of almost like white collar behavior work that you or knowledge work that you all are really capturing through this sort of tooling going forward. And you mentioned software engineering is one potential area. Deep research and sort of analytical jobs is another where there's all sorts of really interesting work to be done that's super helpful in terms of augmenting what people are doing. Are there two or three other areas that you think are the most near-term interesting applications for this, whether OpenAI is doing it or others should do it aside.

14:11I'm just sort of curious how you think about the big application areas for this sort of technology. I guess my, you know, very biased one that I'm excited about is coding and also research in general, being able to, like, improve upon the velocity that we can do research at OpenAI and others can do research when they're using our tools. I think our models are getting a lot better very quickly at being actually useful. And it seems like they're kind of reaching some kind of inflection point where they are useful enough to want to reach out to and use like multiple times a day, for me at least, which wasn't the case.

14:49They're always like a little bit, you know, behind what I wanted them to be, especially when it comes to like navigating and using our internal code base, which is not simple. And it's amazing to see like more recent our models actually really spending a lot of time trying to understand the questions that we ask them and coming back with things that saved me like many hours of my own time. People say that's the fastest potential bootstrap, right, in terms of each model subsequently helping to make the next model better, faster, cheaper, etc. And so people often argue that that's almost like an inflection point on the exponent towards superintelligence is basically this ability to use AI to build the next version of AI.

15:33Yeah. And there's so many like different components of research too. There's, it's not just, you know, sitting off in the ivory tower thinking about things, but there's, there's like hardware, there's, you know, various components of training and evaluation and stuff like this. And each of these can be turned to some kind of task that can be optimized and iterated over. So there's plenty of, you know, room to squeeze out improvements. We talked about browsing the web, writing code, arguably the greatest tool of all, right? Especially if you're trying to figure out how to spend your compute, right?

16:03More efficient code. Generating images, writing text. There are certainly like trajectories of action I think are not in there yet, right? Like reliably using a sequence of business software. I'm really excited about the computer use stuff. It kind of drives me crazy in some sense that our models are not already just like on my computer all day watching what I'm doing. And well, I know that could be creepy for some people. And like, I think you should be able to like opt out of that or have that opted out by default. I hate typing also. I wish that I could just kind of like be working on something on my computer.

16:35I hit some issue and I'm just like, you know, what am I supposed to do with this? And I can just kind of ask. I think there's tons of space for being able to improve on how we interact with the models. And this goes back to them being able to use tools in a more intuitive way. I guess using tools closer to how we use them. It's also surprising to me how intuitively our models do use the tools we give them access to. It's like weirdly human-like, but I guess that's not too surprising given the data they've seen before. I think a lot of things are weirdly human-like. Like my intuition for like, well, why is tool use so impactful to test time scale?

17:07Like why is the combination so much better? Take any role. You can make a decision when you are trying to make progress against a task as to like, do I get external validation or do I sit and think really hard, right? And usually you want to do like one or is more efficient than the other. And it's not always just sit in a vacuum and think really hard with what you know. Yeah, absolutely. You can seek out sort of new inputs. It's like, it doesn't have to be this closed system anymore. And I do feel like the closed system-ness of the models is still sort of a limitation in some ways. Like, you're not necessarily, like, turning this.

17:40I mean, like, I think it'd be great if the model could control my computer for sure. But in some sense, there's a reason we don't go hog wild and say, like, oh, yes, here's, like, the keys to the kingdom. Like, have at it. there are still you know asymmetric costs to like the time you can save and the types of errors you can make and so we're trying to like iteratively kind of you know deploy these things and like try them out and figure out like where are they reliable you know and where are they not because yeah like if you did just let the model control your computer it could do some cool stuff like I have no doubt, but do I trust it to respond to all of the random emails that Brandon sends me?

18:25Actually, maybe for that task, it doesn't require that much intelligence. That's true. Do I trust it to do everything I'm doing? Some things, and I'm sure that set of things will be bigger tomorrow than it was yesterday. But yeah, I think part of this is we limit the affordances and keep it a little bit in the sandbox. just out of caution so that, you know, you don't send some crazy email to your boss or, you know, delete all your texts or delete your hard drive or something. Is there some sort of like organizing mental model for like the tasks that one can do with, you know, increasing intelligence, test time scaling and improved tool use, right?

19:09Because I look at this and I'm like, okay, well, you have complexity of tasks and you have time scale. Then you have like the ability to come up with these RL rewards and environments, right? Then you have like usefulness. Maybe you have some – of course you have some intuition about like diversity and generalization across the different things you can be doing. But it seems like a very large space and scaling RL, like new gen RL is not – it's just not obvious. Like how – to me it's not obvious how you do it or how you choose the path. Is there some sort of organizing framework that, you know, you guys have that you can share?

19:46I mean, I don't know if there's like one organizing framework. I think there are a few like factors, at least that I think about in like the very, very grand scheme of things is like how much like in order to solve this task, like how much uncertainty with the environment do I have to like wrestle with? Like, for some things where it's like this is a purely fat like who was the first president of the United States? Like there's zero like environment I need to interact with to like reach the answer to this question correctly. I just need to remember the answer and say the answer. You know, if I want you to like write some code, you know, that like solve some problem.

20:23Well, now I have to deal with a little bit of like not purely internal model stuff, but also like, okay, I need to execute the code. And like that code execution environment is maybe more complicated than my model can memorize internally. so I have to do like a little bit of like writing code and then executing it and making sure it does what I thought it did and then testing it and then giving it to the user and things get like the amount of that sort of stuff outside the model that you have to like you know you can't just recall the answer and give it to the user you have to like test something and you know run an experiment in the world and then wait for the results of that experiment like the more you have to do that the more uncertain the results of those experiments like in some sense that's like one of the core like attributes of like what makes the tasks hard um and i think another is like how you know simulatable they are um like stuff that is really bottlenecked by like time like the physical world um is also you know just just harder than stuff that we can simulate really well you know it's not a it's not a coincidence that you know so many people are interested in coding and you know coding agents and things um and that like you know robotics is hard and And, you know, it's slower.

21:37And, you know, I used to work on robotics and, like, it's frustrating in a lot of ways. I think both this, like, how much of the external environment do you have to deal with? And then, like, how much do you have to wrestle with the unavoidable slowness of the real world are two, like, dimensions that I sort of think about. It's super interesting because if you look at historically some of these models, one of the things that I think has continued to be really impressive is the degree to which they're generalizable. And so I think when GitHub Copilot launched, it was on Codex, which was like a specialized code model.

22:06And then eventually that just got subsumed into these more general purpose models in terms of what a lot of people are actually using for coding related applications. How do you think about that in the context of things like robotics? So, you know, there's like probably a dozen different robotics foundation model companies now. Do you think that eventually just merges into the work you're doing in terms of there's just these big general purpose models that can do all sorts of things? Or do you think there's a lot of room for these standalone other types of models over time? I will say the one thing that's always striked me as kind of funny about us doing RL is that we don't yet do it on the most like canonical RL task of robotics.

22:43And I personally don't see any reason why we couldn't have these be the same model. I think there are certain challenges with like, I don't know, do you want your RL model to be able to like generate an hour long movie for you natively as opposed to like a tool call? Like that's where it's probably tricky to have you have more conflict between having like everything in the same set of weights. But certainly like the things you see 03 already doing in terms of like, you know, exploring a picture and things like that are are kind of like early signs of something like an agent exploring like an external environment.

23:19So I don't think it sounds too far fetched to me. Yeah, I mean, I think the thing I came up earlier of the also the like, intelligence per cost thing, you know, the, the real world is like an interesting litmus test, because at the end of the day, like, there is a, you know, frame rate in the real world you need to live on. And it doesn't matter if you get the right answer after you think for two minutes, like, you know, the ball is coming at you now, and you have to catch it. Gravity is not going to wait for you. So that's an extra constraint that we get to at least softly ignore when we're talking about these purely disembodied things.

23:56That's kind of interesting, though, because really small brains are very good at that. So you look at a frog, you start looking at different organisms and you look at sort of relative compute. And very simple systems are very good at that ants. You know, like, so I think that's kind of a fascinating question in terms of what's the baseline amount of capability that's actually needed for some of these real world tasks that are reasonably responsive in nature. It's really tricky with with vision, too. We have so our models have some, I think, maybe famous edge cases of where they don't do the right thing.

24:33I think Eric probably knows where I'm going with this. Like, I don't know if you ever asked, like, our models tell you what time it is on a clock. they really like the time 1010. So yeah. It's my favorite time too. So that's usually what I tell people. It's like over 90 % or something like that of all clocks on the internet are 1010. And it's because it looks like, I guess like a happy face and it looks like nice. But anyways, what I'm getting at is like our visual system was developed by interacting with the external world and having to be good at navigating things, avoiding predators. And our models have learned vision in a very different type of way.

25:09And I think we'll see a lot of really interesting things if we can get them to be kind of closing the loop by reducing their uncertainty by taking actions in the real world, just as opposed to thinking about stuff. Hey, Eric, you brought up the idea of what in the environment can be simulated as an input as to how difficult will it be to improve on this. As you get to long-running tasks, like let's just take software engineering. Like there is a lot of interaction that is not just me committing code continually. It's like I'm going to talk to other people about the project, in which case you then need to deal with the problem of like can you reasonably simulate how other people are going to interact with you on the project in an environment.

25:56That seems really tricky, right? I'm not saying that, you know, O3 or whatever set of foundation models now doesn't have the intelligence to respond reasonably. But like, how do you think about that simulation being true to life, true to the real world as you involve human beings in an environment in theory? My spicy, I guess, take on that is like, I don't know if it's spicy, but O3 in some sense is already kind of simulating what it'd be like for a single person to do something with like a browser or something like that. And I don't know, train two of them together so that you have two people interacting with each other.