OpenAI and Codex with Thibault Sottiaux and Ed Bayes

24:41That's great. When I'm often using, for example, cursor, I will also use GPT-5.1 or whatever. And actually, in that context, I found just like the bare model, 5.1 often works better for me than the codex model. So now I'm like, what is it you're doing that's connecting it to that harness? It's actually really thinking about that co-evolution of the harness and the model and thinking about it as one entity and one agent. Fundamentally, what we're building as codex, the codex team is like, it's an agent. And then, you know, we figure out where to put it to work. And the agent isn't just a model itself.

25:11The agent is the model together with the set of tools and the way that it's going to handle its context and be able to think and reason through which actions it should take. And it's pretty clear that if you co-evolve and co-train these two things, you can achieve better results, which is what we're achieving. I think one cool thing as well is Codex, the CLI products, is completely open source. So to your question of what's going on under the hood, you know, the great thing, and we have a really vibrant open source community who contribute a lot of great ideas and issues, and you can just go, you can look at the system prompt.

25:46It was also a funny thing. When we released the new model, there was this tweet which was like, system prompt leaked. It's like, yeah, it's in the open source repo. It's just like nothing to hide. So yeah, so I think like in terms of your new capabilities or tools, you can go and have a look, which I think is super exciting. Yeah, there's a lot of effort and research that goes into like, what are the optimal tools in order to get the results that you want? And oftentimes we're actually quite proud of like how simple the harness is and how simple the set of tools is. This is something that we strive for is that simplicity, being able to have the harness scale with the continued levels of capabilities jump that we expect to see over the coming months and years.

26:29It's something that if you don't optimize for, eventually sort of like comes back to you because you have hyper optimized something in the short term that doesn't scale with continued like capabilities improvements and then by being so close to the codex we run it as like one unit we have product we have engineering we have research we all sit together ideating a lot and using some techniques from research to put in the harness and like using parts of the harness and using that in training and so there's just like this the zeitgeist there and like the sharing of ideas and always zooming in on what will make the agent perform better as one unit.

27:08It's not about optimizing the model in isolation. It's not about optimizing the harnesses. Isolation is finding that combination that works the best together. And that's what the codex series of model offers as well is like that guarantee that we have considered how well it actually works in codex. And that's like the best that we can do. If it's not too much secret sauce, how do you consider that? Is that related to like the reinforcement training that you're doing? Is it a different, you know, initial data set? Like what is actually causing it to behave differently there? It's really about thinking about the model as not just needing to be intelligent, but needing to be an efficient agent.

27:46And if you think about what an agent is, it's going to be a model that gathers its own context and an accident in its environment in order to achieve a goal. And so if you set yourself to train a model to be extremely good at that and be an extremely good coding agent, you're doing different trade-offs. You find that you are able to take different trade-offs at the research level, be it at like the post-training or the RL or the specifics of the training, which we're not going to go into, but the trade-offs are there. And so you're able to achieve efficiency gains and move up in the performance curve.

28:19Now, I mentioned some of the models I see. There's one other model that I see in this list, which is GPT 5.2, which I think was not there when I looked at this a week ago. So what's that about? We just released it yesterday. It's been very successful, more so than maybe we anticipated. We were actually like some of the team was up like all night, like shuffling computer on and making sure that, you know, it kept working and we were achieving the latency target that we have. For agents, the latency of the model and the reasoning and exactly where the compute is is more important than ever before.

28:52this is because you have that latency element between that GPU that we run somewhere and then your computer where the tool calls are run. So you have always this back and forth. And then obviously if the model is able to perform and we're able to sample more tokens per second, that's going to translate into a shorter amount of time to get that result. 5.2 is a particularly exciting model launch, I would say. It's a significantly higher jump than what one might expect compared to like 5.1. GDP eval captures this fairly well. I think a lot of the benchmarks these days are saturated, but a good way to think about it is economical value that you're able to create in the world.

29:34And GDP eval, I think we see more than a 20 % jump there. So definitely recommend trying it out in Codex. It's quite exciting. Depending on when this podcast goes out, you might have something even more exciting to try out, but we'll see about that. I think that is interesting. And thinking about that, like, interaction between local and data center. So, like, how are you all, I mean, some of that I'm sure is proprietary, but like, how are you thinking about locality in this? Are you pushing compute? What does that look like for someone at the scale you're at? The closer the compute is, you know, to your laptop, if that's where you run like Codex CLI, like the better, right?

30:13Because you reduce that round trip. Another way of doing it is to bring the compute environment closer to the compute, right? So to bring, for example, virtual machines and have those effectively be as close to the GPUs as possible. That's the approach that we take with Codex Web. But then if you're running locally within your VS Code extension and the agent runner is effectively running on your machine, then you do want that GPU to be as close to you as possible. So there's an element of like, you know, where in the world is that running for you? And, you know, sometimes you might be better off if you're like, you know, somewhere in the middle of nowhere on an island and we're not like, you know, running a data center there is there's like, you're going to feel that actual latency.

30:55Let's actually dive in a little bit to the guts of the agent, because I think, you know, one of the things that most of the software development world right now is trying to figure out how to build effective agents. And I think coding agents are really at the frontier. They're pushing the edge of what that looks like. So can we kind of break down just like first, the very high level pieces that you think of that go into this agent, the software layer, not the model? Yeah, what are the higher level pieces of, we touched a lot on it already. So you have the model and the inference, that's going to be the intelligence that's driving the rest of the software stack.

31:32And so you have this interesting combination of a piece that is non-deterministic and a piece that is deterministic. At least for now, a lot of the harness is considered to be deterministic. And it's quite simple. If you look at it under the hood, like it's all open source for Codex. It's like there isn't that much magic. It's a for loop and then a bunch of tool calls and then tools that have been designed to work well, like for coding. And it's a pattern that you can apply essentially to any other discipline and any other agent. It's that control going from the model back to its environment, executing an action, and then taking what's been observed in the environment and then pushing that back to the model in order to decide the next action.

32:19and then doing that over and over and over again, you know, maybe hundreds of times until the point where the model believes that the desired outcome has been achieved and decides to stop. So at the very beginning, you have a prompt, you know, or like an intent from a user, then you give control to the model, decides on like the next tool call, goes on and on and on and on, and at some point has, you know, achieved or is unable to achieve and decides to yield that control. And that's when the agent has finished its job. But delightfully simple. It's just a couple of tools, a for loop, and then a model that's given control over that.

32:56But the really exciting thing is not that. I think it's really the products around it that allow you to have control, steer, and supervise those agents. and then as well as thinking about the agent being its little system that will continue to evolve and being increasingly more complicated and being able to perform increasingly complex works. It's not maybe a single agent that's going to be at work. Maybe it's going to be like multiple agents that are going to be at work. But I think like a really exciting thing is like, how do you interface with this ever more complex system that is doing work on your behalf?

33:34Yeah, totally. And if you think, you know, there are some parallels, I think if you look at ChatGPT and when it was released, right, it's like very, very simple. You go online, there's a text input, you type some intent, some message, and the model responds back. But as Teebo says, in this world where we have an agent loop and an agent is carrying out work for you, maybe it's delegating to other agents, it's collaborating with other agents, it's speaking to external agents even, I think the user experience changes. It goes from this back and forth to a little bit more like how we interact with other humans in the world today.

34:08Right. Like if I ask Thibaut to, you know, get me a glass of water, it's going to take him a little bit of time. He's got to go outside. He's got to do a bunch of stuff. Right. If I'm collaborating with a colleague, if I'm collaborating with a colleague, I might ask them to do some significant tasks. So, you know, build some new infrastructure project. It'll take time. they'll have to go out, they'll have to coordinate. So I think we're moving from to kind of longer and longer tasks with more and more complexity. And, you know, models that are increasing in capabilities. And I think the interesting question from a product perspective is then how do you design those interactions in a way that is simple, maintains simplicity, also fits into, you know, everyday workflows nowadays, and also like exposes these incredible capabilities of the model as well in a very simple way.

34:54So one of the things that's interesting to explore in this domain of like user experience of agents, especially as they're going on, is like when I wrote code in the olden days, right? It was doing multiple things for me. It was creating this runnable artifact that somebody could interact with. Okay, that's great. That's useful. It was updating my mental model of the system that I have that I'm working with. And it was also doing some amount of like problem solving and updating of my mental model, probably of the user's problem, or at least how to map that user problem into my system. And so there's these like cascade of mental models that I'm updating, as well as this final artifact that's being generated.

35:31So now as we get into this world where we're delegating more and more of the work of generating the artifact, there is still this like very real need for us to update our mental models across the board. So how do you think about or see that working in this agentic world? How does the product facilitate it? What does that look like? Yeah, I think that's a super important point. and it ties a lot into, in my mind, of what we've seen people use coding agents for primarily over the last, say, like six to three months, which has been a lot to solve and write code for them on their behalf. I think there's a much deeper role that agents have to play in the future, which is to understand, hey, what do you care about?

36:12How can it help you understand the state of the world efficiently around you? Like maybe it should send you something every day, but here's how the code base changed. Here's what users are thinking about the product. Here's how to really explore this topic a little bit more. And so you go much further than just the code generation. You're helping with planning. You're helping with ideating. You're helping with understanding user feedback. You're bringing a lot more context into play than just code itself. And in a way, if you were just to focus on code generation, you would miss out a lot on the opportunity here.

36:50We're thinking about this broader set of things that we can help people with. And I think it's going to be ever more important. Maybe code generation actually will be a very small part of what agents end up doing for you. And we're definitely thinking about this at the product. Yeah, yeah. I mean, it's interesting. A very small maybe example on the team is, I think, when a new starter comes on board, it often takes a long time to get used to a code base. You have to really get to understand it. But yeah, as well as writing code, I've seen new engineers on the team just speaking to Codex and really deeply understanding the code base going back and forth.

37:24And, you know, that means that they don't need to tap on their colleague's shoulder as much anymore. If they do, it's for like some really, you know, high value touch point. But yeah, as you say, I, you know, I've seen people use it for all sorts of things, writing notes, code understanding. So it's, yeah, really beyond just code generation. Yeah. And there's this awesome thing about giving Codex to someone who just started on the team and be like, hey, explore the code base with the help of Codex. And then we barely write documentation like how things work because that's just in the code itself.

37:53What we tend to document more is why things exist. And so I think there's going to be an evolution there as well of like, how do we maintain the knowledge base, how much of it is redundant. definitely when you have sort of intelligence and a little buddy that you know you can just like send off in order to explain something for you so you tend to find that you know maybe it also shifts like what you want to write down yeah absolutely well and there's kind of an interesting thing there like one of the techniques that we found works really well for us with agents is actually documentation that is like maybe transient in some form so it's like here's this problem that I'm solving, gather all the relevant pieces and documentation and link to the relevant files so that I have like one short piece of context.

38:37Okay, now use that to get me to a solution on this particular thing. And so it's like much more temporary documentation than permanent documentation, but giving the agent this like map that it can work with. Yeah. Is this more like a design doc? Where depends? How does it differ? So we can dive into this in a couple different ways. So So I'll use a very quick example of one of my common practices. And I've done this with Codex or other agents. So I have a problem to solve. And I know roughly the area of the code base involved, but I don't want to, maybe I don't know it that well. So I'll say to Codex, for example, hey, I'm going to be wanting to muck around with this subsystem.

39:14Please do an analysis of how that system works today. Write me a document that includes file links and symbols and all these other things. And conceptually, to me, what I'm doing is I'm creating a map of the territory. It's like essentially a context condensation, right? It doesn't need to read all those files all the time, but it needs to know where roughly everything is. So when it needs something, it can pull it. Okay, now I have that subsystem. And I say, okay, I'm looking for a solution that looks something like this. Can you map out like three different variations of that? Have an argument about which ones are better, whatever.

39:41Make some characters, do this. Kind of map out the solution space. Now I have these two very rich documents. And I can say, okay, based on these, look at this, look at this. Pick which is the best solution. Write me an implementation plan. Okay, pretty good. break it down a set of task lists, go. So I'm in some ways manually managing the process of this, but kind of guiding it towards, here's all the relevant parts of the code base with me in the loop often to be like, no, you missed something over here. You got to go look at that again or something along those lines. The workflow that you're describing is extremely powerful.

40:15And it's all based on files and you deciding on like, hey, this workflow is actually very useful for myself. And you discovered it by yourself, maybe talking to other people, and they're just like sort of sharing element right now, like, you know, recipes of like how to work with agents. It's not necessarily that the product is like prescriptive about it. It's delightfully like open-ended actually right now where you can ask Codex to do anything for you. And there's this creative aspect of like, you know, what do you actually ask it? Like, you know, how can he help you? And, you know, maybe it's through this like, you know, complex workflow of like planning and then ideating on like, you know, different options and then going and like performing some implementation of it.

40:55we like that a lot, that flexibility. And we try to also be very mindful when we introduce more opinionated frameworks into the product that could also restrict that flexibility. That's definitely not something that we want. Another interesting angle as well is just seeing how some of the maybe more non-technical people across OpenAI use it or in different disciplines from just traditional software engineering. So there's one person that I know who basically uses it for everything. He uses it to write documents. On the design team, I know a lot of the designers and the product managers, they might do some coding, but they'll do it for lots of other things.

41:32Ideation, as you say, planning. Some very cool things as well on the data science go to market side, a lot of just data analysis, crunching through numbers, looking through a CSV. So yeah, I think from a product perspective, we've been deliberately opinionated about keeping things simple, just like ChatGPT. It's this general purpose interface that you can go to and you can ask it to do anything with chat gpt that might be you know generating images you know answering a question searching the internet and i think for us you know the amazing thing about a coding agent is it's like extremely general purpose so you really want to keep it as simple as possible and then let the user you know that their creativity go run well so on that note and looking forward a little bit one question i'd love to ask is are there any plans for enabling like an inside codex sdk or like hooks or some other way to generate because like for example mentioned I have this workflow which figuring out like oh I want to steer it in this way to fit in my workflow it would be great if for example every time there was a context pull or something like that it reinforced this or other different ways to kind of nudge and control tightly control the context that's going on to fit esoteric workflows that you don't want to have in the core agent yeah when you're getting at it's like really what we're seeing with like the power users even including within OpenAI.

42:50Some of our most prolific users maintain their own fork of Codex. That's one of the awesome parts of it just being code as open source as well. If you want to change it, you can just forward to code. If you happen to be advanced, that shouldn't be too daunting either. Codex can help you change it in a way that's productive to you as well. It is written in Rust. Sometimes we get some comments on that, but we want it to be very robust and like perform it as well. Like it's quite delightful when you just have, you type codex and it opens instantly. That's what we get from like putting like a lot of effort into that.

43:26Hooks are like something that we're debating, like, you know, we'll get there eventually. What we're super excited about right now is like building the right set of like primitives for the agent to be able to perform like increasingly complex work. And so you can think like, you know, what will happen if you're able to run an agent for like, you know, an entire day or like maybe an entire week. and, you know, steer it as it goes. You know, is that like a different thing? Does that require like different, you know, product thinking? And then we touched upon, you know, multi-agent as well. And this is something that we think is extremely exciting and is going to emerge like, you know, in 2026 for sure as something that is not at a prototype stage like you're seeing across the industry right now where like, you know, maybe, you know, folks are like excited about their little sub-agent, but it's going to be like these really robust networks of agents collaborating together in order to achieve like something for you.

44:19That's the kind of stuff that we're really excited about right now. Hooks, maybe at some point. Yeah, nothing massive to add. And just to say, you know, we have a Codex SDK. So, you know, it's possible now we have a documentation page up on it and you can start to play with it. But yeah, to Timo's point, I think there's this tension between, you know, like catering for, you know, very specific workflows and then thinking about what these primitives are in these building blocks so that you can build on top of that and build some, you know, pretty incredible experiences. What would you say the missing primitives right now are?

44:47We have a long list of GitHub issues. Some of them are top voted. Those are actually the ones that we tend to prioritize. So one of the things really that's been requested, like subagents, so we're actively working on how to think about multi-agent networks. And then a lot of it is still product overhang, I think, where it's not about the agent itself, But like, how can we make the product more delightful and more interesting and better suited for managing, steering and supervising agents at scale? That's, you know, what's keeping us very busy right now. Yeah, again, without, you know, going too deep into the roadmap, you know, I think one interesting provocation to think about is as the complexity of using agents or in a multi-agent world, you know, becomes incredibly complex.

45:36how do you stay on top of that? How do you keep track of what different agents are doing, what actions they're taking, and whether you need to give any permissions and the like along the way, if there's any artifacts that they've created, whether that's code or elsewhere. Keeping a track of that and staying on top of it, I think for me as a designer on the team, a really interesting interaction design problem. Say we're moving from this world where you watch a rollout of a minute to, as I say, this 10-hour job. How do you stay on top of that? How do you keep it delightful? How do you meet users where they are?

46:08And so you're not kind of context switching all the time between all of these different things. That's, yeah, as well as the kind of core primitives from an engineering perspective, they're the kind of problems that we can keep on the product side. One other thing you talked about there was all of the non-technical use cases. And I think one of the most amazing things I've seen with, as coding agents and LMs, just as coding assistants have grown, is the extent to which now subject matter experts are able to build at least their own prototypes and often their own applications to help them in their workflows.

46:39Are there any aspects from either a product or technology standpoint that you're thinking about, particularly for those non-technical users looking forward? Yeah, we're thinking about it, especially since it's been sort of like a natural thing that's been happening where we see like increasingly amounts of like non-technical people like inside OpenAI, outside of OpenAI, use codex in their terminal and get it to do cool things for them it definitely got us thinking about how we can do this better and also there's this pull for generality as ultimately the very best coding agent is a general agent that's able to reason across much more than just code the codex agent and models are extremely good at instruction following people find us very useful for data analysis, for editing spreadsheets for doing like market like research and these things and it's definitely something that we want to lean into and you know cater to at some point at the same time right now we're also laser focused on you know making codex the very best tool for you know professional software engineering and you know there's this tension of uh you know hey it's like you know if we want to be really good at this like you know should we also like you know think like a lot about like these other things but ultimately we see it combined very well.

47:58And so it also got us thinking it's very satisfying to see codex used for more and more things, in addition to just being an extremely good tool for coding. Yeah, and from a product perspective, there are some things, we're building an agent, for example, a coding agent. So on the web, you have to set up an environment. There are some things that you just can't get around, which are pretty technical. And if you're a software developer, you need to kind of go through these processes. But I think from a core product experience perspective, There's also more that we can do, and this is what I'm focused on, which is just what's that first experience?

48:29Is it delightful? Is it simple? Can you as a non-programmer kind of rock up and just get involved? And how can that be an on-ramp for you to learn more about coding and to get deeper into it yourself? So this is something in the design team, for example, we had this off-site and we had a few other people on the team who code kind of going around and basically onboarding everyone into Codex, into the CLI product, into the extension, depending where they work. And to be honest, for some of the non-coders, It's a little intimidating to get into the terminal. You have to like, they were installing NPM and these things that might be a little new for them.

49:00But once they got on board, and I think once they kind of saw what the work that the model was doing and started to learn a bit about it, some of the people who just tipped their toes now, I see them more and more coding. So I think it's also like a really cool opportunity to kind of expand the aperture of, what is a software developer and create a really great on-ramp for people to learn more and go and dig deep in themselves. Awesome. Awesome. Well, we are getting close to the end of our time. Is there anything we have not talked about yet today that you think would be important to leave folks with?

49:30One thing we haven't talked about is like the mindset that's important to continue to adopt. And I feel like it's an amazing time to have problems as like solving them, you know, has never been easier. And then there's also the aspect of like, hey, this really helps with answering questions. And there's like this curiosity that gets like super rewarded right now. and definitely like being able to like try and get interested in like changing your approach of like you know how you're going about your day and you know thinking about solving the problems that you have like maybe you had like useful ways of doing that you know that were effective like two years ago and you've stuck to them I definitely feel like it's the right time to like question everything and try new things personally I find this super exciting and you know having like always like many, many ideas and unsolved problems.

50:19It's like, you know, finding that the amount of problems that are unsolved, like, you know, sort of like reduces with time. It's just like, I hope like, you know, someday like, you know, agents will be able to also creatively come up with like super interesting problems that I should be thinking about. It's like, we're not there yet. But, you know, what a time, you know, to just try these things and, you know, get like a ton of like new things done. Yeah, plus one. I think like, you know, also what a time to be a creative as a designer in the team. And, you know, when I'm speaking to young designers or I occasionally teach or kind of mentor young folks, the main thing I say is just kind of just like get involved and give things a try.

50:52Because it's never been there's never been a time where kind of curiosity has been better rewarded by really just getting your hands dirty, pushing yourself out of your comfort zone. And very quickly realizing that, you know, you're able to achieve way more than you might have thought before. Just by, you know, even on a week by week basis, if you look over the past few weeks with all of these model releases, you know, it's just crazy the acceleration that's happening. So, yeah, just kind of, you know, stay curious and get involved. Yeah, it's not long ago where, six months ago, where, you know, you would show, you know, static figmas or slides, like, you know, and just be like, hey, you know, this is an idea of mine.

51:24And then now it's like fully functional, you know, little products. Then I'm like, whoa, you know, this is better than what we have shipped in production. You know, it's like we better get this out soon. And, you know, that step change and like what you're able to achieve like solo as a designer is like, I don't know if like even. referring to you as a designer, you know, does it justice anymore? There's this blurring of, like, you know, roles that's quite delightful. Yeah, there's never been a better time, I think, to be a software engineer or a designer.