Building Codex with Tibo Sottiaux

24:46Tibo Sottiaux:And on top of the open source, One thing that surprised me about codex, and I didn't even know about it until recently, it's not tied to the OpenAI models. You can use other models with codex. You know, like putting myself in a vendor's shoe, it might not be very obvious because, again, all the other vendors I look at when they do a CLI, it's kind of use it with our models. Again, what made you decide to be this permissive about using or allowing to use your harness with other models? It felt quite natural. If you are part of this community and building an excellent coding harness, why would you couple it to your model?

25:31That felt quite disappointing to make that decision. So it didn't feel right. and in general it's like I think you know it's like I kind of try to make decisions that I'm like yes you know it's just like I can just sort of like explain it you know it is correct it's the same reasoning with you know it is open source in the first place it would have been trivial for anyone to fork it and then add support for another thing but then you're just encouraging people to just like you know go and use that fork and then now suddenly you have overhead and the only reason you have a fork is because you know you wanted to change like 10 lines of code to add support for like another model provider.

26:09That feels very silly. So like, you know, why not just support it in the first place? The other thing is we benefit a lot from like being able to just give optionality. So, you know, it's like maybe today, you know, you love using OpenAI models and, you know, you're super productive with them. But like tomorrow, there's a new model that comes out. You want to try it at. Why force you to go and completely change your setup just to try a new model? And then we benefit from the feedback that we didn't get, which is, you know, maybe there's something that you liked about that model. Maybe it actually didn't work well.

26:42But it's sort of like being nice to our users and to the community. It's like, you know, feels like the right thing to do here. And then, you know, we also try like other models, right? So, you know, we try them in the same harness and, you know, it's just all good. And then this is also often like this optionality is very important to companies that we work with. this is something that you know we absolutely lean into this last point i think you know as

27:10Tibo Sottiaux:any serious company you want to have optionality and you want to use a tool that gives you that optionality but i kind of appreciate it because i feel to me like it's kind of honest like look like it forces the whole company to be to compete the best in everywhere in the model layer and the harness layer with open source with with chooseable models and it kind of like doesn't doesn't allow you to like kick back and say like all right we're done we can we can we can hang back for a little bit for now yeah i want us to win users by having you know the best models the most efficient models the best product and then you know if we do all of these things it's like we're gonna have a good time if we sort of like force you to use the product because you know this one thing it's just like then i don't think that will attract you know the the very best people to work on this product either and it's like you know it's like we're doing our best work here we care a lot about the experience it should feel delightful you know like it doesn't irrespective model that powers it, you should fill the light hole.

28:03Tibo Sottiaux:I love the idea of winning based on merit, not based on lock-in. And this is a perfect time to mention our season sponsor, Entire, who also play by the same rules. Like it or not, Git is becoming a bottleneck for modern agent-heavy software development. Devs are creating more code with agents, these agents are pushing more code, many devs are running more parallel agents, these are pushing even more code. GitHub is clearly struggling to keep up and has frequent outages. So what's the solution? Entire was founded by GitHub's last CEO, Thomas Dumke, and he rebuilt Git hosting for the agentic era from scratch.

28:35Tibo Sottiaux:Entire was built to be very fast and to have your repos regionally close to you to reduce latency, allowing for fleets of agents to push in parallel. Some numbers they published. Entire can handle 418 pushes per second. That's up to 89 times faster than every competitor on the market. When GitHub is down, you can still keep working and you don't even need to migrate early from GitHub. You just sign up to Entire and the platform mirrors your repo. And one more neat thing. Have you ever wondered what prompt resulted in this specific code being generated? I find that the prompt and conversation with the agent carries more information than the PR itself, at least for me.

29:11Tibo Sottiaux:Entire captures all the prompt history with your agent right in the repo, easy to check back, and has a pretty innovative UI to show all of this. If you're looking for Git hosting that works even when GitHub is down, head to entire.io slash pragmatic, install the CLI and mirror your repo with a click. I've already done it. Oh, and did I mention that it works with any agent and it's open source? I'd also like to mention our season sponsor, Antizizis. Thibault talked about how the experience of the software you use should feel delightful. Delightful includes no annoying bugs. But when you're using agents to write your code, how do you avoid shipping bugs?

29:45Tibo Sottiaux:Reviewing every line of code is becoming a challenge with the amount of code that agents generate, which is why Antisysys goes well beyond code review. Antisysys runs your whole system in a hostile simulation. This simulation includes both targeted testing and fast testing. By running this simulation it finds every bug before your users do. And because the simulation is fully deterministic it doesn't only find bugs, it gives you a perfect reproduction of every issue which makes it much easier to fix issues. The first thing I thought when I heard about Antisysys is that automated bug discovery and fully deterministic testing sounds like a science fiction, but it's actually hardcore engineering under the hood.

30:23Tibo Sottiaux:Jane Street, Fly.io and the Etsy the community ship agent written code with full confidence because they know it's been verified by Antistesis. To see more case studies and details head to antistesis.com slash pragmatic. And with this let's get back to Thibault and why competition between tools is great. Yeah and I think as an engineer like I always see that whenever there's competition as to someone who's using tools it's always amazing like I remember like when Microsoft had with JetBrains, with the IDE wars. And then there's the clouds battling with each other with all the features. And now, of course, we have the harnesses, we have the models.

30:58Tibo Sottiaux:And as a user, it's great because now we have more choice. They just develop faster. I guess our voice gets heard a bit better. So it's great to hear. Speaking of the harness, can you tell me how it works today in the sense of like, when I start a codex task, does it run always on my machine? Does it choose the cloud? Does it use a sandbox? And how do I control this or know this? Or how much should I know about this as an engineer? Yes. So by default, it runs in, it runs sandboxed. Everything that, if there is like a command that should run with additional permissions outside of the sandbox, it will ask you as a user for permission.

31:45but everything, every tool execution happens within the sandbox by default and it runs entirely on the machine, your local machine. And this has been the case for more than a year now but it is something that is evolving and shifting where you can select to run this in the cloud which then runs in a managed VM where it's the same VM that you get through ChatGPT work and you can sort of like inspect it, but like it runs in a CATA container. It's like a secure environment. And so if everything runs inside of that VM and it doesn't run on your machine and then the only thing that happens in your machine is like your input and then the streaming back of the output.

32:34And so that obviously then is like much nicer on your CPU and your machine and you can scale much, much, much more. And this is like just a step it's going to be much more seamless in the future to use cloud machines and then maybe have a combination of partial execution on your laptop, partial execution on cloud machines. And really the thing that we're thinking about that is very natural is as models just get better and more capable, they can leverage so much more compute and many more resources than are available on your local machine. And so it would be a constraint at some point to just limit execution on your local machine.

33:20Tibo Sottiaux:But one thing that is great about it running locally, and I think the reason I love it when it runs locally, of course, it's a pain because if I'm doing some work, it's like, you know, I have several agents, it's eating CPU. If I want to close my laptop, I cannot kind of leave it like half open, right? When I was in one of the offices of an AI company, I had it half open and they're like, are you running agents? I'm like, yeah, I have one running. I get it. But the reason I do it because I have my local tools. I have my local Postgres database. I have my this and that. How are you thinking about the cloud is amazing, but it doesn't have this setup or it's just a pain to set it up.

33:56Tibo Sottiaux:Are you thinking or are you experimenting with making these setups? And I'm kind of reminded of a topic that we talked about pre-A, which is cloud development environments in like 2022, 23. They're hot. And then we talked about AI more. But I think outside of large tech companies like cloud dev boxes really never took off because there's a very big upfront cost. And then you need to pay like a maintenance cost as well. And you just don't benefit from it as like a solo developer or like a small team with the level of capabilities that we have in agents now is like the setup almost is free. Right.

34:32So like this, the setup cost and this maintenance cost is like if your agent is capable of doing it, you know, it should just do it for you. So for example, if you're saying like, hey, I have my local SQLite or I have a local server and MCPs and whatnot, it's like, how hard is it to actually configure exactly the same setup and keep it in sync on a cloud dev box? Well, maybe it's not that hard if the model just does it for you. And so I think we're going to see a resurgence of fully cloud orchestrated machines, which then frees you from your laptop, right? It's like one thing that we've seen a ton of success with with ChatGPT work is like, it's just available on your mobile.

35:13I start my day just dictating a bunch of tasks into it next to the coffee and it just does it. It has access to my calendar. It has access to my email. It has access to Slack. And it's just so awesome to just be able to walk around and get stuff done without having to carry my laptop everywhere. And I think it's the same thing. We ship like Codex Remote where execution is like still happening on your laptop, but it would be wonderful if, you know, you didn't have to keep your laptop open.

35:43Tibo Sottiaux:Can you tell me a bit on how, in the past, how did you improve Codex? Because I remember when I first used Codex, this was one of the early versions, you know, like you could talk to it, it did stuff. But for example, I said like, all right, make this change. And it did that change. And I had unit tests and it didn't run it. And then later, a few months later, I don't know exactly when, it just started to run it automatically. Were these things, did you improve the script that runs the instructions? I'm not sure how exactly you call the bootstrapping script or whatever that is. Is it improving the model?

36:16Tibo Sottiaux:As a dev, how can I imagine you making each version better between the harness and then between the model? And what's the connection between the two? Yeah, this is a good question. So the harness, in a sense, is always a little bit ahead of the model. Oh, really? How so? What I mean by that is that you have the model, it's capable of certain things, but then you set it up with a couple of crutches so that it can actually do the thing to a level of reliability and in a way that is efficient and also with the behavior that you expect as a user. And so that's the role of the harness, right? It's like, you know, provide guardrails, like safety, make it more efficient, make it more like steerable, controllable.

37:07And then the harness usually is also responsible for, you know, what we call like the developer message, which is sort of like infected in the context at the start of each turn. And so that affects obviously like the purpose of that to affect like the behavior of the agent throughout the turn. a lot of what you have is like the result of the harness and the model. Initially, like maybe you're like, oh, it doesn't run tests. So, you know, you have to remind it to run tests. And then, you know, we train a battle model that is just, you know, capable of like better reflecting on what is it that you really want when you ask for something.

37:49And then, you know, you don't actually have to tell it anymore. So over time, what we see is like the system, the developer message shrinks, and then the hardness also shrinks.

38:00Tibo Sottiaux:Inside of the Codex team, do you have specific goals? Do you say like, all right, right now the Codex as a hardness and model combined is not very good at this or it's kind of doing silly mistakes right here? Or how can I imagine how, as the engineering team, how you're working on the next version of Codex? Because the thing that I don't really get as a dev is like, okay, there's a model, which to me is this magical thing, which will get better. Of course, I'm sure you have some feedback channels, but you also have the hardness, which is the tools that you're building. Like that's probably what the team is responsible for.

38:32Tibo Sottiaux:How do you set even your goals, right? Like in traditional software, you'd be like, we will build this feature and you build that feature because you know how to do it. But it feels a bit more fuzzy to me, this development process. Yeah, it is. And it's why we co-design most things. And it's a process where it's a collaboration between research and the engineering team, like primarily building the core agent hardness. it's always a question of like, okay, we see today that, you know, we are very good at this, but we're not very good at this. And, you know, we have a desire to do like another thing because it would be a very cool product feature.

39:07And then, you know, we always sort of like look at it. It's like, okay, this should this be like a harness change or should this be a model change? And if it's a model change, like how soon can we have it? Can we have it in a month? Can we have it in, you know, three months, six months? And we sort of like work through that. And then depending on, And, you know, how soon we can just fix it in the model at which level of training, then we might decide to not even do something in the harness at all and not and just wait for for for the model to to solve it. You know, it's agents all the way. Right.

39:35So we use agents to analyze like a lot of the feedback to like, you know, come up with themes, you know, to just help us have these conversations and decide on priorities. but we analyze it across all of coding. We analyze it across like all of like, you know, the other domains, like finance, comps, marketing, you know, all the things where our users are using these agents nowadays. And there's like, you know, subcategories within those. And then we roughly know like, you know, how well we perform. And then we're always pushing the frontier. And there's a thing that is interesting is like, as we make, you know, as our pre-training model gets better, as we make the overall model better, like the whole thing lifts up.

40:11But then there are sometimes things that we pay a little bit more attention to.

40:14Tibo Sottiaux:You mentioned, you analyze agents all the way. Can we talk about the software development lifecycle on Codex in the sense of whenever a new engineer joins a team, any team, it's like, okay, how are things done here? And back pre-AI, it would have been you joined a company like Uber or Google and they would tell you like, cool, the way it works is we have an idea or the PM has an idea. We make a plan, we get together, we do some estimations, we break up the work, we code the work, we do tests, we do code reviews, we release, we do feature flags, and then we're on call. That's how it used to be.

40:49Tibo Sottiaux:When someone joins the Codex team, they've clearly been contributing to the open source part. But what do you tell them? How do things get done here? If they're like a total newbie. I introduce them to great people. And then the thing that they hear the most about when they have a question is like, have you asked Codex? And Codex is just by default at OpenAI. is plugged into everything. So it has access to Slack, it has access to all the documents, access to all the code. And it still surprises new starters that you can basically ask it anything. And it will very often just come up with a really good response.

41:29And so the easiest way to understand the state of a project or who's working on something or why a decision was made is Codex knows about it all internally. um so you just you just use all of that we do a lot of work in you know for that reason we do a lot of work in public channels um we open up uh documents with like you know fairly broad permissions and so that you know everyone has access to this information as well and so that you know your agent can go through things and like you know reason through things and then you know we have a couple of other things that are just really uh very helpful for uh team productivity and team collaboration that we haven't released yet but are are going to come like some of it at death day.

42:10Although that just sort of like makes you very grounded and in tune with the rest of the team and allows you to like, you know, just very, very quickly like understand the state of things and produce things yourself. The general recommendation is just like, hey, care about the user, care about the coherence of the product, care about the models and where they're going. If you're doing something and, you know, you're building like this 10 ,000 lines of code crutch to work around the model flaws, you know, you're probably doing the wrong thing. So we have a set of principles. But it's just really sort of like a team culture and ethos at this point.

42:45And, you know, it's just very much like carries on, you know, when people join, it's just like through the rest of the team, just like, you know, sort of like teaching the roots.

42:52Tibo Sottiaux:And then when I have an idea, I think it's a good idea. I talk it through with Codex, maybe I talk with it with some of my colleagues, like, here's a cool new feature I'm going to build as my first contribution or first major contribution to Codex. How do I go about that? But obviously, I code it down with Codex. I obviously test it and make sure that it works. From there on, what's the process? Do you still have the concept of code review or AI code review, a verification of rolling out, a verifying of stage rolled out? You know, the thing is because Codex itself, it goes out to millions of people.

43:23Tibo Sottiaux:Like I just crossed a big 20 million active user mark. But if it's ChatGPT, then it also goes out to like even a lot bigger number of people. Yes. But it's surprisingly like a similar process, whether you ship on Codex or a chat to BT, even though a chat to BT goes out to like a billion active users and growing. You can ship a PR, you know, you can make a change and, you know, get it shipped like the next day or like even the same day. And it just goes out to a billion users and it's fine. We just really instill a sense of ownership and care. So you're like, people are very empowered to make changes, even large changes.

44:02The general thing that is being asked is like sort of like evidence that it's going to be well-received, evidence that is like a worthy addition, evidence that, you know, it's like it is worth maintaining over time, but also like the cost of maintenance is like just really, has, you know, gotten done significantly as well. So we think about these things slightly differently than, you know, say like two years ago or three years ago. The other thing as well is like, you know, we automate as much as possible. So like a lot of like the process of like code review and deploys and, you know, catching regressions.

44:32It's like, you know, all of that is like pretty much automated. And so, like, you know, you get to just focus on just really the idea and, you know, how it's going to help our users. And you care about, you know, the coherence of it all. And so like the overall power of the agent and making things better. And like, we don't, we have a long, long list of things that, you know, we sort of like aspire to do and haven't gotten to yet. And then there's like the sort of like the North Star direction, which is a delightful, simple to use personal AGI that knows everything about you, that it needs to know, has access to the right resources, can take sometimes risky actions on your behalf, but then you get the push notification and then you can verify that.

45:13And it's a thing that you deeply understand as a user, but also it knows about your schedule, it knows about your goals, it can be proactive, and it should be extremely natural. It should be something that you can control through natural language, voice, is, you know, like maybe it should understand, you know, your emotions. Like if it has like a camera feed, it should be the most natural thing on Earth. It's like it should not be like a thing with 10, you know, different buttons and configurations. It's like AGI should be simple to use.

45:39Tibo Sottiaux:You kind of mentioned just briefly the review, the code review, but I wanted to go back to it. You worked at Google on a product used by, you know, like hundreds of millions, which is Google Maps. And Google is very well known for their culture of very strict code reviews. They have, I think, two layers of code reviews. There's a language correctness review. And they've taken, I think they've really perfected it across the industry for a long time. And they do believe that it works and they use it. How do you think that part is changing specifically the human review? Because for a very long time, until maybe a year or two ago, I would have said your code review has all these benefits.

46:14Tibo Sottiaux:Knowledge sharing, the second pair of eyes, removing the bus factor because now someone else understands that when that person is out, they can jump in. Conversations are happening about architecture, not just the code. But now there's, you know, there's a lot more code. And what was the value of code review? What, in what cases, and so on your team, because you guys are so ahead of this, where do you see humans still being or developers being involved in the review stage of valuable? And where is it fine? Did you find it fine to hand it off to an agent? Yeah, the role of code review is changing.

46:54One of the early projects that I did on Codex was working with research on developing a code review model that was going to be to a level where it can spot mistakes in logic and reasoning to a degree where it would require humans potentially multiple hours to capture the same level of mistake. because it requires like really digging like you know three four levels deep into like the dependencies and you don't understand that maybe the documentation actually was wrong and like the implementation of like this third-party dependency is like different from what you expected and so therefore your invariants are not upheld um and these things it's just like you know unless you're an expert in that library you wouldn't know uh and therefore you have a bug and so we developed like these uh code review models and you know we we released them and now they're like the same level of like capability and like ability to spot these mistakes by doing like, you know, deep verification are like just part of the mainline models.

47:52Like when we benchmark them, it's like they're like superhuman in code review. And this is not just true for correctness. This is also true for security, for example, where you're capable of like reasoning across like, you know, very, very complex things. And then, you know, coming up with like, hey, you know, you have a critical security vulnerability here, which is now mandatory across like all of open AI pull requests. Like we block full requests from merging if, you know, we flag them with like a security issue. And this is like all automatic. And the role of code review now is like, I think it was always about correctness.

48:23It was always about, you know, ensuring that things work. But it was also sort of like a little ritual for information exchange and, you know, bringing people on the same page and like, you know, encouraging like a discussion, which ideally would have happened before. But sometimes it just only happens like around the code because once it merged, it just actually runs in production and doing stuff. and then you have to maintain it. So there's like this social aspect to it as well. And so I think all of it is changing, like the correctness, the cybersecurity, the security is like, I think that will be automated.

48:51Really what we see and I see is there's a sort of really discussion around the intent that takes place around the pull request. It's like, what are you even trying to do? And is that the right thing to attempt to do? I think you can have that discussion outside of the pull request. It doesn't have to be around code.

49:07Tibo Sottiaux:So maybe this helps crystallize where a discussion needs to happen versus where we did it because maybe we didn't have the type of tool that we have right now. Yeah, I think this is going to change. And it was like a forcing function because you have to have that discussion or it's good to have that discussion before you merge it and it becomes production code. But I think there are other ways to have these discussions and design things together and make sure that the intent is good. And then the code doesn't matter as much. And it's interesting because when I think back of all my code reviews, like, of course, I have like memories where like it was great.

49:43Tibo Sottiaux:We had a good discussion or I learned something really interesting. But a bunch of times, honestly, it was such a pain in the ass. Like I was trying to get my stuff. You're pinged. Hey, could you review my code? And like, no, right now I'm busy. No, I really need this to unblock me. And then you context switch. And then I feel it's always been like good and bad, right? So I feel whatever we do, there will be always upsides and downside. But now they're just moving. So I guess one upside is, as an engineer, you might have to not give your attention to just kind of basic stuff that doesn't need your input, per se.

50:17Yes, it saves time. And progressively, what we're going to see is also like you have an agreement on the box and the overall contract of what it's supposed to do. And then what is inside the box, as long as you have strict guarantees in terms of resource utilization, data access, security, these kinds of things. It's like what happens inside a box is, you know, it could be literally anything. You just like don't really need to care. And like really what you need to agree on is like, what does the box actually do and what are the invariants that must be satisfied? And I think that is then worth, you know, having like a really good conversation on, you know, maybe assisted by your favorite agent.

50:57But then once you have that and you have that understanding, it's just like changing anything within the box is like, you know, doesn't require for this discussion. And it's like, you know, just really preserves your attention.

51:06Tibo Sottiaux:The cost of maintenance has gone down. you know, maintenance is always such a hot topic. Whenever we build something inside of all these companies like Google, Uber, even startups, like building was the fun part, but then maintenance was the painful. And that's when we learned like, okay, it was not, we're building it, etc. Inside of Codex and OpenAI, what do you see maintenance becoming cheaper changing in terms of instead of what you're building, what the ambition is, the, I guess, custom tooling, those kind of things? Maintenance is really like sort of like a tax that you pay over time just to keep things running.

51:40And it's always been necessary. It will continue to be necessary. But where I think it changes is like a lot of it is just going to be automated. So, you know, it's like, OK, you have you have this third party dependencies. Like you need to upgrade the version. I'm just like, oh, yeah, you can fully automate this, you know, if you have good change log and, you know, and the code is well documented. and the model can just reason through it. It can just blast through your code base, do it in a couple of hours. And previously you would have punted on it because it's not the most fun thing to do. But it's actually really important for your business.

52:17So it's really important for your project, especially for security vulnerabilities. You want to stay up to date, right? You want to apply all these patches. I think that's just going to be fully automated. So a large part of maintenance, it just kind of comes for free, right? And then I think it's awesome to also think about before, like, you know, when you wanted to just completely react, you have to do like a new architecture because you're trying to make space for like a new, you know, different kind of trade-offs, or you have a new understanding of like the workload, or you're trying to fit a new feature.

52:50And like suddenly you realize like your current system is just very limiting and you need to completely re-architecture it. That was like a really, really costly endeavor, right? So, you know, sometimes like multiple years. and I think this is also like super super accelerated now so you like the cost of mistakes uh you know I would say like you know is going down but then at the same time the good old rules I would say of software engineer of like you know having good abstractions like really help like you know going back to this like having the box with invariants like you know if you sort of like draw the right shape you're going to be able to change things much more quickly within the box and like not affect the rest of the services or the rest of your infrastructure.

53:29And I think it's important. It's important to design for very quick iteration and change.

53:33Tibo Sottiaux:I remember when I talked with Peter Steinberger, that was before he joined OpenAI, but about OpenClaw and how he thinks about it. Like, you know, he told me that he doesn't read the code, but he kept thinking about like, I could see that he's holding the architecture in his head and he was telling me how he re-architects a lot and he thinks about how to make it modular, how to allow 100 contributors to each build their thing without stepping on each other's toes. So I'm hearing what you're saying that this care, this planning, this structuring has become maybe just a lot more important to, like, which was something back in the day, you know, it was like the architect or the staff engineer or experienced folks were doing this thing and other engineers around them were kind of building this, you know, smaller parts.

54:17Tibo Sottiaux:But it sounds like now all engineers need to be aware of when you're building your software, right? and plan for it. Yeah, and the GPT models are getting better and better at this as well. Like, you know, thinking about long-term maintenance and like good architecture and like this is like a natural sort of like next step, right? It's like not just about code quality in the sense of like, oh, is this code clean within this file? But like, you know, is the architecture actually correct to reduce maintenance burden over time and like, you know, make space for like future product or feature extensions or changes and just really this act of like, you know, engineering over time.

54:49That's kind of like something that models are starting to become capable of like thinking about very well i think it's just kind of fascinating to understand that the software that we're building is just going through the life cycle much much faster right you know before you had you know you were you were scaling it you were starting it you know maybe as like a small team of you know yourself maybe a couple of engineers and then you would add engineers like slowly and then you know maybe after a year you know it's like if it's very very successful you would have 50 engineers on it or like 100 engineers on it, you would have time to see it coming.

55:21You would have time to see, like, you know, the humans on board and, you know, you can think about the documentation and all that stuff. But now it's just sort of like that explosion of, like, you know, suddenly you have, like, 100 agents contributing to this thing. It's like, you know, that can happen, like, you know, in a weekend. And so, you know, you're just going through it at, you know, major, major speeds compared to before.

55:41Tibo Sottiaux:Okay, but how do you and the folks at OpenAI, like, deal with this? Does it not mess with your mind? Like, you know what I mean in the sense of, like, you've been in this business for quite some time now, like decades or well over. And there was a pace that we kind of got used to. And obviously, it's now a lot faster. But how do you get your head around the fact that A, it's faster, B, the stuff that you've been doing a year ago, right now you're not doing because now the model is good at it. And, you know, like, how do you kind of reconcile that? Because I'm sure there's stuff that you've been really good at related to software that now you can hand off to the agent.

56:17Tibo Sottiaux:Do you not get a little bit of sting? You know, we talked about it stinging for your features to be implemented open source, but it can also sting that I've been really good at, like, I don't know, refactoring or right now it might be architecture, but maybe the model will be really good at that. And now I'm like, oh, OK, damn, like I'm glad, but also like it would have been nice for me to do that. Yeah, I think there's like a craft aspect to it, which occasionally I still, you know, pull up an editor and like write some code. And it's just like it feels nice. And it's sort of like I have fond memories of like late nights sitting in Vim and, you know, just like cranking it out, you know, drinking Coke Zero.

57:00And yeah, just not having to think about anything else other than like the problem in front of me. but really I think it's um it's all about being in the flow and and solving problems and what I find is like you know folks here and also like everyone I talk to is just like adapting very quickly and I think if you if you have a mindset where it's all about code is a tool to solve problems and you can solve so many more problems it's like before you wanted to benchmark something and you weren't quite sure where you were going to net out, it's like, you can just do it. It's going to take you like no more than 30 seconds, you know, to launch something in the background and, you know, get proper numbers and be able to do like a better trade-off.

57:42It makes you, it should make you a better engineer if you just really care about, you know, the outcome and the system working well. And so what it allows us to do at OpenAI, it allows us to run, you know, our inference much more efficiently. It allows us to, you know, get like much more like effective compute and, you know, deploy that to the world. And so like everyone is just like very focused on that and solving important problems at the speed that was not possible before. And like, I haven't yet, you know, encountered someone who's like, oh, that's not good. That's not fun.

58:14Tibo Sottiaux:Do I understand correctly that it sounds like if you have ambitious problems, if you have way more problems than what you can solve today or tomorrow or the next week, sounds like this is not really a problem because when, you know, you get more efficient somewhere, you keep going, which is a lot of startups right like startups are always way more ambitious than that's right they're able to do i don't we're not we're not out of problems for sure right so and i don't think we will be for a while um we have a long long road ahead of us in terms of like mathematical breakthroughs scientific breakthroughs you know making the world a better place like just really building for humans and solving the most important problems that everyone is facing and just doing it in a deeply human way that's that's what we're here for also just going back to coding and like you know these late nights it's like i think there's like it's also like maybe like a glamorous version of it just like i also had very a lot of late nights where i was trying to refactor something um and you know it's just like it would be like three hours deep into the refactor and then realize like actually this is a dead end uh and i must restart from scratch and it was like very frustrating um and so it's like there was like there are like these very very fun times so there's also the time where it's like it doesn't compile and you're just like why is this not compiling yet like i'm sure you had the time where you you go you go later it's now super late you need to go to bed because you need to you need to get some sleep and then you can't really sleep and you have this thing where like you you have some some task that is halfway and it upsets you sometimes I remember dreaming about the code as well.

59:49Tibo Sottiaux:And I guess one thing I don't really have these days when I'm working on my software for my business is I don't really have something that is halfway because I can just do this and then I can leave it at a state where it's kind of like, you know, done, either finished or it's either working or I have proof that it's failed. But it's interesting because, you know, everything's sped up, right? Yeah, maybe. Like I do have, like what a lot of people do and I do myself is like, you know, I have like sometimes like bigger questions that I'm asking myself, like, and I, you know, from conversations I've had during the day or like I haven't yet, you know, just like had the time to just look into it.

1:00:25And so I, you know, I will send off codecs to just like look at it overnight. And then I'm very excited to then wake up and look at the results. And so, you know, it's always like an exciting morning.

1:00:36Tibo Sottiaux:Well, I feel there's an art to doing long running tasks. And of course, you can use the slash goal, which will go and run. And, you know, that's also something that was recently added, like, a few months ago, right? The slash goal command to codex. Yeah. And back to, you know, maybe, like, the harness is a crutch, right? It's a slash goal was, like, necessary to allow, like, you know, to keep the model, like, on track on, like, a singular goal for a very long period of time. And it's, like, it allows the model to literally run for days or weeks if it's, like, a really, really hard problem. but with the new generation of models like what we're seeing is like you know you don't need slash goal anymore you don't need a harness around it you can just tell the model like you know hey go and work for a week and you know it will actually do it.

1:01:20Tibo Sottiaux:Speaking of hard problems and the fact that you're not out of them one of the interesting things that you shipped from the outside I would say it was as an engineer it was moderately interesting is what you call the merge which is codecs appeared inside of chat GPT And the reason I say that as engineers is kind of moderately interesting because we've been using Codex. Like, yeah, it's there. You can now open it in the ChatGPT app. Great. Like, I just went there and I just immediately went to Codex because I don't really use ChatGPT in the app per se. But I talked with folks at OpenAI and people in your team.

1:01:55Tibo Sottiaux:And, you know, they were telling me, like, there was a lot of preparation going on, a lot of engineering challenges. Can you give a sense of how big this project was, what you needed to do? and why was it difficult to pull off and how did Codex and other tools help you get it done in ways that would have been hard before? Because since you've launched the merge, the numbers that you keep sharing of how many people use Codex, it's going up way faster than before. So I assume there's a big scale problem you've solved here. A lot of things were challenging with the merge is, first of all, completely different stacks.

1:02:34ChatGPT is like fully managed cloud-based. Like, you know, you run everything on our systems. We store things like traditional way of like building things, built for scale, built for efficiency. Codex, fully local. And so the merge is just really like, how do you get the same benefits and the same capabilities from this local coding agent and then build a product around it and build it in a way where it can benefit like a much, much broader pool of people, which is also why, you know, all of us joined OpenAI is like to benefit like this very, very broad population across the world. And so it was like a very exciting journey of like figuring out like, how do we build a cloud version of this that in essence is capable of like very, very much the same things, but is also built in a way where, you know, we can serve it through like tens and hundreds of millions of users in a way that is still like, you know, efficient so that we can include it all the way into the plus plan.

1:03:42Challenge GPT work is essentially like running the full codex harness in a cloud, like together with like a cloud computer. It's a very powerful machine, actually. Like people have sort of picked up on it and showed that, you know, what you can do. that, you know, if you are creative with the prompt is that, you know, you can get you can get ChatGPT to, like, you know, train another model in there, you know. Wow. There's, like, pretty wild things. You know, you can get it to install Blender and, you know, do, like, 3D modeling. It's, like, very permissive. It has, like, internet access. It's, like, a powerful machine.

1:04:19And then Codex just works on it. And this is, like, what we ship through ChatGPT work. So a lot of system challenges. The team did it very quickly. Obviously, Codex helped to make it more efficient, look at and build a lot of the infrastructure, and then help resolve a lot of the little differences as well that had been occurring between Codex and Chattapity, like merging plugins, architecture, merging library, and really, really working towards a unified system, which is really the goal. You shouldn't feel like, you know, you can do something in Codex that you can't do in Chagibity or vice versa.

1:04:57Like what we're trying to build is like one unified product that gives you access to the same intelligence, but in the way that you want to use it. And so it was very fun as well, because Codex throughout the whole journey also acted as a journalist to sort of like document all the steps and the debates and the discussions that the teams were having. and it was very animated debate, you know, of how we should do it and how we should name the thing and, you know, when to introduce it in what way and like what to merge into what. There were like many different permutations considered. And so there's like a very fun journalistic element to it where we have a full recounting that Codex did over time.

1:05:35And yeah, it's just it's kind of become known as well as like the toggle arc of OpenAI where, you know, we introduced like the work toggle, which there was also a lot of debate around of like, you know, whether this was like the right thing. And then, you know, it's just like, we kind of grew to just really like it. But over time, we're going to merge things further. So it's like, we're really headed into the direction of like full unification. And, you know, we kind of view this as like a temporary state where, you know, you have like, you have better and stronger capabilities when you're in work mode.

1:06:09But over time, we're bringing this, you know, all the way to like, you know, everyone that uses ChancePG.

1:06:15Tibo Sottiaux:And how do you personally use Codex? Like what's your working setup in terms of agents, in terms of tasks, in terms of what you manage with it? And related to this, I asked Peter Steinberger what I should ask about you. And he said like, you need to ask him, how do you deal with the fact that you're involved with all these projects? Your calendar is like Tetris, but usually you show up pretty cheerful. my calendar is fine um um and it's just i am capable of doing so many more things nowadays because i have the technology like codex and i actually shifted a lot of like my my work on uh on mobile using chat with your work where whenever i have something that i want to take note off i just like fire that off i use dictation a lot um whenever i have a question instead of like writing it down to look into later or delegating to someone i just like fire it off in touch of your work and i get like a report it has like a whole bunch of like custom skills and custom instructions where it's now like very very tailored to like you know produce the kinds of reports and slide decks and uh code explorations you know in the style that i can consume effectively and so every time I'm like between meetings or like you know you'll kind of like see me like you know I was just like dictating to my phone as I said before it's just like we do a lot of work in public channels we have like a lot in Slack we have a lot in Notion and Google Docs as well and so there's pretty much like there's no question really that I feel I cannot ask that you know Codex will be able to sort of like do at least a first pass of thinking through whether it is like public sentiment on a feature looking at production logs for you know how much usage we have on a certain thing making a list of things that we should deprecate because they're not getting traction uh understanding what a certain team is up to it's like any question i have i can get an answer to like you know within 30 minutes and so that's how i use it i use it for everything it's like my personal uh agent in in like all the ways and then oftentimes on on weekends as well i do some like code explorations or like i built some prototypes and I have fun, like sort of like imagining the future of the product in some ways.

1:08:34And I do that with others on the teams. It's not always the same team. And it's just like, in one day, I can build things that I sort of like, I had it in my system, right? It's like, it's like, I woke up one day, I'm just like, we should explore what it means to build this. And then I can just sort of express all of that and get like something in front of people in a day, so that they can think through it and criticize it and hopefully get inspired by it. It's like, by no means, you know, we need to ship it, but it's more like, okay, I flush it out of my system and then, you know, I go on and like, you know, do other things.

1:09:07So it's just like, so, I don't know, it's such a magical time and it's like so empowering.

1:09:12Tibo Sottiaux:And as closing, what would your advice be for a software engineer, such AI engineer, someone who built software, who would want to get the skillset and the experience to have the opportunity to work at a place like the Codex team, like OpenAI or like an AI startup. So like, you know, just become this really great builder with these tools. Because the question that comes up is often like, should I start with the theory? How important are the basics? Should I just get really good at using the tools? Yeah, I think there are two things that are important is a deep, deep curiosity for how things work and an ability to, like, you know, train yourself to understand things very quickly.

1:09:51and so it's it's it is the case that things will continue to change but people that do externally well at open ai are like you know people that just sort of like are able to like grok uh a system quickly and like you know also dive into like a new code base and sort of like you know make sense of it um but obviously like all of that is helped with agents um nowadays right so it's just like there's so much information that you need to absorb and like you know being able to understand and reason through it. I think a lot of that is asking good questions, really, about, you know, how do things work?

1:10:22And just like going into like the five whys, which I think, you know, you can just kind of keep digging and digging and digging. And, you know, you're learning very, very fast through that. The other thing is being in tune with the community or, you know, the people that you're trying to solve a problem for. It's like not everything is like solving a direct problem. Sometimes you're solving a problem that will be useful, you know, to like another group of people in the pursuit of like solving a problem for humans. but just being crisp about the taste or the needs or the requirements and being able to think clearly and like, you know, exercising through this clarity of thought feels really important to me.

1:10:56Like if you can't explain what you're trying to achieve, if you can't explain your intent, if you don't have a tie to a community, if you don't have the taste, it's going to be much harder to do great work.

1:11:06Tibo Sottiaux:Awesome, Thibaut. Well, thanks a bunch for this conversation. This was awesome. Thanks for having me. I've always wanted to get together with Thibaut, and I'm glad that we finally made it happen. I appreciated how Tebow talked about not just the upsides of open source, but also the downsides. Most notably, how competitors can copy features you are just working on in the open right now, and then ship it right before release, and just how much this stings. Plus, you get a lot of low-quality contributions that you still need to somehow deal with. Another interesting one was Tebow saying how the harness is always a step ahead of the model.

1:11:37Tibo Sottiaux:From the inside, the Codex team see their job as building clutches for the model with the harness, the tools, and the setup instruction. and then the next version of the model will be trained to need fewer of these clutches. I'll be honest, as a dev, this sounds a little demotivating that the stuff I build in the next version of the model, it'll just know and we can get rid of it. Plus, I do suspect that it's not just about building these clutches, but also building tools that models will use. And it's not like the next version of the model will reinvent an MCP protocol or scales or plugins. At least I hope not.

1:12:11Tibo Sottiaux:I also enjoyed hearing what the merge, merging chat GPC and Codex, looked like from the inside. It was merging a previously fully local coding agent, Codex, into a managed cloud-based stack, and doing it efficient enough so that it can be included in OpenAI's$20 per month plan, when$20 is not all that much in terms of compute purchase. It was pretty amusing to hear how Codex itself acted as a journalist of the whole project, as it was present in all the Slack conversations and all the documents, and so it could capture all the important debates and decisions. I'm not going to lie, this part felt a little bit of a Big Brother feel to it, where the AI is always watching, but it could well become the new normal in startups in the future.

1:12:51Tibo Sottiaux:I've not yet decided how I feel about this. And finally, I appreciated Thibault's advice for engineers to succeed. Be curious, understand symptoms quickly, and be in tune with the group you are building for. It's reassuring to hear from Thibault as well how much the fundamentals still matter. Do check out the show notes below for deep dives on how Codex, Clockcode and Cursor were built and other related topics. If you like what you heard, please hit a rating on a podcast player that you're using. It means a lot to me and to the show. Thanks, and I'll see you in the next one.