ChatGPT Codex: The Missing Manual

25:38Cool. I think that's a really good set of advice. I think you just gave us our episode title. Like we're just going to call it best practices for using chat GPT codecs. And, you know, I mean, I think people are going to want best practices. So I noticed something that's very interesting, right? Like I think there's always two versions in terms of building agents. One, which is you try to be more controlling. You try to make it more deterministic. And then the other, you try to just prompt it and trust the model. And I think your approach is very much prompted, trust the model. So I see inside of the agents.md system prompt that you just prompt it to behave the way that you want, and you just expect the model to behave it.

26:19Obviously, you have control of the model, so you can train it if it doesn't do well. But one thing that makes me question it is, how do you fit everything in context? Like, what if I just have a super long HSMD? You know, in your live stream, you had it demoing on, like, the OpenAI monorepo, which is just giant, right? Like, so how do you manage caching and context windows and all that? Yeah. I mean, would you believe me if I told you right now that it all fits in the context window? Not the OpenAI repo. No, but sorry, everything that the agent needs. Right, so you reify the agents ID, put it at the top, right?

27:03It's just like another system prompt. No, actually, it's a file that the agent knows how to, like, graph and set for, right? Okay. Because there might be multiple ones. And so you can actually see it in the work log, right? It's, like, going to look for, it very aggressively looks for an agent ID. It's been trained to do that. I'll say it's been really interesting joining OpenAI and seeing how when you're thinking about where models are going and what AI products will look like years from now, you design products in a different way. Before OpenAI, especially when you don't have access to a team of researchers and many, many GPUs, you're building these deterministic programs, a lot of scaffolding around how this operates.

27:49But you don't really let the model operate as full as capacity, right? A lot of it was interesting when I just joined actually I got a lot of pushback saying like, hey, why don't we just like hard code, like, listen you keep using this tool wrong let's just say in our prompt, don't do that and then the researchers will be like, no, no, no we don't do that, we're going to do it the right way, we're going to teach the model why this is the right way to do it and I think that's like related to this overall thought on like, where do you put the deterministic guardrails in and where do you really let the model think, right?

28:23Similar conversation around planning. Should we just have an explicit planning stage where it's like think out loud first, write down what you're going to do, and then go do it? Sure, but what if the task is really easy? All right, do you really want to think this whole time? What if it needs to like replan as it goes? Like do you have all these like if-else conditions, heuristics to do that? Or do you train a really good model that knows how to switch between those modes of thinking? And so it's tough. Like I definitely have advocated for like little guardrails here and there until like the next training run's done.

28:52But I think that's really like we're really building for this future where the model is able to make all these decisions. What's really important is that you give it the right tool, right? You give it ways to manage context, manage memory, manage ways to explore the code base. Those still are really important. Yeah, that's like, yeah, super well said. Like I think building here is like super fun and different. And like, you know, the model isn't all the product, but the model is the product, right? And you kind of, like, need to have this kind of, like, humility in terms of, like, thinking about, like, okay, well, what are the things that, like, we, there's, like, three parties, right?

29:32There's the user, the developer, and, like, the model maybe, right? What are the things that the user, like, just needs to decide up front? And then what are the things that, like, we, the developer, are going to be able to decide better than the model? And then what are the things that the model can just decide best, right? And, like, you kind of, every decision is, like, just has to be one of those three. And, you know, it's not like everything is the model. Like, for instance, we have two buttons in the UI right now, like ask and code. And, like, you know, those probably could get inlined into, like, the decisions the model makes.

30:05But, you know, right now, it was just really, like, it made sense to kind of just give the user choice up front because we spawn a different container for the model first based on what button you press. So, like, if you ask for code, we put all the dependencies in. I'm going to oversimplify here. But if you don't ask for code, if you're just asking a question, we do a much quicker container setup before the model gets any choice. And so that's maybe a user decision. There's some places where user and developer decisions kind of come together around the environment. But ultimately, a lot of agents that I see are really impressive.

30:41But it's basically part of what's impressive is it's a bunch of developers building this really bespoke state machine around a bunch of short model calls. and so then the upper bound of like complexity of problem that the model can tackle is kind of actually just what can fit in the developer's brain. And over time we want these models to capture like or to solve for much more complex problems you know just by themselves on like more and more complex individual tasks and then eventually you could really imagine that you get like a team of agents working together maybe with like one agent that's kind of managing those agents and you know the complexity just explodes and so we really want to like get as much of that complexity as much of that state machine as possible like pushed into the model.

31:19And so you end up with these kind of two modes of building. Like in one place, you're like building product UI and rules. And in the other case, you still have to do work to get the model to learn something. But rather what you have to do is you have to figure out like what are the right things that this model needs to see during its training to like learn something. And so it's still a lot of human work to like figure out how to get that change. But it's like a very different way of thinking of like, we're going to get the model to see this. But how do you build the product to get the signal?

31:47So if you think about the code in Ask, it's almost you're basically getting the user to label the prompt in a way, right? Because they say, Ask, this is an Ask prompt, code, this is a code prompt. Are there any other kind of like fun product designs, like as you built this, of like, okay, we think the model can learn this, but we don't have the data. This is how we architect codecs to kind of help us collect that data? I think file context and scoping is, we don't have great built-in things like that right now. but it's like one of the obvious things that I need to add is another example of this, right?

32:19Like you could have, we're often usually pleasantly surprised that it's, oh, it was able to find the exact file that I was thinking about, but it takes some time, right? And so a lot of times you'll shortcut a bunch of chain of thought by just saying, hey, I'm looking at this directory. Can you go through? So I think that'll probably be there for a bit until you have some better architectural indexing and search capabilities. Yeah, I'll add to this. I'm actually going to double down on my thing about how do we think about it. So one thing we might consider is context window management, right? And should we intervene here?

32:56And so we could do a product intervention, right? Like write some code to intervene. And then kind of the next level of thinking, maybe like a little bit more AGI-pilled, is like, okay, let's get the model to see context window management stuff in this training. I can't even come up with an example now at this point because I'm too AGI-filled, but like, I don't know, we could come up with something that it has to see to like learn how to manage its context. But it's like specifically tasks related to context window. But then the most AGI-filled thing to do is like to be like, we don't actually need to think about this problem.

33:24The model will just figure it out. All we have to do is give it harder and harder problems. And then like it will just have an emergent property of managing its own context because that's the only way it can like solve these problems, right? So I'm kind of slightly like oversimplifying here. But like, you know, basically the model learns to manage its context. And so when you were talking about it working in the monorepo, it learns how to be efficient with the way that it spends its tokens as it's browsing and setting. In your example of there's a giant agent's.md, I guess we would just need to show it some versions where there was that.

33:57And so it learns it shouldn't read the whole thing every time and should first figure out how many lines it has, et cetera. So anyways, summarizing, we just need to keep giving it harder and harder problems. And a lot of these things that we might be very tempted to build a sub-intervention for, it will just have to figure out. And if it doesn't figure it out, maybe it didn't matter. Sure. Yeah, I totally get that. I think, like, we don't really have online models yet, right? And that's kind of what you need for your vision to be real. And for what it's worth, I wasn't thinking about, like, a giant AgentsMD.

34:30I was just thinking about, like, hierarchical nested AgentsMD with, like, a lot of code. um and like i think one issue where you have this version where the model is the product is your your dev cycle as the codex team like the two of you like you have to kind of it's not as tight because you have to be like okay every time there's a bug all right now i need to go get data um and where do you get the data i don't know like maybe employees use it maybe you have like you buy it from vendors and like you hire some human raters or whatever and then you have to train it in and then you have to go test it again it's it's very slow isn't it and like expensive yeah i think it's definitely yeah i think it's definitely like from a from a building perspective you have to do this when you're really willing to play the like the long-term vision of like we're going to build a better model maybe even a better model bespoke for a certain like functional purpose like codex one and then we're going to generalize the learnings from that model into like an even bigger model that's like getting all these other like learnings from other functional purposes and like these together will like become a really powerful thing and that's kind of like the philosophy we have with training models so far and it has been working but it's definitely like a long term play you know like another example we do do this on occasion like for example recently we released GPT 4.1 like really good coding model and again that was like based on like working we were like hey we want to invest better in this area let's hang out with a bunch of developers understand their feedback how things work, you know, create some evals.

36:05And, you know, like you said, this is like, it's a lot of work to do that. But then we end up with a great model and even more exciting, we can then like take those learnings and like put them into our mainline models and then everything benefits. And you kind of, the sort of philosophical view, I don't know if I can like factually prove it or not, maybe someone here can, is that like, if you can like build, do something very specific for like a specific purpose, actually when you bring that and you bring into the generalized model, you might even get outsized returns on that because there's transfer from all these different domains.

36:37Okay, cool. I think we had a couple factual things to wrap up on just the codex itself, and then we wanted to double-click on the compute platform stuff, which I think, Josh, you wanted to cover more on. So I noticed in the details, it was between 1 to 30 minutes in length. Is that a hard cutoff? Have you had it go for longer? Any comment on the task time? Yeah, I mean, I just checked the code base before this. Someone else had a similar question. Our hard cutoff is an hour right now, although don't hold us to that. It may change over time. The longest is I've seen two hours when in development mode and the model went off the rails.

37:17So, you know, but I think 30 minutes is a great ballpark for the kind of tasks that we're trying to solve, right? These are hard tasks that require a lot of iteration. and testing and the model needs that time. Yeah, I mean, yeah, I think actually like our average is like pretty, it's significantly lower than 30. But if you give it a hard task, you'll end up at 30. Yeah, I mean, you know, I think there's a couple analogies here. One, I think the operator team released a benchmark where they had to cut off for two hours. And then the other one is the meter paper, which I don't know if has been circulating, where they estimated that the current average autonomous time is like an hour and it's maybe doubling every seven months.

38:00So like an hour sounds right, but also, I mean, that's the median. So there's going to be some that go longer than that. Yeah, totally. Is this part of the, you had cutoffs for a few, like 23 Sweetbench verified examples that were not runnable. Was that part of it in terms of length or was it just something else? Yeah. To be honest, I'm not exactly sure, but I feel like there's a bunch of Sweebench cases that actually are like, invalid might be too strong of a word, a little bit not sure, but I feel like there's issues with running them, so they just don't work. Okay. And then max concurrency, is there a concurrency limit if I have 5, 10, 100 simultaneous codex?

38:415 and 10 is totally fine. Do we actually have a, I feel like we did introduce a limit for fraud reasons. I don't know what it is. Yeah, I think right now it's 60 an hour. Wow. So one per minute, I'm just going to... Yeah, but look, this is literally the point, right? It is. So long-term, we actually don't want you to have to think about if you're delegating or pairing with AI. If you imagine an AGI super assistant, you just talk to it, and it just does stuff. It answers quickly if it needs to. It takes a long time. And you also don't have to only talk to it. It's also just present in your tools, right?

39:17So that's the long-term thing. but in the near term like yeah this is a tool you delegate to and um the way to use it that we see like you know going back to the i guess that maybe the title of this podcast of like best practices it's like you must have an abundance mindset and you must think of it as like like not using your time to explore things and so like you know often when something a model is going to work on your computer and it's going to work on your computer you'll like really craft the prompt because you know then it's it's going to use your computer for a while and maybe you can't but the way we see people who like love codex the most using it is they don't they think for like maybe 30 seconds max about their prompt it's just like oh i have this idea like boom oh like there's this thing i want to do like boom oh like i just saw this bug or like this customer feedback thing like and you just send it off and so yeah like the more you're running in parallel actually i think that i mean the happier we are and like the happier we think like users are when they see it like that's just the vibe of the product really yeah i would i would pass my own anecdote so i I was on the trusted testers team for this thing, as both of you well know.

40:20And I was using, I found out I was using it wrong. I was using it like cursor. Like I had my chat window open and I watched it code. And then I realized I wasn't supposed to. And I was like, oh, like you guys are just firing the things off and like, you know, going on about your day. And yeah, that was a change of mindset. Yeah, one. Yeah. Real quick, I'll keep it brief. Like one thing that's quite fun is like use it on your phone. because somehow just being on your phone just flips the way people think about things. So we made the website responsive. We'll pull it into the app eventually. So try it.

40:53It's actually super fun and satisfying. Okay, so yeah, it's not... There was a voice... There was one of the videos that was showing the mobile engineer coding with it on his phone, but it's not available in ChatGPT's app. Okay, yeah. Yeah, not yet. Just one question I got from the mobile. I got the notification that I get. When it starts the task, it says starting research the same way the deep research notification is. Is it using deep research as a tool or did you just reuse the same notification? We just used the same notification, yeah. So you mentioned the compute platform, you mentioned how you share some of the infrastructure with RL.

41:33Can you maybe just give people a high level of what the codex has access to, what it doesn't have access to? Like, it doesn't look like people can run commands themselves. They can only instruct the model to do it. Any other things people should keep in mind? Yeah, so, and I'll say it's an evolving discussion as we figure out what parts we can give folks access to and the agent and what we need to, like, hold back for now, right? And so we're learning and it's really, we would like to give humans and agents alike as much access as possible within safety and security constraints. What you can do today, right, is as a human, set up an environment, set up scripts that get run.

42:19These scripts typically will be installing dependencies. I expect that to be maybe 95 % of the use case there. And just really get all the right binaries in place for your agent to use. We actually do have, like, a bit of an environment editing experience where, as a human, you can drop into a REPL, try things out. So, you know, please don't abuse it. But there's definitely ways for you to interact with the environment there. We laugh about that because, like, earlier I mentioned scope creep. Like, we weren't planning on having a REPL, like, to, like, interactively update your environment. But, like, you know, anyway, we tried.

42:54Josh was like, oh, man, we need this. And so that was, like, an example. Scope creep. Thanks for doing it. We do have rate limits in place. and we do monitor that very carefully. But, you know, there's interactive bits of there to, like, to get that going. But once the agent starts running, right, what we actually do today, and we're hoping to, like, evolve on this, is we'll cut off internet access because we still don't fully understand what letting loose an agent in its own environment is going to do, right? You know, for now, the safety tests all have come back very sturdily, like, you know, it's not susceptible to sorts of certain exfiltration attempts on prompt injection.

43:28But there's still a lot of risk to this category, so we don't know. And that's why to start, we're being more conservative there. And when the agent's running, it doesn't have full network access. But, you know, I'd love to be able to change that, right, allow it to give limited access to certain domains or certain repositories. And so all this to say, it's like something we're evolving as we build out the right systems to support that. Not sure that quite touches on your original question. The last thing, though, that I do want to mention is, like, there's an interactivity element with like, you know, as the agent's running, sometimes you're just like, oh, I want to like correct it, tell it to like go somewhere else.

44:07Or, you know, let me maybe fill this part out and then you can take back over, right? We haven't quite solved those problems either. What we really wanted to start was to like shoot for the fully independent, just like deliver massive value in one shot kind of approach. But yeah, we're definitely thinking about how we can weave human and agents together better.

44:34I mean, for what it's worth, I think the one-shot thing is a good angle that the other people, you know, like this is me comparing you to alternatives like Devin and Factory and all the others. there are more focus on multi-shot human feedback all these but like you know i so i have a website i'm working on and i gave it a request and i compared it all the others that was my test for codex and it did one shot it i posted it the screenshot as a tweet just earlier today and it's uh i think it's really good especially if you're running 60 at a time so i think that that really makes sense but it's it is a very ambitious goal because like human feedback is a crutch that we like to use.

45:20It also, I think, makes us write more tests, which is annoying because I don't like to write tests, but now I have to write tests. Fortunately, I'm now getting Codex to write my own tests. And I really like on the livestream as well, you can just kind of ask it to just look at your code base and just suggest stuff to do because I don't even have the energy to figure out what I should be doing. Yeah, and delegated delegation. I thought that was a great line. Yeah. And we're not saying one form factor is better than the others, right? Like, you know, I love using Codex CLI. And it's really, we just want, like, as we talked about in our interview, when I was interviewing OpenAI, like, you really want both modes.

45:59But I think what we see as, like, the role of Codex here is to really push the frontier on that sort of single shot autonomous software engineering. Yeah, I kind of think of the research preview as like our thought experiment. It's like, you know, what is coding agent like in its purest, most AGI-pilled or scale-pilled form look like? And then maybe, I mean, for me personally, I don't know, part of what excites me about working at OpenAI, it's not just solving for developers, but it's just really thinking about how does AGI benefit all of humanity and what does that feel like to non-developers as well?

46:35And so for me, what's really interesting is like thinking of Codex as an experiment for like what it'll feel like to be in other functions, you know, doing work. And like the goal for me to build towards is a vision where it's like, you know, we do the work that's like ambiguous or creative or hard to automate in whatever way. But otherwise we just have like agents like that we're delegating most of the work to. But these agents, they're not like this like long horizon thing versus short horizon. They're just like kind of ubiquitously available with you. So, yeah, we decided to take the purest form to start, which we thought would be the smallest scope thing to ship and probably isn't.

47:14But, yeah, then we're going to bring these things together. Okay, I think we have time for a couple questions. I'm just going to double click on the research preview a bit. It is a research preview. What is left? What do you think would qualify it to be a full release? you know on the live stream Greg mentioned this seamless transition between cloud and CLI is it that or are there other things on your mind? I mean to be completely honest the part of why we believe so much in iterative deployment I can give you some of my thoughts now but also we're really curious to see because this is such a new form factor but you know some of the items that are top of mind for me are like multimodal inputs you know we've talked Yeah, I know you like that, right?

48:01Yeah, like another example would be like, you know, just giving it a little bit more access to the world. You know, a lot of folks have requested for like forms of network access. You know, I also think that right now kind of the UI that we shipped is actually one that we iterated around. It's like a fun story there, but like, and it's one that people find useful, but it's definitely not the final form of what it is. And like we would love for it to be like much closer with the tools that developers spend time in. So like, those are some of the themes we're thinking about. But, you know, to be clear, we'll iterate and figure that out.

48:38I wanted to ask, why did you put finding a typo as one of the onboarding things? Because I used it, and then I saw it, and it's literally just grepping for potential type. It's like searching, grepping for like selenium with an N, or like something with like some TGN instead of NG. but it went through like 50 of these and then finally found will misspelled this w-i-l without the two thing but it was really cool to see what it thought like default spelled as like d-e-f-u-a-l-t it's like it just grabbed all these different things and then eventually got there like why why did you pick that task honestly t when i were talking about it and he was like Like, listen, it would be funny if I had a typo as I type this prompt out and just make it a little bit meta.

49:31You know, nervous fingers on a live stream. So maybe optimize for that. I have noticed it. You know, it likes to do TEH to look for the. And so it's a work in progress. That was great. Any parting thoughts, call to action? Are you growing the team? Do you want specific feedback from the community? Yeah. I think for me, the one thing that is really on my mind for getting better at over the next few months is really helping you customize your environment in a more high-fidelity manner. It turns out the good news is the agent can do a lot of really good work with only the basics, right? It's much like if your dev machine is borked and you're sort of looking at your editor but none of the type checks are working, a lot of folks can still actually do a lot of good work.

50:24But how do you get close at last 30%, 40 %? It's really hard because there's such a wide variety of environments out there, but especially would love feedback from folks on how they would like to see their environment customized. Do they want to just ship us a darker image? Would they rather have us support dev containers? So the form factor of how you do, the DX of how you do environment customization, still very much an open question that we need to improve on. Yeah, big plus one to that. And I think for me, maybe the thing that I'm most interested in is, hey, this is like a new shape of tool to collaborate with.

51:03And I'm just really interested for people to try working with it in as many different ways as possible and kind of figure out where does it work well in your workflow? Like you mentioned earlier, you were trying to use it kind of like your IDE and then you realized it was different. So I would just love for people to take advantage, especially now, like we're very intentionally just providing like very generous rate limits so people can try it. Like we just want you to try it and figure out what sticks, what works, what doesn't, how do you prompt it? And then we want to learn from that and use that to lean in.

51:30So yeah, I guess my parting call to action is like, please go try it out in JatchBT, use it as much as you can, especially now, and then let us know like how you like to hold it basically. I'm worried about the pricing when it happens, but yeah, I'm going to abuse this. Why not? Why not? Yeah. Send us feedback on pricing too. Yeah. Okay. It's too early to talk about pricing, right? Yeah. It's too early now. Yeah. Okay. All right. But yeah, based on cloud code, that's the thing that people are worried about, right? And cloud has started to introduce some kind of fixed pricing and variable pricing.

52:08And I think it's a huge mess. Like there's no right answer. Everyone just wants the cheapest form of code attention they can get. So yeah. Good luck. Thanks. I mean, my take is, I don't know what's going to make it into it. But, like, we aim to deliver a lot of value, right? And it's on us to show that and really make people realize, like, wow, this is, like, doing very economically valuable work for me. And I think a lot of the pricing can fall from that. But I think that's where the conversation should start. Are we actually delivering that value? Yeah, awesome. All right. Well, thank you so much.

52:46Yeah, thanks for working on this. and thanks for sharing your time. It is, it's been, it's been a long time coming, but I think people can start seeing like, OpenAI in general is getting very serious about agents. It's not just coding, but coding obviously is the one loop that is self-accelerating that I think, obviously you guys are super passionate about. It's really inspiring to see. Yeah, super excited to, super excited to just like, shift everyone this coding agent and then yeah, like bring it together and to just like the, you know, the general AGI super assistant. yeah so thanks for having us on thank you guys thank you