The power user’s guide to Codex: parallelizing workflows, planning techniques, advanced context engineering tips, automating code reviews, and more | Alexander Embiricos

24:06And then I got to find something to do with my time and I refuse to add more meetings to my list. So I am with you. Do I have the time and patience for a plan? I mean, a lot of the engineers on the Codex team will basically run two things that they're

24:19Alexander Embiricos:building in parallel, not more than two, like it's usually two. And so they'll kind of like be thinking about what to do on one side. And then, and by the way, this might just be like two different work trees and two different instances of their IDE open. It could be something like that. And they might just be thinking and collaborating with Codex in one while it's working and the other and just juggling two seems to be manageable for sort of normal people. Juggling more than two seems quite hard for normal people. But my view on this from a product direction perspective is we don't really want to ask humans to juggle like that's not fun for many people.

24:56Alexander Embiricos:Some people like StarCraft in code. But quick pause. Quick pause. I love StarCraft, which is why I feel like I'm really good at all this right now. Yeah, yeah, yeah. I think it's actually kind of an app analogy, but I didn't come up with it. I forget who did. But what we're trying to do is just make codecs faster and faster. Yeah. And we are also trying to basically set it up so that you don't have to do the waiting. Like, as the models get smarter and smarter, they can take on harder and harder tasks. Like, I just heard from Naveen at Every this morning, who was sharing a demo of a bug that he, no model could fix.

25:32Alexander Embiricos:And then 5.2 came out yesterday. and he threw 5.2 at it and it fought for 37 minutes and was like, this is the bug. And then in fact, that was the bug and he got the bug fixed. Right. So as we have smarter and smarter models, there is going to be more instances. There are going to be more instances where you want to wait. But I think that's our job as the product builders around the model to make it so that even when the model is thinking, you're not waiting for it to think. Yeah. Or you know that you're waiting and you feel good about it. I think that's one of the challenges I've had with, with some of these where the thinking time is long is I find myself and bless it.

26:10It's like when I, you know, worked with human software engineers, I find myself being like,

26:14Alexander Embiricos:how's it going? How's it going? You still, you still on it? Like you still good. And so I do think it's a really interesting product problem because there is, you know, useful latency in, in these models. But as a product and designer, being able to expose that latency and that reasoning and the progress in a way that makes people not feel antsy about it, I think is still a challenge out there for you all and for everybody else building these kinds of tools. You know, this has been super helpful on sort of like the basics of Codex. I would love to hear one or two integrations with other systems or tools that you found have been really like really multiply the impact you can get out of Codex.

26:55Alexander Embiricos:I think the biggest one by far is going to be GitHub and CodeReview. And then there are some others as well. This is just a nifty graph while I'm here about 5.2, the model. Let's take a quick digression. I'll show you because I just think it's super cool. Basically, what this graph shows is that 5.2, when you give it as long as it wants to think, is an amazingly intelligent model at SweetVenturePro, which is an eval of software engineering tasks. But the x-axis is pretty interesting. Basically, what it shows is the number of output tokens that the model took to perform these tasks. And so it's kind of like, how long did the model have to think?

27:34Alexander Embiricos:And when we say, hey, you can think as long as you want, ish, it's really smart. But the other cool thing is we're able to say, hey, like, we actually don't want you to think that hard, or we want you to like kind of answer quickly. And so it's performing like even higher, you know, results than like, say, this previous model here, 5.1 thinking, but in significantly less time. And so this is kind of what we're trying to build, going back to what we're saying about waiting, right? We wanna just get you the same result much faster and then get you more intelligent results and then you get the model time.

28:01Yeah, get the right result in the appropriate amount of time, right? Yeah, exactly.

28:08Alexander Embiricos:So one thing you can do with Codex is you can ask it for code review. This is actually super easy to do but without integrating with GitHub. We could just be in here. let's just say that it's written some code. I'm going to kind of ignore what happened there. And I'm just going to pretend that I wanted to review this code. So I could type slash review and basically ask it to start reviewing this code. And this is something that people really love. And, you know, right now it does feel like when you put the model in a certain mindset, like, hey, you are a reviewer and you give it a different conversation context than the model that wrote the code, you know, you'll just get like even better critiques than you might get from a human engineer.

28:47Alexander Embiricos:partially because this model has a lot of time to read all the code and maybe even execute code to like validate those changes. So this is just something super useful that I recommend doing. And like many engineers on Codex will like have Codex do work and then multiple times ask it to either review its code or critique its code or just make the code more elegant. And that's just like a massive accelerant. So let's say that you like this, right? And your team has a practice of doing review, something that you can go do is actually enable automated code review in GitHub. And so here, when this PR was pushed, Codex automatically, without anyone having to prompt it, and without anyone having to have a computer running, this is just, you know, in the cloud, Codex went, took a look, and found an issue with the code.

29:31Alexander Embiricos:And the hit rate on these is really high. Like we built this feature so that it only points out issues that it's like very confident are issues because, you know, the sort of the principle here is just like human attention is so scarce. We really want to protect it. But when it finds a really important issue, it'll post here. And this is where you start to feel the AGI a little bit. It found this issue. And then Romain basically replied like, hey, Codex, can you fix it? And then Codex went and fixed it, right? And so we can get into this kind of loop like that. So that would be my number one integration to give.

29:59Alexander Embiricos:The number two might be Slack and linear. Well, so I love this flow. And I think, again, optimizing, I'm actually running a 5.2 branch review right now. I'm not running it in codex, but I will do a comparison on the two experiences. But I'm doing a very much like compare this to base. Tell me what we did here, if there's anything I need to be aware of. I do like these in GitHub code reviews. I do feel like where I have found the highest quality is reducing noise in these. So it's really great that you have kind of focused on confidence, focused on what bugs really are going to matter. And then this loop of kind of can you fix it?

30:44And so are you running this on kind of all your repos? Has this become like how code gets reviewed? Yeah.

30:50Alexander Embiricos:I mean, Codex is just everywhere at OpenAI, which has been really cool to see. You know, I feel like when I hear a story of some tool being used everywhere, I'm always like, a little skeptical. These people are biased. So the thing I can tell the audience here is that earlier this summer, Codex was used by around half the company, which is like, I don't know, pretty low number, right? Half. We're now at the point where basically like all of technical staff, nearly all of technical staff is using Codex constantly. And so it's funny because we don't even have this comparison point since everyone's using codecs.

31:25Alexander Embiricos:It's like hard to compare the people using codecs and the people not. But there was this period of time where we were seeing that like the people using codecs were like 70 % more productive if you looked at PR volume. Obviously, PR volume isn't like the best thing to measure, but it's a thing you can look at. And now that metric doesn't mean anything anymore because everyone uses codecs. Codecs code review itself is enabled on pretty much every single repo of the company and reviews like pretty much all PRs. And, you know, it's one of those features where we were a little bit nervous when we shipped it about how people would feel, but it was just an immediate hit and like people really like it.

31:58Alexander Embiricos:You know, maybe this is a bit of a segue, but, you know, speak product person to product person, like something that we've been thinking about is if our mission is to deliver the benefits of AGI to all humanity, I believe one of the biggest limiting factors is like, do people want to type the prompt or not? you know and like i don't like typing so you know i would be too lazy to do this and so we were thinking a lot about like okay what are things we can do for teams that are just useful without anyone having to do any work and so we tried a few things code review as i showed you here is one of the things we tried big hit um we tried some other things that were pretty interesting like we built a feature that would automatically attempt to revise um the pr when you know you got a code review feedback from someone else.

32:42Alexander Embiricos:And, you know, maybe I'd be interested to try that again. But interestingly enough, that feature was not super popular. The hit rate was like often like PR feedback. I mean, sometimes it's nits, but often it's like kind of in there. You need a lot of human context to understand that PR feedback. And so Codex wasn't acing it. And the sort of hit rate wasn't high enough to be worth the like email you get every time that happens on GitHub. Whereas code review, you know, we were really careful with like how often it did things. And we made sure, for instance, this is one where Codex didn't find any issues, it doesn't even notify you.

33:13Alexander Embiricos:Just thumbs up and motion, you know, you're done. Yeah. I mean, I've had this experience too where it's interesting. I have also used automated code review and I have attempted that like full closed loop. And I have also been dissatisfied with not just the bug fixes. Sometimes they're fine. Sometimes they're not. But I often feel like putting a human in the loop that says, I'm pretty sure this is fine because X, Y, Z, or just remember we did this because A, B, C, and sometimes there's a little context lost. But the other thing from a product perspective, I think is interesting on these like proactive versus reactive agentic experiences is if you have a full agent loop, the human bar for quality is extremely high.

33:56And so dissatisfaction and frustration can bubble up very quickly. It's one thing if like your code review bot, you know, bot says this is broken, you go fix it and then you get it reviewed again and it knits you. You're like, OK, I didn't do exactly what you wanted. But if you have the experience where like your code review bot, you know, raises something, it fixes something and it knits itself, that gets really annoying. And so I do again, this is like a little bit of a agentic product design challenge that we're now going to have that people I think need to pay attention to, which is like, how do you design for latency?

34:28How do you design for perceived quality and bars when humans are involved, when they're not involved? And then one last thing I want to say to your comment, which is, I mean, if our human fingers are no longer valuable and used, what role are we going to play? Like protect the typers. But I mean, I get what you're saying, which is almost all the friction right now in my product development, software development flow is like literally writing the first prompt. Like sitting down and just going, now, let's do the thing now. And it's such a funny shift from where we've been before.

35:08Alexander Embiricos:Yeah, I mean, it's interesting to think about, right? Like, you know, obviously our mission is to just like massively accelerate every single developer and, you know, broadly than anyone who's doing anything where an agent using a computer can help. And so, yeah, the question of what our role is is interesting. Like, I like to joke that the limiting factor is typing speed. I think that's half true. So like, I think for things like review this code, please, the limiting factors is typing speed. There's a lot of just like micro places where an agent could help you. But then I think the other half is actually thinking, right?

35:39Alexander Embiricos:Like now that we can just have ubiquitous code and we can basically prototype things like trivially, the hard parts, I think, become like deciding what actually should make it in. Thinking like what a product should do, like knowing a customer, actually. and then you know if you're building a complex system like being really thoughtful about like the architecture of that system and kind of curating the agent work so yeah you know i think it's i talked to developers who are really motivated by like seeing people use what they build and i think those developers are increasingly just like thrilled right and then i also talked to developers who just love like the feeling of writing code i i don't know i experienced joy from both.

36:24Alexander Embiricos:And I think that's a place where, you know, we think constantly about like, okay, how do we make this feel as fun as possible? Right? Like setting up dependencies kind of sucks, but you have to do that for your agent. So like, how do we make that easy? Reviewing kind of sucks. So like, how do we make that easy? Great. Well, just to recap, because we've done a lot here so far before we get to lightning round, we have shown installing codecs as an extension, setting off tasks, setting up work trees, using plan and in particular the plan on how to plan on the OpenAI blog to generate plans for more complex implementations, especially when you need longer running tasks.

37:05And then automations around code review and bug fixes. The things we didn't call out but I think are really important are you can basically use codex wherever you want. So we showed it in VS Code. We showed it in the terminal. You can get it on the web. You can get it in Slack. You can get it in linear. It can kick off in GitHub. Like I do love this idea of kind of like Codex Anywhere is also really nice. So again, if you're intimidated by or don't understand VS Code, great. Kick it off in the web. If you love your command line tool, great. Let's use a couple of those keyboard shortcuts that we showed.

37:45And so I just think this is a great kind of like starter flow that shows how flexible this platform is and how it can meet, you know, kind of like a bunch of people at a variety of levels of tasks. So I'm actually going to start there with our lightning round questions, which is, you know, you just released 5.2. You showed Sweebench. We're clearly, you know, like model wars every week. I am really curious about Harness Wars. Like, why does the interface to the model like Codex matter so much? And, you know, we've seen a couple things that you've built into the platform, you know, PR review or code review, you know, little like small UX fixes that make it easy to use.

38:33But like, where do you feel like Harness differentiates in your experience using these coding tools?

38:37Alexander Embiricos:I think there's two places. One is just the quality of model work. And then the other is in the user experience. All right. So taking the quality of model work, I presume many people listening to this podcast, you know, tinker with models or have friends who are building models. And like some people just are, you know, they're model whisperers and they know how to use a model. And some people are model whisperers for like a specific model, but then maybe not for another model. And one of the things that's I think very true is that these models are changing all the time, right? Like we've been I've lost track.

39:09Alexander Embiricos:I think we're shipping a new model like every two weeks recently at OpenAI. Right. And each model we ship is better than the last model. It's like super exciting. But they're also evolving. Right. They have new capabilities that are kind of hard to keep track of unless you spend all day on Twitter slash X. And so I think that there's kind of two things here. The first is like you need to know how to get the most out of the model. You need to know, for example, that OpenAI models, we kind of have this like very, again, in an AGI-pilled way that we train models, we kind of just give them access to a shell.

39:40Alexander Embiricos:And we say like, go and do whatever you think you should do in the shell. And so we see these like really interesting emergent behaviors where, you know, sometimes the model will decide to write a Python script to make many, many edits to code. And then we have debates about, well, is that a good idea or not? Like we would prefer it if the model was like plotting through the edits or do we like it running a script? But either way, that's like a thing you should know. And so one of the cool things about Codex is that we're building our furnace in open source. And so you get to see the updates we're making for each model that we ship to make the most out of the model.

40:13Alexander Embiricos:And I can say like every time we ship a model, engineers from the Codex team will go like test it, think about it, talk to the research team. I mean, it's something they're just working super closely together to figure out how to make the most out of the model. And sometimes we'll ship a new capability like models getting better at parallel tool calling. Let's see how that works. Or recently we shipped something called compaction where the model can basically, where we can basically have the model like start a new conversation with itself with a fresh context. And what it'll do is just give itself just enough, just the right information so that to the user, it just feels like it's one conversation rather than like two conversations.

40:47Alexander Embiricos:And so when we build features like that, by building both the harness and the model together, we can be much more opinionated about what to do with the model. And then we can make the actual outcomes like way better for users. And so part of why the Codex CLI is open source is so that anyone who wants to get the best out of Codex models and actually just OpenAI models generally can just go observe how it works. They can use the Codex SDK if they want and just like not even touch the harness, just like delegate to us for that. Or if you want to build your own harness, you can just go copy paste parts of code.

41:21Alexander Embiricos:So we do this all the time. Like I'm at a bunch of like Slack DMs with customers and we'll just send them code partners. Like, oh yeah, this is how we do this. Just like, just copy this code, please. So that's on the just like having higher quality model outputs, but also keeping pace with the pace of innovation from our research team. That would be like why I think the Codex Harness is awesome there. The other side is product overhead. For instance, earlier this year, most of the sort of like really powerful agentic flows that people were using, like Codex and other, were in the CLI. Right. And this is super basic, but like I spoke to many people who don't really like spending all day in terminal.

42:01Alexander Embiricos:Like I love using the terminal, but I spoke to many people who don't. And I spoke to many people who really like seeing the code that is being edited at the same time. That's me. I'm a code reader. I like to read my code. Right. And so, you know, that's a very, very basic thing. But it's a place where just building the right product experience unlocked a ton of growth for us. And so, you know, empirically, I could say that we see many, many more people who like looking at the code that's being written at the same time as the agent than we see like just like running in terminal like on the side.

42:33Alexander Embiricos:So I think, you know, that's a very basic example. And then we were kind of touching on this point of like latency for a more advanced example. Like, I think if we can harness the model right, we can make it so that you can deploy the model to help you with like hundreds of thousands of things a day. but without you having to type, right? But also without it being annoying to like filter these outputs and without you having to wait because whenever it's helping you with something proactively, it's sort of doing so on its own computer and only letting you know when it has something great. So my view is actually that even as all these models are progressing, if like, let's just say that stopped, which it won't, there are many years of product building to do to just like get the harness right and useful for people.

43:13Well, that's great. And I do want to make sure people did not miss this tip, which is if you're trying to figure out how to get the most out of these new models, go peek under the hood at Codex open source and just see what, I mean, I think that's the other thing is what kinds of changes do you have to make when a new model comes out? Especially if you're a builder out there, maybe you're not building a coding tool, but you're building a SaaS product that uses these models. When they come out, being able to observe how the creators of the models actually maximize their unique strengths, I think is a really valuable thing that I think people really underestimate.

43:50Okay, my second question. I spied with my little eye, Atlas. Tell me your favorite Atlas use cases that you think people underappreciate.

44:00Alexander Embiricos:Ooh, the first one is kind of boring, but it's really, really true, which is I have started just asking chat for everything instead of chat being chat to BT. I just ask chat for everything because I get answers that are like really catered to me because I talk to Chad about everything. You know, I'm a weird person. Like if I ask a question and then I like make a decision based off it, I tell it what decision I made because then it remembers. And so then the next question I answer I get is even better. And so just simply like my workflow for anything that's not code, I go to Atlas, I command T to open a new tab.

44:37Alexander Embiricos:And then I just type whatever I want and I get an answer. And I often follow up. Like for me, that's the sort of magic thing about using an LLM. So being able to just like ask your question and then follow up and then maybe navigate to links like super boring. But but I love that. My other sort of favorite feature is that that is side chat. So basically any page you open. Maybe I can show this. Yeah. And while you're doing that, I have to call out that you have settled a debate that I saw today on X, which is like, should you tell your AI when it's done something right or wrong after it's done it?

45:13like once something's fixed a bug, I'm the person's like, great, it looks awesome. Thanks, that fixed. But I do think that in my mind, I've convinced myself that closed loop creates some context that's like, yes, I did this particular thing right or wrong. User has accepted it and that in some future world, it's going to make my life better. Yeah.

45:33Alexander Embiricos:I mean, so I think there's two reasons to do that. The first is just like memory, right? Like if you have memory enabled, which most people do, then you'll get a better answer. A very concrete example is I was on holiday and some plans changed. So we were deciding where to get dinner. And so we just asked chat for dinner recommendations. To be clear, we also we like food a lot. So we also like searched. But it's interesting to get asked chat because it knew like where we were staying and it knew what food we'd had like the day before or whatever. And so it just gave like a really bespoke recommendation.

46:03Alexander Embiricos:And that was cool. So I think my other sort of hot take reason to do this is I think it's important to be polite to AI. I agree. This is not an official company stance, just to be clear. But my sort of meta reason here is I just think it's important to be polite to everyone. And I think that if you start not being polite to chat, I think it can wear off on you and you just start not being polite to other people in your life. And like we're adults. Like imagine kids, right? They hear us like talking to the RAI in some way. They're going to go treat someone who they, you know, their children, they don't know in some like not polite way.

46:36Alexander Embiricos:So that's kind of my my hot thing. I could not agree more. You know, I am a please. Thank you. Good job. And honestly, it's not because of the A.I.'s humanity. It's to protect my own humanity, which is if I get used to being a jerk to anything human like, there is no way that does not bleed into how I think about people, speak to people, clip this. pin it to the top of the YouTube channel. Be polite for you. 100 % that resonates a lot. It's like our humanity is defined by how we treat others, not how they treat us. Right. Exactly. So side chat. So basically, I can go in here, I can click this button.

47:16Alexander Embiricos:And I can ask questions about the page, right? So I could be like, what's great about GPT 5.2. You might joke, like, and be like, why are you asking AI to summarize this article but like oftentimes i'll be at work and someone will send me a thing and be like thoughts and i just like i don't have time so it's like what is this right and you know and then i can have a conversation though right so then i can be like oh like interesting like you know um hey chat what do i think about this i mean when i ask my questions it often grounds itself on what i've talked to it about before it's like well since you are a person who likes you know like this, you probably would be interested in this detail.

47:59Alexander Embiricos:So, I mean, this is not maybe the best example because I'm asking for a summary, but oftentimes if I'm looking at like numbers or math or I need to learn more about like a concept, I'll use this. You can also use this to rewrite something. So if I was like, you know, in a Google Doc, I could like select some text and like be like, hey, like how else might you rephrase this? So I think for me, side chat is a really cool feature. But I think I might be a bit nerd sniped by it, to be honest, in the sense that this is this is what I joined OpenAI to like help build. So I'm very interested in it. I'm like, for me, the idea of an agent that I don't have to, it understands me, I don't have to like map myself to its world is really powerful.

48:39Alexander Embiricos:So side chat is that codex is that too, right? You launch it in the code base, and it's in your environment, you don't have to go to it. I think this is the future. And I'm like really excited for how we can just like take all the best ideas from Atlas and Codex and kind of bring them together into like a basically AGI super assistant. AGI super assistant. Maybe we'll see it next year. It sounds like the name of a product you would really... AGI super assistant 5.1 Mac. I point to you thinking high Mac. Okay, last question. and we maybe already covered this, but we've established you're polite to AI, but when it is not replying, not doing what you want, not remembering, what is your prompting technique to get it back on track?

49:28Alexander Embiricos:Yeah. I mean, so first off, I have a bit of a weird job in that if I notice the AI not replying, I have to go probably file a bug or like start a sev, sev being like a word for an incident. So I, you know, yeah, I have to go do those things. But I think context is everything. So if I see the agent not doing what I want, so I guess one really tactical tip is I don't usually ask for things from the agent without asking for context, without giving context, because I'll say like, hey, I want you to like, like, change this UI from this to this so that, users do this or because we don't want people to be confused about XYZ.

50:09Alexander Embiricos:And I often, it's funny, I think another hot take, I think PMs are the best prompters because we're used to not being the expert in what we're doing and we're used to not being the most intelligent person in the room, right? And so usually we can just maybe suggest, but we don't even know if that's right. So I sort of work with Codex in that way. I'll be like, hey, can you make this more elegant? and I won't say what I want because I actually, it'll look at the code and it'll know better than me. So tip number one is give a lot of context and actually get really good at describing the level of ambiguity of your request.

50:45Alexander Embiricos:Like do not create false precision in your prompt if you don't actually care exactly about what the outcome is. And then the second thing is like, if that doesn't work and you explain why again and it still doesn't work, then I just start a new chat. And you can do things, this is a very like advanced user thing that I don't think anyone listening to this will ever do. But Codex is a very open product. It stores its conversations logs in like your home directory in a.codex folder in a subfolder called sessions. So like.codex slash sessions. So you could just go say like, hey, I started a new session because you got confused.

51:19Alexander Embiricos:I wanted you to do this because of this. Go read your previous session, understand what's going on and then like, you know, continue from there. I love it. ending this episode with a hidden hot tip, which we didn't get to through our codex walkthrough, which is all your sessions are stored locally. So just ask it to go read them. This has been really fun. Alex, where can we find you? And other than reporting bugs, how can we be helpful? I am hiring PMs. So if you're interested, please apply on the job site and also hit me up on socials, we are hiring generally a lot on Codex. We do love bug reports, and we do love feedback.

52:01Alexander Embiricos:And actually, it's already an open source, I don't mind talking about it. We actually are also releasing a bunch of new configuration abilities for Codex, like the ability to allow us commands or skills. So if you wanted to like help build Codex skills, or like tell us what configuration you want, that would be very helpful. And lastly, just check it out. Codex is awesome. So you can find me on Twitter. I'm at Embarico, E-M-B-I-R-I-C-O. And at the r slash codex subreddit, we are there all the time and also love chatting there. Amazing. Well, thank you for joining How I AI. Cool. Thanks for having me.

52:36Thanks so much for watching. If you enjoyed this show, please like and subscribe here on YouTube or even better, leave us a comment with your thoughts. You can also find this podcast on Apple Podcasts, Spotify, or your favorite podcast app. Please consider leaving us a rating and review, which will help others find the show. You can see all our episodes and learn more about the show at howiaipod.com. See you next time.