Why humans are AI’s biggest bottleneck (and what’s coming in 2026) | Alexander Embiricos (OpenAI Codex Product Lead)

24:31You know, you can be much, you can move much faster if you're just optimizing for one of those worlds, right? And so the way that we built codex is that it just uses the shell, but in order to make that like safer and secure, we have a sandbox that the model is used to operating in. So I think one of the biggest accelerants to go all the way back to to answer your question, is just like, we're building all three things in parallel and like kind of tuning each one and, you know, constantly experimenting with how those things work with like a tightly integrated product and research team. How do you think you win in this space?

25:02Do you think it'll always be this kind of like race with other models constantly kind of leapfrogging each other? Do you think there's a world where someone just runs away with it and no one else can ever catch up? Is there like a path to just we win? Again, it comes back to this idea of like building a teammate. And not just a teammate that, you know, participates in team planning and prioritization, not just a teammate that, you know, really tests its code and like helps you maintain and deploy it. But even a teammate, you know, like if you think, again, an engineering teammate, they can also like schedule a calendar invite, right?

25:34Or move stand up or do whatever, right? And so in my mind, if we just imagine that every day or every week, some like crazy new capability is just going to be deployed by a research lab, it's just impossible for us like, you know, as humans to keep up and like use all this technology. And so I think we need to get to this world where you kind of just have like an AI teammate or super assistant that you just talk to. And it just knows how to be helpful, like on its own. Right. And so you don't have to be like reading the latest tips for how to use it. You're just like you've plugged it in and it just provides help.

26:10And so that's kind of the shape of what I think we're building. And I think that will be like a very sticky, like winning product if we can do so. So the shape that in my head, at least I have, is that we build, you know, maybe a fun topic is like, is chat the right interface for AI? I actually think chat is a very good interface when you don't know what you're supposed to use it for. in the same way that if I think of like, I'm like on MS Teams or in Slack with a teammate, chat is pretty good. I can ask for whatever I want, right? It's like kind of the common denominator for everything. So you can chat with a super assistant about whatever topic you want, whether it be coding or not.

26:43And then if you are like a functional expert in a specific domain, such as coding, there's like a GUI that you can pull up to go really deep and like look at the code and like work with the code. So I think like what we need to build as OpenAI is basically this idea of like you have chat, chat-spT, and that is a tool that's like ubiquitously available to like everyone. You start using it even like outside of work, right, to just help you. You become very comfortable with the idea of being accelerated with AI. And so then you get to work and you just can naturally just, yeah, I'm just going to ask it for this.

27:15And I don't need to know about all the connectors or like all the different features. I'm just going to ask it for help and it'll surface to me the best way that it can help at this point in time. And maybe even chime in when I didn't ask it for help. So in my mind, if we can get to that, I think that's, you know, that's how we really build like the winning product. This is so interesting because with my chat with Nick Turley, the head of ChatGPT, I think he shared that the original name for ChatGPT was Super Assistant or something like that. Yeah. And it's interesting that there's like that approach to the Super Assistant and then there's this codex approach.

27:47It's almost like the B2C version and the B2B version. And what I'm hearing is the idea here is, okay, you start with coding and building and then it's doing all this other stuff for you, scheduling meetings, I don't know, probably posting in Slack, I don't know, shipping designs. I don't know. Is the idea there this is like the business version of ChatGPT in a sense or is there something else there? Yeah, so we're getting to the one-year time horizon conversation. A lot of this might happen sooner, but in terms of fuzziness, I think we're at the one year. So I'll give you a contention and a plausible way we get there, but as for how it happens, who knows.

28:21So basically, if we're going to build a super assistant, it has to be able to do things. So we're going to have a model and it's going to be able to do stuff affecting your world. And one of the learnings I think we've seen over the past year or so is that for models to do stuff, they're much more effective when they can use a computer. So now we're like, OK, we need the super assistant that can use a computer or many computers. And now the question is, OK, well, how should it use the computer? Right. And there's lots of ways to use a computer. You know, you could try to hack the OS and use accessibility APIs.

28:56Maybe a bit easier is you could point and click. That's a little slow, you know, and unpredictable sometimes. And another way, it turns out the best way for models to use computers is simply to write code. Right. And so we're kind of getting to this idea where like, well, if you want to build any agent, maybe you should be building a coding agent. and maybe to the user, a non-technical user, they won't even know they're using a coding agent the same way that no one thinks about are they using the internet or not? It's just, they're more just like, is wifi on, right? So I think that what we're doing with Codex is we're building a software engineering teammate.

29:28And as part of that, we're kind of building an agent that can use a computer by writing code. And so we're already seeing like some pull for this. It's like quite early, but we're starting to see people like who are using Codex for like coding adjacent product purposes. And so as that develops, I think we'll just naturally see that like, oh, it turns out like we should just always have the agent write code if there is a coding way to solve a problem instead of, you know, even if you're doing a financial analysis, right? Like maybe write some code for that. So basically like, you know, you were like, hey, is this like the two ends of this product for the super assistant, right?

30:00Of ChatGPT. In my mind, like just coding is a core competency of any agent, including ChatGPT. And so like what really what we think we're building is like that competency. But so here's here's like a really cool thing about agents writing code is that you can import code Right code is like composable Interoperable, right? Because if we you know one very reductive view we could have for an agent is it's just gonna be given a computer and it's just gonna like point and click and you know go around but You know that is the future and then how we get there is Difficult to sort of chart a path because a lot of the questions around building agents aren't like can the agent do it?

30:38but it's more about, well, how can we help the agent understand the context that it's working in? And like the team that's using it, you know, probably has a way that they like to do things. They have guidelines. They probably want certain deterministic guarantees about what the agent can or cannot do. Or they want to know that the agent understands sort of this detail. Like an example would be, you know, if we're looking at a crash reporting tool, hitting a connector for it, every sub team is probably has a different meta prompt for like how they want the crashes to be analyzed right and so we start to get to this thing where like yeah we have this agent sitting in front of a computer but we need to make that configurable for the team or for the user right and let them like stuff that the agent does often we probably just want to like build in as a competency that this agent has that it can do so i think we end up with this generalizable thing that you were saying of like an agent that can just write its own scripts for whatever it wants to do.

31:33But I think that the really key part here is can we make it so that everything that the agent has to do often or that it does well, we can just remember and store so that the agent doesn't have to write a script for that again, right? Or maybe if I just joined a team and you are already on the same team as me, I can just use all those scripts that the agents had written already. Yeah. If this is our teammate, they can share things that it's learned from working with other people at the company. It just makes sense as a metaphor. Yeah. It feels like you're in the Karpathy camp of agents today are not that great and mostly slop and maybe in the future they'll be awesome.

32:09Does that resonate? I think so. I think coding agents are pretty great. I think we're seeing a ton of value there. That feels right. Yep. And then I think like agents outside of coding, it's still like very early. And you know, this is just my opinion, but I think they're going to get a whole lot better once they can use coding too in like in a composable way. this is it's kind of the fun part of like when you're building for software engineers like i at my startup we were building for software engineers too for a lot of that journey and they're just such a fun audience to build for because you know they also like building for themselves and are often like even more creative than we are and thinking about how to use the technology um and so like by building for software engineers you get to just observe a ton of emergent behaviors and like things that you should do and build into the product i love how you you say that because a lot of people building for engineers get really annoyed because the engineers are so they're just always complaining about stuff they're like ah that sucks why'd you build it this way i love that you enjoy it but i think it's probably because you're building such an amazing tool for engineers that can actually solve problems and just you know code for them um kind of along those lines you know there's always this talk of what will happen jobs engineers coding do you have to learn coding all these things uh clearly the way you're describing it is it's a teammate it's going to work with you make you more superhuman it's not going to replace you with the way you just think about the impact on the field of engineering, having this super intelligent engineering teammate?

33:32I think there's two sides to it. But the one we were just talking about is this idea that maybe every agent should actually use code and be a coding agent. And in my mind, that's just like a small part of this like broader idea that like, hey, as we make code even more ubiquitous, I mean, you could probably claim it's ubiquitous today, even pre-AI, right? But as you make code even more ubiquitous, it's actually just going to be used for many more purposes. And so there's just going to be a ton more need for people with this, like humans with this competency. So that's my view. I think this is like quite a complex topic.

34:05So, you know, it's something we talk about a lot and we have to kind of see how it pans out. But I think what we can do, what we can do basically as a product team building in the space is just try to always think about how are we building a tool so that it feels like we're like maximally accelerating people, you know, rather than building a tool that makes it like more unclear what you should do as the human, right? Like, I think like to, you know, give an example right now, like nowadays when you work with a coding agent, it writes a ton of code, but it turns out writing code is actually one of the most fun parts of software engineering for many software engineers.

34:39And so then you end up reviewing AI code, right? And that's often a less fun part of the job for many software engineers. Right. And so I actually think like we see that like this, this comes up, plays out all the time in like a ton of micro decisions. And so we as a product team are always thinking about like, okay, how do we make this more fun? How do we make you feel more empowered? Whereas it's not working. And I would argue that like reviewing agent written code is like a place that today is like less fun. And so, you know, then I think, okay, what can we do about that? Well, we can ship a code review feature that like helps you build confidence in the air written code.

35:10Okay, cool. You know, another thing we could do is we can make it so that the agents like better able to validate its work. And you know it gets all the way down into like micro decisions like if you're going to have the an agent capability to validate work and let's say you have like I'm thinking of Codex Web right now like you have a pane that sort of reflects the work the agent did what do you see first? Do you see the diff or do you see the image preview of the code it wrote? Right and you know I think if you're thinking about this from perspective like how do I empower the human how do I make them feel like as accelerated as possible like you obviously you see the image first, right?

35:42You shouldn't be reviewing the code unless first, you know, you've seen the image, unless maybe it's been like reviewed by an AI and now it's time for you to take a look. When I had Michael Cherell, the CEO of Cursor on the podcast, he had this kind of vision of us moving to something beyond code. And I've seen this rise of something called spec-driven development where you kind of just write the spec and then the code, you know, the AI writes code for you. And so you kind of start working at this higher abstraction level. Is that something you see where we're going? just like engineers not having to actually write code or look at code and there's going to be this higher level of abstraction that we focus on?

36:16Yeah, I mean, I think there's like constantly these levels of abstraction and they're actually already played out today, right? Like today, like coding agents, mostly it's like prompts to patch, right? We're starting to see people doing like spec-driven development or like planned and driven development. That's actually one of the ways when people ask like, hey, how do you run codecs on a really long task? Well, it's like often collaborate with it first to write like a plan on md like a markdown file that's your plan and once you're happy with that then you ask it to go off and do work and if that plan has verifiable steps it'll like work for much longer um so we're totally seeing that i think spec driven development is like an interesting idea it's not clear to me that it'll work out that way because a lot of people don't write like don't like writing specs either but it seems plausible that some some people will work that way you know like a bit of a joke idea though is like if you think of like the way that many teams work today, they often like don't necessarily have specs, but the team is just really self-driven and so stuff just gets done.

37:15And so almost that is like, I'm coming up with this on the spot. So it's, you know, not a good name, but like chatter-driven development where it's just like stuff is happening, you know, on social media and like in your team communications tools. And then as a result, like code gets written and deployed. Right. So yeah, I think I'm a little bit more oriented in that way of, you know, I don't even necessarily want to have to write a spec. Like sometimes I want to only if I like writing specs. Right. Other times I might just want to say like, hey, here's like the customer, you know, service channel and like tell me what's interesting to know.

37:47But if it's a small bug, just fix it. I don't have to write a spec for that. Right. I have this sort of hypothetical future that I like to share sometimes with people as a provocation, which is like in a world where we have like truly amazing agents, like what does it look like to be a solopreneur? um and uh you know one terrible idea for how it could look is that it's actually there's a mobile app and um every idea that the agent has to do is just like vertical video on your phone and then you can like swipe left if you think it's a bad idea and you can like swipe right if it's a good idea and like you can press and hold and like speak to your phone if you want to give feedback on the idea before you swipe you know and in this world like basically what your job is just like plug in this app into like every single like signal system you know system of record and then you just sort of sit back and like swipe i don't know i love this so it's like tinder meets tiktok meets codex it's pretty terrible no this is great so the idea here is this thing is this agent is watching and right listening to you paying attention to the market your users and it's like cool i hear something i should do it's like a proactive engineer just like here we should build this feature fix this thing exactly i think that's really good communicating with you in like the lowest effort.

39:01Yeah, yeah. And like the modern way we communicate. Swipe left to right and vertical feed. And then the Sora video. Okay, so I see how this all connects now. I see. Yeah. To be clear, we're not building that, but like, you know, it's a fun idea. I mean, you know, like in this example though, like one of the things that it's doing is it's consuming external signals, right? I think the other really interesting thing is like if we think about like what is the most successful like AI product to date. I would argue, it's funny, actually, not to confuse things at all, but the first time we used the brand codex at OpenAI was actually the model powering GitHub Copilot.

39:39This is way back in the day, years ago. So we decided to reuse that brand recently because it's just so good. It's codex, codex execution. But I think actually auto-completion in IDEs is one of the most successful AI products to date. And part of what's so magical about it is that when it can surface like ideas for helping you really rapidly, when it's right, you're accelerated. When it's wrong, it's not like that annoying. It can be annoying, but it's not that annoying. Right. And so you can create this like mixed initiative system that's like contextually responding to like what you're attempting to do.

40:15And so in my mind, this is like a really interesting thing for us as OpenAI as we're building. So for instance, when I think about launching a browser, which we did with Atlas, in my mind, one of the really interesting things we can then do is we can then contextually surface ways that we can help you as you're going about your day. And so we break out of this, we're just looking at code or we're just in your terminal into this idea that, hey, a real teammate is dealing with a lot more than just code. They're dealing with a lot of things that are web content. So how can we help you with that. Man, there's so much there.

40:52I love this. Okay. So autocomplete on web with the browser. That's so interesting. Just like, here's all the things that we can help you with as you're browsing and going about your day. I want to talk about Atlas. I'll come back to that. Codex, code execution. Did not know that. That's really clever. I get it now. Okay. And then this chatter, what is a chatter driven development? I had a, no, this is a really good idea, but it reminds me I had John G, Don G on the podcast, CTO of Block, and they have this product called Goose, which is their own internal agent thing. And he talked about an engineer at Block just has Goose watch him with like his screen and listens to every meeting and proactively does work that he should probably want to do.

41:37So ships to PR, sends an email, drafts a Slack message. So he's doing exactly what you're describing in kind of a very early way. Right. Yeah, that's super interesting. And, you know, I bet you, so if we went and asked them what the bottleneck to that productivity is, did they share what it is? Probably looking at it, just making sure this is the right thing to do. Yeah. So like we see this now, like we have a Slack integration for Codex. People love, you know, if there's like something that you need to do quickly, people will just like at mention Codex, like, why do you think this bug is happening?

42:07Right. It doesn't have to be an engineer, even like maybe data scientists often here are using Codex a ton to just answer questions. Like, why do you think this metric moved? What happened? So questions, you get the answer right back in Slack. It's amazing, super useful. But as for when it's writing code, then you have to go back and look at the code. And so the real, I think, bottleneck right now is validating that the code worked and writing code review. So in my mind, if we wanted to get to something like, you know, that a friend you were talking about's world, I think we really need to figure out how to get people to configure their coding agents to be much more autonomous on those later stages of the work.

42:45It makes sense. Like you said, writing code, I used to be an engineer. I was an engineer for 10 years. Really fun to write code. Really fun to just get in the flow, build architect tests. Not so fun to look at everyone else's code and just have to go through and be on the hook if it is doing something dumb that's going to take down production. And now that building has become easier, what I've always heard from companies that are really at the cutting edge of this is the bottleneck is now like figuring out what to build. And then it's at the end of like, OK, we have all this, all 100 PRs to review.

43:12Who's going to go through all that? Right. Yeah. This episode is brought to you by Jira Product Discovery. The hardest part of building products isn't actually building products. It's everything else. It's proving that the work matters, managing stakeholders, trying to plan ahead. Most teams spend more time reacting than learning, chasing updates, justifying roadmaps, and constantly unblocking work to keep things moving. Jira Product Discovery puts you back in control. With Jira Product Discovery, you can capture insights and prioritize high-impact ideas. It's flexible, so it adapts to the way your team works, and helps you build a roadmap that drives alignment, not questions.

43:51And because it's built on Jira, you can track ideas from strategy to delivery, all in one place. Less chasing, more time to think, learn, and build the right thing. Get Jira product discovery for free at Atlassian.com slash Lenny. That's Atlassian.com slash Lenny. What has the impact of Codex been on the way you operate as a product person, as a PM? It's clear how engineering is impacted. Code is written for you. What has it done to the way you operate and the way PMs operate at OpenAI? Yeah, I mean, I think mostly I just feel like much more empowered. I've always been sort of more technical leaning PM.

44:32And especially when I'm working on products for engineers, I feel like it's necessary to like, you know, dog food the product. But even beyond that, I just feel like I can do much, much more as a PM. And, you know, Scott Belsky talks about this idea of like compressing the talent stack. I'm not sure if I'm phrased that right. But it's basically this idea that like maybe the boundaries between these roles are a little bit like less needed than before. because people can just do much more. And every time someone can do more, you can skip one communication boundary and make the team that much more efficient.

45:02So I think we see it in a bunch of functions now, but I guess since you asked about products specifically, now answering questions, much, much easier. You can just ask Codex for thoughts on that. A lot of PM type work, understanding what's changing. Again, just ask Codex for help with that. prototyping is often faster than writing specs. This is something that a lot of people have talked about. I think something that I don't think is super surprising, but something that's slightly surprising is like we see, like we're mostly building codecs to write code that's going to be deployed to production.

45:38But actually we see a lot of throwaway code written with codecs now. It's kind of going back to this idea of like, you know, ubiquitous code. So you'll see, you know, someone wants to do an analysis. Like if I want to understand something, it's like, okay, just give codex a bunch of data, but then ask it to build like an interactive like data viewer for this data, right? You would that's just like too annoying to do in the past, but now it's just like totally worth the time of just getting an agent to go do something. Similarly, I've seen like some pretty cool prototypes on our design team about like if you want to well, like a designer basically wanted to build an animation and this is the coin animation codex and it was like normally it'd be too annoying to program this animation.

46:17So they just vibe coded a animation editor and then they use the animation editor to build the animation, which they then inject into the repo. Actually, our designers are there's a ton of acceleration there. And like speaking of compressing the talent stack, I think our designers are very PME. So, you know, they do a ton of product work and like they actually have like an entire like vibe coded sort of side prototype of the Codex app. And so a lot of how we talk about things is like we'll have like a really quick jam because there's like 10 ,000 things going on. and then the designer will like go think about how this should work.

46:47But instead of like talking about it again, they'll just like vibe code a prototype of that in their like standalone prototype. We'll play with it if we like it. They'll vibe code that prototype into, or vibe engineer that prototype into an actual PR to land. And then depending on their comfort with the code base, like Codex CLIs and Rust is a little harder. Maybe they'll like land it themselves or they'll like get close and then an engineer can help them like land the PR. You know, we recently shipped the Sora Android app And that was one of the most sort of mind-blowing examples of acceleration, actually, because the usage of codecs internally to OpenAI is obviously really, really high.

47:24But it's been growing over the course of the year, both in terms of like now it's basically like all technical staff use it. But even like the intensity and know-how of how to make the most of coding agents has gone up by a ton. And so the Sora Android app, right, like a fully new app, we built it in 18 days. it went from like zero to launch to employees and then 10 days later so 28 days total we went to just like GA to the public and that was done just like with the help of Codex so pretty insane velocity I would say it was like a little bit I don't want to say easy mode but there is one thing that Codex is really good at if you're a company that's like building software on multiple platforms so you've already figured out like some of the underlying like APIs or systems asking Codex to like to port things over is really effective because it has like something you can go look at and so the engineers on that team were basically having codecs go look at the ios app produce plans of work that needed to be done and then go implement those and it was kind of looking at ios and android at the same time and so you know basically it was like two weeks to launch to employees four weeks total insanely fast what makes that even more insane is it was the became the number one app in the app store i don't this just boggles the mind okay so yeah so imagine the number one app in the app store with like a handful of engineers uh i think it was like two or three possibly uh in a handful of weeks yeah this is absurd so yeah so that's a really fun example of acceleration.

49:01And then like Atlas is the other one that I think Ben did a podcast, the engine lead on Atlas, sharing a little bit about how we built there. You know, many, Atlas is actually, I mean, it's a browser, right? And building a browser is really hard. And so we had to build a lot of difficult systems in order to do that. And basically we got to the point where that team has a ton of power users, of Codex right now. And, you know, it got to the point where they were basically, you know, we were talking to them about it because a lot of those engineers are people I used to work with before my startup.

49:37And so they'd say, you know, before this would have taken us like two to three weeks for two to three engineers. And now it's like one engineer, one week. So massive acceleration there as well. And what's quite cool is that, you know, we shipped Atlas on Mac first, but now we're working on the Windows version. you know that so the team now is like ramping up on windows and they're helping us make codecs better on windows too which is admittedly earlier like just the model we shipped last week is the first model that natively understands powershell so you know powershell being uh the native like shell language on windows so yeah it's been it's been really awesome to see like the whole company getting accelerated by codecs like from and you know most obviously also research and like improving how quickly we train models and how well we do it.

50:25And then even like design, as we talked about, and marketing. Like actually we're at this point now where my product marketer is often also making string changes just directly from Slack or like updating docs directly from Slack. These are amazing examples. You guys are living at the bleeding edge of what is possible and this is how other companies are going to work. Just shipping, again, what became the number one app in the app store and just bloved all over. It just like took over the, I don't know, the world for at least a week. built you said in 28 days and like no no 10 days 18 days just to get like the core of it working yeah so like 18 days we had a thing that employees were playing with yeah and 10 days later we were out and you said just a couple engineers yeah two or three okay and then atlas you said was took a week to build no no no so atlas not the whole week but atlas was like a really meaty project yeah um and so i was talking to one of the engineers on atlas um about like you know just how what they use codex for.

51:23And it's basically like, we use codex for absolutely everything. And I was like, okay, well, like, you know, how would you measure the acceleration? So basically the answer I got back was, previously it would have taken two to three weeks for two to three engineers. And now it's like one engineer, one week. Do you think this eventually moves to non-engineers doing this sort of thing? Like, does it have to be an engineer building this thing? Could sort of been built by, I don't know, a PM or a designer? I think we will very much get to the point where, well, basically where the boundaries are a little bit blurred, right?

51:50Like, I think you're going to want someone who's like understands the details of what they're building, but what details those are will evolve. Kind of like how now, like if you're writing Swift, you don't have to speak assembly. You know, there's a handful of people in the world and it's really important that they exist and like speak assembly, maybe more than a handful, right? But that's like a specialized function that like most companies don't need to have. So I think we're just going to naturally see like an increase in layers of abstraction. And then the cool thing is now we're entering like the language layer of abstraction, like natural language.

52:24And the natural language itself is really flexible. Right? Like you could have engineers talking about like a plan and then you could have engineers talking about a spec and then you could have engineers talking about just, you know, a product or an idea. So I think we can also like start moving up those layers of abstraction as well. But, you know, I do think this is going to be gradual. I don't think it's going to go to like all of a sudden like nobody ever writes anything and like, you know, any code and it's just specs. I think it's going to be much more like, okay, we've set up our coding agent to be really good at like previewing the build or like running tests.

52:54Maybe that's the first part, right? That most people have set up. And it's like, okay, now we've set it up so that it can like execute the build and it can like see the results of its own changes. But, you know, we haven't yet built a good integration harness so that it can like, in the case of Atlas, like, by the way, I don't know if they've done any of this or not. I think they've done a lot of this, but, you know, maybe the next stage is like enable it to like load a few sample pages to see how well those work, right? So then, okay, now we're going to set up to that. And I think for some time at least, we're going to have humans kind of curating which of these connectors or systems or components that the agent needs to be good at talking to.

53:27And then in the future, there will be an even greater unlock where Codex tells you how to set it up or maybe sets itself up in a repo. What a wild time to be alive. Wow. I'm curious just the second order effects of this sort of thing, just how quickly it is to build stuff. What does that do? Does that mean distribution becomes much, much more important? Does it mean ideas are just worth a lot more? It's interesting to think about how quick, how that changes. I'm curious what you think. I still don't think ideas are worth as much as maybe a lot of people think. I still think execution is really hard, right?

54:00Like you can build something fast, but you still need to execute well on it. It still needs to make sense and be a coherent thing overall. Yeah, and distribution is massive. Yeah, just feels like everything else is now more important. Everything that isn't the building piece, which is coming up with an idea. getting to market, profit, all that kind of stuff. I think we might have been in this weird temporary phase where, you know, for a while, like you could, you could just, it was so hard to build product that you mostly just had to be really good at building product. And it maybe didn't matter if you like had an intimate understanding of a specific customer.

54:37um but now i think we're getting to this point where actually like if i could only choose like one thing to understand it would be like really meaningful understanding of like the problems that a certain customer has right if i could only if i could only go in with one like core competency so i think that that's that's ultimately still what's going to matter most right like if you're starting a new company today and you have like a really good understanding and like network of customers that are currently underserved by AI tools, I think you're set. Whereas if you're good at building websites, but you don't have any specific customer to build for, I think you're in for a much harder time.

55:14Bullish on vertical AI startups is what I'm hearing. Yeah, I completely agree. There's the general thing that can solve a lot of problems, and then there's like, we're going to solve presentations incredibly well, and we're going to understand the presentation problem better than anyone, and we're going to plug into your workflows, and all these other things that matter for a very specific problem. Okay, incredible. When you think about progress on Codex, I imagine you have a bunch of evals and there's all these public benchmarks. What's something you look at to tell you, okay, we're making really good progress.

55:46I imagine it's not going to be the one thing, but what do you focus on? What's like something you're trying to push? What's like a KPI or two? One of the things that I'm constantly reminding myself of is that a tool like Codex sort of naturally is a tool that you would become a power user of, right? And so we can accidentally spend a lot of our time thinking about features that are very deep in the user adoption journey. And so we can kind of end up over solving for that. And so I think it's just critically important to go look at your D7 retention. Just go try the product. Sign up from scratch again.

56:18I have a few too many Chachapiti Pro accounts that I've just like, in order to maximally correctly dog food, signed up for on my Gmail and they charge me like 200 bucks a month. I need to expense those. but uh uh you know like i think just like the feeling of being a user and the early retention stats are still like super important for us because you know as much as this category is is taking off i think we're still in the very early days of like people using them um another thing that we do that that might be i think we might be the most like user feedback slash social media pill team out there in this space is like a few of us are like constantly on Reddit and Twitter.

56:59And, you know, there's praise up there and there's a lot of complaints, but we take the complaints like very seriously and look at them. And I think that again, because you can use like Coding Aging for so many different things, it often is like kind of broken in many sort of ways for like specific behaviors. And so we actually monitor a lot, just like what the vibes are on social media pretty often, especially I think for Twitter X, it's a little bit more hypey. And then Reddit is a little more negative but real, actually. So I've started increasingly paying attention to like how people are talking about using codecs on Reddit, actually.

57:38This is important for people to know. Which subreddits do you check most? Is there like an R codex? I mean, the algorithm is pretty good at surfacing stuff, but like R slash codex is there. Okay, I'll take. Very interesting. And then if people tag you on Twitter, you still see that, but maybe not as powerful as seeing it on Reddit. Well, yeah. And the interesting, well, the thing with Twitter is it's a little bit more one-to-one, even if it's like in public, whereas like with Reddit, there's like really good upvoting mechanics and like maybe most people are still not bots, unclear. So you get like good signal on what matters and what other people think.

58:09So interestingly, Atlas, I want to talk about that briefly. You guys launched Atlas. I tweeted actually that I tried Atlas and then I don't love the AI only search experience I was just like, I just want Google sometimes or whatever. Like just waiting for it to give me an answer. I'm like, I don't want to. And there was no way to switch. I just tweeted, hey, I'm switching back. It's not great. And I feel like I made some PMs at OpenAI sad. And I saw someone tweet, okay, we have this now. Which I imagine was always part of the plan. It's probably an example of we just ship. We got to ship stuff, see how people use it.

58:41And then we figure it out. So I guess one is that, I don't know, is there anything there? And two, I'm just curious, why are you guys building a web browser? So I worked on Atlas for a bit. I don't work on it now. But, you know, like a bit of the narrative here for me, just to tell my story a bit, was like I was working on this like screen sharing, like pair programming startup. Right. And then we joined OpenAI. And so the idea was really to build a contextual desktop assistant. And the reason I believe that's so important is because I think that it's really annoying to have to give all your context to an assistant and then to figure out how it can help you.

59:16Right. And so if it could just like understand what you are trying to do, then it could maximally accelerate you. And so I would, you know, I still think of Codex actually as like a contextual assistant from a little bit of a different angle, like starting with coding tasks.

59:32But some of the thinking, at least for me personally, I can't speak for the whole product, but was that a lot of work is done in the web. And if we could build a browser, then we could be contextual for you, but in a much more first class way. We weren't hacking like other desktop software, which have like very varied support for it. for like what content they're rendering to the accessibility tree. We wouldn't be relying on screenshots, which are a little bit slower and unreliable. Instead, we could like be in the rendering engine, right? And like extract whatever we needed to, to help you. And also, I like to think of like, you know, video games, like, I don't know if you've played, like, I don't know, say Halo, right?

1:00:11Like you walk up to an object. I mean, this is true for many games. You press, man, it's been a long time. This is embarrassing. Press X and it just does the right thing, right? And I was one of those guys who always read the instruction manual for every video game that I bought. And I remember the first time I read about a contextual action and I just thought it was like this really cool idea. And, you know, the thing about a contextual action is we need to know what you are attempting to do. We have a little bit of context and then we can and then we can help. And I think this is critically important because, you know, imagine this world that we reach, right?

1:00:43Where we have agents that are helping you thousands of times per day. imagine if the only way we could tell you that we helped you was if we could like push notify you so you get a thousand push notifications a day of an ai saying like hey i did this thing do you like it it'd be super annoying right whereas imagine going back to software engineering like i was looking at a dashboard and i noticed some like key metric had like gone down and you know at that point in time an ai could like maybe go take a look and then surface the fact that it has an opinion on why this metric went down and maybe a fix right there, right when I'm looking at the dashboard.

1:01:20That would be like, that would much more keep me in flow and enable the agent to take action on many more things. So in my mind, part of why I'm excited for us to have a browser is that I think we have then much more context around what we should help with. Users have much more control over what they want us to look at. It's like, hey, if you want us to take action on something, you can open it in your AI browser. If you don't, then you can open it in your other browser. Right? So like really clear control and boundaries. And then we have the ability to build UX that's like mixed initiatives so that we can surface contextual actions to you like at the time that they're helpful as opposed to just like randomly notifying you.

1:01:58Hearing the vision for Codex being the super assistant, it's not just there to code for you. It's trying to do a lot for you as a teammate and as this kind of super teammate that makes you awesome at work. So I get this. Speaking of that, Are there other non-engineering common use cases for codecs? Just ways that non-engineers, we talked about, you know, designers prototyping, building stuff. Are there any, I don't know, fun or unexpected ways people are using codecs that aren't engineers? I mean, there's a load of unexpected ways, but I think, like, most of what we're seeing, like, real attraction with people using things are still, for now, like, very, like, I would say coding adjacent or, like, sort of tech oriented.

1:02:38places where there's like a mature ecosystem or, you know, maybe you're doing data analysis or something like that. I personally am expecting that we're going to see a lot more of that over time. But for now, like we're keeping the team like very focused on just coding for now because there's so much more work to do. For people that are thinking about trying out codecs, is there like, does it work for all kinds of code bases? What code does it support? If you're like, I don't know, SAP, can you add codecs and start building things? What's kind of like the sweet spot where does it start to not be amazing yet?

1:03:11I'm really glad you asked this question, actually, because the best way to try Codex is to give it your hardest tasks, which is a little different than some of the other coding agents. Like, you know, some tools you might think, OK, let me like start easy or just like, you know, like vibe code something random and decide if I like the tool. Whereas like we're really building Codex to be the like professional tool that you can give your like hardest problems to. and you know that writes like high quality code in your like enormous code base that is in fact not perfect right now. So yeah, I think if you're going to try Codex, you want to try it on like a real task that you have and not necessarily like dumb that task down to something that's like trivial, but actually like, you know, like a good one would be like you have a hard bug and you don't know what's causing that bug and you ask Codex to like help figure that out or like to implement that, you know, the fix.

1:04:00I love that answer. Just give it your hardest problem. I will say like, you know, if you're like, hey, okay, well, the hardest problem I have is that I need to build a new unicorn business. Obviously, it's not going to work. Not yet. So I think it's like, give it the hardest problem, but something that is still one question or one task to start. That's if you're testing. And then over time, you can learn how to use it for bigger things. Yeah. What languages does it support? Basically, the way we've trained Codex is there's a distribution of languages that we support, and it's like fairly aligned with like the frequency of these languages in the world.

1:04:35So unless you're writing some like very esoteric language or like some private language, it should do fine in your language. If someone was just getting started, is there a tip you could share to help them be successful? Like if you could just whisper a little tip into someone just setting up codecs for the first time to help them have a really good time, what's something you'd whisper? I might say, try a few things in parallel, right? So you could try giving it a hard task. maybe ask it to understand the code base, formulate a plan with it around an idea that you have, and kind of build your way up from there.

1:05:08And like sort of the meta idea here is it's again, it's like you're building trust with a new teammate, right? And so like you wouldn't go to a new teammate and just give them like, hey, do this thing. Here's zero context. You would start by like first making sure they understand the code base. And then you would like maybe align on a plan and approach, and then you would have them go off and do bit by bit, right? And I think if you use codex in that way, you'll just sort of naturally start to understand like the different ways of prompting it because it is, it's a super powerful like agent and model, but it is, it is a little bit different to prompt codex than other models.

1:05:38Just a couple more questions. One, we touched on this a little bit. As AI does more and more coding, there's always this question of, should I learn to code? And why should I spend time doing this sort of thing? For people that are trying to figure out what to do with their career, especially if they're into software engineering computer science, do you think there's specific elements of computer science that are more and more important to lean into? Maybe things they don't need to worry about? Like, what do you think people should be leaning into skill-wise as this becomes more and more of a thing in our workplace?

1:06:10I think there's like a couple angles you could go at this from. I think the, well, the easiest one to think of at least is just like be a doer of things. I think that you know with coding agents getting better and better over time it's just what you can do as even like someone in college or a new grad is just like so much more than what that was before and so I think you just want to be taking advantage of that and definitely when I'm looking at like hiring folks who are earlier career it's like definitely something that I think about is how how productive are they using the latest tools right they should be like super productive and if you think of it in that way, they actually have like less of a handicap than before versus a more senior career person because, you know, the divide is actually getting smaller because they've got these amazing coding agents now.

1:07:01So that's one thing, which is like, I guess the thing, the advice is just like, learn about whatever you want, but just make sure you spend time doing things, not just like fulfilling homework assignments, I guess. I think the other side of it though, is that it's still deeply worth understanding like what makes a good like overall software system. So I still think that skills like really strong systems engineering skills or even really effective communication and collaboration with your team, skills like that I think are important and are going to continue to matter for quite some time. I don't think it's going to be like all of a sudden the AI coding agents are just able to build perfect systems without your help.

1:07:42I think it's going to look much more gradual where it's like, okay, we have these AI coding agents. They're able to validate their work. It's still important. And like, for example, like I'm thinking of an engineer who was working on Atlas since we were talking about it. He set up codecs that it can like verify its own work, which is a little bit non-trivial because of the nature of the Atlas project. So the way that he did that was he actually prompted codecs like, hey, why can't you verify your work? Fix it. And like did that on a loop. Right. And so you still, at various phases, are going to want a human in the loop to help configure the coding agent to be effective.

1:08:16And so I think you still want to be able to reason about that. So maybe it's less important that you can type really fast and you understand exactly how to write. Not that anyone writes a for each loop or something, right? But it is, or you don't need to know how to implement a specific algorithm. But I think you need to be able to reason about the different systems and what makes effective, a software engineering team effective. So I think that's the other really important thing. And then like maybe the last angle that you could take is, I think if you're on the frontier of knowledge for a given thing, I still think that's like deeply interesting to go down partially because that knowledge is still going to be like, you know, agents aren't going to be as good at that.

1:08:56But also partially because I think that like by trying to advance the frontier of a specific thing, you'll actually like end up like being forced to take advantage of coding agents and using them to accelerate your own workflow as you go. What's an example of that when you talk about being at the frontier or something? Codex writes a lot of the code that helps manage its training runs, the game infrastructure. We move pretty fast, and so we have a Codex code review that's catching a lot of mistakes. It's actually caught some pretty interesting configuration mistakes. And we're starting to see glimpses of the future where we're actually starting to have Codex even like be on call for its own training, which is pretty interesting.

1:09:36So there's lots there. Wait, what does that mean to be on call for its own training? So it's running, it's training, and it's like, oh, something broke. Someone needs, and it, does it like alert people? Or it's like, here, I'm going to fix the problem and restart. This is an early idea that we're like figuring out. But the basic idea is that, you know, during a training run, there's like a bunch of graphs that like today, like humans are looking at. And it's like really important to like look at those. We call this babysitting. Because it's very expensive to train, I imagine. and very important to move fast.

1:10:03Exactly. And there's a lot of systems underlying the training run. And so a system could go down or there could be an error somewhere that gets introduced. And so we might need to fix it or pause things or I don't know, there's lots of actions we might need to take. And so basically having Codex run on a loop to evaluate how those charts are moving over time is sort of this idea that we have to how to enable us to train way more efficiently. I love that. This is very much along the lines of this is the future of agents. And its codex isn't just for building code and write. It's a lot more than that.

1:10:34Yeah. Okay, last question. Being at OpenAI, I can't not ask about your AGI timeline and how far you think we are from AGI. I know this isn't what you work on, but there's a lot of opinions, a lot of, I don't know, timelines. How far do you think we are from a human version of AI, whatever that means to you? So for me, I think that it's a little bit about like, when do we see the acceleration curves kind of go like this, or I don't know which way I'm mirrored here, right? When do we see the hockey stick? And I think that the current limiting factor, I mean, there's many, but I think a current underappreciated limiting factor is like literally human typing speed or human multitasking speed on like writing prompts, right?

1:11:19And like, you know, you were talking about, it's like, you can have an agent, like watch all the work you're doing, but if you don't have the agent also validating its work, then you're still bottlenecked on like, can you go review all that code, right? So my view is that we need to unblock those productivity loops from like humans having to prompt and humans having to like manually validate all the work. And so if we can like rebuild systems to let the agent like be default useful, we'll start unlocking hockey sticks. Unfortunately, I don't think that's gonna be binary. I think it's gonna be very dependent on what you're building, right?

1:11:51So like, I would imagine that like next year, if you're a startup and you're building a new pieces, like some new app or something, it'll be possible for you to set it up on a stack where agents are much more self-sufficient than not. But now let's say, I don't know, you mess into SAP, right? Let's say you work in SAP. They have many complex systems and they're not gonna be able to just get the agent to be self-sufficient overnight in those systems. So they're gonna have to slowly maybe replace systems or update systems to allow the agent to handle more of the work end to end. And so basically my sort of long answer to your question, maybe boring answer, is that I think starting next year, we're going to see like early adopters, like starting to like hockey stick their productivity.

1:12:33And then over the years that follow, we're going to see larger and larger companies like hockey stick that productivity. And then somewhere in that fuzzy middle is like when that hockey sticking will be like flowing back into the AI labs. And that's when we'll basically be at the AGI tier. I love this answer. It's very practical. and it's something that comes up a lot on this podcast just like the time to reveal all the things AI is doing is really annoying and a big bottleneck. I love that you're working on this because it's one thing to just make coding much more efficient and do that for people.

1:13:03It's another to take care of that final step of, okay, is this actually great? And that's so interesting that your sense is that's the limiting factor. It comes back to your earlier point of even if AI did not advance anymore, we have so much more potential to unlock as we learn to use it more effectively. So that is a really unique answer. I haven't heard that perspective on what is the big unlock. Human typing speeds review basically what AI is doing for us. So good. Okay. Alexander, we covered a lot of ground. Is there anything that we haven't covered? Is there anything you wanted to share, maybe double down on before we get to our very exciting lightning round?

1:13:43I think one thing is that the Codex team is growing. and as I was just saying, we're still somewhat limited by human thinking speed and human typing speed. We're working on it. So if you're an engineer or a salesperson or I'm hiring for product, a product person, please hit us up. I'm not sure the best way to give contact info, but I guess you can go to our jobs page or do they have contact for you? Actually, do listeners have contact for you? Before they send me like, hey, I want to apply to Codex. I do have a contact form at LennyRichardsy.com. I'm afraid of all the amazing people that are going to ping me.

1:14:18But there we go. We could try that. Let's see how that works. Yeah, or another, maybe an easier version. We can edit all that out or give up to you. But yeah, or I would just say you can drop us a DM. For example, I'm Emberrico on Twitter and hit me up if you're interested in joining the team. What a dream job for so many people. What's a sign they, I don't know, what's like a way to filter people a little bit so they're not flooding your inbox. So specifically, if you want to join the Codex team, then you need to be a technical person who uses these tools. And I think I would just ask yourself the question, hey, let's say, you know, I were to join OpenAI and work on Codex over the next six months, you know, and crush it.

1:14:58What does the life of a software engineer look like then? And I think if you have an opinion on that, you should apply. And if you don't have an opinion on that and have to think about it first, you know, depending on how long you think about it, I guess that would be the filter, right? Like, I think there's a lot of people thinking about the space. And so we're very interested in folks who sort of have already been thinking about, like, what the future should look like with agents. And, like, we don't have to agree on where we're going. But I think we want people who, like, are very passionate about the topic, I guess.

1:15:28It's very rare to be working on a product that has this much impact and is at such a bleeding edge of where it's possible. It's what a cool role for the right person. And so it's awesome that you have an opening. And this audience is a really good fit, potentially, for that role. So I hope we find someone that would be incredible. With that, we've reached our very exciting lightning round. I've got five questions for you, Alexander. Are you ready? I don't know what these are, but I'm excited. Let's do it. They're the same questions ask everyone except for the last one. So probably not a surprise.

1:16:03I should probably make them more often a surprise. Okay, first question. What are a couple books that you recommend most to other people? Two or three books that come to mind. I have been reading a lot of science fiction recently. And I'm sure this has been recommended before, but The Culture. I think it's Ian Banks is the name of the author. Part of why I love it is because it's like basically relatively recent writing about a future with AI. But it's an optimistic future with AI. And I think, you know, a lot of sci-fi is like fairly dystopian. but this is like people sort of the joke at least on the culture subreddit is that let me see if I can get this right it is a like space communist utopia or like I think it's a gay space communist utopia and I just think it's really fun to think about like to use the culture as a way to think about like what kind of world can we usher in and like what decisions can we make today to help usher in that world I don't think anyone's recommended that I know you're reading, you mentioned before I started recording Lord of the Rings right now.

1:17:08If you want another AI-ish sci-fi book, have you read Fire Upon the Deep? No, I haven't. Okay. It's incredibly good. It's like a sci-fi space opera sort of epic tale with super intelligence. Cool. Yeah. Someone, mostly not optimistic, but somewhat optimistic. Okay. Next question. is there a favorite recent movie or tv show that you've really enjoyed yeah there's an anime called jujutsu kaisen which i really like um again it's got a kind of a slightly dark topic of like demons um but what i love about it is that the hero is really nice and i think there's this new wave of like anime and cartoons where the protagonists are really friendly and like people who care about the world rather than being like sort of like if you look at like some older anime like that started the genre like you know there's like Evangelion or Akira and like those characters the protagonists are like deeply flawed like quite unhappy um they didn't start the genre but it was like a trend for a while to sort of poke fun at the idea that in these in these cartoons the protagonist was very young but being given a ridiculous amount of responsibility to like save the world and so there was kind of a wave of like uh content that was like critiquing this by making the character like basically go through like serious like mental issues in the middle of the show um and i'm not saying this is better but at least it's quite fun to have like these like really positive protagonists are just trying to help everyone around them i love how much we're learning about your uh personality hearing these recommendations uh yeah nice protagonists optimistic futures i think you know if you don't believe it you can't will it into existence so you need a balance this is your training data is there a product you recently discovered you really love could be an app could be some clothing could be some kitchen gadget tech gadget a hat yeah so i have been like quite into uh you know combustion engines um and cars actually the reason I came to America initially was because I wanted to work on like US aircraft.

1:19:22But you know, now I work in software. And so for the longest time, I've basically only had like quite old sports cars, old just because they were more affordable. And then recently, we got a Tesla instead. And I have to say that I find the Tesla software like quite inspiring. In particular, it has the the self-driving feature. And, you know, I've mentioned a few times like today, like I think it's really interesting to think about how to build like mixed initiative software that makes you feel maximally empowered as a human, maximally in control, but yet you're getting a lot of help. And I think they did a really good job with enabling sort of the car to drive itself, but all these different ways that you can adjust what it's doing without turning off the self-driving.

1:20:08So like you can accelerate, you know, it'll like listen to that. You can turn a knob to change its speed. You can steer slightly. I think it's actually a masterclass in like building an agent that still leaves the human in control. This reminds me of Nick Turley's whole mantra was, are we maximally accelerated? Yeah. It feels like it's completely infiltrated everything at OpenAI, which makes sense. That tracks. Two more questions. Do you have a life motto that you often think about and come back to in work or in life that's been helpful? I don't know if I have a life motto, but maybe I can tell you about the number one value, company value from my startup.

1:20:45Love it. Which is still something that sticks with me, which is to be kind and candid. That tracks. Kind and candid. Wow. Yeah, and we had to put them together because we as founders realized that we often would be nice and it wasn't actually the right thing to do. We would like delay the difficult conversations and we were not candid. And so every time we would like remind ourselves of this motto and then we would become more candid. And then six months later, we would realize that we were in fact not candid six months ago and we needed to be even more candid. So then the question is like, OK, like how how should we be candid?

1:21:23It's like, OK, well, let's let's think of being candid as an act of kindness, but also think of that both in terms of doing it and willing ourselves to do it, but also in terms of how we frame it to people. That is a beautiful way of summarizing how to how to lead well. What's the book about challenge directly, but care deeply? Radical candor. Yeah, right. Yeah. So it's like another way of thinking about radical candor. Okay. Last question. I was looking up your last name just like, hey, what's the story here? So your last name is Embirikos. And I was talking to JATGPT. And it told me the most famous individuals with the surname are the influential Greek poet and psychoanalyst Andreas Embirikos and his relative, the wealthy shipping magnate and art collector George M.

1:22:08Bjerikos. So the question is, which of these two do you most identify with? The Greek poet and psychoanalyst or the wealthy shipping magnate and art collector? I think it's going to have to be the poet because he loved the island that our family's from. Wait, you know those people. Okay, this is not news to you. Well, I mean, it's an enormous family, but it's like Greek. So, you know, these big families, everyone's your uncle. you know what i mean like my mother's malaysian and also like everyone is my uncle or aunt in malaysia too if that makes sense yeah but yeah he he loved this island that the family sort of like initiated from i believe i don't actually know where that's shipping magnate lived i think it was new york or something but anyway we all came from this island called andros um which is a really beautiful place and it's like there's more like livestock there than than humans uh not too many tourists go there.

1:23:04But I think he like part of what I think is really cool is like he published a lot and a lot of his writing is about like the beauty of that island, which I think is super cool. Wow, that was an amazing answer. Two more questions. Where can folks find you if they want to follow you online and, you know, maybe reach out and then how can listeners be useful to you? I'm one of those people who has social media only for the purposes of having work. You know, my phone turns black and white at like 9 p.m. at night. But yeah, so Twitter or X at NB Rico. And yeah, if you post in r slash codex, I'll probably see it.

1:23:37So you can go there. How can listeners be useful? I would say, please try codex. Please share feedback. Let us know what to improve. We pay a ton of attention to feedback. I think it's like, honestly, like the growth has been amazing, but it's still very early times. So we still pay a lot of attention and hope to do so forever. And also I would say, if you're interested in working on the future of coding agents and then agents generally, then please apply to our job site and or message me in those social media places. Alexander, this was awesome. I always love meeting people working on AI because it always feels like this very, I don't know, sterile, scary, mysterious thing.

1:24:20And then you meet the people building these tools and they're always just so awesome. And you especially, just so nice. And as you like the examples you shared, optimism and kindness, you know, this is what we want to be. These are the kinds of people we want to be building these tools that are going to drive the future. So I'm I'm really thankful that you did this. I'm grateful to have met you. And thank you so much for being here. Yeah. Thanks so much for having me. This is fun. Thank you so much for listening. If you found this valuable, you can subscribe to the show on Apple Podcasts, Spotify or your favorite podcast app.

1:24:56Also, please consider giving us a rating or leaving a review, as that really helps other listeners find the podcast. You can find all past episodes or learn more about the show at Lenny's podcast dot com. See you in the next episode.