How OpenAI Built Its Coding Agent

21:13And we're going to have simple model loop to help write it and so forth. But we just cut scope and like ship the really early research preview. So there's a much of that expected feedback. Now one of the things that really surprised me is that there was one feature that we didn't expect people to use. And in fact, we used it so little internally that it just had a bunch of bugs we hadn't caught before releasing. And that was multi turn. So basically, like I was saying, like we, and we told our alpha users, I guess to do this, we basically said, hey, just like repropt, like far from any prompts, and like maybe you can go back and forth.

21:43It turns out that if you go back and forth more than once, so you do like you return to all right, right? The product was completely broken and that we were not like correctly, like carrying over the diffs from the prior steps. And it's just a lack of context, but persistent context essentially after the third time. Exactly. And this is just like a plain the terministic bug. It's not like a weird model behavior thing. It's just like we implemented the code wrong because no one everybody just got to door four basically. Yeah, yeah, yeah, and so for me that was really interesting to see that like people had his intuition for how they wanted to use the product and that wasn't like the reprompted tuition It was the hey like I'm gonna get like this main thing and then I kind of want to you know get babies It that across the way to like actually landing it without it of attaching my computer and that like we kind of knew that might be a thing but it was much more of a thing than we expected.

22:30And you think that's basically because internally, opening up employees are sophisticated enough to know that you do all this upfront context building work for the agent to try to get as much as you can in the first turn. But a user, once you made it fully cloud -connected, so the marginal cost of doing kicking off an agent was so low that they just quickly got to the third, fourth turn without too much thinking. It's funny. I almost feel like, in a way, we're less sophisticated because we understand too much about the models. Right. like their expectations are lower than the average. Yeah, because we're like, oh, you know, this is a reasoning model, like works great, like, especially when you like prompt it in this way.

23:03Right. And then like, you know, folks outside open air, just like, why does it not? This is how I want to use it. This thing is like basically, like, you know, obviously it's not easy yet, but it's like, oh, is it like, it's this like super smart model. I can't just like, all I want you to, you wrote this amazing PR. I just want you to change one thing. Why can't you do it? Right. And so, you know, obviously the bug that I mentioned we fixed, but that's something now we're thinking more about, like, okay, how do we enable that kind of multi -tune interaction? How do we make it faster as well? Like container startup, just for example, takes time.

23:31Yep. And there's a lot of optimization we can do. But for now, if you need to incur a full container startup to like change one variable name, that's super frustrating. So there's a bunch of things like that that we want to improve around. Okay. That iteration loop. Do you think that the, is the arc of product development of agents such that you think the shape of the industry will be more and more Apple S square? where you'd go, well, cold starts are a problem for containers, because that's a really terrible user experience. So instead of like outsourcing containers to some third party vendor, we're then lying down for providing us cold start.

24:05We're just gonna bring this all in -house. Is the most magical experience gonna be the full stack and the integrated experience where all the dependencies, all the middleware is all done in -house? Or do you think that this is going to be more Android -esque where you guys, a company like OpenEI has an opinionated experience, owns the agent, sort of interface, but everything else is mostly like a collection of different tools orchestrated by different vendors. It's a great question. I think it's going to be a bit of both maybe an annoying answer, but or rather where do you think the line? Where would you build versus buy?

24:38Right. No, totally. So I think it's actually more like for whom or who will use what? I think that the average user or maybe like the new startup that is building with agents from scratch will just do things in a very different way. And they'll basically have a bunch of agents with this computer environment that scales really well that has all the credentials they need, but is also protected with the right forms of sandboxing applied at the right times, with the right monitors on all network egress and all this stuff. And maybe this kind of computer, I think of it as a laptop, although it's not, is actually the thing that many agents use.

25:14And it contains many tools, not just the terminal, but it has a browser and it has whatever, right? You know, API access and it's like, it gets piped the right credentials at the right time and so like you kind of think of yourself when you're hiring like your new agent for your new startup what you might do before you bring on a co -founder even you know. Right. You think of yourself as just like setting up that environment and it's and you're just getting like this like fairly generalist employee that can code. Right. Like if you think of codex right now it's like it basically takes prompts and turns them into messages and dips and that's like not general.

25:44I can't be like, oh yeah, hey, can you move engineering sink? It's a 30 minutes later because I have a conflict. But like, my real software engineer can do that, right? A real software engineer can go peruse any source of data, can find out they don't have potential. I mean, they can just use the internet. Right. So I think we'll get towards that. And I think we'll be able to build a really nice managed system for that. That lets you use more capabilities safely. And with some product pushes from us on how to make the most of it. So for example, recently we shipped best of N and like, you know, it's very simple feature.

26:17But in our minds is like kind of just the beginning of like taking advantage of the fact that we're not running under laptops. So we can explore like four versions of the same. Right. And then you have is what there's some evaluator model of the best of actually the evaluator is the human right now. Ah, like, you know, you know, the roadmap is like fairly obvious. If you just imagine like what we're thinking. They just throw like O3 pro. So right. So so yeah. So there's that. However, also, you know, the majority maybe of valuable code is actually written by enterprises who rightly so are like really locked down all their IP and their code.

26:49Right. And so something we've been thinking about as well as like how do we meet these enterprises in a way that we can like provide value to them as well in a way that they like. Right. And so I think what we're going to get towards is like there's this like default way of working with things. And then we'll basically have some flavor of on -prem or bring your own compute that we support, where it's like, hey, here are all the things we manage for you when you use our compute. If you're gonna use your compute, then we can work with you and provide you as much of a harness as possible to automate things, but you're gonna have to wanna manage that compute for the agent, basically, that environment for the agent, here the tools it should have, here's how you should sandbox it, or bring your own R -back or whatever you're doing.

27:29And so like the codex CLI, which we haven't talked much about, but in my mind, like the codex CLI might evolve into that where it's like, hey, if you want to like run the agent loop in your own environment, then we can help you do that and you can use something that's an evolution of the CLI. I think you should, what let's talk about CLI versus the interface. What are the two differences between codex and codex CLI? Yeah, so the place where I want this to get to is just like there's GitHub, right? And GitHub has a website and a CLI and a mobile app and it's not confusing. Right now it's a little bit confusing in that they are just completely distinct experiences.

28:01We have Codex in ChatBT, which is an interface that you can write a prompt and then we run Codex in the Cloud and then you get back a different answer or an answer to your question. Then we have the Codex CLI and that's a completely distinct experience with all the same ideas which is basically you can run this tool in your terminal and we'll hit our model from VAPI. And basically this agent will work locally with you in your computer. So right now I kind of think of it as you delegate to Codex and chat SPT remotely. Right. And then you pair with Codex CLI on your computer. And what is the moment where the CLI journey integrates into the cloud workflow?

28:37Yeah. And so where I think we want this to go is there's just like one idea of Codex and it's just like where do you want it working? Right. And there's going to be times where it's just simply easier. Like you don't have to set up an environment when it runs locally. So maybe if you're trying something for the first time, that's the prototype. Yeah, just whatever. Yeah, or like you don't even know if you like Codex yet. You're just a new user. Like maybe you just want to use the CLI or something. Right. And then maybe then you're using it and you realize, hey, I want all this cool parallelization and all this stuff.

29:04Like you have this run in the cloud and you set up the cloud environment. And then from then on, you should still be able to interface with that in the CLI if you want. except now it's running in cloud environments, so it's more powerful. Yeah, so I think we kind of want to construct that and bring these things together, but obviously we're in this temporary state of completely disnaked. Yeah, I think so it's interesting hearing you talk about how there was this evolution from like the moment where you were using the tool as this like very precious first iteration tool where you put a ton of sort of weight and context into it hoping to get back a really useful answer the first time around.

29:40And then there was not a moment where you'd like, actually this is more like a slot machine because other modalities in AI have played out very similarly. So this was the case with image models, for example, right? Two years ago, people were trying really hard to get the first version of image model, which are like GANS, you know, just generally about adversarial networks, even pre -lexed able to fusion, to be to produce useful sort of image coherent images. And they just weren't there, right? They would produce these like artistic renders, which were great for like artistic exploration, but they weren't sort of useful because they didn't have the concrete coherence of a graphic design, out, you know, a piece of graphic design, for example.

30:13And then if you remember the first, like, era of diffusion models, like Dolly and Midjourney1, they started to get more coherent, but there was this trick that a lot of product people started using. And David from Midjourney was one of the first to do this, where he added four generations in the Discord bot not one. Because the idea was, the insight was like, this is a slot machine. This is a stochastic process. And you never really know which one the user's going to like best, especially for a super subjective domain, like art and like images. And so human preferences is super subjective. So let's just give them all four and we'll figure out which one they like.

30:49Now over time, if you collect enough human preference, you can kind of nudge the distribution to be more aesthetically pleasing or you can nudge it to be more like better typography or whatever. You can nudge these distributions, but by and large, that this day, the best UIs for image models are still ones that give you like for outputs if not more and then allow the user to select the best of N. You know, and for a long time, people were like, that's going to work for these super creative domains where like verifiability or accuracy is not an issue. Like images like video, like music, audio. But what's surprising is you're actually describing that same for pretty verifiable domain like coding.

31:26Because at the end of the day, it sounds like there's still enough stochasticity in the sampling of a model, even as it gets better at reasoning. That makes sense to try, use it like a best of N machine. And this is led to the, I guess, a popular set of critiques against reasoning models that like, they're not, you know, RL from verifiable rewards doesn't actually introduce new capabilities. It's just really good at pulling out capabilities that are already in the model. It's good really good at sampling. Do you think that this is just an interim awkward phase? We're like, yes, the best of N is better at getting sort of the right answer from the existing model.

32:03it's not adding new capabilities yet, but where we are going, are you from now? There will be actually new capabilities that come from running Verify. But you know, Aril on all the codex usage that is about to happen from users. Where do you, how bitter lesson build basically? Are you roughly on that dimension? Yeah, I mean, basically, I think an unsolved problem, and it's both a research on a product problem is like, how do we steer agents? Right. What that are working independently. And you know, you're talking, you mentioned like, hey, like, is best of end there too? to, you know, so the model has more shots on goal, basically, to, you know, to sample correctly.

32:36And I think, you know, that might be part of it, but actually one of the things we've learned working on codecs is that, well, the human also doesn't know what they want. Right. Right. And, you know, so if I ask you to fix a bug, like there might actually be four reasonable ways to fix that bug with sort of different architecture implications. And I might, I haven't explored the solution space myself. That's why I'm delegating this. So I kind of want to know what the ways are. and then I want to, you know, maybe I would pick the one that the model thinks is best to, but it's like helpful for me to see, like maybe that sucks in some way.

33:08Yeah, but it's helpful for me to see the other ways that have like larger trade offs, so then be confident in the right one. Yeah. So that's for like fixing a bug, which is like a very verifiable type thing. If I ask you a model to like, you know, the classic example, implement TikTok till or something, right, you know, I might not know what I want either, like maybe there's different styles and different like approaches you could take at various steps along the way. Right. Right. And so, you know, it's kind of funny. You were talking about, you know, generating four emitters and CMOS in the grid.

33:35And like in my mind, like for a front end change, you could totally imagine a UI for different. Where it's like the model of some work. And then we like run the stuff. We take, you know, the model in its environment runs the app and then like takes four screenshots. And you actually just like have this like similar curatorial UI. Right. That's like just pick the one you like most. We had a Rick Rubin on the podcast a few weeks ago and Rick's a legendary music producer. And he recently used a plot code to create a new vibe coding book. And so we were talking to him about how he, what's his, what was his observation about how creating with AI?

34:06How is it creating with AI? Code Gen tools different from creating music. And he was like, oh no, it's the same. It's like going into a studio and he was talking about this story about, you know, what, going into the studio with Johnny Cash and watching Johnny just pick up a guitar and start jamming. And often the process of creating a great song is you just pick up a tool like a guitar, and then you just do four different iterations in completely different directions, and then you usually have a creative partner like a producer or somebody going, not that one sucked, go this way. And it's that constant sort of best -of -end process in creating, like in the process of creating music that often results in the best output.

34:45And often the quality of the end song is a determinant of the taste decisions you make along the tree of best event. And so what's giving me hope about here and you talk about it is if you read the hacker news thread, for example, when you guys launched Codex somewhere down, I forget, I'm about halfway down the page was like a tree of discussions about how does this mean coding is going to get much less fun because all of the interesting parts are being delegated to the agent and all the humans having to do now is just sit and review. But actually what you're saying is there parts, the workflow where you get to almost entirely the offload the plumbing parts of software engineering and focus on the taste exploration, which is sometimes the most fun part of software engineering.

35:28Is right? You're getting a front end UX or even when you're speccing out like a really great schema for a database, you know, some of the most fun times I've had is when I'm sitting with an infer engineer and we're spec - spec - spec - spec - spec - spec - spec - spec - spec - spec - spec - spec - spec - spec - with, you know, a bunch of pseudoconieralized - actually, that's not the right one, but it gave you an insight that then allows you to try another schema out. Is that where we think we go? Is that the silver lining or are we actually destined for World War? We're just all reviewing PRs and all the creative parts of software are gone.

35:58Totally. Yeah. So this is just an opinion here, but I think you're right in that coding might be a little more painful for some number of months because you have to do things like environment setup. Right. These are the teenagers. Yeah. These are the teenagers. I think like to be real, like that's true. Maybe you don't get to write as much of like the code yourself right now. But I think we will get to that more exciting place pretty quickly because turns out, environment setup is probably something that an agent can also massively help with. Right. And we can close that loop where you're not comparing four deaths or something like that, but we've figured out the interaction model with the agent, so you're making decisions in a way that feels more like talking to another human who's just really smart and fast.

36:41And then also that you're making these decisions not based on reading raw code in the case of and at least, but maybe you're making decisions based on the outcomes. In the case of frontend, you're just choosing screenshots or clicking around a preview. Or if it's back in, maybe there's some tests you agreed on and you're just looking at test outputs to decide. The other thing that's interesting is that, well, if you were to guess, I'll give you a few things that people use Codex for and I'm curious what your guess would be the most the biggest ones are. are like, let's say it's like building you features, asking questions, planning, debugging, and fixing bugs.

37:18Like, what do you think people would use codecs for more? I think they would like to use it for debugging. They probably aren't using it yet for that because there's often my knee jerk when I'm using an agent is that it just doesn't have enough context to fix. For routine tasks, like, you know, some piece of boilerplate react is broken, like debugging you're totally fine. But I find I use it more and more for well -defined, well -scoped, well -contained tasks like create this new UI element that does blah or a refactor that's like where the atomic unit is very well constrained. But I'm curious what are you actually seeing?

37:55Yeah, I mean, so my intuition is that people would use codecs for fixing bugs a lot because you know, bugs are somewhat well -defined, you know, you can kind of tell if it's fixed. You might even have like some logging data or telemetry data that you could just paste into the model is excellent. If it's the end, there's some of our earliest delight moments where we're like dumping in the stack trace and then just figure it out. But actually by far the thing that people use codecs for is building new features. And I don't know. That was just like slightly surprising to me because that is some of the most fun stuff to do.

38:27And if you read like you know, blog posts, like folks who are using codecs in that way, it does look like they're having quite a lot of fun because it's just the sheer speed they're experiencing. The speed to prototyping is basically collapsed completely with something like Codex. And broadly speaking, this is the explosion of vibe coding. I think that makes sense to me because when you're prototyping a new idea, I find the most rewarding is when you actually, if you can get to the first draft really fast and then kind of iterate from there, that's fun. Sometimes the worst is when you have an idea, you kind of want to see it and then you lose steam between like firing up your IDE and seeing the first version of it.

39:05right compiling. This is why hackathons have proven to be this like I think magical sort of you know type of event where you get people together and commit to getting over the hump of the first prototype. But in many ways I think something like codex or you know broadly speaking really good coding agents have turned every day into a hackathon because they've collapsed the energy you need to get over the hump of all the plumbing, all the environment setup to take test an idea. When I was at Discord, we used to have this ritual across the company that was an annual tradition called Hack Week. And some of the entire company would just stop for like a week.

39:44And it wasn't just engineering. It was product marketing, sales, ops, the entire company could hack on anything they wanted. And some of the most enduring and popular features that made it into production, the company over the years came from Hackathon projects. And it begs the question of, well, if there's a whole team called the product an engineering team whose job it is to ship great features, why did it take this like special thing called a hack week to produce such great features. And there is something about when you reduce the the cost of prototyping new ideas, you end up getting things that don't make it to the usual PRD flow.

40:18And it sounds like that's what a lot of users are using codex for now is like that first to reduce the time to magic essentially the time to first prototype. Let's change tack for it because there's this elephant in the room, right, which is that if, you know, Mark famously wrote an op -ed in 2011 to 2012, which is like software is eating the world. And after I saw that chart, you mentioned of the GitHub merge success rates of AI agents starting 35 days ago, hitting 80%. And as of this morning, the volume being 350 ,000, it sounds like AI is eating software engineering. Does it even make sense to study software engineering anymore to get a CS degree if you're a freshman at Stanford today, or just a freshman grad, you know, somebody graduating high school and you're broadly interested in software.

41:00Does it even make sense? The major in CS? So my take is that it's two things. First of all, I think still a great time to major in CS. I think there's like going to be so much more software created and therefore so much more software engineers needed. But I also think figure out how to be using AI constantly while you do it and hopefully you're at a university that's like very forward leaning and so they're kind of embracing it. I hear about policies like, hey, use AI as much as you want, but you just have to say how you use AI as part of your assignment. Right. It's great. Right. If you're on a place where the main place where I would be worried if I was a student right now as if I was studying CS and my college didn't allow the use of any AI, because then I would just feel like I'm like falling behind.

41:41If you went to college but you were only allowed to write assembly and you could not write C back in the day, that would just be deeply worrying, I think. But yeah, my take is, you were talking about this, right? Like we can do so many more things now. And we hear this from customers too, and from users, they're just like, hey, I would never have bothered doing this before, but I threw the idea into Codex, just for the sake of it. And I do this all the time, and a lot of the time I do that, and then I see that output. And I'm like, I just still don't really care to do this. But then sometimes, this thing that they would not have even bothered doing, Codex either straight shots it, or gets it to 90%, and they're like, you know what, I'm excited enough to do the last 10 % here to get this merged.

42:21And then the thing that would never have happened now happens. You know, some of my favorite examples, like internally or like when people build like new internal tools that accelerate the rest of their team. And like it's the kind of thing like someone's complaining and slacking. Like I wish we had this tool to like, I don't know, look at these logs in a better way. And right. No, you know, it just can't be bothered. Everyone's too busy. And then you right now you have this like great parser. Right. So I think that there are so many places where we could use software and that software could be more personalized to small groups or even individuals that we just are missing out on.

42:50And so, yeah, now I believe that like with just the acceleration we're seeing in software development, I think we'll have many more of those tools existing and they'll be much cheaper to maintain as well. Like that's the thing we're on the tip of now as well where you're starting to see AI agents getting plugged into like GitHub or like Slack or linear as the agents feature. And I think that that will make it much more efficient to actually have some like app out there and running. Right. Similarly, you know, even we're seeing there's like, this is not codex, but we're seeing products out there that will like write the app for you and then deploy it for you as well.

43:23Right. And so it's just like all in one full stack basically. Yeah. So it's just like it's long story short. It's much easier. I think to bail build software to deploy that software and to maintain it. I think that's just going to we're just at the beginning of this change. Let's talk about that. It's been 35 days now. As a product leader, you've had a chance to actually see the best -late plans rarely survive contact with reality. So now, what priors have you updated the most and what comes next? Where does Codex go in the V2? Because this was just a research preview. But what are the biggest improvements and what's the shape of the product in the future?

43:56Yeah. So I think there's one sort of conviction that is deepened and then one prior that's like being slightly updated. So the conviction that deepened is that this form factor of an agent working on its own computer in the cloud is the future and is incredibly powerful and worth figuring out how to get right. So we're continuing to invest in making that environment set up faster and making performance just way better. First time users on boardying. Yeah, first time users on boardying, but also just once you're running, things should just be faster. Sure. Speed is actually always the underrated feature.

44:26And is that the biggest gains in speed you think on a come from doing things like model distillation or do you think that comes from just better orchestration of tools? Where do you think it comes? Honestly, I think the low -hanging fruit is just plain old deterministic, like, dev -on -se type stuff. Okay. You know, like right now, we clone your repo every time you do a task, even if it's a follow -up. And then we run your set -up scripts from scratch every time. And so if you have a large repo and a lot of dependencies to install, like that thing is slow. Okay. You know, we start with gashing.

44:53Yeah, we can just like, we can fix these things. Yeah. And again, like, I love that we didn't, I love that we shipped without those things. Yeah. To be zero. Yeah, exactly. So there's like that. And I think, like I mentioned best event, And I think thinking about how to make the most, like basically, how do we spend more compute for you on your behalf is like very exciting. And then how do we bring this closer to the tools you work in? For me, the interface and chat chat, it's actually very functional, but it's not where developers go when they want to write code. Where do you go when you want to write code?

45:23If you're either your terminal or your IDE, similarly, where do you go when you want to triage issues? Well, you go to your Azure Manager, right? And so forth. So I think we want to bring it much closer to the tools people work in. And if actually, you know, the goal is to get to an agent that is like, basically a teammate and it's like seeing what's going on your team and like picking stuff up for you. Right. That's okay. Is this just, is Codex just going to be a slack teammate? I can just ping and I interact and slap it. Yeah. It should just like, I kind of think of it as like, it's just, it should be sort of a ubiquitous teammate.

45:51Right. You know, it's just in your tools. In the tools you want it to be in at least. Right. You know, and we'll start very gentle, just like, hey, you decide when codex does work and then over time we'll figure out how to like kind of like more proactively chime in and you know we had a jam about this recently like you know that it's kind of an interesting point like I don't think we want it to proactively like DM you all the time every five minutes and something happens so I think there'll be some evolution of tools where we come up with like if you if anyone here has hit play video games you know there's always like press X to like and it like if your next to a door it opens a door if you like our next to some object to pick up the object.

46:27It's a conceptual action. Yes, right. Yeah. Convectoral correctiveness. Yeah. We'd wait for the hint that you want to do something and then jump in. Yeah. Right. And this is kind of like when we're getting to like interactive agents, I think that's just like a big open area. But it's like how do we have agents who understand what your team is trying to do and respond to like stuff in your team work spaces. Right. And then how do we have an agent that understands what you are trying to do? And it's almost like this agent is like both in all your tools, but like sitting next to you while you're working on your computer and like kind of just being like, oh yeah, like I can help you here.

46:54Right. So that's like actually the conviction that is deepened right we're like yes all of this works when you give it It's own computer and we need to figure out how to create this infrastructure for ecosystem and operations and like make that safe and so forth Then the other thing though that is bit of an update is like just thinking about how people Like learn to use these tools I think right now there's some things that are pretty clunky Obviously we've talked a lot about environments set up. I think also some of the things that you know you have to do like like updating agent .mp is very manual and you have to like commit to your repo to get that context of the agent.

47:26And so for me, I just think a lot now about like, okay, how do we make this like way easier to try? I reduced the cognitive burden of the onboarding, fewer decisions to get to the magic. Yeah, he's okay, got it. What has it changed most about research and the frontier of where frontier models are going? Right? As in your mind, does this mean that is the efficacy of how good Codex is as a post -strain version of O3 Pro at using tools that, like, lugging to this workflow, does it make you go, well, it just makes sense to, for an unlimited amount now of compute on post -straining models to get better and better at being autonomous coding agents.

48:05Or do you think there's some marginal plateau point at which you go, you know, after this point, there's not really much the user's getting from better and better tool usage. You know, how does this change the trajectory your progress might come to the front to your research. Yeah, that's a really interesting question. I don't, I definitely don't know if I have the answers to this. But what I can say is that one of the best parts of doing, you know, an optimized version of O3 was that we got to make a bunch of like hybrid research product decisions very quickly. And I think that is incredibly exciting for thinking about how to make something useful.

48:38So you know, if I imagine we would have had this idea of like, you know, it's like really important that the agent knows how to write really good like PR descriptions and, you know, tests code in a certain way that's used to working in varied environments. And you know, when it runs some tests, it doesn't just tell you that it did, but it cites deterministically in the logs that outputs so you can verify that yourself. Those are a bunch of product ideas, really. And they're not, like those ideas I just mentioned are not higher model intelligence, nor even really a higher ability to call the right tools.

49:07It's just this understanding that I like into the first few years of job experience of a software engineer. You start, you have a three -seq this incredibly precotious college grad, like they're smart, but like doesn't actually know how to be a software engineer, just it's not a code. And like this is some like transfer, so it kind of knows a bit of software engineering. And then like that's fine, but you could make it way more useful for, you know, the human trying to use the agent if it has those first few years of job experience. Right. So I think that there's no reason that those that knowledge couldn't be infused into the model exactly.

49:38Exactly. I've dreamed in some of these, but I think that having the freedom to like go and like exploit these ideas like relatively cheaply and see what sticks and what doesn't. It is really powerful. So, frankly, like, I don't really know to what extent it makes sense to like have like a bunch of custom post -trains for like absolutely everything that matters, but I think for something as important as like coding to us, I think that I think we're willing to say like, hey, for coding, we really care about this. Let's just do everything we can to like have the best product. So like, we actually did a similar thing with GPT 4 .1 where we basically were getting a bunch of feedback developers, we said, okay, let's go talk to a bunch of developers, like make custom e -vals for them, right?

50:15Deeply understand, like what are models great at? What they want to get better at? And then we release the custom model, right? And then the goal should always be, okay, whenever we do this, like we have for one, okay, the next version of our, like, sort of general model. Should just integrate that. Yeah, should integrate everything. Right, yeah. We have friends who are different levels of AGI build. did working on Codex update your priors on, you know, 2027? Okay, so I'm very agi -filled. I'm aware of my, my like, slightly joking, or, but I can't tell if I'm joking 100 % take is that if you took a model today and ran it in the right loop, we're basically there.

50:56Would it have rights? That's the question I sometimes wonder. And should they be able to turn themselves off and go take a vacation if they want? Yeah, so you know, that's kind of where I have are you are you pro neighbor rights for O3 pro? I am pro thinking about it You know, I mean like I don't think we're at a point where it's obvious, but I It sounds kind of crazy, but I feel like it's a question worth considering every now and then and not or more concrete How far are we from full recursive self -improvement? Okay, okay, sorry Basically, I think working on codex made it very clear How we can have agents just like omnipresent in our lives being incredibly useful?

51:32because what I realized is that obviously we need to do a lot of model improvement, but I also saw how this like just concretely a lot of like normal product work to do, right, to set them up in the right way, and then that normal product work will then like pull the models into, you know, into being more and more useful. So I think like by 2027, like agents will just be absolutely ubiquitous in the workplace. I think in personal life it might be a little bit slower because in personal life there's less of these like constant pipes of like signals of things to respond to. The reason this matters is that if you think of Chatch B .T., you just have this input box, right?

52:06And most people, including myself, probably use it for like 1 % of the things that I could use it for, because I just don't even know to use it in that way, or I don't profit, right? Right. The intention just isn't there yet. Yeah. But like, it's similar. Like, if imagine you hire a teammate and then the only time they do work is if you specifically tell them to do a task, then they would just be very underutilized, right? But what makes a great teammate great is that they, you kind of tell them what their job is and they just start responding for actors. It's self -starters. Yeah. So I think that is the big unlock for agents at work because there's like streams you can subscribe into, like, you know, your communications tool.

52:38Right. And in personal life, I think that might be a bit slower, but we'll see. Do you think that what percentage of all GitHub beards? Do you think we'll be written by an AI agent 12 months from now? That's a really tough question. I sort of changed my mind every time I answered. So maybe a slight cop out and then curious for your answer to would be that there will be teams for whom 90 % of their peers are written by agents. But I don't know how quickly that will spread. This is the common thing with AI. We live on the bubble, you could call it on the cutting edge. And so we're just adopting everything rapidly, but then it takes a while to defuse or do things.

53:19But I think the cutting edge will be at 90 % on teams. Right. No, I think that's right. There's I don't think people often talk about the coding economy as one homogenous economy in the realities. There's multiple sub economies, but they're at least two big economies, which is there's the Fulacve a better word, you know, there's the digital native companies, right? These are technology companies usually born in the post internet era where they grew up Where either the founders or most the vast majority of the team has grown up natively understanding how to do modern software development. The default assumptions when a code base is initialized is that it's gonna be, you're gonna use Git for version management.

53:57There's going to be branching, there's going to be good review process and so on. Like sort of modern software types. And then there's the vast majority of actually the world's mission critical code, which we talked about earlier is Fortran, Coball, like running on -prem in these massive ETL systems like in Virginia or in parts of Europe that were set up in post World War II and the Cold War, with a default assumption that everything had to be locked down. Often, these code bases are running big parts of critical infrastructure like the railway system or the air traffic control system. So the very high impact and high stakes code, they're not modernized whatsoever, and they're constantly rotting because of that technical debt.

54:43And I think one of the most exciting things is that But the one time migration costs to modernize these code bases now has collapsed precipitously because agents can do so much of the plumbing work that typically would hire some system integrator or Accenture Deloitte for a 10 -year contract where they'd come in. This is part of the founding thesis of DOGE, which is just vast parts of the American government in IT infrastructure. It's super legacy and we're getting overcharged as a country to modernize it. And agents go in and are, as long as we can get enough distribution, training data on like Fortran and Kobol and so on.

55:16Then the one time like upgrade costs should fall and we should see an, like ideally, this is my hope is that tools like Codex modernize that entire sort of legacy code economy. And then we get to upgrade everybody onto like modern software engineering, right? It's then, it's standing to happen from what I can see now in countries that get to leapfrog legacy infrastructure because it's starting from day one. And very, it's very similar like civil infrastructure, like roads and highways and so on. So if you go to a country like Singapore, which is a much more modern country because it's barely 60 years old You know, it's only got its independence in the 1950s then they didn't have to build The roads and so on that Britain did and then upgrade them all which is like refactors suck and they take way more time If you could just start from sort of a clean slate It's much easier to modernize and so what I'm finding is that it is easier for Countries that are whose IT infrastructure is just newer to adopt agents They're still legacy.

56:13I mean, that's majority of us running off an on -prem and it's not modern, you know, it's certainly not TypeScript, but it's easier to upgrade from, you know, systems that were written in C++ to Python, then it is to go from cobalt to Fortran, whatever to Python. But if there's anything that makes me super excited that these economies will merge, it's autonomous agents, right? Doing all the plumbing work and doing it for fraction of the cost and time, that these mega sort of consulting companies have started to charge. And frankly, many of them don't end up ever completing a project to just turn into a boondoggle.

56:49So I'm very excited about that part. And that's why I think AI is going to eat software because software did the modern sort of startup economy and digital economy, software it re -fast. But there were other parts of the world, especially mission -critical industries, where it was like a one -time software upgrade you largely driven by military scenarios, and then we never modernized all of that infrastructure since then. So that's why I think the cyber security side of this, the safety evils that you're talking about, I think over time, would come to be seen as having been very prudent because the thing that puts all of that adoption at risk is having like one terrible incident, then that then changes the risk posture for a bunch of enterprises.

57:28I have a question about that, actually, I'm kind of curious. So when, you know, a lot of the larger companies that we talk to, their use case is very different. It's not like building new features, which is what we see most of our users using us for, but it's re -factors, large re -factors, and re -platforming. So I'm curious, if you mentioned some of these companies or governments or systems that you're thinking about, kind of had this one time upgrade for military reasons, and never upgraded from there, I am curious if there was a specific reason that they all want to upgrade now that you're seeing, or if actually we're still kind of in the state of like there's no forcing function.

57:59So like although it's easier to do, there's still no impetus. Right. So for sure, the geopolitics has accelerated like adoption for a bunch of governments, right? In Europe, the Ukraine crisis has forced a lot of governments in that region to go, wait a minute, like our air traffic control systems, especially in age of unmanned sort of drone warfare. It is it is crazy that when there's a bug, we need to call in some legacy contractor who built it like 20 years ago to come and do some onsite maintenance, right? That's been a wake -up call. And so you're seeing these like there was a sort of an $800 billion defense bill that you were passed, you know, six months ago.

58:36And the most urgent adoption is certainly happening at the intersection of like legacy code not working and battlefield needs and drone warfare code bases that interact with air traffic control systems with like UAV planning with mapping. Those are the code bases that are like most urgently being upgraded. I think in other parts of the world, there's just a desire to modernize. So if you look at the UAE or the Kingdom of Saudi Arabia, we talked about how the UAE role is rolling out chat GPT to the entire country. I think that's coming mostly from a top -down directive to just embrace the AI future that's coming rapidly.

59:09Basically, the more AI build I find ahead of state is, the more rapid the adoption is, certainly for chat GPT -like tools, but also coding. That's not driven by some military function. But then there are other regions like where for sure geopolitics excel, reading all that. And you know, you don't have to talk about this before, but usually those scenarios often need a slightly different ergon, like the ergonomics of code are different. They're very on -prem. They're very, they require a level of air gapping from cloud systems that like the modern software engineering workflow doesn't lend itself to.

59:43And so we may see this like bifurcation of codex as a family. Like I'm curious over the next few years, you know, the military require, oh, let's the critical industry needs of modern autonomous coding agents might require some pretty basic architectural differences than the, you know, let me ship the latest and greatest of our next version of our software product on GitHub. I think I don't think it's a coincidence that the last time we saw a huge adoption and IT infrastructure on the world was the Cold War. And now we're living through some pretty unstable times, both in the Middle East. And I think that is causing governments.

1:00:20I think the US has always been somewhat forward leaning, posture -wise, on adopting the latest and greatest technology. We make other governments look, you know, like rightly so, like dinosaurs, and those folks, nothing forces dinosaurs to wake up like an impending comet hitting them and impending it extinction. So that's definitely happening. Yeah, I think it's interesting, like for me, playing this through a line as we're working on codex, I do think there needs to be an answer for like, you know, how do you use this agent in air -gapped environment? Right. How do you use this agent? Like, you know, this critical industry is, and then there's just many like large companies who have like incredibly stringent security needs.

1:00:58Right. It's kind of the way we've kind of thought about building is the most important thing is to, you know, build to AGI, right, and then distribute it the benefits of that to all humanity. And so we're kind of like leaning towards the like, okay, the primary thing is the like fully self -hoved, you know, the thing where we hosted for you. Right. you know, containing environment and everything. And instead of in parallel, we have this like sidetrack of like, okay, and like, how are we going to make sure that like today, you know, you can use codec CLI. You could use that in a, I guess, relatively airgapped way.

1:01:24Obviously, it needs to sample the model. And then as we build new capabilities into codecs and chat to BT, how do we just make sure that if you're running like something like CLI, right, and like get the most of all, you know, the capabilities as they went out of it. But it might, oh, it might be a little bit like, okay, we build it in the like fully self -contained system. First and then we push down. Right. You know, there's this narrative violation. I keep hearing about, I keep hearing from folks in San Francisco that, you know, openly I is all in on consumers. Cause it's, cause the rise of chat GPT as a consumer companion has been so extraordinary.

1:01:58But clearly our entire conversation is an exception to that story, right? Because almost everything we've talked about has been focused on developers and governments. So why is that misconception there? I think Chatch B .T. is in fact an amazing and large business and it's super cool to work at a company that is really distributing AI to a giant number of people. But yeah, we are incredibly serious about coding. And in fact, we always have been since the first Codex product that was powering GitHub Co -Pilot on all the way through with our models. I will say though, I think people are noticing we've always been very serious about coding models and we're now getting very serious about coding products as well.

1:02:41Whereas before, we had these amazing models that could use them in whatever tool that you want to use them in. Now definitely, a lot of the stuff that I'm working on is thinking about, hey, actually, there's a lot of, as we build agents, there's a lot of value we can provide by not only thinking about the model, but also thinking about how the model is useful to you in a certain form factor and actually the form factor really affects everything. And so yeah, we're spending a lot of time on effort building like even better coding models and even better coding products Particularly focused on agents, but even beyond.

1:03:12So you've been a founder before One of the scary things about hearing opening and going from being serious about models to all the products is If you're a founder in the space and you want to build something interesting in the coding space There's this extension looming, right? Which is anything I'm going to build Just going to be subsumed by opening as products next year. So how would you think about that? But if you were leaving OpenAid starting a company today, what would you do and what would you not do? Okay, so if I was leaving OpenAid today, probably the sort of the market changed. I would be thinking the most about or one of them would be agents.

1:03:42Okay, great, not super controversial. Then I would think, okay, like we were talking about earlier, an agent is basically like a really good model that I'm probably not going to build at my startup. And then I need to give that model access to tooling in an environment. And then I need to like figure out what tasks it needs to be good at. And then obviously give it to customers. And the interesting thing about it is that those latter three things, right, the tooling, the environment and the task distribution, what I guess I'm the customer, so four things, whatever. All of those things are very much based in like knowledge of a customer.

1:04:13And those aren't things that like open AI is going to like, you know, generally do for like every industry, right? Like coding happens to be of particular importance to us, just broadly, but even you know, within coding, there's a lot more specifics, specific areas. So just to really spell this out, you know, if you think of the environment, you know, Like it's really, you know, training codecs was like really non -travel to like figure out how to give the environments different, how to give the model different environments to train in, you know, with like different kinds of realistically dependent, realistic dependency setups.

1:04:40Right. Various amounts of dependencies even installed, like varying amounts of unit tests. Like we actually the startup that, you know, I sold to OpenAI was like, multi, that's how I joined. And we had very few unit tests on a lot of our code. And it's like kind of funny and like that. But that's realistic. That's like a real startup code base. Right. So actually, if you wanted to do that for some specific function, I don't think it would be easy for us to open AI to create that many environments for the agent to use and train on and then use it test time. So that's hard. And then I think the task distribution is also really interesting.

1:05:13Codex, we have a lot of intuition for what a good code and task could look like and where to draw the boundaries. Today, it's like provide prompt and then you get an answer or a diff that you can turn into a PR. But like those are some decisions we had to make around what the boundaries of the agent are. And then we had to go collect a bunch of those typed at the work tasks or invent those tasks. So again, train the agent how to do it and evaluate how all it was doing. So I think that again, for a very specific industry, I don't know, I'm trying to come up with an example. Let's say accountants, but in the specific region of the world, with a specific set of rules, they might have very specific tooling that's provided by the state for doing that accounting.

1:05:51right? There might be very different kinds of like based like knowledge and documents available and then like the way you need to do the work on different. So I mean, I think it is a very good question and I'm not 100 % sure what I would do as a browser founder right now. But I think that I would try to lean really hard on like very good customer knowledge and less hard on like product. That makes sense. Right. It sounds like the last mile connective issue between a industry where you have deep domain expertise that comes more valuable. Whereas the first mile of all the general purpose parts of an agent's flow, you should assume you should offload that to open AI.

1:06:29Yeah. Yeah. And then I think the other thing I might do is I might keep my company really small. So rather than doing the classic hyper scale thing, I would try to use agents as much as possible, company as small as possible. So that we're just agile and nimble. I guess this is probably like just an age -old advice. But let me push back on that for a second because it turns out that in many industries, serving the customer deeply like you're describing often requires human, a human touch. That might be sales, it might be solutions engineering, it might be customer support and so on. It does sound like what you're saying is you would certainly keep your engineering team very small and minimal.

1:07:05But if servicing the domain required more of the human touch, then that you would, you know, you would scale because if you'd required, often my experience is that getting an agent to actually work in the enterprise in the legacy industry requires going in and doing a fair amount of integration work, at least upfront. So maybe it's a setup thing, upfront, you parachute and somebody who understands how to get an agent up and running. And then you can leave because it's really just for them for the customers like consuming teammate like you were saying earlier, but maybe to where you do need people is that integration point.

1:07:41Now ideally over time, I guess you're saying the model should, the products should just get good enough at integrating into the customer's environment, but sometimes for regulatory reasons or otherwise, you just need a human there. You know, are there some industries that like clearly do you feel like out of bounds for opening eye because that just is not on the path to AGI, but that would still would interact with coding agents? First off, it's a good point on like the actual like integration work, probably where we're at cars. Humans, I would say yeah, if it's especially if it's in -person type integration work or like complex, then I think you're spot on there.

1:08:11Industries that are out of bounds. I think it's like, it's like a hard question to reason about because like we are building like general products. Right. And so you can like kind of use like chat to PT to answer any question like already today. So I wouldn't say there's like bounds, but it's more like focus. I would say, you know, right now, open AI, we're very focused on like serving consumers generally and like being really good at coding. Right. You know, there's some other things too. So I would just say, yeah, the more, maybe we should just not even have this answer. Yeah, we can take this part out of you.

1:08:44Give it 10 minutes time check as well. Perfect. Great. Oh, great. Yeah. But, Rob. Stop me. That was a good one. I'm like, I don't know, man. I've not found a right one. You don't have to speak on behalf of Sam about which why world domination is not complete and total. I can feel sorry. So slightly different topic. A question I get from a lot of parents, especially with kids who are approaching the end of high school, and in that phase where they're picking careers or thinking about what they want to do is this immense anxiety, especially for folks in tech, for whom, you know, for the last, for the vast majority of the like 20, 30 years, it's been a fairly stable assumption that like if you went, if you were smart and generally oriented towards technical fields, if you went and studied software engineering, you'd have a pretty great career and safe and sort of rewarding time in the knowledge economy, and it seems like like coding agents like Codex are taking a violent hammer to that assumption.

1:09:42How would you advise friends who are parents who are trying to figure out how to help their kids choose a career for the future? So I'll answer this with humility because I don't have kids, but I do think about this. And actually, I think my point of view would just be that the world has always been changing. It's changing now, but it was changing before that. maybe is changing a little faster, but that's the main thing to notice is actually the pace of change, not the specific change. And so, I think the most, if I had a kid at late high school now, I would probably just be trying to encourage them to just be excited it is about whatever they're doing and be incredibly curious and constantly learning.

1:10:21I studied CS, did you study CS as well? I started with CS, and then transferred to bioinformatics because I was more interested in healthcare. Right. And now you do investigate. And I studied mechanical engineering and then I changed to CS and now I work in product in an AI at OpenAI. But the startup that I had started was not an AI company. So things are constantly changing. And I think the most important thing is to be agile, curious and have some foundation that you can build upon as the world evolves around you. So I think similarly, if I had a child in late high school, I would just want them to crush whatever it is that they're doing.

1:10:53And it wouldn't really matter what specific thing they've chosen. you know, Eileen technical, so that would be cool, but like, maybe even that is optional. And then I would just raise them with the expectation that they'll probably have like many career transitions throughout their lives. And if you were having seen what you have with Codex, knowing what you do, where it's going, let's say you have the chair of the Computer Science Department at University, what would you do differently now versus before when Codex launched? Well, one is you'd allow kids to use the AI tools, but let's hear thinking about the future future of computer science education and how that should be taught over the next five, 10, 15, 20 years.

1:11:30How would you do differently? Yeah, again, just opinions here, but I think I would have, you know, like Stanford, it was a class for really grow assembly, forget the name of that class. That was cool. We had one class. So yes, 140, I think it was. Yeah. And then, you know, similarly, I would have like a handful of classes where people folks do things like very manually to understand what's going on behind the scenes and also to build a confidence that they can. But then generally, I would move towards having students trying to deliver some kind of outcome, be it like they've learned something or they've built something or something.

1:12:02Project based learning. Yeah, and then I would probably encourage them to use these various tools so that they're picking up the skills. And I don't know, this is just an idea in my head, but if we could help them speed run through that arc, then maybe every quarter that they're using a different set of tools, and so they're becoming very mentally plastic in terms of how they get things done. and I think that would be the best simulation of what G -Shirt work would look like. I'm not sure what would you do. Well, at G -Shirt class, CS143 at Stanford every year, this year we taught it in winter quarter, and we had about 300 students.

1:12:35And I was, you know, thinking through what was a, in previous years, we had a mid -term and, you know, we had, like, problem sets. And this year we decided just to do, have it be a combination of speakers who are CTOs or folks, researchers in AI come in and talk about the infrastructure problems of building AI products at scale. And then we had one final project where everybody had to build an agent and ship it. And they were all allowed to use any coding tools. Obviously, in fact, we gave folks some credits to a Mistral models and Black Forest models and the founder of Curse, they came by and kind of talked about the IT and why they should all be using it.

1:13:11And what was extraordinary, right, was it was so clear that the distribution of the final projects followed this power law where the top four or five teams that really adopted wholeheartedly the coding cursor and the AI models and did a fully sort of AI assisted workflow of their final project, like produced software that was like production grade ready. If I was still running the platform work at Discord, I would have totally shipped four or five of those was on the front page of the app store we had. In fact, I sent some of them to the founders of Discord and they were like, we should probably ship this.

1:13:45The quality bar was just extraordinary for something they were able to build in a basically a 10 -requarter. Then there was this sort of, you know, usual sort of middle of the pack that had made a half -hearted attempt but enough to get a good grade to customize the templates we'd given them. But clearly hadn't like asked, what is something that now I can create that I couldn't before? Now that I have access to extraordinary coding agents, and then there was just the classic sort of bottom of the class that I think just didn't accept those tools and think deeply about like trying them, using them, learning with them, developing a field for like what they're good at and what not good at, and kind of turned in a final project that would have been totally possible to build a year ago.

1:14:30Why do you think they didn't want to use the tools you were giving them? Look, it's hard to parse out from just a final project, but I did office hours with a lot of the students every week. And you could very clearly think the number one predictor of their success was their mindset. It was just about like, did they were they curious and hungry to learn outside of like a traditional textbook? And look, some of them, some of the students just had a lot going on, you know, being a college student is a stressful thing today. And so I don't, I have a lot of empathy for... there's definitely this awkward moment you're describing right now where a number of this graduating seniors from who are graduating with college degrees this year started out as freshman in a very different economy right right when they picked CS the assumption was hey if I like do well in the core CS curriculum if I take a four if I get a 4 .0 GPA and I do like one or two good internships you know somewhere along the way and I apply for a job I'm going to get a job at a pretty good debt company.

1:15:33That's just not happening anymore. And it might be because there's a set of layoffs or some overhang from the Zerpa era, or it might be because a lot of engineering teams are reducing their footprint of entry -level jobs. But I was definitely shocked by how many Stanford CS grads they were looking for, you know, graduating seniors, still looking for full -time jobs, you know, come winter senior year. And I think that's anxiety inducing, it's stress inducing, that has bleed over effects on, And can you concentrate on this project -based class when you're like, also, the number of the students were also juggling interviews.

1:16:06And we're coming to office hours. When I thought they were going to be coming to ask about the code, we're asking for career advice, which is totally fine. But I do think there's a transition phase right now, which can be very stressful for computer science students. And I think you're right, the faster they're able to onboard to using these tools rapidly and realizing that the gap on what they can create now is extraordinarily high, the faster I think they're going to transition into the new economy better. Because I do think there's an expectation, certainly for modern software teams, certainly at OpenAI, that you're just fluent in all of these tools now relative to four, five years ago.

1:16:43It was crazy. When we graduated through Stanford, I didn't take a single class that required the use of Git, which is absurd. I happen to pick it up in an internship, but there's no class that actually requires you at least at the time, require you to know how to use Git. Yeah. And so I think I do think the computer science departments around the country have to recognize that and change and do the kind of make the changes you're talking about. And my hope is that in the interim, you know, students will won't wait around for their deans and their professors to do that for them because you can just go and use codex, you know, for free.

1:17:16I think the research per year is literally free. Is that right? Well, you have to you have to have a plus encounter of pro -count beta. Yeah, it's a good point. Maybe we should do something for students. Student license. Yeah. You know, I will say that like we, so we're hiring for codex, please. which I said, if you're interested in working at Codex, DM at EmbiRico on Twitter, it's eMBI. What's the idea on the show notes? Yeah, I don't know if I'm allowed to plug myself here, but yeah, we're hiring, but we mostly are hiring very senior, but we actually are, we decided that we're pretty interested in hiring with a couple of new grads.

1:17:44Oh, that's interesting. Yeah. And so it's been interesting just looking at new grad profiles, and I totally feel you on the, yeah, I mean, it's definitely a tough time to be graduating. I don't know if this is advice, but what I can say is that when I look at new grad profiles, For me, the thing that I take the most signal from is if they've built something, right? And if they've built something that's linked from their profile, I can just like click to it. Projects. Yeah. And, you know, like, it's just like a cool website. Right. Like, gray, it's matter of much less now. Yeah. I don't even look, I actually, now that you, I didn't even realize that I haven't looked at anyone's grades.

1:18:17You know, like I just like, because, you know, admittedly, we're only hiring a few new grads. Right. But that is the single largest signal. I just have built. Right. And is there some way for me to validate that? Like, maybe it's because I can click to the website or maybe you just have some stats on like how many people used it. Right. And then when I talk to them, I'm just like, yeah, talk, let's talk about what you built and how you thought about that. So maybe that's somewhat helpful for folks who are looking for something. You know, I kind of reflect on my journey here to OpenAI, which I'm really grateful for.

1:18:45And I viewed as a privilege to be working here. But, you know, when I looked back to when we were working on the startup multi, I was like, not an AI company and we saw like chat be come out and we started to follow all this alum stuff. I remember just feeling like, wow, like there is a chance that if we don't do this right over the next couple of years, like my co -founder and I were talking, there's a chance that we actually just end up like dinosaurs. Right. And so at the time, we actually made like a very explicit decision to like heavily prioritize getting us and the entire company like wrapped on AI things.

1:19:14And to some extent, like, I don't know if I could have like gotten the job that I have here at OpenAI. I was just applying randomly. I think it's because we had built something that was interesting that we were able to get that attention and have that conversation. So I guess if there's one takeaway here, it's just like, just gotta build. It's time to build. Yeah. Thanks for listening to the A16z podcast. If you enjoyed the episode, let us know by leaving a review at ratethispodcast .com slash A16z. We've got more great conversations coming your way. See you next time. As a reminder, the content here is for informational purposes only.

1:19:49Should not be taken as legal business, tax, or investment advice, or be used to evaluate any investment or security and is not directed at any investors or potential investors in any A16Z fund. Please note that A16Z and its affiliates may also maintain investments in the company's discussed in this podcast. For more details, including a link to our investments, please see A16Z .com forward slash Disclosures.