How OpenAI Builds for 800 Million Weekly Users: Model Specialization and Fine-Tuning

22:30In practice, it's mostly been the format. In practice, it's mostly been the offline data that they've already created or they are creating with experts or something and using their product that they're able to use here. But the main thing I was trying to say around the reinforcement fine-tuning APIs, it kind of changes the paradigm away from just small incremental tone improvements, which is what SFT did, to actually improving the model to potentially SOTA level on a particular use case that you know about. Like that's where people have really started using the reinforcement fine tuning API. And that's why it's gotten more uptake.

23:05Because if the discussion is less like, hey, I can make this model, you know, not like speak in a certain way better. It's less compelling. But if it's like, hey, for like, you know, medical insurance coding or for like coding planning, agentic planning or something, you can create the world's best model using your data set with RFT. Then it becomes a lot more. And will you ever, or maybe do you, will you ever find ways to get access to that data? Yeah. If I had the data and I wanted cheap GPUs, I'd trade you for it. I don't know. Yeah, I mean, we've talked about this and we've actually been piloting some pricing here too where it's like, because this data is really helpful and it's kind of hard to get.

23:42And if you actually build with a reinforcement fine-tuning API, you can actually get discounted inference and potentially free training too if you're willing to share the data. It's always kind of, you know, it's up to the customer there. But if they do, it is helpful for us, and there will be benefits for the customer as well. That's awesome. Okay, you said that the use on prompt engineering have changed. Yeah. Actually, I wasn't aware of that. All the other things I wasn't aware of, this one I wasn't. Yeah, I mean, I think the prevailing view, this is back in 2022. I remember I was talking to so many people, and they were basically, I mean, this is similar to, like, the single model AGI view as well, which is, like, prompt engineering is just not going to be a thing, and you're just not going to have to think about what you're putting in the context window in the future.

24:22Like the model would just be good enough and it'll just like know. It'll know what you need to do. Yeah, that's definitely not a thing. Yeah, but like, I don't know, maybe people forget it, but like that was like a very common belief back then because like the scaling laws or whatever, something with the scaling laws and like you'll just mind meld with the model and like you just like, like prompting and like instruction following will be so good that you won't really need to do it. And if anything, like, yeah, it's like clearly been wrong. And, but it is interesting because I think it's a slightly different world that we're in now where the models have gotten really, really good at instruction following relative to the GB3.5 or something.

24:54But I think the name of the game now is less on prompt engineering as we had thought about it two years ago. It's more of the context engineering side where it's like, what are the tools you give it? What is the data that it pulls in? When does it pull in the right data? Well, this is very interesting. I mean, to reduce it to an almost absurdly simplistic level, the weird thing about RAG, for example, the classic use of RAG, is you're using cosine similarity to choose something that you're going to feed into a super intelligence. So, like, you know, you're like, I'm going to randomly grab this thing based on, like, fucking embedding space.

25:28It doesn't really, you know, and then, you know, when you want the super intelligence to decide the thing to do, and so it's like pushing intelligence in that retrieval clearly is something that makes a lot of sense. It's almost like pushing the intelligence out in a way. Exactly. And to be fair, I think, like, RAG was kind of introduced when the models were like, it was like pre-reasoning models. so it was like you only had to kind of like one shot to like do this and it wasn't that smart but now that we do have the reasoning models now that we have I mean if you like one of my favorite models is actually 03 because it was like one of the most diligent models it would just like do all these tool calls and it's like really the intelligence itself trying to like do the you know tool calls or reg or anything like that or write the code to execute and so the paradigm has shifted there but yeah because of that I think like context engineering prompt engineering what you put what you give the model is like extra important yeah yeah Okay, so you have the API, which is horizontal.

26:18You've got ChatGPT and other products, which are vertical. We haven't even talked about pixels. This is all just language. Are agents a new modality? Is that something else? Like, you know, like a codex or... What do you mean by modality here? Like, I mean, they feel both vertical and horizontal to me in a way. Like, to me, ChatGPT is a product, right? It's like it's a product and like my mom uses it, right? Yep. And an API is a dev thing. You kind of give it to a developer, and a CLI is kind of somewhere in between to me. It's like, is it a product? Is it horizontal? How is it handled internally?

26:55Is it a totally separate team that does agents? No, so it's... Yeah, it's interesting because I think the way that you frame it just now almost seemed like agents was this singular concept that might have its own particular team. Maybe a better question is, what is an agent to you? Yeah, yeah, yeah. Even getting a language is important for this conversation. So I actually don't even know if it would be helpful for me to share, but my general take on agents is it's an AI that will take actions on your behalf that can work over long time horizons. And I think that's the pretty general. Pretty utilitarian.

27:30Yeah, yeah. But if you think about it that way, yeah, I mean, maybe this is what you mean by modality, but it is just a way of using AI. And I guess it could be viewed as a modality, but we don't view it as like a separate thing separate from API. Let me just try and kind of give you a sense of where this question is coming from. Like I know how to build a product and we know how to go to market for products. We know how to do like, you know, we know the implications of turning them into platforms. Like it's just we've been doing this for a very long time, right? We know how to do the same thing for APIs, right?

28:03We know how to do billing. We know like the tension of like people build on top of it and all of that stuff. And like what I've been trying to, and this is just maybe a personal inquiry, it's just not clear for me for an agent if it sits in one of those two camps. Is it more like the product camp? Is it more like the... Because it's kind of both. I could literally give you coding as a user and then you just talk to it or I could build in a way, kind of embed it in my app. But then that means something to you as far as like, you know, how do you price it and what does it mean for ecosystem? Like, for example, like, would you be fine if I started a company and just like built it around Codex?

28:45Is that a thing? Starting a company and building it around Codex? Yeah, yeah. I actually think that'd be great. Like, it's a, we like release like the Codex SDK and we like want people to be able to build it and hack on it. Yeah. Actually, I think this might be what you're getting at, which is, and this is like a kind of a unique thing about OpenAI and kind of reflects on how it's run, which is at the end, like at the end of the day, OpenAI is like an AGI company. is like an intelligence company. And so agents are just like one way in which this intelligence kind of be manifested. And so the way that I'd say we actually think about internally is all of our different product lines, Sora, Codex, API, ChatGPT, are just different interfaces and different ways of deploying this.

29:21So you don't really... And so there's no like single teams like this is, you know, like thinking about agents. I would say the way that it manifests itself more is like each product area thinks about like what is, you know, this intelligence is actually turning into a form where like it can actually, agentic behavior is more possible. What would that look like in a first-party product like ChatGPT, what would that look like? This is actually why Codex ended up becoming its own products. What would it look like in a coding-style product? We explored it and ChatGPT kind of worked there, but actually the Klai interface actually makes a lot more sense.

29:49That's another interface to deploy it. And then if you look about the API itself, it's like this is another interface to deploy it. You're thinking about it in a slightly different way because it's a developer-first mindset. We're helping other people build it. The pricing is slightly different. But it's all these different manifestations of this core intelligence that is the Asian behavior. It is so remarkable how much of this entire economy is basically just token laundering. In a sense, it's literally like anything I can do to get like English in or like a natural language in and then like, you know, the intelligence out.

30:20And I mean, it's because things are so resistant to layering. It's so hard to layer a language out. Like, you know, I could even do it pretty easily with like Codex. I could just like use it, you know, as a component of a program and just, you know, basically launder intelligence. I mean, of course, you know, I'd be charged to do that. So I actually, my view of this and having seen now so many kind of launches of different products, I've seen agent launches and the definition that you have, I've definitely seen APIs, and I've seen products on these, it's like they're actually quite different than like what we're used to.

30:55Like the COGS is different, the defensibility is different, like all of this. So we're kind of rewriting it. And so it's kind of like, you know, you came from a kind of pricing background. I mean, you were working on a model for pricing. Now you have the API. So I just love your thoughts on like, I mean, how have you evolved your thinking and how do you price these, you know, access to intelligence where, you know, you don't know how many people are going to use it. It's almost certainly usage-based billing, not something else. Like, can you talk just a bit about like philosophy around pricing on these things?

31:26Is it different for product versus API? Yeah, I think that the, The honest truth here is it's evolved over time as well. And I actually think the simplest, the reason why we've done usage-based pricing on the API, honestly, is because it's closest to how it's actually being used. And so that's kind of how we started. I actually think usage-based pricing on the API has surprisingly held strong. And I actually think this might be something that we'll keep doing for quite a long time, mostly because... I don't know how you don't do usage-based. Yeah, yeah, yeah. I just don't know how that... Yeah, and then there's also the strategy of how we price it.

32:02And internally, one thing we do is we always make sure that we actually price our usage-based pricing from a cost-plus perspective. We're actually just trying to make sure that we're being responsible from a margin perspective. By the way, this is a huge shift in the industry in general just because I remember the shift from on-prem to recurring. Yeah. That was a big, big deal. That created Zora. It created a whole company. It was like the whole books on it. A bunch of consultants on how you do this. It changed. And I think the shift to usage is as big or bigger. And it's also even a really hard technical problem.

32:36I can't even imagine 800 million wow. How do you build? Yeah, well, 800 million wow is a little easier because it's not usage-based pricing. It's subscription. So that was way easier. But I mean, there's still a lot of users on the API that we need to manage all the billing side. There's some overages or stuff you've got to deal with on that? What do you mean by overages? I don't know. I guess I don't know. Most people have quotas and then we'll kind of like, you know, they're like max quotas that we don't let people go over. But like in practice, these quotas are like pretty, pretty massive. And that would literally be like one of the most complex systems somebody's ever built if you would do usage-based at like that scale.

33:11I mean, these are very, very, very, and like you have to be correct. Like these are very hard systems to scale. Yep, yep, yep, yep, yeah. Yeah, I mean, we have a whole team thinking about this now internally. Yeah, I mean, usage-free pricing is also interesting. So there's, we acquired this company called Roxette a while ago. A founder, his name is Benkot. He's right here at the org now. Yeah, Benkot's awesome. Benkot's incredible. He's one of the best. Benkot, if you're listening, we're huge fans. I'm a huge fan. He's going to love this. Yeah, he's great, man. He's a legend. Anyways, I was talking to him about pricing as well.

33:41And his take is that pricing is kind of like a one-way ratchet. And basically, once you get a taste of usage-based pricing, you're never going to go back to the per seat, the per deployment type pricing. And I think that's definitely true. and I think it's just because it's getting, it gets closer and closer to like your true utility. You're getting all this thing. The main pain point is like, you have to maintain all this infra. Yeah, to like get it to work well. But if you do have it, he thinks it's like a one-way ratchet where like there's just like no going back. And then, and I think the hot new thing now is like, oh, with AI, you can now kind of measure like outcomes.

34:10And so that's like another, you know, like step forward. And if that works, like maybe it's a one-way ratchet. So we thought about that. It's like, you know, is there some type of like outcome-based pricing? This is more on the first party side on an API. It's kind of hard to measure that. very hard. I mean, that's hard because you end up having to price and value non-computer science infrastructure, right? Like, you're literally going into verticalization now. Like, you're like, I mean, listen, if it's like porting a code base, maybe you'd have some expertise, but if it's like, whatever, like increasing crop yields.

34:40Yeah, yeah. Like, at some level you need to, like... But there could be a world where, like, the AI is, like, I don't know if it can actually, you know, make judgments of these and do it in an accurate enough way where it can tie it to billing. I think this is a problem with AI conversations because like at any point in time you're like, but it could get good. Yeah, yeah, yeah. It's not a problem anymore. Yeah, yeah. At some point it'll be solved. It's so much like the prompt engineering and the single AGI I think from before. Yeah, it's like when you reach that level of, when you push it that far everything's kind of solved.

35:07On outcome-based pricing it sounds very appealing. Like if it can work it can work. But one thing that we've started realizing is it actually ends up correlating quite a bit with usage-based pricing especially with test time compute. Like if the thing is just like thinking quite a bit. Like actually, you know, if you charge just by usage based and not outcome based, you're like basically approximating outcome based at this point. If the thing is like thinking for like so long, it's like highly correlated with what it's doing. It's just adding more value. Yeah, yeah, exactly. And so like maybe at the end of the day, like usage based pricing is all you need and it's like we're just going to like, you know, live in this world forever.

35:44But yeah, I don't know. It's constantly evolving. I think our thinking has evolved here as well. I personally am like keeping track of if the outcome-based pricing setups can actually work here. But at least on the API side, I think, you know, it's such a usage-based setup. We have to get infrastructure around this. And so I think we'll probably stay with that for a while. So how do you think about open source? I mean, you know, I think you're the only big lab that's releasing open source, is that? No, Google has some of theirs. Okay. Yeah, it's mostly smaller models on their side. Yeah, yeah, yeah.

36:13That's right, yeah. So how do you think about open source vis-a-vis, you know, competition, cannibalization, you know like what's the strategic goal what's the complexity yeah yeah so I personally love open source like I think it's great that there's a all of us grew up with it right like the internet wouldn't exist without it like you know so much of the world was built in half of it nothing would exist without it except for maybe Windows and so it was interesting because I felt like over the last years before we launched the open source model I know Sam feels this way as well it's like there's this like weird like you know mindset where because OpenAI hadn't launched anything it just seemed like it was super like anti like OpenAI was like super anti open source but I'd actually been having conversations with Sam ever since I joined about open sourcing a model we were just trying to think about like how can we sequence it what compute is always a hard thing it's like do we have the compute to kind of like train this thing so we've always wanted to kind of do this I'm really glad that we were able to finally do it I think it was earlier was it earlier this year I lost that time.

37:16AI time is so crazy. Yeah, I was like, was it last year? No, it was this year. Yeah, when GPOSS came out. And so I was just really glad that we did that. The way that I generally think about it is, one, I think as a, this is also particularly true for OpenAI because, as you said, we are a vertical and a horizontal company. It's like we want to continue investing in the ecosystem and just from a brand perspective, I think it's good. But then also, I think from OpenAI's perspective, if the AI ecosystem grows more and more it's like a rising tide it's all really helpful for us and if we can launch an open source model and it helps unlock a whole bunch of other use cases in the other industries I think that's actually not good for us Also what people talk about a lot is how well these open source AI business models actually work because the cannibalization risk is actually very low Yeah.

38:11And, like, you don't really enable competitors a lot because, I mean, when we say open source, you really mean open weights, right? It's not like they can recreate it, right? You know? And, like, if I can distill your API as well as I can distill, like, you giving me the weights in some way, like, and so, like, it doesn't really change that dynamic a lot. But, like... Yeah, I mean, to be clear, like, we have not seen cannibalization at all. Yeah, of course not. I mean, yeah, yeah. It's, like, it seems like a very different set of use cases. The customers tend to be, like, slightly different. The use cases are very different.

38:37And, by the way, it turns out inference is super hard. to actually have scalable, fast, performant. That's a hard, hard problem. Yeah, so I'd say the way that I personally think about open source in relation to the API business in particular is, well, one, it hasn't shown cannibalization risk, so I'm not particularly worried about that. But also, especially for all these major labs, there are usually two or three models where that is where you're making all of your impact, all of your revenue. And those are the ones where we're throwing a bunch of resources into improving the model, and these tend to be the larger ones that are extremely hard to inference.

39:08We have a really cracked inference team at OpenAI. And my sense is like, even if we just like, you know, open source them, like if we just literally open sourced GPT-5 or something, it would be really, really hard to inference it at the level that we are able to get it to do. There's also, by the way, like feedback loop between the inference team and like the training team too. So like we can kind of like optimize all of that. Can you, like, is it possible to verticalize models for products? You like train models specifically for products? Yeah, I mean, to actually, yeah. I think, I mean, we've kind of done this with GPT-5 Codex, right?

39:39Or do you mean like even more verticalization? I mean, like deep, deep, deep verticalization where like, you know, like the released model wouldn't, you know, it's like actually part of a product. I think we're like basically starting to move in that direction. I think there's a question of how deeply you verticalize it. I think most of what we've done is mostly at like the post-training, like the tool use level. Like Codex is particularly good at using the, sorry, GBD5 code is particularly good at using the codex harness. But there's like even deeper verticalization you can do. Yeah, so I mean, that one I think is more of an open question.

40:14Yeah, so like a lot of my, I mean, a lot of my mental model of this comes from the pixel space, which is like, you know, you can Laura a bunch of image models, right? And you can do a bunch of stuff to make it better and more suitable for some products, for example. but like these open source models are really really good and like you would believe that you could like verticalize a model for like editing or cut and paste or this or that you know like that's actually part of this but you actually don't see that happen yeah it's almost always like you're just kind of exposing like a model not something like specific to a product yeah I think there's a distinction to be made between like the image model space and the text model space also because the image models tend to be way smaller and you can iterate on it a lot faster.

41:02That's why you get that crazy cool proliferation of the image model side. Whereas I don't know, for the text models, there's always going to be this really big, fat free training step that you have to invest in. And then even the post-training side is like, it's not the easiest thing. Just from a compute perspective, obviously it's much smaller, but it's still pretty heavy to do a full mid-training or a post-training run. And so I actually think that's one of the bigger bottlenecks. because I think you are right that like on the image side, yeah, you can like fine tune and like image diffusion model will be like extremely good at like editing faces.

41:35Yeah, like something very specific. And then you build a product around that. Yeah, yeah, yeah. And it's like, yeah, you can just kind of put all these resources and iterate on that one specific model whereas it's a much heavier emotion on the tech side. I got to say, it is a bit of an anti-pattern to do both languages, like language-based models and diffusion-like pixel models in the same company. like most that have tried like it sounded very clunky to do it but I mean you and Google are the two kind of counterexamples for this and so like is it possible to even like converge the infrastructures on these things like I mean is it totally different orgs is it shared infrastructure like how do you operationalize yeah I think I think you're totally right it's an antipattern it's pretty tough to pull off I think honestly like props to Mark on our research team for structuring things in a way we're able to do it.

42:26From my perspective, I think the biggest thing is I think our image, I think we call it the world simulation team, the team that builds Sora and all that under Aditya, is just extremely solid. It's like the highest concentration of talent that I've seen in a while. But is it the same? Are they totally separate infrastructure? Do they use the same infrastructure? Yeah, yeah, yeah. So it's actually pretty separate. And I think that's part of the reason why we're able to kind of do this. Well, one is the team needs to be extremely strong, which they are. And then two is they're run very separately.

42:59They're kind of like thinking about their own particular roadmap. They think about productization very separately as well, which is how the Sora app kind of came out of that as well. And then, yeah, even the inference stacks are kind of like different. They own a lot more around their inference stack and they optimize their inference stack pretty separately. And so I think that contributes to helping us run things in parallel, but it's pretty hard to pull off for sure. Maybe you can educate this on me. So I think about APIs as mostly text-based for OpenAI. Do you guys do actual pixel-based stuff?

43:34Yeah, yeah, we do. We have a bunch. So Dolly 2 is in the API, the OG model. Dolly 2 is in the API. That was like the first real text image model, right? Yeah, yeah, yeah. That was actually the model that got me to go to OpenAI because it was the summer when I was looking for, I was thinking about something new. It's when Dolly 2 came out and it just completely blew my mind. Wow. And I distinctly remember, I was like asking it to do the simplest thing, like draw a picture of a duck or something. It was like the simplest thing now. And it just like, it generated a picture of a, you know, like a white duck.

44:04And so that was actually the thing that kind of got me to open it in the first place. But yeah, we have a bunch in our API, the image gen model as well as in our API. And then Sora 2 is in our API. We launched it at DevDay. It's actually been a huge hit. I've been very, very surprised. Need more GPUs for that. But the amount of use cases... And then from your standpoint, you can converge that, like the API infrastructure probably. Yeah, I'd say on the API side, a lot of the infrastructure is shared for those, but once you reach the inference level, they're separate, right? Because you've got to inference them differently.

44:36And it is that team that has just been really laser-focused on making that side particularly efficient and work well separate from the text models. But yeah, we have image gen, we have video gen, and we'll continue adding more to the API there. So it feels like we've been evolving our thinking as an industry on a bunch of stops, right? Like one of them for sure is like the models like we've talked about. The other one is like context engineering. It seems to me that like actually how you build agents and expose them has evolved too. So maybe you can talk a bit about that. Yeah, yeah. I think, so at Dev Day this year when we launched our agent builder, I got a bunch of questions around this because the agent builder is like, Yeah, it's like the bunch of different nodes and it's like the deterministic thing.

45:18And I was like, oh, is this really like the future of agents? And we obviously put a lot of thought into this when we were thinking about building that product. But the way I think about it is... Do you think they came from a point of being constrained? By the way, they're like, oh, this is too constraining. Yeah, I think people are like, it's too constraining. It's not like AGI forward. You know, like at the end of the day, the AGI will do everything. And so like, why not? Why have nodes in this like node builder thing? Just tell it what to do. Yeah, and so I think there's like two things at play here.

45:43One of them is like there is a like, practicality components. And then the other thing is, I think there are actually like different types of work that exist out there that could be automated into agents. And so on the practicality side is, yeah, like the models today, just like maybe in some future world, instruction following would be so good that you just like ask it to do this four-step process. And it like always does the four-step process exactly. We're still not there yet. And in the meantime, you know, this entire industry being born and a lot of, you know, people still want to use these models.

46:10What can you build for them. So there's a practicality component of it. When did you launch that? Dev Day. So it feels like forever ago. Earlier this month, October 6th or something. So less than a month ago. It's been crazy seeing the reception to it, by the way. I think the video where Christina and my team demos Agent Builder is one of the most viewed videos on our YouTube channel now. I will say just anecdotally from my perspective, people love it. That's great. But I also saw the dissonance too. Like I saw when it came out people were like wait what is this? Yeah exactly. Is this no code, low code?

46:47Yeah exactly. It's another low code thing. And now people love it. Yeah yeah. Yeah so there's a practicality piece. There's another piece which is like when we were talking to our customers we've realized that there's like because at the end of the day a lot of this the agent work is just trying to automate work and like what people do in their day to day jobs. I realized there's like actually like two different types of work. There's the work that we think about which is like maybe what like software engineers do which is like It's very undirected. There's a high-level goal, and then you have your cursor, and you're just writing code, and you're exploring things and going towards an objective.

47:19That's more like knowledge-based work, like data analysis, maybe like that, like coding's kind of like this. But then there's another type of work, which is actually what we realize is maybe even more prevalent in industry than software. We're just not aware of it, which is work tends to be very procedural, very SOP-oriented. Customer support is a good example of this. Like customer support, there's like very clear policy that these agents and people have to follow. And it is actually not great for them to deviate from this and like try something else. It's like the team really, the people running these teams just really want these SOPs to be followed.

47:50And this pattern actually generalizes a ton of different work. A standard operating procedure. Yeah, sorry. So it's like the way in which you need to operate the support team. But like this extends to like marketing, this extends to like sales, this extends to like a bunch, way more than it has any right to. And what we realize is like there's a huge need on that side to have determinism here. Yeah. Of which an agent builder with nodes that kind of like helps enforce this thing ends up being very, very helpful. But I think a lot of us, especially in Silicon Valley, don't really appreciate that there's like a ton of work that actually falls into this camp.

48:21I got to say like there's a pattern that's similar to this. I'm wondering if you've seen it that I've seen where some regulated industries actually can't let any generated content go to a user. Yeah. Right. And so what they do is I think it's so interesting. They'll either pass in a conversation tree, and you can choose something from here. Yeah, so there's some human element to it. So as part of the prompt, they're like, here are the viable things you can say, choose which one to say. So the language reasoning has happened by the model, but nothing generated comes out. Interesting, interesting.

48:52Does that make sense? Yeah, yeah, yeah, yeah. And then another one I've seen is actual pseudocops. I'll pass in a Python function. And then it'll ask a human to use a pseudocode to write actual code that makes it in? or it actually has a response catalog as part of it and it has like the logic to apply. And then... And so like the model takes the language in from the human user and then, well, like, you know, the logic of how to respond is like in Python code because it just turns out that like there's been a lot of code written for these types of things and then it actually includes the responses that you would send out.

49:27Does that make sense? Actually, a lot of NPCs are done this way, like actually video game NPCs. Yeah. Because the way that I think about it is like, you know. So that way with the NPCs, the actual code being generated by the model is not what ends up making it to the end user. The code is not being generated by the model. It's the prompt has the code. So let's say that I have an NPC, and I want the NPC, like let's say you're the gamer. And so you're coming in and you're talking to my NPC, but my NPC has some logic that it needs to do. Like if you say a certain thing, I'll give you a key, or maybe a little barter.

50:02like describing the game logic in English just doesn't work actually if you try and do it and then like actually scripting the output doesn't work either if you needed to use it in a game context like you would have to know like give like a specific direction or a specific this or that so how do you make these things behave in a more constrained way people pass in functions like they'll actually describe the logic in Python so like my prompt will be like you're an NPC in a video game the user just asked you a question here's the logic you should go through. If the user says this, then do this.

50:33It's like the pseudocode. Like if the user has this, you know, in the belt, do this. Like whatever, whatever, whatever. And then here are the set of valid responses. And so you're almost constraining. I see, I see, I see. And then when it actually does do a response, you can validate that it's one of those responses. I see, I see. It's like highly structured. Yeah, yeah, okay. So the MPC still only exists in that, like the space that it can act in is still only within the space of the program that you gave. Yeah, well, the logic is in there. So it can have a normal conversation, but like in as much as you're trying to guide the logic for like game design or game logic.

51:04I see. So you see this with NPCs but you also see this with regulated industries where like I literally can't have it like. Yeah, I was going to say what you described kind of sounds like, you know, giving the SOPs to like your set of human operators to like have to stick to it, please. Yeah, you must say these three things and here's like the discussion. And like you cannot give a refund if it's like less than this amount. Yeah, yeah, yeah. Very interesting, yeah. I mean, yeah. I don't want to equate them to NPCs but like this is similar to similar. I'm just saying it's actually like if you want to really guarantee what happens, there's like a set of techniques that you do.

51:36And like there's some situations where you want to constrain what they do. It could be from a regulatory standpoint. It could be because you want it to run for a long time. And it also could because I actually have game logic. And my game logic is a traditional program. Like I have like a monetary system. I have an item system. I have a battle system. Like you can't describe that in English. Like you have to kind of give it to them so they can behave within that. Yes, and that is exactly the problem I think we're trying to solve here. That's so awesome. If you do not give it any of this, like it can just kind of go off and do whatever.

52:01And yet there are like regulatory concerns around this. And that is the exact use case that I think we're trying to target with the Asian Builder. That's awesome. Well, listen, we're running out of time and there's a million more things I want to ask you, but listen, I really appreciate your time to come in. It was a great kind of surveying like what's going on and particularly like teasing apart horizontal versus vertical in this page. Yeah. Which I really want to do. So thank you so much. Yeah, thank you. Thanks for listening to this episode of the A16Z Podcast. If you liked this episode, be sure to like, comment, subscribe, leave us a rating or review, and share it with your friends and family.

52:35For more episodes, go to YouTube, Apple Podcasts, and Spotify. Follow us on X at A16Z and subscribe to our Substack at a16z.substack.com. Thanks again for listening, and I'll see you in the next episode. As a reminder, the content here is for informational purposes only. should not be taken as legal business, tax, or investment advice, or be used to evaluate any investment or security, and is not directed at any investors or potential investors in any A16Z fund. Please note that A16Z and its affiliates may also maintain investments in the companies discussed in this podcast. For more details, including a link to our investments, please see a16z.com forward slash disclosures.

53:23Thank you.