GPT-5 and Agents Breakdown – w/ OpenAI Researchers Isa Fulford & Christina Kim

0:00I mean, I think it's pretty unique at OpenAI to be able to work on something that's so generally useful. I mean, it's like everything they tell you not to do at a starter, but it's just like your user is anyone. You just kind of take it for granted that you literally have this like wizard in your pocket. We're trying to make the most capable thing, and we're also trying to make it useful to as many people as possible and accessible to as many people as possible. I think we hear this with GBD5, but internally when people were testing, they're like, oh, I thought I asked like a really hard question.

0:26I feel like we're all being salted that he has it in seconds. I'm like, when he doesn't even want to think at all. Today's episode was recorded the day GPT -5 launched, a major milestone not just for OpenAI, but for the entire AI ecosystem. Joining me in the studio, fresh out the launch livestream for three people who are instrumental in making this model a reality. Christina Kim, researcher at OpenAI, who leads the core models team on post training. Issa Fulford, researcher at OpenAI, who leads deep research in the chat GPT agent team on post -training and A16Z General Partner Sarah Wang, who's helped lead our investment in OpenAI since 2021.

1:03We talk about what's new in GPT -5 for major leafs and coding and creative writing to meaningful improvements in reasoning, behavior, and trust. We also get into training, our environments, and why data quality is more important than ever. We also cover agents, what that word actually means, the paradigm shift for Acing workflows, and the golden age for the idea guys. Let's get into it. As a reminder, the content here is for informational purposes only. Should not be taken as legal business, tax, or investment advice, or be used to evaluate any investment or security, and is not directed at any investors or potential investors in any A16Z fund.

1:40Please note that A16Z and its affiliates may also maintain investments in the companies discussed in this podcast. For more details, including a link to our investments, please see A16Z .com forward slash disclosures.

1:56So, slow news day. Not much going on for you guys. Thank you for coming. I know, obviously, you know, Tina, you were just on the livestream. We're recording a day of congratulations. Thank you. For those who are unfamiliar, why don't you introduce what you guys do at OpenA? Yeah, I'm Christina. I lead the core model team on post training. I'm Issa. I lead the deep research, like, chat GPT agent team on post training. And Tina, you've been here for, you both been here for, for one hour on you. Do you know what one you'd be a little bit of your history at the company? Yeah, I've been in opening eye for about four years now.

2:30I've originally worked on Web GPT, which was the original first LLM using tool use, but it was just one question. So the model learned how to use the browser tool, but you only asked one question, you got to answer back. And then we kind of just had this realization like, oh, you normally when you have questions, you have more questions after that. And so they started building this chatbot, and then eventually became chat GPT. And what would have been the reactions so far, you know, it's only been a few hours, but in your live stream, like, what are any reflections, any what can you, what can you tell us the day of?

3:02I'm honestly really excited. I think that obviously we have some great e -velle numbers and numbers are always really exciting, but I think the thing I'm like really excited about this model is just it's way more useful, like in cross like all the things that people actually use chat for. And it's not just like, and it's I think the e -velle numbers look good, but then also the way when people use it, I think they'll notice a quite big of a difference when the utility of it. I mean, this is my personal use cases. I use it for coding and writing all the time and it's just a huge step change. Sorry.

3:30You've been involved in helping lead our investments since 2021. What when you share more or tee up how you've been thinking about this as a relates to coding or more broadly. Yeah. Well, actually just on the topic of coding, it was a huge deal to have Michael Troll come on there and not only showcase the capabilities, but also say this is the best coding model in the market. And so just curious to the extent that you can share what did you do differently to get these results? Yeah, I think huge shout out to the team, especially Michelle Pokeris, like me, I think to get these things right, and like e -vail numbers is one thing, like I said, but to get the actual usability and like how great it is at coding, I think it takes a lot of detail and care.

4:14I think the team put a lot of effort into data sets and thinking about the reward models for this. But I think it's just literally just caring so much about getting coding working well. And maybe actually just a double click on front end web development. I mean, we've seen as sort of investors in the ecosystem that's obviously taken off in the last six to eight months. If you could pinpoint the improvement to that piece specifically, is it around, is it more around aesthetics? or is there sort of another capability leap forward in terms of what we can do with front end web development? I think there should be a lot more we can do with front end.

4:53I think the way we've gone this big leap, I mean, if you compare to O3's front end coding cable, this is just totally next level. Totally. It feels very different. And I think it kind of just goes back to what I was saying. The team just really cared about like nailing front end. And that means like getting the best data, like thinking about the aesthetics of the model and all of these things. I think it's just all those details that are really coming together and making the model like great effort and really exciting to see. Loved the demos in the live stream too. I wanted to ask about model behaviors because I know you worked on that too.

5:24But how did you guys think about that for GBT5? And there are a lot of things that we've talked about in prior models of like syncophancy and characteristics like that. How did you guys think about for this? What did you guys change or tweak? Yeah, the design of this model has been very, very intentional for model behavior, especially with the sick of and see issues that we had like a few months ago with Foro. And we've just spent a lot of time thinking about like, yeah, what is the ideal behavior? And I think for post -training, what's really, or one of the reasons I really like post -training is it feels more like an art than maybe even like other areas of research because you could have to make all these trade -offs, right?

5:59Like you have to think about like, for my rewards, like all these different rewards, I could be optimizing during the run, like how does that trade off against it, right? I want the assistant to be super helpful and engaging, but maybe that's a bit too engaging and getting too engaging gets to the overly effusive assistant that we have. So I think it's really a balancing act of trying to figure out what are the characteristics and what do we want this model to actually feel like. And I think we were really excited with GPT -5 because it's kind of a time to reset and rethink about, especially since it's so easy to make something, I think very engaging in the sense that in an unhealthy way, how can we make this a very healthy, helpful assistant?

6:38Same more about how you've achieved such as reduction in hallucinations, but also deception, what's the relationship between those? I guess for me, I find hallucinations, deception's pretty related. And we kind of saw this a lot with the reasoning models. The reasoning model would understand that it didn't have some ability, but then it still really wanted to respond. I think we really baked it into the models that they want to be helpful. And so they're like, whatever I can say, to be helpful in that moment. And that's kind of what we consider for like deception versus hallucinations. Sometimes the model like literally it seems that they will just say something quickly.

7:10And we kind of see a lot of this reduction with the thinking with when the models are able to stay step by step, they actually can like pause before blurting out any answers kind of what I feel like with a lot of the previous models for hallucinations. Over the next few weeks as you're evaluating, what are the biggest questions that you're having or that you're sort of anticipating being potentially answered? I'm just really curious to see how all of these things reflect in usage, right? Like, I think coding is way, way better. Like, what is this actually unlocked for people? And I think we're really excited to be offering these models at the price points that we have because I think this actually, like, unlocks like a lot more use cases that really weren't there before.

7:46Maybe like previous competitor models were, are good at coding, but the price point is not as exciting. And so I think with this number of capabilities that we have in this model and the price point, I'm kind of excited to see like all the new startups and like developers like doing things on top of it. Yeah, we're excited too. But by the way, just on the topic of usage, you obviously have a lot of products with a ton of usage already. And since we have one of the deep research gurus here too, how did deep research, chat, GPT operator, sort of your existing products and form how you went about approaching GPT 5?

8:21One thing that's interesting is with reinforcement learning, training a model to be good at as a bit capability is very data efficient. You don't need that many examples to teach it something new. So the way that we think about it on my team is we're trying to push capabilities and things that are useful to people. So deep research, it was the first model to do very comprehensive browsing. But then when O3 came out, it was also good at comprehensive browsing. and that's because we're able to take the data sets that we've created for the, you know, frontier agent models and then contribute it back to the frontier reasoning models.

9:00We always want to make sure that the capabilities that we're pushing with agents makes it into their flagship models as well. Yeah, that's great. Very self -reinforcing. You mentioned all the startups that you're excited to see come as flush out what you think that that could look like or even just highlight some opportunities you're more excited about because of this. I mean, people always say vibe coding. I think basically like non -technical people have such a powerful tool at their hands. I think really you just need some good idea and like you're not gonna be eliminated by the fact that like you don't know how to code something.

9:30Like you saw two of our demos, which were front end coding or in the beginning and that's just literally took minutes. I think that would have honestly taken me like a week to actually build like fully interactive. And so I think we're just gonna have a lot more. I would expect like maybe a lot more like indie type of like businesses built around this because of the fact that like you just need to have the idea right of simple prompt and then you get the full flesh out. It's the world of the idea as a guy. Yeah, I think so. Finally. Yeah. How about in the broader sort of AGI discourse, like what does this mean or accelerator or not, or like how do we think about sort of the broader AI discourse in terms of what does GBT5 mean here or change the conversation in any sort of way?

10:12I think with GBT5, because it's like a new, it's obviously state of the art and all the things we talked about. But I think if you're showing that we're can continue pushing the frontier here. I feel like there's always people who are like, oh, we're hurting a wall, things aren't actually improving. I think the interesting thing is I feel like we've almost saturated a lot of these e -vails in the real metric of how good our models are getting is I think and be usage. What are the new use cases that are being unlocked? How many more people are using this in their daily lives to help them across multiple tasks?

10:45So I feel like that's actually like the ultimate usage in terms of like that I'm excited about for terms of like are we getting to age you guys? Yeah, actually I think Greg made this comment about how he was comparing the last model to this model and the benchmark went from 98 to 99 It's like clearly we saturated the benchmarks I least on that that front -end is instruction following What benchmarks do you pay attention to like how do you guys think about e -vals right because given you're already saturating what's out there to a large extent or doing very well along those dimensions. What actually gets you to push the frontier is that so usage would be kind of post the model release, but before you get there, what are you guys looking to internally to help guide you?

11:27Is it a lot of internal e -vails that you created? Is it early access to start -up, seeing what they think, maybe it's a combo of all the above, but how do you weigh all those things? Yeah, I mean, I think on our team, we really work backwards from the capabilities we want the models to have. So maybe we want it to be good at creating slide decks or something or spread, as it's in spreadsheets. And then if eVals for those things don't exist, we try to make eVals that are representative measures of that capability in a way that's actually going to be useful for users. And then a lot of those in -tunnel will collect them maybe from human experts or trying some zazipi create examples, or we'll actually look at usage data.

12:12And then for us, we'll just try and heal climb on those. Yeah, I think we make this joke a lot internally that like if you want a nerd -type someone into working on something, you just need to make a good e -vow, and then, so happy it is trying to heal climb that. Yeah. I like what you said about starting with the capabilities first. How do you prioritize which you actually are shooting for? Let's say there's this dimension of maybe deeper into everyday use versus getting much deeper into the expert use cases How do you think about that trade -off? What does that trade -off mean practically speaking and what do you guys prioritize when?

12:48I mean, I think it's pretty unique at OpenAI to be able to work on something that's so generally useful. I mean, it's like everything They tell you not to do it a starter because it's like your user as anyone like for deep research We wanted it to be good across every single domain. Someone might want to do research in, and I think you only have the privilege of doing that if you work at a company that has huge distribution and all different kinds of users. So yeah, I mean, I think if you choose a capability that's quite general, like online research, you just have to make sure that you represent like a distribution of tasks across nodes of different domains if you want to get good at all of them.

13:26But then, yeah, sometimes it's hard to decide to focus on one specific thing because there are just so many different vessels that you could choose from, but I think in some cases maybe coding will be really important. So then a specific team will focus on coding, but I think in general, because the capabilities are so general, usually like the next model improvement just kind of improves performance on a pretty broad range. Yeah, I think we've kind of seen this like with the progression of even the models that we've haven't chat with you, like as a model gets smarter, it's better at instruction following, it's better at tool use.

14:03And like, some more things get unlocked as we just continue to make smarter models. So I think like a good chunk of our team also, like, does focus on just getting general intelligence up because I think the wins that we get from there are like, like, East is saying, like, pretty great. Whenever we get a new base model, it's just saying, like, oh, wow, suddenly this clicks, it works. And I think we kind of saw that moment with like, operator, because we have been working on computer usage, but I think it was hard to finally get the model to actually, without the multiple total capabilities to really support it, you couldn't have something like operator when it launched.

14:34Yeah, it's the same thing with everyone was talking about agents, but we didn't really have a way of actually training useful agents. I mean, I think everyone was talking, they're all these agent demos, but nothing that actually really works. But I think when we saw the reinforcement learning algorithm working really well on math and physics problems and coding problems, it became pretty clear, just from reading through the chain of thought, like, okay, this thing's actually like thinking and reasoning and backtracking and to build something that's able to like navigate the real world, it also needs to have that ability.

15:04So we realize, okay, like this is a thing that's going to actually let us get to useful agents. And so I think it's interesting at opening eye because you have people pushing, like, you know, foundational algorithms, getting really good at math, getting a gold medal in the IMO. And then on post training, we'll often take like those methods and try and figure out how to make things that are most useful and usable to all of our users. How much do the improvements are coming from the architecture versus the data versus the scale? How do you sort of think about that? My opinion, I'm very data -filled.

15:36I think data is very important. I think deep research was so good because ESA puts so much thought and careful attention to the data curation that they did and thinking about all the different use cases she wanted to have represented. So I'm on team data. Yeah, I mean, I think all are very important, but especially now that we have such an efficient way of learning, data is even higher quality data is even even more important. Maybe on the data topic, we've been talking a lot about RL environments. It's a popular space for startups who all want to work with you guys. And I was curious just to get your thoughts on this, you've been data, or your data -pilled.

16:19But what are the bottlenecks that you see for the next stage? Is that, I mean, maybe tying it to RL environments, is there sort of a lack of good, realistic RL environments that that's sort of the next frontier, which maybe creates an opportunity for these startups that once you sort of are able to really work within a environment that takes a long time to build, these are not, you know, sort of built in in a day or two that you can actually automate labor to the full extent of like compute, you know, the way that you and you computer used to do. Yeah, I think in my opinion, I do think there's a lot of value in getting really good tasks and getting really good tasks requires really good our own environment.

17:01I think the more complicated and with the more realistic, the more simulated we can make them, I think the better we'll get. And I think we're kind of seeing that like tasks matter just like task matter more at this point given the fact that we have such a strong algorithm So I think the data creating data and figuring out like the best tasks to train on is like the one of the big questions We have yeah like there's some generalization from training on like one website to another But you want to get really really good at something the best thing to do is just like train on that exact thing right so Yeah, I think we're definitely just constrained by how things that we can represent in a way that we can train on.

17:39The Charge EBC agent, for example, has such a general tool. It has a browser and a terminal. Between those two things, you can basically do most of the tasks that a human does on a computer. In theory, you can ask it to do anything that you can do on your computer. It's obviously not good enough to do that yet, but with the tools that has in theory, you can push it really, really far. So now we just have to like make it really good at all those things by you know training on training on way more things Let's talk about creative writing Maybe you talk about the improvements there. How do you think about it?

18:13That's one of my favorite improvements in GBT5 The writing I honestly find it's very tender and touching especially for a lot of the creative writing that we want to do We were thinking through like a bunch of different samples for the livestream and like every time I was like Oh, that's actually like, that hits like, it's like, it's like, it's like, it's like, it's like spooky. And I'm just like, oh, this feels like someone, like someone should have written this. Um, but I think it's really cool because you can actually really use it for, um, like helping you with things like, like, like my example I did in the live stream was like, writing, helping me write the uology.

18:46Something that like, that's like kind of hard to write, especially if it's writing. It's really something a lot of people are good at. Like, I'm personally a very, very bad writer. I'm not sure. I think it's a better story. I think it's a better story. But it's so great to have this tool to help me craft whenever I use it literally for a simple thing. Slack message to figure out how to phrase as well. It'll help me give me some iteration. How to say something to the team. I want to see those prompts. We're now just looking for M dashes. That was good. I was like, where do you stand on the M -dash?

19:24I like M -dash. I do that normally, now people think I'm just the same as D. I know me too. Going back to the discourse for a second, Sam said in his interview with Jack, he said, if you had said 10 years ago that we would get in models at the level of sort of PhD students, I would think, wow, the world looks so different. And yet we've basically taken it for granted. Do you think basically the improvements or similar, like as soon as we get them, we're just going to be like, oh, yeah. Now, now this is the standard, or do you think at some point, this is gonna be like, oh my God, this is like, how do you think about sort of people's ability to sort of acclimate or adjust or?

20:04Yeah, I mean, it seems like people adjust to really quickly, don't you think? I really, it's a tragedy to you, you got released in everyone, I was like, wow, that's so cool. But then you just kind of take it for granted that you literally have this like wizard in your pocket, you're like, ask it whatever random thought you have. And it just pops out like a good essay and you're like, oh, okay, cool. That's what's happening. I guess people adapt to things rather quickly. In my opinion, with technology, it is really easy. And I think because the form factor is so easy, even with like new tools, like deep research and chat, you be the agent, it's like presented in such like a, like easy way that people already know how to interface with.

20:36Like I think as long as that's true, even with the models getting like much smarter than us, like I think it would be, it's still going to be like quite approachable to people. Do you think the jump from GPT 4 to 5 was bigger or maybe 3 and a half to 4? I mean at least one thing for me and my usage of it is sometimes I'm wondering if I have hard enough questions to ask it to actually highlight the difference. Right. Because when it gets to a point where it's just answering what you need so well, it's like almost harder to tell the difference in some areas. But with writing, yeah, I've been using it for a few weeks and it's just kind of blown me away by any way that models previously haven't.

21:16Maybe I'm biased, I'm a recency bias, but I think this jump to four to five is most impressive for me, because I guess with 3 .5 when we first released it, the most common use case for me then also was still just for coding. And but now even though four was better at coding, I feel like the jump between four and five in terms of breadth of ability to do things is just way different and way more. And you can just handle a lot more complex things than before, with the context being much longer as well. I think the jump to 4 to 5 to me is much bigger. Is there anything the model categorically can't do?

21:51I guess for 5, we don't really take action in the real world yet. We're going to team up with Asian for that. Yeah, as I said, you could ask the agent to do anything, but it's not capable enough to do everything you want it to do yet. We take a conservative approach, especially with asking the user for confirmation before doing any kind of action that's irreversible, so like sending an email or ordering something, booking something. So I think I can imagine quite a number of tasks where you'd want to take the bulk actions, which you might not be able to do right now, because it would ask you every single time, but I think as people get more comfortable using these things and as they get better and you trust them more, you might allow it to do things for you without checking in with you as much.

22:37Maybe just to build on that question, And in terms of what it can't do today, but what you would sort of direct future research toward, if you look at coding something like end -to -end DevOps, for example, that feels like the logical next set of capabilities, do you guys think we'll get there in, I don't know what you'll name it, but 5 .5 or GPT -6, how far away from something like that? Yeah, I don't know about the exact thing of DevOps, but I do feel like with the models getting much smarter, One other thing that came to mind when you asked me the question is longer running tasks. And I think GBG5 is great because within a couple of minutes maybe you get a full -fledged app, but then what would it look like if you actually gave it like an hour, like a day a week?

23:21What can it actually get done? I think that's, there's gonna be a lot of interesting stuff. We're interested to see what will happen there. Yeah, I think a lot of it is not just about the model capability, but it's actually like how you set it up in a way to do things. like I'm sure that you could build something that's like monitoring, you know, your humio or like data dog whatever, like with these current models, it's just like setting up the harness like to make that possible. And same for, for like agentic tasks, I think a lot of things that will be quite useful will be when the agent like proactively does something for you, which I don't think is impossible today.