OpenAI’s CPO on what’s coming next: Hardware, GPT-5, Jony Ive, agents, more

26:30There's no there's no answer So how do you grade that right? That's what that's one problem The other is like as you start to take on more complex tasks. You're not just answering questions you're actually trying to automate some multi-step workflow, there may be ambiguity in the right way to do that. If I'm an AI booking a flight for you, there's not a single way to grade which correct flight. You also get into these really interesting, challenging, subjective ways of how do we actually grade this particular task? And part of having an eval, if you want to at least automate it, is you need to also have a grader for it so that you can very quickly understand how you're doing on that eval.

27:11So it is interesting. It's one of the skills that I think is going to be more and more important for PMs over time is the ability to actually create evals for the products that you're building. Yes, I mean, that's one. And the other one is the prompt at the front end. Because what we are starting to see with these leaks, and I don't know how real they are or not, but there are a number of X accounts that say I've just had the leaked system prompt of and then insert your favorite foundation model or coding tool. And the system prompts, which are the sort of structured instructions that go out with every query, are really, really quite complex.

27:49I mean, they're product in of themselves now, right? They run to thousands of words. They're highly structured. There's clearly strategies being applied as they get put together. So how important is that skill and capability when you're shipping products to people like me? I mean, actually, more than people realize, I think. I would love to make it, over time, less of a thing. And I think over time it is. Like, if you go back a year or two, everybody was talking about prompt engineering and it was going to be the skill that everybody had to master in order to do anything with AI. You don't hear it talked about quite as much like that.

28:27And I think that's a good thing. You know, ideally, it matters less and less that for any particular user, if they have a question, they want an AI to do something for them, you shouldn't need to get into like Arcana around, did I use the exact right word? And did I give my exam? You know, it should just work. I think that's part of increasing intelligence is the model can understand what you're looking to do and do a good job of it without you having to work super hard at it. That said, prompts still do matter and the models are very controllable with prompts. And so, you know, we still find we'll launch something and we'll find that it's not behaving in certain ways the way we want it to.

29:08And we can adjust it with a prompt a lot of times. You don't need to go back and like retrain the model. So it's both that I want to make it less necessary over time and that it is still a powerful vector. Well, these are two of the vectors of the direction of the product. But the third one is the idea of agents and what agents will bring to us. I think probably the first agent product that OpenAI launched that I used was deep research. And the word agent is being thrown about quite a lot. I mean, I'll use the word agent. And what I mean is I've strung a bunch of prompts together through your API.

29:43And there's a bit of logic to move a document through a series of steps to the other end. What does an agent mean to you? We think of an agent as something that can do independent work. So it's not just a quick, you know, you ask a question, you get an answer, but it's actually off doing tasks for you in the real world. So another, I think deep research is a great one where it's off doing, you know, hundreds of searches and putting together a complex report for you that might have taken you a week. I think another is Codex, which is our software engineering agent that we just launched. You know, what you can do is if you have a code base that you're operating against, you're building a new feature in a code base or debugging something, you can just give this agent the prompt like, hey, I need you to fix this thing.

30:28I want you to do this to the background of my web page. I want you to build this new feature. And it will go and look through your entire code base, understand all of the context. If you're fixing a bug, it'll go try and figure out where that bug exists. And then it will write new code for you and create a pull request like a diff. you know, here's the set of changes we need to make to the code. And then you can go review the code and the agent did the whole thing. And so you know, I used to be an engineer, I still write code a little bit in my spare time, but I haven't written a single, you know, line of code for open AI.

31:03But with Codex, I was this was like, you know, a few days before it launched, I was, you know, it was like 11 at night or something, I was doing a bunch of work that I had to do before I could go to bed. And I was like, you know what, I bet I could fix a bug right now. And so went and found a bug that looked relatively simple, and just, you know, pasted the context into codex said, can you go off and fix this bug? By the way, it was in a language that I had actually never worked with in my life. So it would have taken me even more time if I had to do it myself. And 10 minutes later, I had a pull request, it looked reasonable, I submitted it, an actual legitimate engineer looked at it and said, yeah, this looks right.

31:43And now there's a few lines of code shipping today that came from me using Codex. It speaks to the power of this thing when you can just have this software agent off actually solving real world tasks for you. And in the meantime, I was writing email and following up on Slack and doing all the things that I do in my day job. So it was just purely additive, which I think is really cool. Yes, because the codex process takes a little bit of time. It has a lot of material it has to read and understand and then make the changes. And I'm curious about, this is a question that everyone who's built a product that does code automation or developer augmentation is asked is, so today what portion of the OpenAI code base is in the first instance produced by codex rather than by a human engineer?

32:35Yeah, it's pretty meaningful and it's increasing quickly. Right, okay. Somewhere in the meaningful, I'll go up and ask 03 what meaningful means in percentage terms. And I'll get a good distribution there. The cool thing is you can fire off 10 of these tasks at once, right? So we try and actually give you the value of all this parallelism where it's not just you can do one thing. But if you have a codex agent working for you, why not have 10 codex agents working for you on 10 different tasks? And by the way, just to connect it to the previous topic on evals, there's a really important kind of subtlety to them too, where they have to be tailored to the product that you're trying to build and the problem that you're trying to solve.

33:19where coding isn't one thing. Coding is a small vertical of the entire world. But even within coding, you can be good at lots of different kinds of coding. And with Codex, that was a great example of going and saying, okay, what kinds of coding really matter to us? What kinds of tasks? And all the tasks that a developer does, what kinds of tasks do we really want to be good at? And we created evals for those. And then we made sure to monitor as we trained the model, is it getting better and better and better at these? and you go and accumulate tasks and examples for the model to learn from, but you do it against a specific set of evals that correspond to a specific set of problems you want to solve.

33:58It's very capability-driven in that respect, right? And then it speaks to how do you actually do enough testing both to make sure that you are getting to the right level of score that you want, but also making sure that it doesn't go off the rails, right? And I think that as these agents get more and more complex, given more complex tasks. That's something to bear in mind. I mean, in one of my workflows, which was a very, very simple one where I wanted an agent to go through and grab some data from a series of web pages and populate an Excel spreadsheet. And I was using some third-party agent framework.

34:38And it was so diligent, Kevin, that it said, I must check my work, which it did about 400 times and left me with a$75 bill and had got it correct the first time, right? Got it right and got stuck in this strange loop. So that's one of the things I think that I hear when I go around people saying, well, how are we going to be able to control these things? Not from a humanity out of control measure, but from a enterprise reliability. How can I make sure that this isn't like the sorcerer's apprentice and the thing runs out of control when I simply asked it to book one flight to Italy and it's booked me 200.

35:19And how do you test for all of that? Yeah, I think part of this is about making sure that like we talked about earlier that the user is in control here. So you should be able to at some point be like, hey, you know what, you've checked enough, like, you're good. And the other interesting thing in all of this is the technology is evolving so quickly, like much more quickly than I think we're used to with technology. We're used to things taking like decades to deploy and to really achieve scale. One of the phenomenons you see with AI technology is there'll be some benchmark, some eval that AI just can't crack.

36:01And people are like, oh, AI just can't do that. And then one day, somebody ships a model that gets like 5 % on that eval. Still mostly can't do the job, but just like begins to get it. And then what you inevitably find is like two months later, there's a model that's at 30 on that eval. And then four months later, there's a model that's at 60. And then, you know, within six months, it's completely saturated and like models are great at that new skill and will forever be. And so you go very quickly from like, proof of existence to like, oh yeah, of course AI models can do that. That like rate of development is still, I think, something that we're not totally used to.

36:47It's that first one or two percent, right, that becomes hard, that proves it can be done. It's the Kitty Hawk flyer. And then within 30 years, we're moving large numbers of passengers across the Atlantic. But in this case, it's within 30 days. So I want to ask about coding and coding agents. So if you look at the growth of Gen AI applications and SimilarWeb had some data come out a few weeks ago, the baseline was that the generalized chatbots are growing at 25 % per quarter. That's the chat GPTs and so on. Virtually every other product category, image generation, video generation, sound generation, is growing slower than that or declining in size.

37:29And I view that as the black hole that is the capability of your core models. The one category where growth was faster, 75 % a quarter according to SimilarWeb, was in coding. I was curious, have you selected coding from a kind of commercial perspective because you can really see the demand and developers are always willing to experiment? Or do you select coding because being a testable, structured, verifiable set of outputs, it's a slightly easier challenge than the sort of fuzzy amorphous tasks that occupy the rest of the world? Yeah, it's a really good question. And actually, coding is this vertical that kind of hits all of these things.

38:11You know, for one, it's really important to us because if we can speed up coding, if we can make, you know, every engineer more effective, we also make ourselves more effective. And so we can build even faster and we can bring AGI to the world faster. So it's interesting to us from that perspective it's a clear kind of milestone or step on the way to AGI itself, because it's a very sort of general purpose reasoning. It's also a relatively gradable task. Like you can tell like, like in math or other things, if you get the answer right, it's also something that our engineers are familiar with. So it's a problem space that they understand and have good intuition for.

38:50It's also a huge market, as you were saying. It's also a market full of early adopters. technologists leaning into this, it's also relatively sort of open and unregulated. It's not like trying to go into health or something where there are all kinds of other things you have to do. And so it's this aggregation of all of these interesting things that make coding a really interesting market. And I haven't seen that data, but I totally believe it. And would you say that within coding, are you already seeing signs of serving people like you who are not technically engineers anymore. In other words, we're seeing that expansion of the market through these tools.

39:31Oh, yeah. And I think there's going to be so much value in democratizing coding out to the world. There's like, what, 30 million developers or something worldwide, depending on how you define it, which is great. That's a lot of people. But imagine if a billion people can write code. I was talking to somebody the other day who was just telling me they were, during COVID, They were working for their local county, trying to get, you know, vaccinations and stuff out to people. And they were trying to put together a website to track so people could like sign up and just do basic stuff. And the whole world was busy and they just, they couldn't do it.

40:09They couldn't create a website. They didn't have the skills to do it. And as a result, they were, you know, managing things less efficiently, doing a bunch of manual work at a time when everybody was slammed. And he was just saying, like, can you imagine if I had these tools, we would have been able to create a website overnight. It would have just worked. And, you know, they would have been able to do their work more effectively. And you have that times a million as you look around the world. And so, I mean, that is actually the other thing about coding that I think is super interesting. That's maybe like the, you know, ninth reason why coding is a good, it's such a general purpose technology.

40:46If you can create code, then you can create all kinds of things. And so there's something really powerful to the idea that a billion people might be able to write code. But it also speaks, though, I think, to how this may fundamentally change the software industry in the way that the internet changes the software industry, not just because of packaging and distribution, but the way we interacted with social technologies, right? My Microsoft Word, when it was on floppy disk, never allowed me to sort of exchange notes with somebody else on like Google Docs. One of the big questions that's out there is, as a platform company that is building the most performant models out there, how much space do you leave for startups?

41:34I remember when Microsoft introduced disk compression in Windows 95 or 97 or something, and there were a whole load of third-party companies that offered disk compression that essentially went out of business there and then. And this is something that happens on X every time you release a new foundation model. It's like every time Kevin tweets, another 50 startups die. Where is their space, right? Where is their space in the software world, in the AI software world, which is safe from the increasing capabilities of the foundation models that you and other companies are building? So Stephen Sanofsky told me an interesting story about this one time.

42:12Stephen used to run Windows at Microsoft and Office and, you know, everything. and he was telling me this story about like the transition from windows 93 to windows or whatever it was called back then to windows 95 windows 3.1 maybe where it was just like the beginning of the internet and and so most people weren't using the internet and if you were going to actually get on the internet with windows 3.1 you had to like go to some you know university of Oregon professor's website and download a TCP IP stack, compile it yourself, and like, you know, install some device drivers, and then you could actually go on the internet.

42:51And then in Windows 95, of course, the internet was happening. And they were like, okay, we need to ship this stuff with Windows. And so they did. And it was like you were saying, there was this, you know, there were a bunch of people who were like, hey, now that, you know, you've just like, put that university of whatever a professor, he did all this work and now you just shipped it. Come on. And Stephen's point was that you would never want to live in a world where today you still had to go to like some professor's website and download a TCP IP stack and compile it yourself to get it going. You just want to use the internet.

43:23And basically the expectations of a platform, the consumer expectations of a platform are an increasing function of time. And if the platform can provide more of the technology that, you know, if you see something where in order to build the actual thing that people want to build, you've got 10 different companies having to go build the exact same piece of like foundational infrastructure, you should probably just provide that. And then those 10 companies can go do like more interesting stuff. And I've always remembered that story. It's really stuck with me because I think the fact that people are just going to expect more and more out of these platforms is very real.

44:02But the upside is all for third parties in this, for developers in this world, because if the platform provides more of the building blocks, then they can spend less time re-implementing the wheel on these building blocks and more time doing the thing that they actually uniquely add value in. And AI is going to change absolutely everything in our life. Any industry, any vertical, any geography that you can imagine, AI is going to touch. And so there's so much opportunity for developers to reinvent and reimagine. I think anything that we can do on the platform side to help accelerate that by making more of the building blocks easy, we should be doing.

44:43So let's imagine my son, 20 years old, and he wants to build a product on top of OpenAI. Where is a good place for him to go and build it? I mean, almost everywhere. There's so much opportunity. Sam said this one time and it stuck with me. He said, if you're building a company and you're building at the frontier of the model capabilities, if you're building something that really just barely works and you can't wait for our next model because you know it's going to make your product sing, then you're probably building in the right place because you're introducing something new to the world. You're making something possible that wasn't possible before.

45:28and that's where you want to be. If instead you're building like some sort of scaffolding around that covers up the weaknesses of a current model and you're actually afraid of our next model because it might not have those same weaknesses, that's a bad place to be building because on average, models are going to improve really fast and what's a weakness from one model will not be a weakness of the next. So like the, I think the thing to be building is like what we talked about at the beginning, re-imagining use cases from first principle building them from the ground up with AI. And if you're in a place where you're excited about the next model that comes out, because that's what's going to make your product sing, that's a great place to be.

46:07That's a fantastic heuristic, which is actually if you're a founder out there, think about something that you want to build that the models are not yet capable of, but will be capable in a little bit of time and you can build on top of that capability. We can't talk about products without talking about your new product buddy, Johnny Ive. So tell us a little bit about what the mood in the office was when that lovely black and white photo was released. Oh, people are incredibly excited. I mean, how could you not be? I use products that Johnny designed my entire day. He's been a part of building some of the most sort of cherished products and hardware that we use every day.

46:45How could you not want to work with him? And getting to know him through the process and other things, he's also such a lovely human being. For somebody who's accomplished so much, he is so humble, thoughtful, kind, soft-spoken. And then, you know, he'll say something sometime and you'll be like, oh my God, like that is a completely different way of looking at the world. And that opens my eyes to something that I just had never thought about. And so is this combination of like genius and also wonderful human being, how could you not be excited to work with him? And of course, he's a Brit. How is he going to work, his group and your group going to work?

47:27How are you guys going to interface? Well, I mean, he's coming in focusing on these sort of consumer hardware products. And then also over time, I think will play a very significant role in design as a whole at OpenAI. And again, I'm very excited about that. How could you not be? Johnny Ive coming in to take charge of a lot of your design, you know? you know he'll tackle the the drop down hopefully at uh at some point it's like we've got the eye the eye touch there and and so so i think that it's interesting to talk a little bit about hardware and how that interacts just in the last couple of minutes with the overarching vision so the overarching vision in a way you start at the beginning you talked about ai systems that will act a little bit like employees um i guess for people in their domestic lives that's more like like helpers.

48:18We don't generally have many. We think of it as a super assistant. Like a super assistant. And so what is the relationship between that and the need to have a hardware device alongside? I already have a hardware device. It's pretty good. I'm talking to it right now. It's more the opportunity. As we've said a few times, AI is going to touch every part of our lives and every part of our days in every part of the world. And that means, I think that there's an opportunity to reinvent and reimagine a lot of the services and the products that we use every day. You know, in some cases, there are a lot of products that I use every day that are great.

49:00They probably need to fundamentally change and they should with AI. And if they're not, they're not taking advantage of all these amazing new capabilities that we have, especially where those capabilities are going to be in 12 months, 24 months, 36 months. So I think there's an opportunity to reinvent and reimagine here. And that's true both on the software side and on the hardware side. So, you know, we have some thoughts about how that might occur. Obviously, Johnny has thought deeply about this, and we're excited to see what we can build together. And I'm sure there will be lots of others building in the space.

49:32And that's one of the reasons that we have, like, we put so much care and attention to our APIs and our developer platform. Because, The world is not just open AI. There's going to be incredible startups and incumbents and everybody else building really cool things using AI. And we'd like to power it in any case. Some of these will be first-party products that we build, and some will be other products that others build that leverage our models. And both of those things are really important to us. I mean, I hear that. I hear what you're saying as well, because I've already started to realize the limitations of the phone as the form factor for working with the models.

50:11You can't really put in a longish prompt to O3. I'm really reliant on talking to it. And if I'm in a noisy place, that won't work. The idea of having an ambient intelligence around me, I always have an AI model listening into my meetings and I'm talking to them regularly to do my work. So you start to see the limitations of something that's got the power draw of the phone and the size of the phone and does other things. So that will be a really exciting opportunity. And please sign me up for the alpha test. well before GA. We've got a couple of minutes. I just want to throw out some questions.

50:44How far behind are the top Chinese AI firms in core foundation model capability? Not as far behind as they used to be. And I think as US AI labs, we need to be very cognizant of that. I think it's really important that the leading models, the models that we all use, are models that are built off of democratic principles, not authoritarian ones. And we take that really seriously. Is there an AI app out there, whether it's in China or coming somewhere else, that's not built by OpenAI that you quite like and you like to use and play around with? I mean, I think a lot of the video apps are super fun.

51:19I also think, I also find Waymo magical. That's my go-to example of the way AI is touching our lives. You know, again, like, self-driving was like two years off for 10 years. And now suddenly it's here and it works and it's going to change a lot. It's absolutely magical. You are a keen runner, and I'm curious about whether you have a Garmin or a Suunto and what you would want from your exercise tracker that AI could bring that it isn't today. Ooh, that's a good question. Actually, I have an Apple Watch that I mostly use, and then if I'm doing like a 100-mile race or something, this doesn't quite have the battery, so I'll use a Garmin.

52:00What do I want? I think actually one of the things that I would love is better coaching. just like a little bit. And I think the AI is totally capable of doing it. I think Strava has some things that they're working on around this. But I would love to see better coaching and like AI analyze workouts and things like that. So that I got the, I think it's possible to get the kind of analysis that you would get from like a professional coach today from an AI for most users. And I feel like that's the kind of thing that, you know, five years from now, we're going to be like, oh yeah, I can't even imagine when that didn't exist.

52:34But it's only sort of peeking through right now. Just a little bit. But I do hear you. I think the possibility of having that personalized coaching would be quite sublime. So last question, when are you going to ship AGI? We're working on it. Every day we get a little closer. Every day. I mean, when will we know? Will we know? I think, look, I think it's one of those things that we talked about intelligence being multifaceted. There are already a bunch of places today where AI is way better than a human. And there are places where AI is like laughably worse than a human. But every, you know, every so, every like month or whatever, when there's new models, the baseline creeps up.

53:16And more and more things, AI is becoming superhuman at more and more things. And at some point, it's going to be superhuman at, you know, substantially everything. And we're going to call it, but it doesn't happen all at once. I think it's not like we go to bed one night and there's no AGI and we wake up in the morning and there's AGI. It's an incremental process of AI, you know, getting better and better at more and more things. Well, with that thought, Kevin, you keep climbing that hill. Thank you so much this morning for giving us your time. It's great to have you. Thank you for having me.