OpenAI’s CPO on how AI changes must-have skills, moats, coding, startup playbooks, more | Kevin Weil (CPO at OpenAI, ex-Instagram, Twitter)

24:55And so it's unclear to a lot of people just like, should I build a startup in the space or not? Is there any advice you have or any guidance for where you think OpenAI or just foundational models in general likely won't go and where you have an opportunity to build a company? Well, one of my, so this is something that Ev Williams used to say back at Twitter that's always stuck with me, which is no matter, no matter how big your company gets, no matter how like incredible the people are, they're way more smart people outside your walls than there are inside your walls. And that's why we are so focused on building a great API.

25:31We have 3 million developers using our API. No matter how ambitious we are, how big we grow, by the way, we don't want to grow super big, there are going to be, there are so many use cases, places in the world where AI can fundamentally make our lives better. We're not going to have the people, we're not going to have the know -how to build most of these things. And I think like I was saying, the data is industry specific, use case specific, behind certain company walls, things like that. And there are immense opportunities in every industry and every vertical in the world to go build AI -based products that improve upon the state of the art.

26:13And there's just no way we could ever do that ourselves. We don't want to, we couldn't if we did want to. And we're really excited to power that for 3 million plus developers and way more in the future. Coming back to your earlier point about the tech changing constantly and getting faster, not exactly knowing what you'll have by the time you launch something in terms of the power of the model. I'm curious what allows you to ship quickly and consistently and such great stuff. And it sounds like one answer is bottoms up and powered teams versus a very top -down roadmap that's planned out for a quarter.

26:49What are some of those things that allow you to ship such great stuff so often so quickly? Yeah, I mean, we try and we try and have a sense of where we're trying to go. You know, point ourselves in a direction so that we have some rough sense of alignment. Like thematically, I don't for a second. And we do quarterly road mapping. You know, we laid out sort of a year -long strategy. I don't for a second believe that what we write down in these documents is what we're going to actually ship, you know, three months from now, let alone six or nine. But that's okay. There's a, I think it's like an Eisenhower quote, plans are useless.

27:25Planning is helpful, which I totally subscribed to, especially in this world. It's really valuable. If you think about quarterly road mapping, for example, it's really valuable to have a moment where you stop and go, okay, what did we do? What worked? What went well? What didn't go well? What did we learn? And now what do we think we're going to do next? And by the way, everybody has some dependencies. You know, you need the infrastructure team to do the following things, partnership with research here. And so you want to have a second to kind of check your dependencies, make sure you're good to go and then start executing.

27:56We try and keep that really lightweight because it's not going to be right. You know, we're going to throw it out halfway because we will have learned new things. So the moment of planning is helpful, even if you're only going to, you know, it's only partially right. So that's I think just expecting that you're going to be super agile and that there's no sense writing a three month road map, let alone a year long road map because the technology is changing underneath you so quickly. We really do try and go like very strongly bottoms up kind of subject to our overall directional alignment. We have great people.

28:33We have engineers and PMs and designers and researchers who are passionate about the products they're building and have strong opinions about them and are also the ones building them. And so they have a they have a real sense of what the capabilities are too, which is super important. And so I think you want to be more bottoms up in in this way. And so we operate that way. We are happy making mistakes. We make mistakes all the time. It's one of the things I really appreciate about Sam. He pushes us really hard to move fast. But he also understands that with moving fast comes, we didn't quite get this right or you know, we lost this thing.

29:09It didn't work. We'll roll it back. You know, look at our naming. Our naming is horrible. There's a lot of questions people have for you. Yeah, model names. Yeah. It's absolutely atrocious and we know it. And we'll get around to fixing it at some point, but it's not the most important thing. And so we don't spend a lot of time on it. But it also shows you how it doesn't matter. Again, ChatGPT, the most popular fastest -growing product in history, models are, it's the number one AI API and model. So clearly it doesn't matter that much. And we name things like O3 Mini High. Oh man, I love it. Okay, so you talked about road mapping and bottoms up.

29:51And I'm really curious how you, is there like a cadence or ritual of aligning with you or Sam or he or you review everything that's going out? Like is there a meeting every week or every month where you guys see what's happening? On key projects, so we do product reviews and things like that like you would expect. There isn't a ritual because there isn't a, I would never want us to be blocked on launching something. You know, waiting for a review with me or Sam if we can't get there. If I'm traveling or Sam's, you know, busy or whatever, that's a bad reason for us not to ship. So obviously for the biggest, most high priority stuff, we have a pretty close beat on it, but we really try not to, frankly.

30:33Like we want to empower teams to move quickly. And I think it's more important to ship and iterate. So we have this philosophy that we call iterative deployment. And the idea is like we're all learning about these models together. So there's a real sense in which it's way better to like ship something, even when you don't know the full set of capabilities and iterate together like in public. And we kind of co -evolved together with the rest of society as we learn about these things and where they're different and where they're good and bad and weird. I really like that philosophy. There's also a bit of, I think the other thing that bit like ends up being a part of our product philosophy is the sense of like model maximalism.

31:22The models are not perfect. They're going to make mistakes. You could spend a lot of time building all kinds of different scaffolding around them. And by the way, sometimes we do because sometimes there are things, you know, kinds of errors that you just don't want to make. But we don't spend that much time building scaffolding around the parts that don't match that because our general mindset is in two months, there's going to be a better model and it's going to blow away whatever, you know, the current set of limitations are. And so if you're building and we say this to developers too, if you're building and the product that you're building is kind of right on the edge of the capabilities of the models, keep going because you're doing something right because you give it another couple months and the models are going to be great.

32:07And suddenly the product that you have that just barely worked is really going to sing. And you know, that's kind of how you make sure that you're really pushing the envelope and building new things. I had a the founder of Bolt on the podcast stack Blitz is the company name and he he's sure the story that they've been working on this product for seven years behind scenes and it was failing. Nothing was happening. And then Nelson, it was sorry to mention a competitor, but Claude came out or a sonnet 3 .5 came out and Nelson, everything worked. And they've been building all this time and finally worked.

32:40And I hear that a lot with YC just like things that never were possible now are just becoming possible every few months with the updates and the models. Yeah, absolutely. Let me actually ask this. I was implanting to ask this, but I'm curious if you have any quick thoughts. Just why why is a sonnet so good at coding and kind of thoughts on your stuff getting as good and better at actual coding? Yeah, I mean, Kudro's to Anthropic, they built very good coding models. No doubt we we think that we can do the same. Maybe by the time this podcast is shipped, we'll we'll have more to say, but either way all credit to them.

33:19I think this intelligence is really multi -dimensional. And so I think there's that the model providers, it used to be that OpenAI had this like massive model, you know, 12 months or something ahead of everybody else. That's not true anymore. You know, I like to think we still have a lead. I'd argue that we do, but it's certainly not a massive one. And that means that there are going to be different places where you know, the Google models are really good or where Anthropics models are really good or where we're really good and our competitors are like, ah, we got to get better at that. And it actually is easier to get better at a certain thing once someone's proved it possible than it is to, you know, forge a path through the the jungle and doing something brand new.

34:03So I just think, yeah, as an example, it was like nobody nobody could break four minutes in the mile. And then finally, somebody did the next year, 12 more people did it. I think there's that all over the place. And it just means that competition is really intense. And consumers are going to win and developers are going to win in businesses are going to win in a big way from that. It's part of why the industry moves so fast. But, you know, all respect to the other big model providers models are getting really good. We're going to move as fast as we can. And I think we've got some good stuff coming.

34:36Exciting. This makes me also think about in many ways other models are better at certain things. But somehow, ChatGPT is like the like if you look at all the awareness numbers and usage numbers, it's like no matter where you guys are in the rankings, people seem to just like think of AI and ChatGPT almost as as the same. What do you think you did, right? Kind of win and consumer mindset, at least at this point, and awareness in the world. I think being first helps, which is one of the reasons why we're so focused on moving quickly. We like being the first to launch new capabilities, things like deep research.

35:11We've also, our models are very, they can do a lot of things, right? So they can take real -time video input. You have speech to speech. You can do speech to text and text to speech. They can do deep research. They can operate on a canvas. They can write code. And so ChatGPT can kind of be this one -stop shop where all the things that you want to do are possible. And as we go forward in it, we have more agentec tools like operator where it's browsing for you and doing things for you on the web. More and more, you're going to be able to come to this one place to ChatGPT, give it instructions and have it accomplish real things for you in the world.

35:52There's something fundamentally valuable in that. We think a lot about that. We try to move really fast so that we are always the most useful place for people to come to. What would you say is the most counterintuitive thing that you've learned after building AI products or working at OpenAI? Something that's just like, I did not expect that. Maybe I should have expected this, but one of the things that's been funny for me is the extent to which you can kind of reason. When you're trying to figure out how some product should work with AI, you can often, or even why some AI thing happens to be true, you can often reason about it the way you would reason about another human.

36:37And it kind of works. So maybe a couple of examples. When we were first launching our reasoning model, we were the first to build a model that could reason. Instead of giving you just a quick system one answer right away at every question you asked, it was the third emperor of the Holy Roman Empire. Here's an answer. You could ask it hard questions and it would reason. The same way that if I asked you to do a crossword puzzle, you couldn't just snap, fill in everything. You would be, well, this one across, I think it could be one of these two, but that means there's an A here. So that one has to be this away.

37:14Backtrack, kind of step by step, build up from where you are. Same way you answer any difficult logistical problem, any scientific problem. So this reasoning breakthrough was big, but it was also the first time that a model needed to sit and think. And that's a weird paradigm for a consumer product. You don't normally have something where you might need to hang out for 25 seconds after you ask a question. And so we were trying to figure out, we know what's the UI for this because it's also not like with deep research where the model is going to go and think for 25 minutes sometimes. It's actually not that hard because you're not going to sit and watch it for 25 minutes.

37:53You're going to go do something else. You're going to go to another tab or go get lunch or whatever. And then you'll come back and it's done. When it's like 20, 25 seconds or 10 seconds, it's a long experience. It's a long time to wait, but it's not long enough to go do something else. And so you actually need, and so you can think, like if you asked me something that I needed to think for 20 seconds to answer, what would I do? I wouldn't just like go mute and not say anything and kind of shut down for 20 seconds and then come back. So we shouldn't do that. We shouldn't just like have a slider sitting there.

38:28That's annoying. But I also wouldn't just start babbling every single thought that I had. So we probably shouldn't just like expose the whole chain of thought as the models thinking. But I might go like, huh, that's a good question. All right, I might approach it like that and then think, you know, you're sort of like maybe giving little updates. And that's actually ended up what we ended up shipping. You have similar things where you can like you can find situations where you get better thinking sometimes out of a group of models that all try and attack the same problem. And then you have a model that's looking at all their outputs and integrating it and then giving you a single answer at the end.

39:06I mean, sounds a little bit like brainstorming, right? I certainly have better ideas when I get in a room and brainstorm with other people because they think differently than me. And so anyways, there's just like all these situations where you can actually kind of reason about it like a group of humans or an individual human and it sort of works, which I don't know. Maybe I shouldn't have been surprised, but I was. That is so interesting because when I see these models operate, I like, I never even thought about you guys designing that experience. Like to me, just feels like this is what the LLM does.

39:37It just sits there and tells me what it's thinking. And I love this point you're making of like we like, let's make it feel like a human operating. And well, how does human operate? Well, they just talk a lot. They think here's the thing I should explore. And I love that deep sequined like to the extreme of that, right? Where they're just like, here's everything I'm doing and thinking in. And people actually like that too, I guess was that was that surprising to you like, oh, maybe that could work too. People seem to like everything. Yeah, we learned from that actually because we, when we first launched it, we kind of gave you like the subheadings of what the model was working about, but not much more.

40:12And then deep seek launched and it was a lot. And we kind of went, you know, I don't know if everyone wants like that. There's some novelty effect to seeing what the models really think about. We felt that too when we were looking at it internally. It's interesting to see the models chain of thought, but it's not, you know, I think at the scale of like 400 million people, you don't want to see the model kind of like babble a bunch of things. And so what we ended up doing was summarizing it in interesting ways. So instead of just getting the subheadings, you're kind of getting like one or two sentences about how it's thinking about it.

40:43And you can learn from that. So we kind of tried to find a middle ground that we thought was an experience that would be meaningful for most people. But you know, showing everybody like three paragraphs is probably not the right answer. This reminds me something else. You said at the summit that has really stuck with me, this idea that chat people always make fun of like chat is not like the future interface for how we interact with AI. But you made this really interesting point that may argue the other side, which is like as humans, we interface by talking and the IQ of a human can span from really low to really high.

41:17And it all works because we're talking to them. And chat is the same thing and it can work on all kinds of intelligence levels. Maybe just share, maybe I just shared it, but I guess anything there about just why chat actually ends up being such an interesting interface for all of us. Yeah, I don't know if maybe I'm maybe this is one of those things I believe that most people don't believe, but I think chat is an amazing interface because it's so versatile. People tend to go, oh, yeah, well, that's just like, you know, we'll figure out something better. And I kind of think, I kind of think this is, it's it's it's incredibly universal because it is the way we talk.

41:51Like I can talk to you verbally, like we're talking now, I can, you know, we can see each other and interact. We can talk on WhatsApp and, you know, be texting each other. But all of these things is this sort of like unstructured, you know, method of communication. And that's how we operate. If I had to, and if I had some more rigid interface that I was allowed to use when we spoke, I would be able to speak to you about, you know, far fewer things and it would actually get in the way of us having like maximum communication bandwidth. So there's something magical. And by the way, in the past, it never worked because models, there wasn't a model that was good at understanding all of the complexity and nuances of human speech.

42:34And that's the magic of LLM's. So to me, it's like an interface that's exactly fit to the power of these things. And that doesn't mean that it always has to be just like, I don't necessarily always want to type. But if you do want that very open -ended flexible communication medium, it may be that we're speaking and the model speaking back to me. But you still want that like that very sort of lowest common denominator, no restrictions way of interacting. That is so interesting. That's really changed the way I think about the stuff is that point that chat is just so good for this very specific problem of talking to a superintelligence basically.

43:13By the way, I think there are like, it's not that it's only chat either. Like there are, if you have high volume use cases where they're more prescribed and the, you don't actually need the full generality, there are many use cases where it's better to have something that's less flexible, more prescribed, faster to specific tasks. And those are great too. And you can build all sorts of those. But you still want chat as like this baseline for anything that falls out of whatever, you know, vertical you happen to be building for. It's like a catch -all for like every possible thing you'd ever want to express to a model.

43:50I'm excited to chat with Christina Gilbert, the founder of one schema whenever a long time podcast sponsors. Hi Christina. Yes, thank you for having me on, Lenny. What is the latest with one schema? I know you now work with some of my favorite companies like Ramp, Vanta, Scale and Watershed. I heard that you just launched a new product to help product teams import CSVs from especially tricky systems like ERPs. Yes, so we just launched one schema file feeds, which allows you to build an integration with any system in 15 minutes as long as you can export a CSV to an SFTP folder. We see our customers all the time getting stuck with hacks and workarounds.

44:27And the product teams that we work with don't have to turn down prospects because their systems are too hard to integrate with. We allow our customers to offer thousands of integrations without involving their engineering team at all. I can tell you that if my team had to build integrations like this, how nice would it be to be able to take the software roadmap and instead use something like one schema and not just to build it but also to maintain it Absolutely Lenny. We've heard so many four stories of multi -day outages from even just a handful of ad records. We have laser focused on integration reliability to help teams end all of those distractions that come up with integrations.

45:00We have a built -in validation layer that stops any bad data from entering your system and once schema will notify your team immediately of any data that looks incorrect. I know that importing incorrect data can cause all kinds of pain for your customers and quickly lose their trust. Christina, thank you for joining us and if you want to learn more, head on over to oneschema .co. That's oneschema .co. I want to come back to the you talked about researchers and their relationship with product teams. I imagine a lot of innovation comes from researchers just like having an inkling and then building something amazing and then releasing it and some ideas come from PMs and engineers.

45:39How did those teams collaborate? Does every team have a PM? Is it a lot of research -let stuff? Just like what give us an extensive where ideas and products come from mostly? It's an area where we're evolving a lot. I'm really excited about it frankly. I think if you go back, you know, a couple years when chat GBT was just getting started. Obviously, I wasn't an open AI, so but we were more of a pure research company at the time. A chat GBT, if you remember, was a low -key research preview. It was for many years. Yeah, it wasn't a thing that the team launched thinking it was going to be this massive product.

46:19Oh, ChatGPT. Yeah. It was just a way that we were going to let people play with and iterate on the models. We were primarily a research company, a world -class research company. As ChatGPT has grown and as we've built our B2B products and our APIs and other things, now we're more of a product company than we were. I still think we can't work. Open AI should never be a pure product company. We need to be both a world -class research company and a world -class product company. The two need to really work together. That's the thing that I think we've been getting much better at over the last six months.

47:00If you treat those things separately, and the researchers go do amazing things and build models, and then they get to some state, and then the product and engineering teams go take them and do something with them. We're effectively just an API consumer of our own models. The best products, though, are going to be, it's like I was talking about with deep research. It's a lot of iterative feedback. It's understanding the products you're trying to solve, or the problems you're trying to solve, building e -vals for them, using those e -vals to go gather data and fine -tune models to get them to be better at these use cases that you're looking to solve.

47:35It's a huge amount of back and forth to do it well. I think the best products are going to be end product design and research working together as a single team to build novel things. That's actually how we're trying to operate with basically anything that we build. It's a new muscle for us because we're kind of new as a product company, but it's one that people are really excited about because we've seen every time we do it, we build something awesome. Now, every product starts like that. How many product managers do you have at OpenAI? I don't know if you share that number, but if you do. Not that many, actually.

48:13I don't know. 25. Maybe it's a little more than that, but my personal belief is that you want to be pretty PM -light as an organization, just in general. I say this with love because I am a PM, but too many PMs cause this problem. We'll fill the world with decks and ideas versus execution. I think it's a good thing when you have a PM that is working with maybe slightly too many engineers because it means they're not going to get in a micromanage. You're going to leave a lot of influence and responsibility with the engineers to make decisions. It means you want to have really product -focused engineers, which we're fortunate to have.

48:57We have an amazingly product -focused high -agency engineering team. When you have something like that, you have a team that feels super empowered. You have a PM that's trying to really understand the problems and kind of gently guide the team a little bit, but has too much going on to get too far into the details. You end up being able to move really fast. That's the philosophy we take. We want product -the -engineers all the way through. We want not too many PMs, but really awesome high -quality ones. So far, that seems to be working pretty well. I imagine being a PM at OpenAI is a dream come true for a lot of people.

49:40At the same time, I imagine it's not a fit for a lot of people. There's researchers involved. Very product -minded engineers. What do you look for in the PMs that you hire there for folks that are like, maybe I shouldn't go work there, I shouldn't even think about that? I think, I've said this a few times, but high -agency is something that we really look for. People that are not going to come in and wait for everyone else to allow them to do something. They're just going to see a problem and go do it. It's just a core part of how we work. I think people that are happy with ambiguity because there is a massive amount of ambiguity here.

50:17It is not the kind of place. We have trouble sometimes with more junior PMs because of this because it's just not the place where someone is going to come in and say, okay, here's the landscape. Here is your area. I want you to go do this thing. That's what you want as an early career PM. No one here has time and the problems are too ill -formed and we're figuring them all out as we go. High -agency, very comfortable with ambiguity. Ready to come in and help execute and move really quickly. That's our recipe. I think also happy leading through influence because, I mean, it's usual as a PM. People don't report to you.

51:03Your team doesn't report to you, etc. But you also have the complexity of a research function, which is even more self -directed. It's really important to build a good rapport with the research team. I think the EQ side of things is also super important for us. I know that most companies at PM comes in and they're just like, why do we need you? As a PM, you have to earn trust and help people see the value. I feel like it's probably a very extreme version of that where they're like, why do we need this person? We have researchers, engineers. What are you going to do here? Yeah, I think people appreciate it, done right.

51:42But you bring people along. I think one of the most important things a PM can do well is be decisive.

51:50There's a real fine line. I mean, it's kind of like, I don't love the PM as the CEO of the product, illusion all the time. But just like Sam and his role would be making mistakes if he made every single decision in every meeting that he was in. He would also be making mistakes if he made no decisions in any meetings that he was in. It's understanding when to defer to your team and to let people innovate. When there is a decision to be made that people either don't feel comfortable with or don't feel empowered to make or a decision that has too many different disparate pros and cons that are spread out across a big group and someone needs to be decisive and make a call.

52:39It's a really important trade of a CEO. It's something Sam does well. It's also a really important trade of a PM at a more microscopic level. Because there's so much ambiguity, it's not obvious what the answer is in a lot of cases. Having a PM, they can come in and by the way, this doesn't need to be a PM. I'm perfectly happy if it's anybody else. But I kind of look to the PM to say, if there's ambiguity and no one's making a call, you better make sure that we get a call made and we move forward. This touches on a few posts. I've done of just where is AI going to take over work that we do versus help us with various work.

53:14So let me come at this question from a different direction of just how AI impacts product teams and hiring things like that. So first of all, there's always talk of LM's doing our coding for us and 90 % of code is going to be written by AI in a year. Dario and Anthropics said that. At the same time, you guys are all hiring engineers like crazy, PM's like crazy. Every function is dead, but you're still hiring every single one. I guess just first of all, let me just ask this, how do you and the team like say engineers, PMs use AI in your work? Is there anything that's like really interesting or things that you think people are sleeping on and how you use AI in your day -to -day work?

53:51We use it a lot. I mean, every one of us is in chat GPPT all the time. Summarizing docs using it to help write docs with GPPT's that write product specs and things like that. All the stuff that you would imagine. I mean, talk about writing e -vals. You can actually use models to help you write e -vals and they're pretty good at it. That all said, I still don't, I'm still sort of disappointed by us and despite, I really mean me. If I were to just like teleport my five -year -old self leading product at some other company into my day job, I would recognize it still. I think we should be in a world certainly a year from now, probably even more now that I almost wouldn't recognize it because the work clothes are so different and I'm using AI so heavily.

54:41I'd still recognize it today. In some sense, I'm not doing a good enough job of that. Just to give an example, why shouldn't we be vibe coding demos right left in center? Instead of showing stuff in Figma, we should be showing prototypes that people are vibe coding over the course of 30 minutes to illustrate proofs of concept and to explore ideas. That's totally possible today and we're not doing it enough. Actually, our chief people officer, Julia, was telling me the other day, she vibe coded an internal tool that she had at a previous job that she really wanted to have here at OpenAI and she opened, I don't know, Windsor for something and vibe coded it.

55:26How cool was that? If our chief people officer is doing it, we have no excuse to not be doing it more. That's an awesome story. Some people may not have heard this term vibe coding. Can you describe what that means? I think this was Andres' term.

55:47You have these tools like cursor and Windsor that get up copilot that are very good at suggesting what code you might want to write so you can give them a prompt and it will write code and then as you go to edit it, it's suggesting what you might want to do. The way that everyone started using that stuff was give it a prompt, have it do stuff, you go edit it, give it a prompt, and you're really going back and forth with the model the whole time as the models are getting better and as people are getting more used to it, you can just let go of the wheel a little bit and when the model is suggesting stuff, it's just like tap, tap, tap, tap, like keep going, yes, yes, yes, yes, yes.

56:28Of course, the model makes mistakes or it does something that doesn't compile, but when it doesn't compile, you paste the error in and you say go, go, go, go, go, and then you test it out and it does one thing that you don't want it to do so you enter in an instruction and say go, go, go, go, go, and you just kind of let the model do its thing and it's not that you would do that for production code that needed to be super tight today yet, but for so many things, you're trying to get to a proof of concept, you're getting to a demo and you can really take your hands off the wheel and the model will do an amazing job and that's what that's that's fine coding.

57:05That's an awesome explanation. I think like the pro version of that, which is I think the way Andre even described it as you talk, you do like a there's a step like whisper, super whisper, something like that where you're like talking to the model, not yeah, not even typing. Yeah, totally. Oh man. So let me let me just ask I guess when you look at product teams in the future, you talked about how you guys should be doing this more instead of designs having prototypes, what do you think might be the biggest changes in how product teams are structured or built, where do you think they're going in the next few years?

57:36I think you're definitely going to live in a world where you have more where you have researchers built into every product team and I don't even mean just it at like foundation model companies because I think the future, I actually, frankly, one thing that I'm sort of surprised about about our industry in general is that there's not a greater use of fine tuned models. Like a lot of people, you know, these models are very good. So our API does a lot of things really well. But when you have particular use cases, you can always make the model perform better on a particular use case by fine tuning it.

58:12It's probably just a matter of time. You know, folks aren't like quite comfortable yet with doing that in every case. But to me, there's no question that that's the future every, you know, models are going to be everywhere just like transistors or everywhere. AI is going to be just a part of the fabric of everything we do. But I think there are going to be a lot of fine tuned models because why would you not want to more specifically customize a model against a particular use case? And so I think you're going to want sort of quasi researcher machine learning engineer types as part of pretty much every team because fine tuning a model is just going to be part of the core workflow for building most products.

58:50So that's one change that maybe you know, you're starting to see a foundation model companies that will propagate out to more teams over time. I'm curious if there's a concrete example that makes that real. And I'll share one that comes to mind as you talk, which is when you look at cursor and Windsor for something I learned from those founders is that they they use like a sonnet. But then they also have a bunch of custom models that help along the edges that make the specific experience. That's not just generating code, even better like auto complete and looking ahead to where things are going.

59:20So is that one or any other examples of what you what what what is a fine tune model there? And I think these things will be building with these researchers on their teams. Yeah, I mean, so when you're fine tuning a model, you're basically giving the model a bunch of examples of the kinds of things you want it to be better at. So it's here's a problem, here's a good answer, here's a problem, here's a good answer, or here's a question, here's a good answer, you know, times a thousand or ten thousand. And suddenly you're you're teaching the models to be much better than than it was out of the gate at that particular thing.

59:55We use it everywhere internally. We also we we use ensembles of models much more internally than people might think. So it's not here is I have ten different problems. I'll just ask, you know, baseline GPT 40 about a bunch of these things. If we have ten different problems, we might we might solve them using, you know, 20 different model calls, some of which are using specialized fine tune models. They're using models of different sizes because maybe you have different latency requirements or cost requirements at different for different questions. They are probably using custom prompts for each one.

1:00:35Like basically you want the to teach the model to be really good. It you want to break the problem down into more specific tasks versus some broader set of high level tasks. And then you can use models very specifically to get very good at each individual thing. And then you know, you have an ensemble that sort of tackles the whole thing. I think a lot of good companies are doing that today. I still see a lot of companies kind of giving the model single generic broad problems versus breaking the problem down. And I think there will be more breaking the problem down using specific models for specific things, including fine tuning.

1:01:15And so in your case, because this is really interesting, is that you're using different levels of chat GPT like a 103 and that's really super. They'll be parts of our internal stack. So we do, if you give you an example, customer support with 400 plus weekly, 400 plus million weekly active users, we get a lot of inbound tickets, right? I don't know how many customer support folks we have, but it's not very many. 3040, I'm not sure. We're way smaller than you would have at any comparable company. And it's because we've automated a lot of our flows. We've got, you know, most questions using our internal resources, knowledge, base, you know, guidelines for how we answer questions, what kind of personality, et cetera.

1:02:03You can teach the model those things and then have it do a lot of its answers automatically. Or where it doesn't have, you know, the full confidence to answer a particular question, it can still suggest an answer, request a human to look at it. And then that human's answer actually is its own sort of fine tuning data for the model. You're telling it the right answer in a particular case. And we're using it various places, you know, some of these places you want a little bit more reasoning is not super latency sensitive. So you want a little more reasoning and we'll use one of our O series models.

1:02:38In other places, you want a quick check on something. And so you're fine to use like four O mini, which is super fast and super cheap. And in general, it's like specific models for specific purposes. And then you, you, you ensemble them together to solve problems. By the way, again, not unlike how we as humans solve problems. A company is arguably an ensemble of models that have all been, you know, fine tuned in based on what we studied in college and what we have like learned over the course of our careers. We've all been fine tuned to have different sets of skills. And you like group them together in different configurations.

1:03:15And the output of the ensemble is much better than the output of anyone individual. Kevin, you're blowing my mind. That sounds exactly correct. And also different people, you pay them less. They cost less to talk to. Some people take a long time to answer. Some people hallucinate. This is, I'm telling you, this is like, this is a mental model, but it really does work in thinking about it. This is great. Some people are visual. They want to dry out their thinking. Some people want to talk word cell. Wow. This is a really good metaphor. So again, coming back to your advice here, because I love that we circled back to it.

1:03:52It's, you're finding a really good way to think about how to design great. Yeah, experiences and LMs, I guess specifically think about how a person would do this. Well, it's, it's maybe not always the answer is to think about how a person would do it, but sometimes to gain intuition for how you might solve a problem, you think about what an equivalent human would do in those situations and use that to, to, you know, at least gain a different perspective on the problem. Wow. This is great. There's just like, you know, because some of this really is talking to a model, there's a lot of prior art because we talk to other humans all the time and encounter them in all sorts of different situations.

1:04:31And, and so like, there's a lot to learn from that. Okay. So speaking of humans, I want to chat about the future a little bit. So you have three kids and someone, a community member asking me this hilarious question that I think gets at something a lot of people are thinking about. So this is Patrick Srell. I worked at him with a mid -air beam B hasks. He says, ask what he's encouraging us kids to learn to prepare for the future. I'm worried my six year old by the year 2036 will face a lot of competition trying to get into the top roofing or plumbing programs and you get back up plan. That's funny.

1:05:04So our kids are, we have a 10 year old and eight year old twins. So they're, they're still pretty young. They're kind of, I mean, it's amazing how AI native they are. They just, it's completely normal to them that there are self -driving cars that they can talk to AI all day long. They have full conversations with JGPT and Alexa and everything else. I don't know. I think who knows what the future holds? I think, you know, things like coding skills are going to be relevant for a long time. Who knows? But I think if you teach your kids to be curious, to be independent, to be self -confident, you teach them how to think.

1:05:47I don't know what the future holds, but I think that those are going to be skills that are going to be important in any configuration of the future. And so, it's not like we have all the answers, but that's how Elizabeth and I think about our kids. And do you find that AI? There's a lot of talk about AI tutoring. Is that something you guys are doing? Anything you're, I know they're using JGPT. I love all the photos you post. They're playing with prompts and stuff. But I guess is there anything there you're experimenting why a third thing is going to become really important? This is something that it's maybe the most important thing that AI could do.

1:06:25Maybe that's a grand statement. There are lots of important things that AI can do, including like speeding up the pace of fundamental science research and discovery, maybe is actually the most important thing AI can do. But one of the most important things would be personalized tutoring. And it kind of blows my mind that there is still, I know there are a bunch of good products out there. Like, you know, Khan Academy does great things. There are wonderful partner of ours. Vinod Kostola has a nonprofit that's doing some really interesting stuff in the space and was making an impact. But I kind of want, like I'm kind of surprised that there isn't like a two billion kid, you know, AI personalized tutoring thing because the models are good enough to do it now.

1:07:13And every, every study out there that's ever been done seems to show that when you have, you know, classrooms is still classroom to use. Like education is still important. But when you combine that with personalized tutoring, you get like multiple standard deviation improvements and learning speed. And so it's just, it's uncontroversial. It's good for kids. It's free. Chat GPT is free. You don't need to pay for it. And the models are good enough. Like, it still just kind of blows my mind that there isn't something amazing out there that, you know, our kids are using and your future kids are using and like people in all sorts of places around the world that aren't as lucky as our kids to be able to like have this sort of built -in solid education.

1:07:58Again, chat GPT is free. People have Android devices everywhere. Like this could, I really just think this could change the world and I'm surprised it doesn't exist and I want it to exist. This kind of touches on something I want to spend a little time on, which is a lot of people also worry a lot about AI where it's going. They worry about jobs. It's going to take. They worry about, you know, the super intelligence squashing humanity in the future. What's kind of your perspective on that and just kind of the optimistic case that I think people need to hear. I mean, I'm a big technology optimist.

1:08:29I think if you look over the last 200 years, maybe more, technology has driven a lot of the advancements that have made us the world and the society that we are today. It drives economic advancements. It drives geopolitical advancements, quality of life, longevity, advancement. I mean, technologies are the root of just about everything. So, I think there are very few examples where this is anything but a great thing over the longer term. That doesn't mean that there aren't temporary dislocations or where there aren't individuals that are impacted and that matters too. So it can't just be that the average is good.

1:09:12You've got to also think about how you take care of each individual person as best you can. So, it's something that we think a lot about and as we work with the administration as we work with policy, we try and help wherever we can. We do a lot with education. One of the benefits here is that ChatGPT is also perhaps the best reskilling app you could possibly want. It knows a lot of things. It can teach you a lot of things if you're interested in learning new things. But these are very real issues. I'm super optimistic about the long run and we're going to need to do everything we can as a society to ensure that we make this transition as graceful and as well supported as we can.

1:09:59To give people a sense of where things might be going, that's a big question. A lot of people mind so someone asked this question that I love, which is AI is already changing, creative work in a lot of different ways, writing and design and coding. What do you think is the next big leap? What should we be thinking is the next big leap in AI assisted creativity specifically and then just broadly? What do you think things are going to be going in the next years? Yeah. This is also an area where I'm a big optimist. If you look at Sora, for example, we talked about ImageGen earlier and the absolute like, Fount of Creativity that people are putting across Twitter and Instagram and other places, I am the world's worst artist.

1:10:41The worst. Maybe the only thing I'm worse at than art is singing.

1:10:49Give me a pencil and a pad of paper and I can't draw better than my five and then our eight year old. It's just like it's, but give me ImageGen and I can think some creative thoughts and put something into the model and suddenly have output that I couldn't have possibly done myself. That's pretty cool. Even you look at folks that are really talented, I was talking to a director recently about Sora, someone who's directed films that we would all know. And he was saying, for a film that he's doing, take the example of some sci -fi -ish, think of Star Wars and you've got some scene where there's a plane zooming into some death star -like thing.

1:11:37So you've got the plane looking at the whole planet and then you want to cut to a scene where the plane's at the ground level and all of a sudden you see the city and everything else. How are you going to manage that cut scene and that transition? He was saying, in the world of two years ago, I would have paid a 3D effects company 100 grand and they would have taken a month and they would have produced two versions of this cut scene for me and I would have evaluated them. We would have chosen one because what are you going to do? Like pay another 50 grand and we had another month and we would have just gone with it.

1:12:20And it would be fine. Movies are great. I love them and obviously we can do great things with the technology that we've had. But you now look at what you can do with Sora and in his point was now I can use Sora, our video model and I can get 50 different variations of this cut scene. Just me brainstorming into a prompt and the model brainstorming a little bit with me. I've got 50 different versions and then of course I can iterate off of those and refine them and take different ideas. Now I'm still going to go to that 3D effects studio to produce the final one. But I'm going to go having brainstormed and I much more creative approach with an outcome that's much better.

1:13:04I did that assisted by AI. So my personal view on creativity in general is that no one's going to, you don't type into Sora, like make me a great movie. It requires creativity and ingenuity in all these things. But it can help you explore more. It can help you get to a better final result. So again, I tend to be an optimist in most things but I'm actually I think there's a very good story here. I know Sam Altman, I think it was him who tweeted recently the creative writing piece that you guys are working on where to say he's very bad at writing creative stuff and he shared an example where it's actually really good.

1:13:40Imagine that's another area of investment. Yeah, there's some exciting stuff happening internally with some new research techniques. So we'll have more to say about that at some point. But yeah, Sam sometimes likes to show off some of the stuff that's coming. This is hard. By the way, it's like very sort of indicative of this iterative deployment philosophy. We don't have some breakthrough and keep it to ourselves forever and then bestow it upon the world someday. We kind of just talk about the things we're working on and share when we can and launch early and often and then iterate in public.

1:14:19And I really like that philosophy. I love all these hints that a few things coming. I know he can't say too much. You talked about how there might be a coding leap coming in the near future. Maybe by the time this comes out. Is there anything else people should be thinking about might be coming in the near future? Anythings you can tease that are interesting, exciting. Man, this hasn't been enough for you. Only everything is getting better every day. Yeah, I'm like, man, I hope we get some of the stuff out before the episode launches. It's your new time box. I don't piss people off.

1:14:54The amazing thing to me is we were talking earlier about how far models have come in just a couple years. If you went back to GPT -3, you'd be like disgusted by how bad it was even though any of two years ago was mind blown by how good these were. And for a long time, we were iterating every six to nine months on a new GPT model. It was like GPT -3, GPT -3 .5, four. And now with this O -series of reasoning models, we're moving even faster. Where every roughly three months, maybe four months, there's a new O -series model. And each of them is a step up in capability. And so the capabilities of these models are increasing in a massive pace.

1:15:47They're also getting cheaper as they scale. You look at where we were even a couple of years ago. The original, I think the original, I don't know what was it, GPT -3 .5 or something, was like 100x the cost of GPT -4 on mini today in the API. So a couple of years, you've gone down two orders of magnitude in cost for much more intelligence. And so I don't know where there's another series of trends like that in the world. Models are getting smarter. They're getting faster. They're getting cheaper. And they're getting safer too. They hallucinate less every iteration. And so there's just the Moore's law and transistors becoming ubiquitous.

1:16:34That was a law around doubling the number of transistors on a chip every 18 months. If you're talking about something where you're getting 10x every year, that's a massively steeper exponential. And it just, it tells us that the future is going to be very different than today. I still, the thing I try and remind myself is the AI models that you're using today is the worst AI model you will ever use for the rest of your life. And when you actually get that in your head, it's kind of wild. I was going to actually say the same thing. That's the thing that always takes me when I watch this thing. Like you're talking about Sora.

1:17:13And I imagine many people hearing that are like, no, no, it's not actually ready. It's not good enough. It's not going to be as good as a movie. I see in the theater, but the point is what you just made, but this is the worst it's going to be. It will only get better. Yeah. Model maximalism. Just like keep building, you know, building for the capabilities that are almost there. And the model's going to catch up and be amazing. Escape to where the puck's going to be. Yeah. This reminds me. I was just using as jibbly fighting everything the other day. And I was just like, what has taken so long?

1:17:42Just like that. Yeah. What was that? I said, as one does. As one does these days, I was just like, it has taken a minute to generate this image of my family in this amazing way. Like, come on, let's take it so long. You just get so used to magic happening in front of you. Yeah, totally. Okay. Final question. This is going to go in a completely different direction. A lot of people asked about this. So famously, you led this project at Facebook called Libra, which is now called Novi. A lot of people are always wondered what happened there. That was a really cool idea. I know some people have a sense.

1:18:16There's regulation, challenges, things like that. I don't know if you've talked about this much. So I guess just, can you just give people a brief summary of just like, what is Libra? They this project you work on and just what happened? And how you feel about it? Yeah. I mean, David Marcus led it and I happily worked for him and with him. I think he's a visionary and also a mentor and a friend. You know, honestly, Libra is probably the biggest disappointment of my career. When I think about the problems we were solving, which are very real problems, if you look at, for example, the remittance space, people sending money to family members in other countries, it is maybe, I mean, it's incredibly regressive by people that don't have the money to spend or having to pay 20 % to send money home to their family.

1:19:03So got rages fees. It takes multiple days. You have to go then pick up cash from, it's just, it's all bad. And here we are with like three billion people using WhatsApp all over the world, talking to each other every day, especially friends and family, exactly the kind of people who send money to each other. Why can't you send money as immediately, as cheaply, as simply as you send a text message? It's just, it's one of those things when you, when you sit back and think about it, that should just exist. And that was what we set out to try and do. Yeah, I don't think we played all of our like cards perfectly.

1:19:44If I could go back and do things, there are a bunch of things I would do differently. You know, we, we tried to kind of get it all at once. We tried to launch a new blockchain. It was a basket of currencies originally. It was integration into WhatsApp and Messenger. And I think the whole world kind of went like, oh my god, that's a lot of change at once. And you know, it happened also to be at the time that Facebook was at the absolute like nature of its reputation. And so that didn't help, right? It was, it was also not the Messenger that people wanted for this kind of change. We knew all that going in, but we, we went for it.

1:20:20I think, if we, I think there are a bunch of ways that we could do that that would have introduced the change a little bit more gently. You know, maybe still gotten to that same outcome. But few or new things at once and introduced the new things one at a time, it, who knows? You know, those were decisions we made together. So we all own them. Certainly I own them. But it just fundamentally disappoints me that that this doesn't exist in the world today, because the world would be a better place if we'd been able to ship that product. I would be able to send you, you know, 50 cents in WhatsApp for free.

1:20:54It would settle instantly. Everybody would have a balance in their WhatsApp account. We'd be trans, I mean, it was just, it should exist. I don't know, to be honest, like, I mean, the current administration is super friendly to crypto. Facebook's reputation, Meta's reputation is in a very different place. Maybe they should go build it now. I was looking at the history of it. And apparently, they sold the tech to some private equity company for 200 million bucks. Yeah. Yeah. And we had to buy back their upload of current blockchains that are built on the tech, because the tech was open source to make, aptos and mistin are two companies that are built off of this tech.

1:21:35So, you know, at least the all of the work that we did did not die, but and lives on in these two companies and they're both doing really well. But still, you know, we should be able to send each other money in WhatsApp and we can't today. Here, here. Well, thanks for sharing that story. Kevin, is there anything else you want to share or maybe a last negative advice or insight before we get to our very exciting lightning round? Oh, the lightning round. Let's go do that. Let's do it with that. Kevin, where you start very exciting lightning round, are you ready? Yeah. Let's do it. Okay. What are two or three books that you find yourself recommending most other people?

1:22:12Co -intelligence by Ethan Mollick, a really good book about AI and how to use it in your daily life as a student as a teacher. He's super thoughtful. Also, by the way, a very good follow -on Twitter. The accidental superpower by Peter Zion. Very good. If you're interested in geopolitics and the forces that sort of shape the dynamics happening. And then I really enjoyed cable cowboy.

1:22:41I don't just fascinating if you like business, especially if you want to get into like, I mean, the man was an incredible deal maker and shaped a lot of the modern cable industry. So that was a good biography. These are all first -time mentions, which is always great. Oh, good. Next question. Do you have a favorite recent movie or TV show that you really enjoyed? I wish I had time to watch a TV show. So I'm just so -or -of -it -os. Yeah, right. I don't know. I read when I was a kid, I read the Wheel of Time series. And now Amazon has it as they're like the third season of it. So I want to watch that.

1:23:20I haven't yet. Top Gun 2 was an awesome movie. I think that's no longer new, but you know, that shows when last time you watched the movie was. But I like the idea. I want more like Americana. I want more like being proud of being strong. And I thought Top Gun 2 did a really good job of that. Pride and patriotism, I think the US could use more of that. Is there a favorite product that you recently discovered that you really love other than your super intelligence internal tool that you all have access to them? I'm just joking. Right. Internally, DR. Well, I think like vibe coding with products like Windsorf is just super.

1:24:06I'm having a great time doing that. I still just love that our chief people officer vibe coded some tools. Maybe the other one is Waymo. Every chance I get I'll take a Waymo. It's just a better way of riding and it still feels like the future. So they've done an amazing job. It's awesome. By the way, I had the founder of Windsorf on the podcast. They might come out before this or after this. And also Cursors CEO is coming on the podcast either before or after this. Oh, cool. I have a ton of respect for what those guys are doing. They're those are awesome products. Just changing the way everyone builds product.

1:24:38No big deal. Yeah. A couple more questions. Do you have a favorite life motto that you often repeat yourself and release full and work in life? Yeah. So actually, this is interestingly enough. It's more of a philosophy. But then I thought Zuck encapsulated it one time on a on a Facebook earnings call. So I actually had this made into a poster sits in my room. But but somebody was asking Mark. This is literally on an earnings call. So it's like an analyst on an earnings call asking him, you know, it was some quarter where Facebook had grown a lot. This was back in the 20 teens sometime, I think.

1:25:15But it's like, you know, so what what did you do? What, you know, what was it that you launched? That was the one thing that drove all this growth for you. And he said something to the effect of, you know, sometimes it's not any one thing. It's just good work consistently over a long period of time. And that's always stuck with me. And I think it is, I mean, you know, I run all Tremarathons. It's like, it's just about grinding. I think people too often look for like the silver bullet when a lot of life is and a lot of like excellence is actually showing up day in and day out, doing good work, getting a little bit better every single day.

1:25:52And you know, you may not notice it over a week or even a month. And a lot of people then, you know, kind of get like dismayed and stop. But actually, you keep doing it. The gains keep compounding. And over the course of a year, two years, five years, it adds up like crazy. So good work consistently over a long period of time. Damn, I love that. I got to make a poster of this now. That is, I resonate with that. Okay, I'll take it. That is so good. Okay. Final question. I'm going to ask if you have any prompting tricks. And I'm going to set it up first. But think about if you have a trick that you could recommend to people for prompting LMS better.

1:26:28There's this, I had a guest Alex Kamorowski come on the podcast. He's from Stripe and writes as weekly reflections on what's happening in the world. A lot of them are AI related. And he wants described in LMS as a zip file of all human knowledge. And all the answers are in there. And you just need to figure out the right question to ask to get the answer to every problem, basically. And so just reminded me how important prompt engineering is. And knowing how to prompt well, you're constantly prompting chat GPT. What's one tip? One trick that you found to be helpful and helping you get what you want?

1:27:00Well, I'll say first of all, I want to kill the idea that you have to be a good prompt engineer. I think if we do our jobs, that stops being true. It's just one of those sharp edges of models that experts can learn. But then you just over time, you shouldn't need to know all that. The same way you used to have to get deep into like, you know, what's your storage engine? And my sequel, are you using NODB 4 .1? And you know, there's still use cases for that if you're at the deep edge of my sequel performance. But most people don't need to care. And you shouldn't need to care about my details of prompting if AI is really going to become broadly adopted.

1:27:39But today, we're not totally there. I think by the way, we are making progress there. I think there is less prompt engineering than there was talking about in the importance of giving examples. You can do effectively poor man's fine tuning by including examples in your prompt of the kinds of things that you might want and a good answer. So like, here's an example and here's a good answer. Here's an example. Here's a good answer. Now, go solve this problem for me. And the model really will listen and learn from that. Not as well as if you do a full fine tune. But much more than if you don't provide any examples.

1:28:21And I think people don't do that often enough. That's awesome. One tip that I heard, I'm curious if this works, is you tell it, this is very, very important to my career. Make it like really understand, like someone will die if you don't answer the correct thing. Does that work? It, you know, it's really weird. There's probably a good explanation for this, but you can also say things. So yes, I think there is some validity to that. You can also say things like, I want you to be Einstein. Now answer this physics problem for me. Or you are the world's greatest marketer, the world's greatest brand marketer.

1:28:58Now here's a naming question. And it's, there is something where it sort of shifts the model into a certain mindset that can actually be really positive. I use that tip all the time actually. I always, when I'm coming up with questions for interviews, and I use it occasionally to like come up with things I haven't thought of. I actually type, you're the world's best podcast interviewer. Right. I have Kevin, Kevin Wheel coming on the podcast, yeah, and actually works. Yeah, by the way, back to our other point that we made a few times, like you do do that sometimes with people, right? You sort of put them, you frame things, you get them into a certain mindset and they have a lot of different.

1:29:38So I think there are like human analogs of this one more time. Kevin, this was incredible. I was thinking about a way to end this, the way I feel like, I feel like not only are you at the cutting edge of the future, like you're in the team are kind of like actually the edge that is creating the future. And so it's a real honor to have you on here and to talk to you and to hear how you think things are, where do you think things are going and what we need to be thinking about. So thank you for being here, Kevin. Oh, thank you so much for having me. I feel, I get to work with the world's best team and you know, all credit to them.

1:30:13But really appreciate you having me on. It's been, it's been super fun. I forgot to ask you the two final questions. We're looking for exciting online if they want to reach out. And well, how can listeners be useful to you? I am at Kevin Wheel, K -E -V -I -N -W -E -E -I -L on pretty much every platform. You know, I'm still a Twitter DAU after all these years. I guess an X DAU. I've linked in wherever. And I think the thing I would love from people, give me feedback. People are using chat GBT. Tell us where, tell me where it can be, where it's working really well for you and where you want us to double down.

1:30:46Tell me where it's failing. I'm very active and engaged on Twitter. I love hearing from people what's working and what's not. So don't be shy. And I learned following you helps you figure out all the stuff that you're launching. Like you share all the things that are going out of you day or week. So that's also benefit. And by the way, 400 million weekly active users, all you know you feedback. Here we go. Yes, let's do it. You're going to work that great. Okay. Well, thank you, Kevin. Thanks for being here. All right, man. Thanks so much. See you soon. Bye, everyone. Thank you so much for listening.

1:31:18If you found this valuable, you can subscribe to the show on Apple Podcasts, Spotify, or your favorite podcast app. Also, please consider giving us a rating or a leaving review, as that really helps other listeners find the podcast. You can find all past episodes or learn more about the show at Lenny'sPodcast .com. See you in the next episode.