ChatGPT – The Super Assistant Era | BG2 Guest Interview

27:56And, you know, it took a while, but I know you were thinking about it a lot. I know you guys are thinking about pricing quite a bit. Maybe tell us a little bit about pricing. You know, right now, pricing is pretty simple. Is there a path for folks who are getting a lot of great value to price that product differently and, you know, meet them where they are and the other way on the other side? And pricing is, there's no world in which pricing doesn't significantly evolve when the technology is changing this quickly, right? ChatGPT originally was entirely free, and the reason for that was that it was intended to be a demo, and we were going to wind it down after a month.

28:34We then realized that the demo went viral, and people loved the demo, and it was actually a product. But we realized to be a product, you can't take the product down every time you're at capacity. So we shipped subscriptions simply because it could shape the demand. It was a way of gracefully turning users away when we had to turn away someone, and it felt like the fairest and most equitable way of doing so is saying, hey, if you really need this product, pay a subscription fee and you got it. Then we figured out how to make the product stable, and we had the choice of do we keep the subscription thing or do we go back to free?

29:04And we realized we had consistently more tech that we couldn't scale, GPT-4 being the first example because we had way too many free users to serve GPT-4, and we put it behind the plus plan. And so the way we stumbled into subscriptions was sort of accidental by trying to just solve for the user. And it felt like the right way at the time to provide maximal access to our tech. Since then, we've had so many other breakthroughs, including test time compute, where you can scale up intelligence kind of as much as you want, more or less. And it took us, in the entire industry, a little bit of time to turn that into product value, but we're here now where, you know, our power users want to use more and more and more intelligence.

29:55And, you know, it's possible that, you know, in the current era, having an unlimited plan is like having an unlimited electricity plan. You know, it just doesn't make sense because, like, you know, people may need a lot, a lot of electricity, and they're getting a lot of value out of that. There's a reason you can't buy that, right? So, obviously, I want to be really thoughtful about the way that we evolve our plans and SKUs and subscriptions, But you'd be incredibly surprised if it didn't change given the magnitude and profoundness of the technical breakthroughs that we've had and the product breakthroughs that follow.

30:28Yeah And you know relatedly so so I imagine you're gonna have something for the power users What about the other side? How do we get? the casual users into the into the wheel and and still monetize them? As mentioned, our business model will evolve in the North Stars access. We would like to pick a way of providing an offering that maximizes the number of people who can access our most powerful tools. I think for the longest time, that has been subscriptions. Subscriptions have the downside of the fact that in many markets, people don't have credit cards or they don't use credit cards to subscribe to software.

31:15And we're interested in other ways that can maximize access of the tech. Our ads pilots are in that spirit. We really view it as a tool of bringing Chachapiti and our intelligence most broadly to anyone around the world. And it is an example of how we constantly need to evolve and figure out the best way to bring the demand in line with what we are able to offer. Makes sense. Makes sense. You know, the ads piece has been a tricky one because, you know, Sam has historically expressed reluctance about ads. And, you know, you've got to maintain a lot of trust while delivering that. So I guess what changed?

31:59I think we've talked about this several times in my history at OpenAI. and every time it came up, we said, if we were to do ads, we'd have to be really thoughtful about the way we do it. So the first thing we did, you know, starting, you know, end of last year, was to really engage the company on, if we put ads in ChatGPT, how should we approach it? What should the principles weigh? How do you preserve the things that are magical about ChatGPT while getting the benefits of ads, which, you know, is our ability to bring our most advanced tech to anyone, regardless of their ability to pay. And I really love where we ended up on the principal side.

32:42On the experience side, we're very, very early. But on the principal side, I feel really proud because it's very important that the answer of chat should be to be independent, as an example. Respecting user privacy is very important, and there's a lot to learn from the way that tech has evolved over the last few years, or really last decade.

33:03And I like that the principles are out there before we've even really gotten started. Like we're very early with our pilots. It's kind of interesting. I was obviously very anxiously and eagerly looking at our support in bounds and data. And the most common inquiry about ads is not how do I disable ads or turn off ads, but it's like how do I run an ad? because the entire ecosystem is really excited to be part of the story and to figure out a way to talk to ChatGPT users. So there's a lot more to come, but I'm very eager to get this right. Yeah. Yeah, I'm sure you guys will. Switching gears, Nick, something you and I have spoken about a little bit is distribution and partnerships.

33:52There was a couple of big partnerships last year. Apple Reliance with Gemini. Those are two big user bases, right? A lot of India, a lot of the iOS users. Doc, tell us a little bit of how you think about partnerships for ChatGPT to meet the user base and maybe specifically on those two as well. Look, I think partnerships are a great way to bring two products together and to expose something like ChatGPT to people who might not otherwise have encountered it. The thing that I care about most when considering something like a partnership is what is the user experience and can we make it amazing? Because at the end of the day, when you look at what's going on in the market, you can get users to click on things.

34:50You can get them to tap any sort of product, especially if it looks like a product they recognize, etc. But if the experience isn't truly awesome, people will churn or they will at least not retain in the way that we've been lucky to retain them on ChatGPT. So for that reason, I'm super interested in tasks like that, but it needs to be great. It needs to recruit the user. We are very lucky to have a great brand and a recognizable product for many folks. And I want to make sure that anything we do is accretive to all that. Nick, you are a master of trade-offs. You must be making a lot of trade-offs right now.

35:31Tell us about some of the trade-offs you're making. Tell us about a trade-off that you might be making that people don't appreciate from the outside. There are a lot of trade-offs indeed for different reasons, right?

35:49What I encounter a lot is trading off, delivering on the people, on the use cases that exist in the product today and making them better versus productizing step change technology that's going to generate a whole other set of use cases. Because when you think about how ChatGPT came to be, it was a totally open-ended product. It was basically a user experience around a technical breakthrough. and we couldn't have told you all the ways that people find it valuable but putting it out there was really important because it allowed us us to discover in the world to discover what the way you can do and then post chat chibiti we can obviously very systematically go and improve on the things that people actually want to use it for and when you're at a company in this moment where you both have such amazing traction with what exists today and the most mind-bending breakthroughs on the research side the balance you have to strike is making the core product you have better today with all the things that matter latency reliability making the use cases really great that people come to with you know providing access to to the step change and we try to get the balance right over a small team and we don't always get it right and for that reason, that's one of the most difficult tradeoffs that I have to deal with.

37:14Nick, I imagine one of the hardest tradeoffs you guys make here is those GPUs that are melting between chat GPT, between codecs, research. How do you guys allocate the GPUs? That is a very good question, and I'll let you know when I figure it out. Just kidding. We've gotten a lot better at this. I really hope, by the way, to be at a point one day, and I've yet to reach that point where we don't have to face this trade-off because it's really painful to have real user demand for products that you can't serve. If you only ever worked in software, that's an entirely unusual dynamic, right, where you are limited by this zero-sum resource out there.

37:57The marketplaces have it, but I think pure software doesn't really have the dynamic, right? Yeah. So one thing we try to do, obviously, we prioritize our existing users first. We want to provide a fast, reliable product, and that is critical and table stakes. Then when you look at new capabilities, the sort of naive business school thing to do would be to probably look at revenue, incremental revenue per GPU or something like that. But this is where it's more an art than a science because we often have new breakthrough capabilities that are entirely zero to one. Deep research was one of those.

38:35We couldn't have told you, is there going to be demand for a, you know, consumer demand for a research product. But if you don't productize it to find out, you will never know. So, you know, this is where we have to be a little bit thoughtful on how we balance, you know, things that are no brainers that people are really going to love with things that are brand new ideas. And then obviously on the research side, there's a reason that Mark has the job he has because a big part of his job is figuring out what research to fund. And, you know, obviously GPU is a big part of that. So very nuanced topic that we're continuously getting better at.

39:11But for me, the priority is always on our users. Yeah. The other takeaway that I had is you don't have line of sight to a time when you won't have that problem. It's been so fascinating because we obviously have been incredibly lucky to encounter more and more users who want to use our technology. But then the value that we're able to provide for each user is going up as well. And GPU consumption correlates pretty well with that value. and when you just look at token consumption per user, especially in the enterprise too, which is a massive opportunity, you see a lot of very GPU hungry workflows and yes, demand keeps going up even as prices go down.

40:01This is a fascinating insight. People used to think that humans were, you can't kind of make more humans. Well, it takes nine months and then 19 years. but you're saying that's actually less finite resource than GPUs. Yeah, on the human side you can hire more humans and obviously we've been busy doing that and bringing the best talent across functions to OpenAI. In the world with agents you can also get more leverage per human and make your humans very effective at their job to do more. But GPUs are zero sum and if you don't have more GPUs You really have to figure out how do you make very very hard trades and hate making hard trades for our users?

40:46hence the desire to Have more GPUs, but it's it's useful to start with the most zero-sum trade-off when you do your planning So I think starting working backwards from GPUs is usually pretty pretty good idea Yeah, you know one of the we have all these external data sources for charts of users and usage and Activity and retention all those things where we don't have is tokens per user over time And I bet that chart is like a sweet line going this way. I think internal is pretty good. Our internal employees is a pretty good indicator for what's about to happen. And yes, the charts are mind boggling.

41:22Yeah, yeah, yeah. Fascinating. Okay, a couple of quick ones on the present before we go into the landscape, which is, you know, shopping. You know, we just moved into a new house. We took some photos and we were hoping that all our furniture would magically appear that chat GPT helped us paint but you know a lot of a lot of Recent updates on chat GPT shopping. Tell us tell us about it. What are you thinking? Yeah On shopping as a chat GPT as a shopping assistant the shopping is one of those use cases that exist organically in chat GPT Today and they work You can ask chat GPT about any purchase you might be planning and get pretty excellent advice but it's also one of those cases where the experience that exists in chat today it's not it's not the perfect experience that you would want because shopping is very visual for example so you're going to want to actually see products and images and be able to compare and contrast not just read you know walls of text people care about the sources of you know you know where can I learn more about a given product, et cetera.

42:32And so there's a lot of work to do to make this discovery really, really good and allowing people to use ChatGPT as an assistant to find the right product to buy. And that's where our focus lies, is making that really great and making that really great in a way that works for our retail partners as well. Because as I mentioned earlier, there's a huge appetite from the ecosystem to be part of the ChatGPT journey. and nailing the discovery piece has been the most promising focus area to date. Nick, on ChatGPT, you must see a breadth of information. You must see a breadth of use cases that people are doing with ChatGPT.

43:12And tell us something about, you know, what does the world underestimate about ChatGPT that you have maybe been surprised by or a listener might be surprised by? There's been a real change in the way that people think of ChatGPT. you over the last year or so where it's increasingly like a true thought partner to people. It's not just a thing that, you know, answers your question, but it's a thing that you can, it's a sparring partner that you can actually think things through with. And that shows up in all kinds of demands ranging from life advice, where, you know, if you've got a relationship problem, you can actually get a lot of value to ChatGPT just helping you think through how to handle it and how to talk to your partner about it, all the way to a work setting where you're working on an analysis or you're trying to figure out how to frame something or you're trying to build something.

44:07And chat GPT really shows up as a second brain of sorts. And I think that's qualitatively different in terms of how the mental model it occupies with people. And you see in the usage patterns and the use cases that exist. And I think the more we nail things like proactivity, which we talked about earlier in tasks, et cetera, I think the more it's going to feel like a teammate in the workplace and like a super assistant at home. And I think that's going to meaningfully change the use cases that people come for. Yeah. You know, I've been, the most high stakes thing I do with ChatGPT is we have a new baby.

44:45And when the baby's crying at three in the morning, ChatGPT, you know, what's going on? First of all, congrats. Second of all, I've heard this from all parents in my life. The Chachupiti has become indispensable as a thought partner. And it makes sense, right? If you have a really specific scenario or you think it's a specific scenario to you, Chachupiti really comes through and can help you build confidence. And I think that's such an empowering thing, right? And I imagine new parents aren't always the most confident about what is the right thing to do. and if ChatGPT can make you feel like you have agency and control, I think that's really valuable.

45:28Yeah, it's huge. Well, thank you for making ChatGPT. It's literally getting me an extra hour of sleep every day. It took a village, but that is a great metric. That should be the North Star metric is incremental hours of sleep. That's a great one. Incremental hours of sleep, incremental hours of joy. There you go. I mean, you joke, but we talk about this a lot And because spiritually that is pretty close to what we hope we can do, right, is help you reach whatever you consider self-actualization. Yeah. Whether or not that's sleep or joy or any other goal you might have. Yeah. Yeah, yeah. Well, thank you to the village.

46:02We're going to switch gears and talk about the landscape. Sure. There's a lot going on on the field. you know how would you frame Chad GPT's differentiation to people out there there's a lot of different products out there look it's the best time in history to be a consumer of technology it is indeed because you got options and the competition is intense and I think that's beautiful and it's actually good for us too because if you were to pre-mortem why a company like OpenAI does not achieve its mission, it's probably focus because of the sheer number of opportunities that become possible when you approach AGI, right?

46:46And having competition and options out there, I think it forces us to focus on our customers too and on things that really matter, which aren't always the most flashy things, right? Sometimes it's latency, reliability, the quality of the user experience. So I think it's a really good thing. I think the biggest differentiation of Jatchabed is the team behind it because we're not static, right? Anything we build will get copied sometimes in ways that are high craft sometimes in ways that are sort of check boxes and it's really important to us that we evolve the category and build the super system that we've always imagined and I think the reason that I have confidence that that's possible at a speed that outpaces, you know,

47:36the dynamic of being copied is that we have an amazing team. And that we have an amazing team across research and engineering and design and all the different functions that it takes to make something amazing. And I think our unique ability has been to bring those functions together to build something that is sort of at the intersection of useful and possible right in that moment. So, you know, my best answer for you is we keep pushing forward and we hope to be expanding what people think of this product as. You know, last winter we had obviously, you know, what was called Code Red. Google had a great model.

48:10There was a lot of, you know, talk about it, Mark Benioff switching very vocally to Gemini and us delaying ads and health agents and shopping. Basically hit pause on everything, making chat really better. Talk to us about that moment, both about what led to that and what was happening in that moment? Yeah. So first off, Code Reds are a tool we use to create focus. And as you can imagine, when you're in a place like OpenA, and this is what makes us special to work here, is there are so many different things going on. It's a research lab. We are pursuing many different ideas, right? And there's been these moments where we've wanted the company to come together to solve a problem across boundaries, no matter what your project might have been.

49:01And at the end of last year, we had one of those moments where we felt like we need to show up for our users. We need to focus the things, focus on the basics, like reliability, performance, the way that talking to the model feels, making personalization really great, all these elements that our users care about. And I loved it because it was really an opportunity to work with a bunch of folks who I don't normally get to work with on making the product great. And we just exited the Code Red, which we knew we would with the launch of 5.3, which is a great model for the everyday user. It's great to talk to.

49:37And 5.4, which is a workhorse if you're trying to do real knowledge work. And undoubtedly, we're going to continue to use the tool of a Code Red whenever we want to create focus. but I'm excited because I think ChatGPT is in a great spot. Yeah, so Code Red is over now. That's correct. It's not the new normal. It's not the new normal. We want it to be a special thing, but it is a tool I suspect we will continue to use. That's great. That's great. And maybe tangibly, if you were to point out, how did Code Red change ChatGPT, or maybe the ops or how the team operates? The thing I try to get, you know, foster with the team is focus.

50:19So we're certainly more focused than we were six months ago on, you know, the things we really want to nail. And some of those things are very behind the scenes, like latency, reliability, those kind of things. And some of those things are, like, very conservative efforts, like involving ChatGPT into the Super Assistant. And so focus is the main lasting artifact. And as you imagine, it's hard to stay focused sometimes when there's so much going on in the space, but that's the hard job. And you asked me about trade-offs earlier. Getting the team to focus on the things that really matter to users is certainly one of them.

50:56That's always worth it. Yeah, you know, in the back of my mind that I asked you that question is all the other founders that are in the arena right now. And just a reminder that, hey, Code Red is a tool. for you. Wartime and penalty, as you used to call it, is a tool. Yeah, I think, you know, every company does it differently in terms of how you get stuff done. But I think it's really valuable to have terminology that, you know, means something that, you know, signals to people, it's okay to drop your other stuff and it's okay to, you know, focus on this thing together even if that wasn't your original job.

51:30Yeah. So I think it works really well at a place like OpenAI. But I imagine startups would have an equivalent. Yeah. You know, one of the things that caught everybody's imagination on our team was what Peter was doing at OpenClaw. Incredibly potent to put all the tools together. Obviously, Peter is a great builder. Congrats on bringing on Peter to the team. Tell us a little bit about what Peter is working on and when might the billions on ChatGPT have something to see there? Well, first of all, I'm very excited for Peter to be here. I was excited to have another German speaker in the house. He's Austrian, I'm German.

52:11So we're exchanging guten morgens. But the Open Claw is so inspiring because it brought to life in many ways a vision that we'd had in different forms, you know admittedly around this kind of AI that is fully embodied that you know exists across different UIs that can do stuff for you that has state that has an interaction pattern that feels a little bit more like talking to a human you know because you know you know open call allows you to interact in a very very natural way where you can send many texts back and forth and it's very curt and so there's a lot of elements of of open claw that I think were very clarifying to folks across the industry and um but the best you know i i'm super excited to just like learn learn from peter and bring in uh into the company and figure out what we can do together so there's there's a lot more to come all right so now on to the most fun section rapid fire all right you ready sure we'll start with my most my favorite game which is long short uh pick an idea a startup a business a product that you love, you think you're very bullish on?

53:27Yeah, I'm, if I were starting a company today, I'm really excited about these companies that are going into companies and getting extremely hands-on and doing effectively professional services with AI because we've saturated all the evals and you need to get proximate to the problems. So that's, it's those companies that I'm paying attention to. Fascinating. So this is, you know, this is an example. It's just be like, hey, you're going and either acquiring or going inside an operating form that has scale and a humming engine. Exactly. And making that a more efficient engine. Yeah, or just like, you know, you're doing contracts for customers that have really hard problems.

54:07And you're actually going in and committing to solving the problem. And doing outcomes. Yeah, because, like, you know, there's a reason, I think, that we've made so much progress on math and coding, but not on many other domains. Because those are domains we are approximate to, we as people who work in labs. And there's all kinds of other domains that we are not as proximate to. And if you get proximate, I think you can build something transformative. And I think this is more important now precisely because the easy problems have been solved. The obvious problems have been solved by the models.

54:40Credit where credit is due. I think Notebook LM is awesome and differentiated and helps me learn new stuff. I think it's great. It's so good. It's so good. I think this is the example of you can innovate and you can build something totally different. It's awesome. Yeah, yeah, yeah. It's so good, particularly for, I found it for some more technical learning to be a very approachable way to learn. Totally. And it's really cool. I feel like AI, an underrated capability of AI is to just transform things into a different medium. And I think that's so important for learning. We just launched these dynamic math blocks, which allow you to visually understand math inside ChatGPT.

55:22Learning is obviously a big use case for us, too. And I think just being able to transform things from text to visual, soon from visual to video, and all these different media is amazing because people have such different ways of processing information. And some people are auditory learners, some people are visual, some people are reading. So I think that's really magical and a great, great angle to take. Yeah, amazing, amazing, amazing. You know, one of the things I think about a lot is education and education for kids now in school. The world's changing so fast. I'm not sure our education system's changing that fast.

56:00Yeah. What advice would you have for students who are in school now, you know, who might have to adapt faster than the system around them might adapt? It's a really good question and something that I've thought a lot about myself. And, you know, I think the most important PERMA skill in this era is curiosity, I think, because if the machine can answer all your questions, you better have good questions. and the only way to have good questions I think is to pursue the things you were actually excited about from an early age and throughout your entire life and I reflect on this because the only reason I'm here and working on this stuff is because I thought it was neat when I got nerd sniped in the interview process and it's like this is so cool and so no matter what you're doing I think that's an important skill is to be curious and learn to stay curious.

57:03And I think I'm confident that if you foster that skill, you will know how to adapt to, you know, an evolving landscape of tools and AIs and jobs. So that would be my advice. Yeah. Curiosity has always been the Porma scale. Our friend Bill Gurley wrote about it in his book, Running Down a Dream. But you're going to have to check that out. Yeah. What is a job that gets more valuable, not less, as AI gets better as AGI arrives? Well, I think maybe the easy answer is being an entrepreneur because it's the best time to build ever in terms of being able to self-actualize your idea. Maybe one that is maybe non-obvious is I think writing, actually, is very important.

57:55and it's not because the AI can't write. You know, AI will become amazing at writing just like any other domains, but because I think the skill of writing forces you to be very clear on what you have to say. And even though prompt engineering is obviously going to go away and has gone away to much extent, the idea of expressing what you want to a machine requires you to be a pretty good writer and a very precise writer. So I would say that that is a, you know, in any profession that involves very clear writing and therefore thinking, I think is well set up. Yeah, 100%. Honestly, I mean, this is the whole thing about Sloth, right?

58:41There's just so much. That's the other thing. I think there's going to be a permanent need for high quality, trusted, authoritative content. And tools like ChatGPT can help you discover that content. Yeah. But I think the need for amazing content is also here to stay. And final question, what has been your AGI, feel the AGI moment? When did you feel it? I've had so many, honestly. And it's definitely not stopped. A few weeks or so after I joined OpenAI, GPT-4 had finished training. And I remember trying it out, and it actually, it didn't impress me at all, nor anyone else that week, because it kind of didn't work.

59:29And it's because we hadn't figured out how to post-train it. And I think seeing it go from kind of, wait, is this really a thing? Or was GPT-3 kind of it? To, wow, actually this is an entire step change with what felt to me at the time, who didn't understand much about AI at all, as like just some tweaks or some a little bit of final stretch work was profoundly humbling because you can realize that you might it might not look like we are close to really powerful useful ai but we probably are um and then the moment that you know really you know there was two things that gpd4 did that felt like agi to me uh one is they could do poetry and i didn't think it was possible for an animal to do poetry just kind of fundamental philosophically it just didn't feel like in scope And then the other one was it could produce code that actually worked and compiled.

1:00:24And then my next moment where I stared at the ceiling just in awe was when I realized GPT-4 could just simulate an entire computer terminal. Like a full computer with commands, etc. And I'm like, wait, how would this be imbued in a language model? And there's been so many moments since then, honestly. Reasoning was a moment. One of the moments was when I think Mark and I were giving a demo of reasoning in front of the whole company. And this was a moment where we were still trying to kind of find use cases that were hard enough for the reasoning to make a difference. We're way past that point.

1:01:03We know. But at the time, I think we were having to do a puzzle in front of everyone. And I think one of the moments that maybe totally feel the EGI is like, we were in the middle of the demo and everyone started laughing. I was like, wait, what is funny? And then I stared at the screen because we're showing this chain of thought as it was streaming out of the model. And the model swore and said like, oh, damn it, may have to adjust because they realized I had made a mistake in the puzzle. And the fact that it did that, but particularly the fact that it did that in a way that was entirely emergent from the, you know, RL process, completely blew my mind and made me feel quite humble about what else these models might be able to do.

1:01:45So that was one of those moments. And then most recently, watching people use codecs, like watching people walk around with their computer open because they don't want the task to end. Watching people who have never coded in their life make stuff and bring ideas to life, That feels like an AGM. Honestly, it's just accelerating for me, and it doesn't wear off at all. And everyone has a different thing, obviously, but those are worth some of mine. Yeah. You know, it's 10 years ago, there was a product called Kite. I don't know if you remember. It was for software engineers. It was like an AI coding product.

1:02:25That's when I felt the hunger for personal AI, and nothing happened for 10 years, and then everything happened in the last 10 months. The timing thing is really hard because it's actually quite possible to predict where things will end up, I think, in terms of the kind of product-perform factors you're going to have. But to know when it happens, it's really hard for me to make statements on anything between sort of eventually and in three months. Yeah. Because of all the ambiguity around, you know. Well, that's a tight enough window. Now in three months is a tight enough window. Three months is pretty okay.

1:02:59Try to stick to the three-month plan more or less. though my team would probably tell me we don't but I try but yeah anything in between three months and eventually is difficult well thanks for doing it you've got a lot going on this was a total treat we're so excited to see all the great products you release for us if we can do anything to be of help let us know awesome thanks very much thanks for having me of course man this was fun

1:03:35As a reminder to everybody, just our opinions, not investment advice.