OpenAI cofounder Greg Brockman on the scaling hypothesis and refactoring as a killer AI use case

0:00This is totally backwards from how you're supposed to do a startup, right? You're supposed to have a problem. And we had no idea what the problem was. Is there a world where the AI becomes the manager and that, you know, give you ideas and give you some time to do? This was probably the hardest project I've ever done because it felt totally doomed, right? It's like, I know like every instinct, every builder instinct of mine, actually, he filled him. Oh, it was it. Greg in 2010 dropped out of MIT to become our first engineer. When we went on to become Stripes CTO in 2015, after he left, he co -founded OpenAI.

0:29Open AI is really cooking at the moment and Greg is one of the most productive people I know. Cheers!

0:37Wow, you really have some... What's called photos? Some deep cuss. GDV Tril, yeah. So... Yeah, we figured we'd put people at ease, you know, make people feel at home. Very nice, thank you. Okay, well I just drive straight into all the questions I have which is lost. Alright. If you were not working in AI, how could one have known that something was about to start working? People were telling you that AI was the future in the 1970s and the 1980s and the 1990s, and then very quickly in the late 20s, I was never considered to have met. Well, I was someone who was not in the field, and so I remember very much what it was like.

1:102013, 2014, it felt like every day on Hacker News, there'd be a new deep learning for X article. And I remember being like, what is deep learning? And I knew like one person in the field, and I asked them to introduce me to more people in the field. And I just kept getting introduced to a bunch of my smartest friends from college. Now, if you actually look at the work that was being done, you're 2012 basically, you know, image recognition for the first time, you could solve with the neural net much better than anything else. And I just blew all these traditional computer vision to approach it out of the water.

1:42It's like this like learned system that is able to outperform 40 years worth of let's like write down all the rules and try to like, you know, sort of handcraft the algorithm for the task. And it's very easy to then be like, okay, well, this approach sure works for computer vision, but it's never gonna work for machine translation in, you know, 2014 suddenly you're getting great results in machine translation. And I think that this pattern was applied in subfield after subfield. One thing I've been wondering about is so many different things are finally working at the same time. And so we have LLNs, which are obviously amazing.

2:14Then we also separately have image models really working And we also have text -to -speech -to -text working way better than they were before. And so, what's the common factor behind everything starting to work at the same time? Well, it's deep learning. Right? I think deep learning is the core for a long time. Like, you know, why didn't deep learning work in the 1980s? Well, so, okay. So, you look at the number of orders of magnitude of compute that we've gone through from 1940 to today. I mean, it's just astounding. Using their all -explain by compute scale -ups, I think. I think that's the right algorithms.

2:44Of course, the type of algorithm changes in some of those results aren't even deep learning, but I think that fundamentally it is about compute, and you needed an algorithm that is scalable that can actually absorb that compute. Was open AI the first company to take the scaling hypothesis really seriously? I think that claiming the first is always difficult, but I think that it is clear that we are sort of succeeded much more wildly sooner than anyone else. And so I think that we have real conviction behind what we needed to do. Some people think that OpenAI set out to prove the scale hypothesis, whereas it was almost the other way around of the scale hypothesis is what we observed as the thing that was working for us.

3:19And you know, we really saw it for the first time actually during our Dota 2 project. We started out with 16 cores to train a little agent on Yacob and Shimon who were getting the project from an ML perspective on their desktop. And then they scaled the 32 cores. And it felt like every week I'd come back to the office and they scaled up by another 2X and we had 2X performance. And just so clear, you just need to keep going. Where does this thing peter out? and it just never did. Founders get too much credit because you have an initial product that's a pretty reasonable idea. And then you listen to the customers and you follow what's working on.

3:52So you were saying that was kind of open AI with the scaling hypothesis where you started trying to make Dota AI's work and you noticed that adding more compute worked really well. And you said, where else will just throw more compute at it, yield benefits? I think that's like to first order correct. And I think one thing that distinguishes OpenAI from, you know, sort of the typical startup is we did everything in reverse. Right. It's like you're supposed to have a problem to solve. No one cares about the technology. Form DNC up front. Exactly. Yes. And for us, we really chased the technology without any idea of how it would be applied.

4:29Yes. And a lot of pursuing the technology really is, you have to let reality hit you cold hard in the face. There's just no other way to achieve results. You can't sort of will it into existence. You can't convince people that this is the thing. You have to actually make the system work. We have to figure out what is the right frontier, what are the problems, what are the things during the edge of working and to really double down on those. What else do you take away from the data work? How else does this? You could say you could just start with how it ends. We could have skipped that period in the wilderness, but it sounds like it was somewhat formative for the OpenAI organization.

5:03Well, so I think Dota had many lessons, one of which actually was in management lesson for me. I remember when he started out the project, I tried to set a list of milestones, right? It's like, okay, this date we're going to be this player, this date we're going to be this player. And it did not work at all. I remember our first milestone came and you realized that you cannot control the outcome, right? You cannot set outcome based milestones. What you can do is you can control the inputs of, we're going to try these experiments by this state. We're going to implement this feature by this state.

5:37And that is what actually worked. And I remember it was one of those things that was just like a story that I could not have sort of written in any better if we'd intended to, where we had a, we bid our in -house like best player. And then we were playing as a semi -pro, and he was just trouncing us, trouncing us. And then suddenly we were starting to get pretty good. So we showed up at the international, this tournament blind. First day, we had three players that we played against. We went 30, 30, and then two won. We were like, oh no, we lost. What happened? It turned out that this pro that we were playing against, that he had used an item we never trained against.

6:14And we were like, oh no, we're totally going to be hosed. So what do we do? Well, we just need to change the training. And so people stayed up all night to get this done. They added this extra item in there, 4 a .m., they finally got the job running. that Wednesday we're supposed to play against the number two and the number one player in the world. And our semi -pro plays against it. And he's like, this bot is totally broken. And we're like, oh no, we like, clearing head of bugs, something terrible has happened. And you're like, look, it's taking all this damage it doesn't need to, I'm gonna go kill it.

6:42It goes into kill it, he loses. And he's like, that was weird. He'd realized that what it happened was it had learned a baiting strategy. And then we realized, well, we have a super bot, but it's so bad at the beginning, because it's trying to do the baiting. So what if we just stitched the two bots we have together? And then that bot was just undefeatable. And we played against this number one player in one. And to me, this is like the story of how deep learning works. It's like you can't control where you're gonna go. You can control everything that goes in. You can put these metrics and these measurements and you can have sort of the evaluations and that being able to gauge where you're at is almost as important as being able to make the forward progress.

7:18But if you get all those elements right, then you can do true magic. And you're also describing something that works really well for an organization where it was motivating to stay up all night. Like, you know, if the prize was impossibly far away, it wouldn't have been as motivating, but the fact that there was a near -term reward function and you were able to show concrete progress. I think so. Yeah, I mean, I think some of my favorite engineering stories have the same character. I mean, when I stay up all night to get our ISO 85, the 83 integration stuff. Something that's staying up all night for like critical projects that actually have important history in my hands out.

7:49Yeah, all startups. I'm glad to hear the tradition of the live -o -out on the wild opening. So Dota's fallen, Chess's fallen, Go has fallen. We've passed the Turing test. I think by anyone's measure with people commenting how the little fanfare wouldn't be dead, but we seem to have pretty good. What's a good new Turing test? Well, I'll tell you two things. One is that if you look at the strict version of the Turing test, I would actually claim we haven't done it yet. So no one's really gone that extra mile to say, can we actually have an AI that is fully indistinguishable for me human? And it's not clear if it's even a good task, right?

8:25But I think that the right question to your point is like, well, what is the milestone that we should be chasing in terms of capability? Like I remember talking to one of our board members in 2018, and he said, look, I get that you were outside about near -term age AI, but it just doesn't feel like it's on track. And I asked, well, what do you mean? He said, in a world with the near -term age AI, you would expect massive economic value to be delivered by AI already. And where is it? And in 2018, I think it was a very fair criticism. And clearly that's starting to change now. If it was like one thing that's may really change the AI Marcus is personalization, up to quite recently, when you asked Chacha Dea question, it was like walking into, you know, a shop off the street, they've never met you before, they know nothing about you, whatever.

9:12That's obviously not ideal for this, you know, close part of your digital life. I'm curious how you're thinking about personalization from a product point of view. It feels to me like the most meaningful change since the chat interface. Two and a half years ago. Two and a half years ago. I mean, I think it's absolutely critical. And I think it is very rightly considered to be kind of a next frontier. Like I am someone who always, when I just Google something, I go into incognito mode. Because I don't even want my computer to remember that history. And I always used to go for contemporary chats on chat GPT.

9:45But now my usage has totally reversed. I want Chatchee, BT to know to remember everything. I want to remember all of my interactions, because it's useful. Okay, so you guys figured out from a product point of view how to make the memory actually work better. And this is actually a product point of view, but also really research point of view. And I presume there's a flip -flop between the product and research where when you find something that's useful for a product point of view, then the product people say, and I'm just a product person, like, you researchers go actually make this good, and then that kind of kicks off more research.

10:10It's like, well, we actually, so to some extent, that's a failure mode in our mind, right? that I think that we really don't want to have that kind of silo. We really want to blur the lines and have people cross -collaborate. So it's very different mindsets from how you would traditionally build a product versus how you do research. Part of what had happened actually was that we had GPT -3. We knew we needed to build a product in order to be able to see and raise funding. We were like, well, what product do we build? We wrote down a list of like 100 different products. We could do a medical thing.

10:40Then you're like, well, now we have to sell the hospitals. You can have to hire doctors and you just realize you give up on the G and AGI, right? You're gonna like go for a specific thing. And so someone had the idea of saying, well, why don't we just make a API and let people figure it out? And again, this is totally backwards from how you're supposed to do a startup, right? You're supposed to have a problem and we had no idea what the problem was. Yeah, yeah. And so we're gonna back into the problem. And so this actually felt like this was probably the hardest project I've ever done because it felt totally dimmed.

11:10Right, it's like I know like every instinct every builder instinct of my actually feel didn't oh felt it wasn't just like Open ended or something no fault. You're still doing it. I mean, it's like at some point if you have there's no There's definitely no other path. There was no other path was the only shot we had I remember someone also saying like I can't imagine anyone paying for samples from this model and I was like might be right And I'm still trying to imagine myself. And it was just not clear where we above threshold, or below threshold. And we showed it to people and people were interested, but the very different from people being like, I will build my company on top of this.

11:48And I suppose the first use case to get any traction. So AI Dungeon. And what was that again? There you go. So AI Dungeon was a tech -based, tech -based adventure game. Oh, sure, yay. Yeah. OK, but that was real revenue. or that was zero revenue. Yes, and in fact, I believe they were our first paying user. And I could you confuse where you're like, oh, clearly the future of OpenAI is gaming. I know, I know, I know. I'll go to our roots. Exactly. And it's interesting too, because yet we had dreaming of all of these applications and medicine and all these things. And you start with the gaming application.

12:25But we could see signs of life on so many other things. I think in many ways, JPT3 was like the best world's best demo machine, right? when we bring these to the API, people were coming with all these cool things you could do, but making them reliable was so hard. It really wasn't until the next generation of GPD 4 until we started to figure out how to do post training well, that then you were actually able to build real businesses on top of these things. Bill Gates was saying recently that GPD 4 was the best demo he'd ever seen since Xerox Park. He said it to me. The night he saw it, yes. So that's high praise.

13:00I want to say to the medicine thing you mentioned this. So like you said, I think your family has personal stories you've talked about getting very valuable, diagnostic help. We ourselves actually, it's a bit more minor in our family, but we managed to fix a cat thanks to debugging it within the LLM. And I think this is an interesting example, right? Because so many people that I know have had some kind of experience like this. And maybe it's because you actually don't get that much time from a doctor. Are there other examples like this medicine application where you're seeing a lot of success that many people have similar stories, but we just hear less about it.

13:42Yeah, I think it's a great question. And by the way, I think medicine is an example of one where I kind of thought it was going to be one of the last domains that we successfully built to add value in. but it turns out that the bar is so low and you just need to exceed WebMD. And so I think that we have seen sort of other areas that are like a real common theme. Like one that's very interesting right now is like the life coach, like life advice kind of application. Or you just talked about it. You're actually really taking off? Yeah, it really is. And so I think that there's things like education is another area that just like clearly like is really having an impact.

14:15And there are studies coming out now that actually show that people are able to learn better through the use of these tools. Let's be expected, right? Like it is the bloom to sigma effect in a product. Yes. And that, for example, is like why, you know, Sal Khan, that's why he started Khan Academy, was to think about if you can give personalized tutoring to everyone. Yes. And we showed him GPT -4. He's like, this is the thing. Yeah. Like, we're going to become a GPT -4 app. And so I think that there are these really amazing applications that are affecting everyone's like daily lives. You know, obviously programming is another one.

14:46I think that people are seeing all across the board in professional context. We're heading to a world where just like, you know, if you want to do productive work, you don't have access to computer, like you're going to be hampered, and so similarly, not having access to AI is heading in the same direction. Okay, so speaking of not having access to AI, I will posit that these days, it feels like AI product development is mostly OS limited. Is that how you feel? Like are we stuck at the moment? I do feel a little the stuckage, but it's not to worry. It is overcomable. But yeah, I think it is true.

15:25Look, like two years ago, we released plugins in ChadGPT. Do you remember them or those? Yeah. Yeah. And that was like trying to make it so anyone could write apps that then ChadGPT could access. And the models were just not that good. Right. Then we limited it to like three plugins at a time, because there's only so many functions and stuff. And it just wasn't that reliable. And now we're in a world where MCP basically, is really taking off and is a way to hook up your AI to different tools and very much like kind of try and take that same type of idea and really make it work. Now the world that we're in is very similar where there's certain interfaces we don't have or being able to access your phone and all those APIs.

16:07And there's a question of is the model above threshold to actually use them or not? And my observation has been that basically I think that there is maybe a lag of like six months of different interfaces that are hard to access. But once we have a model that's good enough, we will find a way, people will find a way. And so I think that we're in a world where I have every expectation that we will get the future that has been promised, but it's just gonna take some work. I feel like there are many moments where I'm using my phone and I want a single button where it's just like, you know, check to you.

16:38What do you think of this? Like I need your comment, I need your fact check, I need your explanation, something like that. And you take a screenshot and you're like, go into chat GBT and you click upload photo and it feels like very 1993 versus the button on my phone that just says, hey, chat GBT, what do you think about this? And obviously you guys are not empowered to go build best. That's what I mean by it feels somehow like we're a little operating system. Well, I definitely get it, but I'll say, I think that there are two dimensions. So this is kind of how I've been thinking about things since we released the API back in 2020.

17:08There is capability and convenience. What you're referring to is the convenience, right? but it's pretty inconvenient to do the screenshot and paste it. But the thing is, if the capability is good enough, you are willing to accept any sort of inconvenience. It's like if this, by taking the screenshot, showing you the chat, you could give you amazing insight. You can tell you how to build stripe in some way, and it takes you a month to do it. You have to crawl to the top of the mountain. You'll do it. The convenience will not stop you. And so the point that I'm trying to make is that if the capability is high enough, people will start doing a specific thought.

17:45They'll discover the use cases and the convenience will just catch them. And then in the convenience, there's so much pressure. There's pressure on the phone manufacturer. There's pressure on us. There's pressure on everyone in order to bring down the security. So it seems to be patient. It'll be great in three years time. Yes. Really in it. New is the AI. A criticism people like to love you of AI is, yeah, it's great and handy and all, but it hasn't come up with a single novel advance in mathematics. Have you? or science. Well, you could have if you had become a mathematician, but humanity has for keeping the score board.

18:20Why don't you make it that criticism? Just wait. Okay, so you think like it would take one of the millennium prizes, you think we possibly will see that. I think for sure. I mean, there's no question. Like two years, five years, ten years? I think that is the question. It's just timing, right? That is my question. I would put it two to five years as the right number. And I think ultimately, this comes back to the question of benchmarks, right? Is that actually being able to solve a linear problem is pretty high bar. Yeah. Pretty high bar. And once you can do that, there's so many other things that will definitely be possible.

18:56And I think that we're starting to see the leading edges of this. And to me, if we look at our definition of AGI, we recently started talking about the framework of thinking about levels of AGI, starting from chat bots to reasoners to agents to innovators to organizations, five levels. And we're basically somewhere in level three right now. Yeah. And level four, this innovators, like that's going to be different. You know, I recently posted some pictures of our visit to Appling, Texas, where we're building these big data centers together with our partner Oracle. And imagine taking that whole data center and just thinking hard about one problem.

19:34Imagine just thinking about how to solve a Millennium problem or how to cure a specific kind of cancer. Maybe it needs access to some apparatus. Maybe it needs access to robotic wet labs. Maybe it needs access to different tools in the world. But that level of computational power couples with the ability to experiment and learn from your ideas. That is going to be something the world has never seen. Okay, so yet again, we just we haven't put a a respectable amount of compute from these problems compared to what we will be doing. We're still least tiny little computers. So that actually gets to, in terms of these scaling laws, do they eventually run out because we just run out of compute or like do you eventually get to the point where we're inventing new kinds of nuclear energy and so that is what unlocks the next level.

20:18A lot of energy that comes online now is for data centers, which is not true when you guys started training GPT -2. And so, isn't that just the upcoming bottleneck? Right. As it should be, right? It really should be that it's energy manufactured into intelligence and that's your only bottleneck. But I'm saying that'll be like quite a plateau compared to the exponential growth we've seen over the past few years. Well, I think, unless the key change in terms of promising and plans for building and everything like that, this is, I think, the core, right? is that if you look at every trend in this field, there is these exponentials, these S curves that sum up to exponentials.

20:57Sure, but these exponentials were mostly existing in like tech, Silicon Valley space, where it's pretty easy to have exponential growth. It's pretty hard in permitting and real estate and damning rivers and building nuclear power plants. It's harder to have exponential growth. Let's see how fusion pans out. Yeah, okay, but even fusion, most industry observers would say is still kind of five years away. And so, what it's the next five years of power growth. So, look, I think that it is very possible that we end up bottlenecked on energy. And that's actually one reason that we've been spending a lot of time really trying to advocate for the fact that we just need far more power.

21:33And I think that my observation of the market is that ultimately the capitalist markets do provide. Yeah, I think there's this absolute tsunami of demand that is coming our way. But I feel some confidence that, again, And when there's enough pressure, when there's enough clarity of this is the bottleneck, and it's not just for any company, right? It's really for national competitiveness. And you look at other countries that are just building huge amounts of power far more than we are. I think that actually for America to remain competitive, there's just no choice but to build out for real power.

22:06We do. Speaking of bottlenecks, everyone was talking about the data wall in 2023. I think this is an interesting thing where knowing it's talking about the data wall anymore, And yes, it doesn't feel like AI progress has slowed down. Just test time confused, is this, like, people were wrong about the data wall and where it presented a bottleneck. Like, these are actually still a data wall, but it's two years away. It's basically all of these things, right? It's really, it truly is, right? It's like, you keep changing the paradigm, right? That is the real core of the Kurzweil view of the world, right?