Scaling Test Time Compute to Multi-Agent Civilizations — Noam Brown, OpenAI

23:55I didn't know it at the time, but like he and others at OpenAI had also been thinking about this. They'd also been thinking about RL. they've been working on it and I think they had some success but like you know with most research like it does you have to iterate on things you have to try out different ideas you have to yeah try different things and then also as the models become more capable as they become faster it becomes easier to iterate on experiments and I think that the work that they did even though it didn't like result in a reasoning paradigm it all builds on top of previous work right so So they built a lot of things that over time led to this reasoning paradigm.

24:31For listeners, no one can talk about this, but the rumor is that that thing was codenamed GPT-0 if you want to search for that line of work. I think there was a time where basically RL kind of went through a dark age when everyone went all in on it and then nothing happens and they gave up. And now it's sort of the golden age again. So that's what I'm trying to identify. Why? What is it? And it could just be that we have smarter based models. and better data. I don't think it's just that we have smarter base models. I think it's that, yeah. So we did end up getting a big success with reasoning.

25:04But I think it was, in many ways, a gradual thing. To some extent, it was gradual. You know, like there were signs of life. And then we like, you know, iterated and tried out some more things. We got like better signs of life. I think it was around like November, 2023, or October, 2023, when I think I was convinced that we had like very conclusive signs of life that like, oh, this is the paradigm and it's going to be a big deal. That was in many ways a gradual thing. I think what OpenAI did well is like when we got those signs of life, they recognized it for what it was and invested heavily in scaling it up.

25:40And I think that's ultimately what led to reasoning models arriving when they did. Was there any disagreement internally, especially because like, you know, OpenAI kind of pioneered pre-training scaling, you know, and kind of like computers all you need. And then you're kind of saying, maybe that's not how we get there. Was it clear to everybody that like, okay, this is going to work? Or was it controversial? There's always different opinions about this stuff. I think there were some people that felt that pre-training was all we need, that we scaled it up to infinity and we're there. I think a lot of the leadership actually at OpenAI recognized that there was another paradigm that was needed.

26:10And that was why they were investing all this research effort into this RL stuff. And I think that's also to the credit of OpenAI that like, okay, yes, they figured out the pre-training paradigm and they were very focused on scaling it up. In fact, the vast majority of resources were focused on scaling that up. But they also recognized the value that something else was going to be needed. And it was worth putting researcher effort into other directions to figure out what that extra paradigm was going to be. There was a lot of debate about, first of all, what is that extra paradigm? So I think a lot of the researchers looked at reasoning and RL was not really about scaling test-time compute.

26:49It was more about data efficiency because, you know, the feeling was that, well, we have tons and tons of compute, but we actually are more limited by data. So there's the data wall and we're going to hit that before we hit limits on the compute. So how do we make these algorithms more data efficient? They are more data efficient, but I think that also like they are also just like the equivalent of scaling up compute also by a ton. That was interesting. There's like a lot of debate around like, OK, what exactly are we doing here? And then I think also, even when we got the signs of life, I think there was a lot of debate about the significance of it.

27:19That like, okay, how much should we invest in scaling up this paradigm? I think especially when you're in a small company, like, you know, OpenAI, like, in 2023, was not as big as it is today. And compute was more constrained than it is today. And if you're investing resources in a direction, that's coming at the expense of something else. And so if you look at these signs of life on reasoning, and you're saying like, okay, well, this looks promising. saying we're going to scale this up by a ton and invest a lot more resources into it. Where are those resources coming from? You have to make that tough call about where to draw the resources from.

27:49And that is a very controversial, very difficult call to make that makes some people unhappy. And I think there was debate about whether we're focusing too much on this paradigm, whether it's really a big deal, whether we would see it generalize and do various things. I remember it was interesting that I talked to somebody who left OpenAI after we had discovered the reasoning paradigm but before we announced O1 and they ended up going to a computing lab. I saw them afterwards, after we announced O1, and they told me that at the time, they really didn't think this reasoning thing like these O-series, the strawberry models were that big of a deal.

28:29They thought we were making a bigger deal of it than it really deserved to be. And then when we announced O1 and they saw the reaction of their coworkers at this competing lab about how everybody was like, oh crap, this is a big deal. And they pivoted the whole research agenda to focus on this. Then they realized, oh, actually, this maybe is a big deal. A lot of this seems obvious in retrospect, but at the time, it's actually not so obvious and can be quite difficult to recognize something for what it is. I mean, OpenAI has a great history of just making the right bet. Yeah, I feel like GPD models are kind of similar, right?

29:07Where like it started with games and RL, and then it's like, maybe we can just scale these language models instead. And I'm just impressed by the leadership and obviously the research team that keeps coming up with these insights. Looking back on it today, it might seem obvious that like, oh, of course, like these models get better with scale. So you should just scale them up a ton and it'll get better. But it really is, the best research is obvious in retrospect. And at the time, it's not as obvious as it might seem today. Follow-up questions on data efficiency. This is a pet topic of mine. It seems that our current methods of learning are so inefficient still, right?

29:44Like compared to the existence proof of humans, we take five samples and we learn something. Machines, 200, maybe per whatever data point you might need. Anyone doing anything interesting in data efficiency? Or do you think there's just a fundamental inefficiency that machine learning has that will just always be there compared to humans? I think it's a good point that if you look at the amount of data these models are trained on and you compare it to like the amount of data that a human observes to get the same performance, I guess pre-training, it's a little hard to make an apples to apples comparison because like, I don't know, how many tokens does a baby actually absorb when they're developing?

30:21But I think it's a fair statement to say these models are less data efficient than humans. And I think that that's an unsolved research question. And probably one of the most important unsolved research questions. Maybe more important than algorithmic improvements, because we can increase the supply of data out of the existing set of the world and humans. I guess that's good. So a couple of thoughts on that. One is that the answer might be an algorithmic improvement. Maybe algorithmic improvements do lead to greater data efficiency. And the second thing is that it's not like humans learn from just reading the internet.

30:55So I think it's certainly easiest to learn from just like data that's on the internet, but I don't think that's like the limit of what data you could collect. The last follow-up before we change topics to coding, any other just anecdotes or insights from Ilya, just in general? Because like you've worked with him, so there's not that many people that we can talk to that have worked with him. I think I've just been very, very impressed with his vision that I think like, especially when I joined and I saw, the internal documents at OpenAI of what he had been thinking about back in 2021, 2022, even earlier, I was very impressed that he had a clear vision of where this was all going and what was needed.

31:36Some of his emails from 2016, 17, when they were founding OpenAI was published. And even then he was talking about how he thinks like one big experiment is much more valuable than 100 small ones. That was like a core insight that differentiated them from brain, for example. It just seems very insightful that he just sees things much more clearly than others. And I just wonder what his production function is. Like, how do you make a human like that? And how do you improve your own thinking to better model it? I mean, I think it is true that, I mean, one of OpenAI's big success was betting on the scaling paradigm.

32:10It is just kind of odd because, you know, they were not the biggest lab, you know, it was difficult for them to scale. Back then, it was much more common to do a lot of small experiments, more academic style. People were trying to figure out these various algorithmic improvements, and OpenAI bet pretty early on large scale. We had David Luan on, who I think was VP Eng at the time of GPT-1 and 2, and he talked about how the differences between Brain and OpenAI was basically the cause of Google's inability to come out with a scaled model. Just structurally, everyone had allocated compute, and you had to pool resources together to make bets, and you just couldn't.

32:47I think that's true that OpenAI was structured differently. And I think that really helped them. Like OpenAI functions a lot like a startup and other places tended to function more like universities or, you know, research labs as they traditionally existed. The way that OpeningEye operates more as a startup with this mission of building AGI and superintelligence, that helped them organize, collaborate, pull resources together, make hard choices about how to allocate resources. And I think a lot of the other labs have now been trying to adopt paradigms more like that, setups more like that. Let's talk about maybe the killer use case, at least in my mind, of these models, which is coding.

33:27You released Codex recently, but I would love to talk through the Noam Brown coding stack. What models you use, how you interact with them. Cursor, Windsurf. Lately, I've been using Windsurf and Codex. Like actually a lot of Codex. I've been having a lot of fun. You just give it a task and it just goes off and does it and comes back five minutes later with a pull request. And is it core research task or side stuff that you don't super care about? I wouldn't say it's side stuff. I would say basically anything that I would normally try to code up, I try to do it with Codex first. Well, for you, it's free, but yeah, for everybody, it's free right now.

34:04And I think that's partly because it's the most effective way for me to do it. And also, it's good for me to get experience working with this technology and then also seeing the shortcomings of it. It just helps me better understand, okay, this is the limits of these models and what we need to push on next. Have you felt the AGI? I felt the AGI multiple times, yes. Like, how should people push Codex in ways that you've done? And, you know, I think you see it before others because obviously you were closer to it. I think anybody can use Codex and feel the AGI. It's kind of funny how, like, you feel the AGI and then you get used to it very quickly.

34:43So it's really, like... Dissatisfied with, like, where it's lacking. Yeah, I know. You know, it's magical. one day. I was actually looking back at the old Sora videos when they were announced. Because remember when Sora came out, it was just magical. You look at that and you're like, it's really here. This is AGI. But if you look at it now and it's kind of like, oh, you know, the people don't move very organically. And there's a lack of consistency in some ways. And you see all these flaws in it now that you just didn't really notice when it first came out. And yeah, you get used to this technology very quickly.

35:17But I think what's cool about it is that because it's developing so quickly, you get those feel-the-AGI moments every few months. So something else is going to come out and it's magical to you. And then you get used to it very quickly. What are your windsurf pro tips now that you've immersed in it? I think one thing I'm surprised by is how few people... I mean, maybe your audience is going to be more comfortable with reasoning models and use reasoning models more. But I'm surprised at how many people don't even know that O3 exists. like I've been using it day to day it's basically replaced Google search for me like I just use it all the time like and also for things like coding like I tend to just use the reasoning models my suggestion is like if people are not have not tried the reasoning models yet because like honestly like we do like people love them people that use it love them obviously a lot more people use GPT 4.0 and just like the default on chat GPT and that kind of stuff I think it's worth trying the reasoning models like I think people would be surprised at what they can do I use Windsurf daily and they still haven't actually enabled it as like a default in Windsurf.

36:21Like I always have to dig up, like type in 03 and then it's like, oh yeah, that exists. It's weird. I would say like my struggle with it has been that it takes so long to reason and actually break out of flow. I think that is true. Yes. And I think this is one of the advantages of Codex that like, okay, you can give it a task that's kind of self-contained and like it can go off and do its thing and come back 10 minutes later. And I can see that if you're doing, if you're using this thing as more like a pair of programmer kind of thing, then yeah, you want to use GPT 4.1 or something like that.

36:51What do you think are the most broken part of the development cycle with AI? In my mind, it's like pull request review. I use Codex all the time, and then I got all these pull requests, and it's kind of hard to go through all of them. What other thing would you like people to build to make this even more scalable? I think it's really on us to build a lot more stuff. These models are very limited in some ways. I think I find it frustrating that, you know, you ask them to do something and they spend 10 minutes doing it. And then you ask them to do something pretty similar and then they go spend 10 minutes doing it.

37:25And like, you know, it's, I think I described them as like, they're geniuses, but it's their first day on the job, you know? And that's like kind of annoying. Like even the smartest person on earth, when it's their first day on the job, you know, they're not going to be like as useful as you would like them to be. So I think being able to get more experience and act like somebody that's actually been on the job for six months instead of one day, I think would make them a lot more useful. But that's really on us to build that capability. Do you think a lot of it is GPU constrained for you? If I think about Codex, why is it asking me to set up the environment myself?

37:58When the model, if I ask Go3 to create an environment setup script for a repo, I'm sure it'll be able to do it. But today in the product, I have to do it. So I'm wondering, in your mind, could these be a lot more if we just, again, put more test time compute on them? Or do you think there's like a fundamental model capability limitation today that we still need a lot of like human harnesses around it? I think that we're in an awkward state right now where like progress is very fast and there's things that are like, clearly we could do this in the models to be better. We're going to get to it. It's, you're just limited by how many hours there are in the day.

38:31So progress can only proceed so quickly. We're trying to get to everything as fast as we can. And I think that O3 is not where the technology will be in six months. I like that question overall. There's a software development lifecycle, not just generation of the code, like from issue to PR, basically, is the typical commentary of that. And then there's the Windsurf side, which is insider IDE. What else? Pull request review is something that people don't really... There are startups that are built around it. It's not something that Codex does, and it could. And so then there's like, what else is there?

39:06that is sort of rate limiting the amount of software you could be iterating on. It's an open question. I don't know if there's an answer. Anything else on ASWE in general? Like where do you think this goes just in form factors or what will we be looking at this time next year in terms of what models were able to do that they're not able to today? I don't think it's gonna be limited to ASWE. You know, I don't think it's gonna be limited to software engineering. I think it's gonna be able to do a lot of remote work kind of tasks. Yeah, like freelancer type Upwork. Yeah, or just like even things that are not necessarily software engineering.

39:43So the way I think about it is like anybody that's doing a remote work kind of job, I think it's valuable to become familiar with their technology and kind of get a sense of what it can do, what it can't do, what it's good at, what it's not good at. Because I think the breadth of things that it's going to be able to do is going to expand over time as well. I feel like virtual assistants might be the next thing after ASU then. Because they're the most easily like, you know, virtual assistants, like hire someone in the Philippines, someone who just looked through your email and all that, because that is entirely, you can intercept all the inputs and all the outputs and train on that.

40:16And maybe opening, I just buys a virtual assistant company. Yeah. I think what I'm looking forward to is that for things like virtual assistants, the models, like if they're aligned well, they could end up being like really preferable for that, for that kind of work. You know, if there's always this like principal agent problem, where if you delegate a task to somebody, then are they really aligned with doing it as you would want it to be done? Or just as cheaply, as quickly as they can. And so if you have an AI model that's actually really aligned to you and your preferences, then that could end up doing a way better job than a human could.

40:52Well, not that it's doing a better job than a human could, but it's doing a better job than a human would. That word alignment, by the way, I think there's an interesting overriding or homomorphism between safety alignment and instruction following alignment. And I wonder where they diverge. Okay, so I think where it diverges is like, what do you want to align the models to? Like that's, I think, a difficult question, you know? Like you could say like you wanted to align it to the user. Okay, well, what happens if the user wants to build a novel virus that's going to wipe out half of humanity?

41:22That's safety alignment, yeah. So there's a question of like, I think alignment, I think they're related, you know? And I think the big question is like, what are you aligning towards? It's, yeah, there's like humanity goals and then there's your personal goals and everything in between. So that's kind of, I guess, the individual agent. And you announced you're leading the multi-agent team at OpenAI. I haven't really seen many announcements. Maybe I missed them on what you've been working on, but what can you share about interesting research directions or anything from there? Yeah, there hasn't really been announcements on this.

41:53I think we're working on cool stuff and I think we'll get to announce some cool stuff at some point. I think the team, in many ways, This is actually a misnomer because we're working on more than just multi-agent. Multi-agent is one of the things we're working on. Some other things we're working on is just like being able to scale up test time compute by a ton. So, you know, we get these models thinking for 15 minutes now. How do we get them to think for hours? How do we get them to think for days, even longer? And be able to solve incredibly difficult problems. So that's one direction that we're pursuing.

42:22Multi-agent is another direction. And here, I think there's a few different motivations. we're interested in like both the collaborative and the competitive aspect of multi-agent. I think the way that I describe it is people often say in AI circles that humans occupy this very narrow band of intelligence and AIs are just going to like quickly catch up and then surpass like this band of intelligence. And I actually don't think that the band of human intelligence is that narrow. I think it's actually quite broad because if you compare anatomically identical humans from, you know, caveman times. They didn't get that far in terms of like, you know, we would consider intelligence today, right?

43:01Like they're not putting a man on the moon, you know, they're not like building semiconductors or nuclear reactors or anything like that. And we have those today, even though we as humans are not anatomically different. And so what's the difference? Well, I think the difference is that you have thousands of years, a lot of humans, billions of humans cooperating and competing with each other, building up civilization over time. The technology that we're seeing is the product of this civilization. And I think similarly, the AIs that we have today are kind of like the cavemen of AI. And I think that if you're able to have them cooperate and compete with billions of AIs over a long period of time and build up a civilization, essentially, The things that they would be able to produce and answer would be far beyond what is possible today with the AIS that we have today.

43:56Do you see that being similar to maybe like Jim Phan's Voyager skill library idea, resaving these things? Or is it just the models then being retrained on this new knowledge? Because the humans then have a lot of it in the brain as they grow. I think I'm going to be evasive here and say that we're not going to... Until we have something to announce, which I think that we will in the not too distant future, I think I'm going to be a bit vague about exactly what we're doing. But I will say that the way that we are approaching multi-agent in the details and the way we're actually going about it is, I think, very different from how it's been done historically and how it's being done today by other places.

44:38I've been in the multi-agent field for a long time. I've kind of felt like the multi-agent field has been a bit misguided in some ways in the approaches that the field has taken and the way that's been approached. And so I think we're trying to take a very principled approach to multi-agent. Sorry, I got to add. So you can't talk about what you're doing, but you can say what's misguided. What's misguided? I think that a lot of the approaches that have been taken have been very heuristic and haven't really been following the bitter lesson approach to scaling and research. Okay. I think maybe this might be a good spot.

45:10So obviously you've done a lot of amazing work in poker. And I think as the reasoning model got better, I was talking to one of my friends who used to be a hardcore poker grinder. And I told them I was going to interview you. And their question was, at the table, you can get a lot of information from a small sample size about how a person plays. But today, GTO is so prevalent that sometimes people forget that you can play exploitatively. What do you think is the state? as you think about multi-aging and kind of like competition, is it always going to be trying to find the optimal thing? Or is a lot of it trying to think more in the moment, like how to exploit somebody?

45:46I'm guessing your audience is probably not super familiar with poker terminology. So I'll just like explain this a bit. A lot of people think that poker is just like a luck game. And that's not true. It's actually, there's a lot of strategy in poker. So you can win consistently in poker if you're playing the right strategy. So there's different approaches to poker. One is game theory optimal. This is like you're playing an unbeatable strategy and expectation, like you're just unexploitable. It's kind of like in rock, paper, scissors. You can be unbeatable on rock, paper, scissors if you just randomly choose between rock, paper, and scissors with equal probability.

46:17Because no matter what the other guy does, you know, they're not going to be able to exploit you. You're going to win. You're going to, like, not lose an expectation. Now, a lot of people hear that and they think, like, well, that also means that you're not going to win an expectation because you're just playing totally randomly. But in poker, if you play the equilibrium strategy, it's actually really difficult for the opponents to figure out how to tie you. And they're going to end up making mistakes that will lead you to win over the long run. It might not be a massive win, but it is going to be a win.

46:46If you play enough hands for a long enough period of time, you're going to win in expectation. Now, there's also exploitative poker. And the idea here is that you're trying to spot weaknesses in how the opponent plays. You know, maybe they're not bluffing enough, or maybe they fold too easily to a bluff. And so you start adapting from the game theory optimal balance strategy of like, you bluff sometimes, you don't bluff sometimes, to then playing a very unbalanced strategy that's like, oh, I'm just going to like bluff a ton against this person because they always fold whenever I bluff. Now, the key is that there's a trade-off here, because if you're taking this exploitative approach, then you're opening yourself up to exploitation as well.

47:22And so you have to choose this balance between playing a defensive game theory optimal policy that guarantees you're not going to lose but might not make you as much money as you potentially could versus playing an exploitative strategy that can be much more profitable but also it creates weaknesses that the opponents can take advantage of and trick you. And there's no way to perfectly balance the two. It's kind of like in rock, paper, scissors. If you notice somebody is playing paper for five times in a row, you might think like, oh, they have a weakness in their strategy. I should just be throwing scissors and I'm going to take advantage of them.

47:51And so on the sixth time you throw scissors, but actually that's the time when they throw a block, you know, and you never really know. So you always have this trade-off. The poker AIs that have been extremely successful, and like my background is like I worked on AI for poker for several years during grad school and made the first superhuman no limit poker AIs. the approach that we took was this game theory optimal approach, where the AIs would play this unbeatable strategy and they would play against the world's best and beat them. Now, that also means they beat the world's worst, like they would just beat anybody.

48:19But if they were up against a weak opponent, they might not beat them as severely as a human expert might, because the human expert would know how to adapt from the game theory optimal policy to be able to exploit these weak players. And so there's this kind of unanswered question of like, how do you make an exploitative poker AI. And a lot of people had pursued this research direction. I dabbled in it a little bit during grad school. And I think fundamentally, it just comes down to AI is not being as sample efficient as humans. We discussed earlier, if a human's playing poker, they're able to get a really good sense of the strengths and weaknesses of a player within a dozen hands.

48:56It's honestly really impressive. And back when we were working on AI for poker in the mid-2010s, these AIs would have to play like 10 ,000 hands of poker to like get a good profile of like who this player is, like how they're playing, where their weaknesses are. Now, I think with more recent technology, that has come down. But still, the sample efficiency has been a big challenge. Now, what's interesting is that after working on poker, I worked on diplomacy. I think we talked about this earlier. And diplomacy is this, you know, it's a seven player negotiation game. And when we started working on it, I took a very game theory approach to the problem.

49:33I felt like, okay, it's kind of like poker. You have to compute this game theory optimal policy and you just play this. You're going to not lose an expectation. You're going to win in practice. But that actually doesn't work in diplomacy. And it doesn't work, again, for a question of how much of a rabbit hole do we want to go down on this? But basically, when you're playing the zero-sum games like poker, game theory optimal works really well. When you're playing a game like diplomacy, where there's like you need to collaborate and compete and there's room for collaboration, then game theory optimal actually doesn't work that well.

50:02And you have to understand the players and adapt to them much better. So this ends up being very similar to the problem in poker of like, how do you adapt to your opponents? In poker, it's about adapting to their weaknesses and take advantage of that. In diplomacy, it's about adapting to their play styles. It's kind of like, if you're at a table and everybody's speaking French, you don't want to just keep talking in English. You want to adapt to them and speak in French as well. That's the realization that I have with diplomacy, that we need to shift away from this game theory optimal paradigm towards modeling the other players, understanding who they are, and then responding accordingly.

50:37And so in many ways, the techniques that we developed in diplomacy are exploitative. They're not exploitative. They're really just adapting to the opponents, to the other players at the table. But I think the same techniques could be used in AI for poker to make exploitative poker AIs. If I didn't get, you know, AGI pilled by the incredible progress that we were seeing with language models and like shifting my whole research agenda to focusing on like general reasoning, probably what I would have worked on next was making these like exploitative poker AIs. It would be a really fun research direction to go down.

51:09I think it's still there for anybody that wants to do it. And I think the key would be taking the techniques that we use in diplomacy and applying them to things like poker. I think to me, the core piece is when you play online, you have a HUD, which tells you, you know, all these stats about the other player and like, you know, how much they participate, preflop, blah, blah, blah. And to me, it's like a lot of these models, from my understanding, are not really leveraging the behavior of the other players at the table. They're just kind of looking at the board state and kind of working from there.

51:36That's correct. The way the PokerEyes work today, they're just kind of like sticking to their pre-computed GTO strategy. and they're not adapting to the other players at the table. And you can do various kind of hacky things to get them to adapt, but they're not very principled. They don't work super well. Okay, any grad students listening? If you want to work on that, I think that is a very, very reasonable research direction that'll at least get in front of you and get some attention at least. The other thing that this conversation brings up for me is, well, yeah, well, one of the hypotheses for what is the next step after test time compute is world models.

52:15Is world modeling importance or worthwhile research direction? Like, Jan LeCun has been talking about this nonstop. But, like, basically no LLMs have, like, they have internal world models, but, like, not explicitly a world model. I think it's pretty clear that as these models get bigger, they have a world model and that world model becomes better with scale. So they are implicitly developing a world model. And I don't think it's something that you need to explicitly model. I could be wrong about that. When dealing with people or multi-agents, it might be because you have entities that are not the world and you're resolving hypotheses of which are the many types of entities you could be dealing with.

53:00You know, there was this like long debate in the multi-agent AI community for a long time about, and it's still going on, about whether you need to explicitly model other agents, like other people, or if they can be implicitly modeled as part of the environment. For a long time, I took the perspective of, of course, you have to explicitly model these other agents because they're behaving differently from the environment. They take actions, they're unpredictable, they have agency. But I think I've actually shifted over time to thinking that actually, if these models become smart enough, they develop things like theory of mind.

53:33They develop an understanding that there are other agents that can take actions and have motives and all this stuff. And these models just develop that implicitly with scale and more capable behavior broadly. So that's the perspective I take these days. So like what I just said was an example of a heuristic that is not bitter lesson filled and you just, it just goes away. Yeah. It's really all come back to the bitter lesson. Got to cite them every, every podcast. So one of the interesting findings and most consistent findings, you know, I think you were at ICLR and one of the hit talks there was about open-endedness And this guy, Tim, who gave that talk, has been doing a bunch of research about multi-agent systems, too.

54:12One of the most consistent findings is always that it's better for AI to self-play and improve competitively, as opposed to sort of humans training and guiding them. And you find that with like, you know, AlphaZero and R10, whatever that was. Do you think this will hold for multi-agents? Like self-play to improve better than humans? Yeah. So, okay. So this is a great question. And I think this is like worth expanding on. So I think a lot of people today see self-play as like the next step and maybe the last step that we need for superintelligence. And I think if you're following, you know, you look at something like AlphaGo and AlphaZero, we seem to be following a very similar trend, right?

54:52Like the first step in AlphaGo was you do large-scale pre-training. In that case, it was on human Go games. With LLMs, it's pre-training on, you know, tons of like internet data. And that gets you a strong model, but it doesn't get you, you know, an extremely strong model. You know, it doesn't get you superhuman model. And then the next step in the AlphaGo paradigm is you do large-scale test time compute or like large-scale inference compute. And in that case with MCTS, and now we have like reasoning models that also do like this large-scale inference compute. And again, that like boosts the capabilities a ton.

55:24Finally, with AlphaGo and AlphaZero, you have self-play, where the model plays against itself, learns from those games, gets better and better and better, and just goes from something that's around human-level performance to way beyond human capability. It's like these Go policies now are so strong that it's just incomprehensible. What they're doing is incomprehensible to humans. Same thing with chess. And we don't have that right now with language models. And so it's really tempting to look at that and say, oh, well, we just need these AI models to now interact with each other and learn from each other, and they're just going to get to super intelligence.

55:56The challenge, and I kind of mentioned this a little bit when I was talking about diplomacy, the challenge is that Go is this two-player zero-sum game. And two-player zero-sum games have this very nice property where when you do self-play, you are converging to a minimax equilibrium. And I guess I should take a step back and say, in two-player zero-sum games, two-player zero-sum games are chess, Go, even two-player poker, all two-player zero-sum, what you typically want is what's called a minimax equilibrium. This is that GTO policy, this policy that you play where you're guaranteeing that you're not going to lose to any opponent in expectation.

56:35I think in chess and go, that's pretty clearly what you want. Interestingly, when you look at poker, it's not as obvious. In a two-player zero-sum version of poker, you could play the GTO minimax policy, and that guarantees that you won't lose to any opponent on earth. But again, I mentioned there's, you're not going to beat a weak player. You're not going to make as much money off of them as you could if you instead played an exploitative policy. So there's this question of like, what do you want? Do you want to make as much money as possible? Or do you want to guarantee that you're not going to lose to any human alive?

57:06What all the bots have decided is like, well, what all the like AI developers in these games have decided is like, well, we're going to choose the minimax policy. And conveniently, that's exactly what self-play converges to. If you have these AIs play against each other, learn from their mistakes, they converge over time to this minimax policy, guaranteed. But once you go outside of two-player zero-some games, like in the case of diplomacy, that's actually not a useful policy anymore. You don't want to just have this very defensive policy, and you're going to end up with really weird behavior if you start doing the same kind of self-play in things like math.

57:40So for example, what does it mean to do self-play in math? you could fall into this trap of like, well, I just want one model to pose really difficult questions and the other model to solve those questions. You know, that's like a two-player zero-sum game. The problem is that like, well, you could just like pose really difficult questions that are not interesting. You know, you could just like get, ask it to do like 30-digit multiplication. It's a very difficult problem for the AI models. Is that really making progress in the dimension that we want? Like not really. So self-play outside of these two-player zero-sum games becomes like a much more difficult, nuanced question.

58:17So I think Tim kind of like basically said something similar in his talk that there's a lot of challenges in really deciding what you're optimizing for when you start to talk about self-play outside of two players or some games. My point is that like, this is where the AlphaGo analogy breaks down. And not necessarily breaks down, but like, it's not going to be as easy as self-play was in AlphaGo. What is the objective function then for that? What is the new objective function? Yeah, it's a good question. Yeah. And I think that that's something that a lot of people are thinking about. Yeah. I'm sure you are.

58:52One of the last podcasts that you did, you mentioned that you were very impressed by Sora. You don't work directly on Sora, but obviously it's part of OpenAI. I think the most recent new updates or in that sort of generative media space is autoregressive image gen. Is that interesting or surprising in any way that you want to comment about? I don't work on ImageGen, so my ability to comment on this is kind of limited, but I will say I love it. I think it's super impressive. It's one of those things where you work on these reasoning models and you think, wow, we're going to be able to do all sorts of crazy stuff, like advanced science and solve agentic tasks and software engineering.

59:29And then there's this whole other dimension of progress where you're able to make images and videos now, and it's so much fun. And that's getting a lot more of the attention, to be honest, especially in the general public. and it's probably driving a lot more of the subscription plans for ChasBT, which is great. But I think it's just kind of funny that we're also, I promise, we're also working on super intelligence. But you can make everything Ghibli. I think the delta for me was, I was actually harboring this thesis that diffusion was over because of autoregressive emission. There were rumors about this end of last year, and obviously now it's not to come out.

1:00:05Then Gemini comes out with text diffusion, and diffusion is so bad. And this is two directions, and it's very relevant for inference, of autoregressive versus diffusion. Do we have both? Does one win? The beauty of research is you've got to pursue different directions, and it's always going to be clear what is the promising path. And I think it's great that people are looking into different directions and trying different things. I think that there's a lot of value in that exploration. And I think we all benefit from seeing what works. Any potential in diffusion reasoning? Let's say your channel.

1:00:42I can't answer that. Okay. So you did a master's in robotics too. We'd love to get your thoughts on one, you know, OpenAI kind of started with the pen spinning trick and like the robotic arm they wanted to build. Is it right to work on this humanoid likes? Do you think that's kind of like the wrong embodiment of AI outside of the usual, you know, how long until we get robots, blah, blah, blah. Is there something that you think is like fundamentally not being explored right now that people should really be doing in robotics? I did a master's in robotics years ago. And my takeaway from that experience, first of all, I didn't actually work with robots that much.

1:01:16I was like technically in a robotics program. I played around with some Lego robots my first week of the program. But then honestly, I just like pretty quickly shifted to just working on AI for poker and was kind of nominally in the robotics master's. But my takeaway from like interacting with all these roboticists and seeing their research was that I did not want to work on robots because the research cycle is so much slower and so much more painful when you're dealing with like physical hardware. Like software goes so much more quickly. And I think that's why we're seeing so much progress with language models and like all these like virtual coworker kind of tasks, but haven't seen as much progress in robotics that like physical hardware just is much more painful to iterate on.

1:01:56On the question of humanoids, I don't have very strong opinions here because this isn't what I'm working on, but I think there's a lot of value in non-humanoid robotics as well. I think drones are a perfect example where there's clearly a lot of value in that. Is that a humanoid? No. But in many ways, that's great. You don't want a humanoid for that kind of technology. I think weekly, I think that non-humanoids provide a lot of value. I was reading Richard Hemmings, The Art of Doing Science and Engineering, and he talks about how when you have a new technological shift, people try and take the old workloads and like replicate them just in the new technology versus you actually have to change the way you do it.

1:02:37And, you know, when I see this video of like, you know, you're a humanoid in the house, it's like, well, the human shape is kind of has a lot of limitations that can actually be improved. But I think people, what's familiar, you know, it's like, would you put a robot with like 10 arms and like, you know, five legs in your house? Or would it be Yuri? And like, when you get up and you see that thing walking around and is that why we use humanoid? So I think to me, there's almost like this local maximum of like, you know, we got to make it look like a human. but I think like what's like the the best shape in-house I'm terrible at product design so I am not the person to ask on this I think there is a question of like is it better to make humanoids because they're more familiar to us or is it worse to make humanoids because they're more similar to us but not quite identical like I don't know which one I would actually find creepier yeah yeah the thing that got me humanoid pilled a little bit was just the argument that most of the world is made for humans anyway.

1:03:32So if you want to replace human labor, you have to make a humanoid. I don't know if that's convincing. Again, I don't have very strong opinions in this field because I don't work in it. I was weakly in favor of humanoids. And I think what really persuaded me to be weakly in favor of non-humanoids was listening to the physical intelligence CEO and some of his pitches about why they're pursuing non-humanoid robotics. And conveniently, their office is actually very close to here. So if you wanted to... They're speaking at the conference I'm running. Okay, perfect. So I'm looking forward to that. So I'd say listen to his pitch and maybe he can convince you that non-genote is the way to go.

1:04:08Awesome. The other one I would refer people to is Jim Phan recently did a talk on the physical tiering test, which he did at the Sequoia conference, which was very, very good. He's such a great educator and explainer of things. It's very hard, especially in that field. Cool. We're done asking you about things that you don't work on. So these are just more rapid fires to sort of explore some of your boundaries and get some quick hits. How do you or top industry labs keep on top of research? Like what are your tools and practices? It's really hard. I think that a lot of people have this perception that like academic research is irrelevant and that's actually not the case.

1:04:45I think that we do. We look at academic research. I think one of the challenges is like a lot of academic research shows promise in their papers, but then actually doesn't work at scale or even doesn't replicate. I think if we find interesting papers, we're going to try to reproduce that in-house and see if it still holds up and then also doesn't scale well. But that is a big source of inspiration for us. Whatever hits archive, literally you do the same as the rest of us. Or do you have a special process? Especially if I get recommendations. We have an internal channel where people will post interesting papers.

1:05:20And I think that's a good source of like, okay, well, this person that is more familiar with this area thinks that this paper is interesting. So therefore I should read it. And similarly, I'll keep track of things that are happening in my space that I think are interesting. And if I think it's really interesting, maybe I'll share it. For me, it's like WhatsApp and Signal group chats with researchers. And that's it. I think it is like, I mean, a lot of people look at things like Twitter. And I think it's really unfortunate that we've reached this point where things need to get a lot of attention on social media for it to be paid attention to.

1:05:50That's what the grad students are trained. They're taking classes to do this. I do recommend to like, you know, I've worked with grad students, I work with fewer now because we don't publish as much. But when I was at FAIR publishing papers, like I would tell the grad students I was working with that like, you need to post it on Twitter. And you need to, and we go over like the Twitter thread about like how to present the work and everything. And there's a real art to it. And it does matter. And it's kind of the sad truth. I know when you were doing the ACPC, like the AI poker competition, you mentioned that people were not doing search because they were limited to like two CPUs at inference.

1:06:25Do you see similar things today that are like keeping interesting research from being done? That might be, it's not as popular. It doesn't get you into the top conferences. Like, are there some environmental limiters? Absolutely. And I think one example is for benchmarks that you look at things like humanity's last exam, Like you have these incredibly difficult problems, but then are still very easily gradable. And I think that actually limits the scope of what you can evaluate these models on. If you, if you stick to that paradigm, it's very convenient because, you know, it's very easy to like then score the models, but actually a lot of the things that we want to, you know, evaluate these models on are kind of like more fuzzy tasks that are not multiple choice questions and making benchmarks for those, for those kinds of things is so much harder and probably also like a lot more expensive to evaluate.

1:07:11But I think that those are really valuable things to work on. And that would fit the same moment. GPT 4.5 is like a high taste model in a way. There's kind of like all these like non-measurable things about a model that are really good that maybe people are not. Well, I think there are things that are measurable, but they're just like much more difficult to measure. And I think that a lot of benchmarks have kind of stuck to this paradigm of posing really difficult problems that are really easy to measure. So let's say the pre-training scaling paradigm took about five years from discovery of GPT to scaling it up to GPT-4.

1:07:45And then we give test-time compute five years as well. So if test-time compute hit a wall by 2030, what would be the probable cause? It's very similar to pre-training where you can push pre-training a lot further. It just becomes more expensive with each iteration. I think we're going to see something similar with test-time compute. We're like, okay, we're going to get them thinking instead of three minutes, they're for three hours and then three days and then three weeks. Oh, you run out of human life. Well, there's two concerns. One is that it becomes much more expensive to get the models to think for that long or scale up test time compute.

1:08:19As you scale up test time compute, you're spending more on test time compute, which means that there's a limit to how much you could spend. That's one potential ceiling. Now, obviously, not obviously, but I should say that we're also becoming more efficient. These models are becoming more efficient in the way they're thinking, as they're able to do more with the same amount of test-time compute. And I think that's a very underappreciated point, that it's not just that we're getting these models to think for longer. In fact, if you look at 03, it's thinking for longer than 01 preview for some questions, but it's not like a radical difference, but it's way better.

1:08:45Why? Because it's just like becoming better at thinking. Anyway, yeah, these models, you're going to scale up test-time compute. You can only scale it up so much. That becomes a soft barrier. In the same way that pre-training, it's becoming more and more expensive to train better and better pre-trained models or bigger pre-trained models. The second point is that as you have these models think for longer, you kind of get bottlenecked by walk-clock time. If you want to iterate on experiments, it is really easy to iterate on experiments when these models would respond instantly. It's actually much harder when they take three hours to respond.

1:09:15And what happens when they have three weeks? It takes you at least three weeks to do those evaluations and to then iterate on that. And a lot of this, you can paralyze experiments to some extent, But a lot of it, you have to run the experiment, complete it, and then see the results in order to decide on the next set of experiments. I think this is actually the strongest case for long timelines, that the models, because they just have to do so much in serial time, we can only iterate so quickly. How would you overcome that wall? It's a challenge. And I think it depends on the domain. So drug discovery, I think, is one domain where this could be a real bottleneck.

1:09:49I mean, if you want to see if something like extends human life, it's going to take you a long time to figure out if like this new drug that you developed, like actually extends human life and doesn't have like terrible side effects along the way. Side note, do we not have perfect models of human chemistry and biology by now? Well, so this is, I think, the thing. And again, I want to be cautious here because I'm not actually a biologist or a chemist. Like I know very little about these fields. Last time I took a biology class was 10th grade in high school. I don't think that there's a perfect simulator of human biology right now.

1:10:18And I think that that's something that could potentially help address this problem. That's like the number one thing that we should all work on. Well, that's one of the things that we're hoping that these reasoning models will help us with. Yeah. How would you classify mid-training versus post-training today? All these definitions are so fuzzy. So I don't have a great answer there. It's a question people have. And you're like opening eyes like now explicitly hiring for mid-training. And everyone is like, what the hell is mid-training? I think mid-training is between pre-training and post-training.

1:10:50It's like, it's not post-training. It's not pre-training. It's like adding more to the models, but like after pre-training. Interesting ways. Yeah. Okay. All right. Well, you know, I was trying to get some clarity. Is the pre-trained model now basically like just the artifact that then spawns other models? and it's almost like the core pre-training model is never really exposed anymore. And it's the mid-training, the new pre-training, and then there's the post-training once you have the models branched out. You never interact with an actual, just like raw pre-trained model. Like if you're going to interact with the model, it's going to go through mid-training and post-training.

1:11:30So you're seeing the final product. Well, you don't let us do it, but we used to. Well, yeah, I mean, I guess there's open source models where you can just like interact with the raw pre-trained model. But for OpenAI models, like they go through a mid-training step and then they go through a post-training step and then they're released. And they're a lot more useful. Like, frankly, if you interacted with the only pre-trained model, it would be super difficult to work with and it would seem kind of dumb. Yeah. But it'd be useful in weird ways, you know, because there's a mode collapse when you post-trained for it for, like, chat.

1:11:59Yeah. And in some ways, you want that mode collapse. Like, you want that collapse of, like, distribution. Yes, to be useful. Yeah, I get it. Yeah. We're interviewing Greg Brockman next. You've talked to him a lot. What would you ask him? What would I ask Greg? I mean, I get to ask Greg all the time. What should you ask Greg? To evoke an interesting response that he doesn't get asked enough about, but you know this is something that he's passionate about, or you just want his thoughts. I think in general, it's worth asking where this goes. What does the world actually look like in five years? What does the world look like in 10 years?

1:12:33What does that distribution of outcomes look like? and what could the world or individuals do to help steer things towards the good outcomes instead of the negative outcomes. Okay, like an alignment question. I think people get very focused on what's going to happen in one or two years. And I think it's also worth spending some time thinking about what happens in five or 10 years and what does that world look like? I mean, he doesn't have a crystal ball. But he certainly has thoughts. That's, yeah, so I think that's worth exploring. Okay. What are games that you recommend to people, especially socially?

1:13:12What are games that I recommend to people? I've been playing a lot of this game called Blood on the Clock Tower lately. What is it? It's kind of like Mafia or Werewolf. It's become very popular in San Francisco. Oh, that's the one we played in your house? Yeah. Okay, got it. It's kind of funny because I was talking to a couple people now that have told me that it used to be that poker was the like way that like the VCs and tech founders and stuff would socialize with each other. And actually now it's shifting more towards Blood on the Clock Tower. Like that's the, that's the thing that people use to like, um, you know, connect in the Bay Area.

1:13:46And I was actually told that a startup held a recruiting event that was a Blood on the Clock Tower game. Yeah. So, uh, I guess it's like, it's really catching on, but it's a fun game. and I guess you lose less money playing it than you do playing poker. So it's better for people that are not very good at these things. I think it's kind of like a weird recruiting event, but it's certainly a fun game. What qualities make a winner here that is interesting to hire for? That's the thing. It's like, okay, I guess you get the ability to lie. Deception and like picking up on deception. Like, is that the best employee?

1:14:20I don't know. So my slight final pet topic is Magic the Gathering. So you have, we talked about some of these games, Chesco, and they have perfect information. Then you have Poker, which is imperfect information in a pretty limited universe. You only have a 52 card deck. And then you have these other games that have imperfect information, like a huge pool of possible options. Do you have any idea of how much harder that is? How does the difficulty of this problem scale? I love that you asked that because I have this huge store of knowledge on AI for imperfect information games. This is my area of research for so long.

1:14:57and I know all these things, but I don't get to talk about it very often. We've made superhuman poker AIs for No Limit Texas Hold 'em. One of the interesting things about that is that the amount of hidden information is actually pretty limited because you have two hidden cards when you're playing Texas Hold 'em. And so the number of possible states that you could be in is 1 ,326 when you're playing Heads Up, at least. And that's multiplied by the number of other players that there are at the table, but it's still not a massive number. And so the way these AI models work is they enumerate all the different states that you could be in.

1:15:32So if you're playing like six-handed poker, there's five other players, five times 1 ,326, that's the number of states that you'd be in. And then you assign a probability to each one. And then you feed those probabilities into your neural net and you get actions back for each of those states. The problem is that as you scale the number of hidden possibilities, like the number of possible states you could be in, that approach breaks down. And there's still this very interesting unanswered question of what do you do when the number of hidden states becomes extremely large? You know, so if you go to Omaha poker, where you have four hidden cards, there are things you could do that's kind of like, that are kind of heuristic that you could do to reduce the number of states.

1:16:09But actually, it's still a very difficult question. And then if you go to a game like Stratego, where you have 40 pieces, so there's like close to 40 factorial different states you could be in, then all these like existing approaches that we used for poker kind of break down and you need different approaches. And there's a lot of active research going on about like, how do you cope with that? So for something like Magic the Gathering, the techniques that we used in poker would not out of the box work. And it's still an interesting research question of like, what do you do? Now, I should say this becomes a problem when you're doing the kinds of search techniques that we used in poker.

1:16:43If you're just doing model free RL, it's not a problem. And my guess is that if somebody put in the effort, they could probably make a superhuman bot for Magic the Gathering now. Yeah, there's still some unanswered research questions in that space. Now, are they the most important unanswered research questions? Right. I'm inclined to say no. I think there's like, the problem is that like the techniques that we used in poker to do this kind of search stuff were pretty limited. And like, if you expand, if you expand those techniques, maybe you get them to work on things like Stratego and Magic the Gathering, but they're still going to be limited.

1:17:12They're not going to get you like superhuman and code forces with language models. So I think it's more valuable to just focus on the very general reasoning techniques. and one day as we improve those I think we'll have a model that just out of the box one day plays Magic the Gathering at a superhuman level and I think that's the more important and more impressive research direction Cool. Amazing. Yeah. Thanks for much for coming on, Noam. Yeah. Thanks for your time Yeah. Thanks. Thanks for having me