OpenAI's Dan Roberts: Why AI Can Now Make Discoveries

0:00One of the things that ChatGPT was able to do was assume it was false. When you go against the grain and do something contrarian like that, you really have to have strong conviction in what you're doing in order to persevere down a really long calculation path. I feel really excited that we will get to really answer a lot of fundamental questions in the field of science that we care about, with the aid or the models being the driving force. And so that's just really thrilling. Hi, I'm Matt Turck. Welcome to the Matt Podcast. It's been yet another extraordinary last few days in AI, with OpenAI, DeepMind, and Anthropic cracking some of the most famous long-insolved questions in mathematics, known as the Erdos problems.

0:37A moment many view as a stunning breakthrough and yet another signal that AI is moving from doing the work we ask of it to autonomously making deep science discoveries. To unpack the moment and the fundamental advances in model reasoning that make it possible, I'm excited to welcome Dan Roberts, a top AI researcher at OpenAI, who comes from a deep background in theoretical physics and has a particular interest in the intersection of science and AI. In this conversation, we go deep on what reinforcement learning actually is, why it's the most important paradigm in AI right now, and what's ahead for AI and science.

1:12Please enjoy my conversation with Dan Roberts. Hey Dan, excited to do this. Thanks for taking the time. Of course, very happy to be here. You are the lead of the foundations of reinforcement learning team at OpenAI. So what does that mean? What does the name mean? The larger team that we're on is called foundations. And we think about reinforcement learning. So very boring foundations of reinforcement learning. But the team comes from a mandate of thinking about the science of reinforcement learning. and a long time ago, which in AI speak is like six months ago, maybe a year, I guess now two years.

1:54So before we released a one and thinking reasoning models, we were studying this internally. And, and one of the advantages to being first, or at least to being forced and spending a lot of resources on scaling things up is that you can empower a group of people to not just work on making the thing work, but work on understanding how it works. And then beyond that, how do we scale? How should we think about scaling reinforcement learning versus scaling pre-training? So what are scaling laws look like? But then going beyond that, what sort of things does this kind of training teach us? What doesn't it teach us?

2:32We're very interested in at the frontier for exploratory scenarios. How do we either improve or understand better what reinforcement learning is doing? We have all this compute that famously we are in the process of acquiring and we would like to turn that compute into intelligence. And to do that, we need to make thinking models and somewhere along the way, we interact with that process, usually at the earlier stage for models, not the next model, but things that are like the next model or the next next model. Great. And quickly, what was your path to OpenAI? So how did you go from studying physics to being where you are today?

3:16I did a PhD in theoretical physics from MIT, thinking about the intersection of quantum gravity and quantum information. Thought a lot about black holes, quantum chaos kind of thing of what if you throw something into a black hole? What happens to the information? Does it come out? If we think about black holes as computers, how fast are they? I was very interested in this fundamental question in theoretical physics, which is how do you find a quantum theory of gravity? I also got very interested in this interplay between computation and the laws of physics. Any computer exists in the universe and behaves according to physical law.

3:56So the sort of computations you can do are bounded by the laws of physics and there's some sort of interesting relationship there black holes are pretty interesting because they sort of saturate some conjectured bounds around processing of information and from there i did a postdoc at the institute for advanced study and around that time i'm pretty old now for at least for this field so that was about 2016 uh was when the dqn atari paper from deep mind happened in 2015 and then alpha go was in 2016 and i got very excited about the possibility of machine learning and then deep learning was statistical science that lived in a similar framework to the sort of frameworks that we use to study the rest of the universe.

4:44And so there's always this question of, how does everything work? This three-year-old question of, I'm curious about everything. And if you look outward and you care enough, you end up philosophy maybe. If you're quantitative, you end up in physics. Very, very crude characterization. And AI that works is very fascinating. Or AI systems that work are fascinating because they are simple examples that do things that humans do. And then if it lives in the same framework that we use to understand everything else, then, you know, it's sort of like you can draw the parallels between how does the universe work and how do I work or how does intelligence work.

5:22So I got extremely interested in AI and deep learning. Then I went to FAIR, Facebook's AI research lab, around 2017 to basically try and use the tools from theoretical physics to understand deep learning. Deep learning was supposed to be this really difficult thing that you couldn't understand. And I thought maybe the tools of physics could be helpful. this actually culminated in a book that I wrote with a collaborator who now is still a collaborator of mine now at OpenAI working on the same thing as Shoyeta but we wrote this book The Principles of Deep Learning Theory that is a culmination of these sets of ideas of can we sort of use the statistical ideas of understanding statistical systems like the gas in the room, we can characterize them with some simple laws of thermodynamics like the ideal gas law, and maybe we can make similar progress in understanding deep networks.

6:24So that was sort of my transition. I also had a startup along the way and spent some time at Sequoia Capital as an entrepreneur in residence. So there's some tension between am I a scientist and am I an entrepreneur, but about two years ago, after thinking about whether I want to start another AI company, I realized that the thing that was most exciting right now was what was happening at the frontier, that there's some amazing scientific progress happening in AI. And to really get at the questions and understand what's going on, you need to be there and you need to participate. And that meant joining lab.

7:01So I joined OpenAI two years ago. Great. Thank you for that. Where do you think we are in the evolution of AI being increasingly able to solve difficult scientific problems? I mean, certainly something that we've been talking as an industry about for a while now, but it seems to be accelerating, perhaps just like everything else in AI. But where do you think we are? I think one of the interesting things is that this process is smooth. there's no sharp point, or I don't think there will be a sharp point where we'll say that systems weren't able to be useful for the scientific process to their fully-fledged scientists.

7:43There will be sort of a gradual shift. If you had to point to one moment, maybe it would be the release of O1 by OpenAI and the sort of paradigm of test time compute and reasoning. But I'm sure if I tried to make that claim, you could go and look at GPT-4 and see that there's glimpses of that sort of useful behavior for the scientific process were already present. As a general point, you know, the models are very good at certain types of things that clearly are amenable to making progress in math. They're not open loop, fully fledged scientists in any domain, although, you know, neither am I. It seems like it's just this really nice gradual process.

8:21So it feels like a particularly fun week to be having this conversation because over the last few days, there were a number of different announcements in the general field of AI and mathematics around the AirDosh problems. OpenAI came out first with this progress, but like almost within a few hours, Google DeepMind had a claim as well on different problems and anthropic had some claims however from what i understand the open ai approach and the deep mind approach were very different and that may be very interesting in terms of what that means for ai as a research scientist this conjecture everyone assumed was true and uh but could not prove it one of the things that chat gpt was able to do was uh assume it was false.

9:15And when you go against the grain and do something contrarian like that, you really have to have strong conviction in what you're doing in order to persevere down a really long calculation path. Because there's a lot of choices that you can make along the path. And if you get any of those choices wrong, if your ideas don't work, then you find out that you didn't make any progress. And so you need this really strong persistence. And then you need expertise in this other field, which is like algebraic number theory, some sort of generalization of number theory on things that sort of generalize the integers and the real numbers.

9:51You go down that path really far, you can refute this conjecture. So that was the big result. The big result was that this conjecture of this lower bound for the number of pairs that you can make is false. Not only is it false, it was false due to a really interesting connection to another field of mathematics. And so you would have to be somebody who is aware of this problem as interesting, which sounds like your expertise is one thing, and then be an expertise in something else, and then also be super contrarian and go down this really long path. And then you would have identified the solution.

10:25The OpenAI approach and the deep mining approach were very different. Do you want to compare and contrast the two approaches? One of the approaches that GDM takes is to take problems, present them in a formal language called Lean, and then use methods to search for proofs in that language. And some problems, for problems to be representable, there's this process called auto-formalization, where you take English version of the problem and you translate it into rigorous formal statements, and then you conduct your proofs there. And it's designed so that the proofs can be airtight. No one has to go and check for some hidden assumption or some weird thing.

11:08I guess it's usually hidden assumptions or definitions that are not airtight. But in that setting, which is a setting that DeepMind has cared a lot about, they were able to formalize some problems and use their system to prove them. So that's one approach. Another approach is to just take the problem in English with mathematical expressions as well, but just the English statement of it, which is informal, and understand what is meant by that and solve that in informal language, presenting a proof much like the way a human mathematician would or a human mathematician who's not using Lean. And then you have to check it.

11:49The verification problem is harder because it's not something that auto-checks. And that second approach was OpenAI. Most of our results that we publicize, as far as I can think, are all in the informal setting. We have language models that we've taught them to reason at test time. And one of the applications or benchmarks for that is reasoning in mathematics. Okay, great. All right. So let's get into reinforcement learning. To make this broadly accessible, let's start from the top. What is the one, two, three sentences definition for reinforcement learning? And perhaps give us a simple non-technical analogy.

12:33for people to understand? Maybe a simple thing to do would be to give you two examples of how you could try to learn something, you as an individual. And maybe we can take a game or even, say, a video game. I'm old enough where I played the original 8-bit Mario Brothers, the Super Mario Brothers. And so here are two ways you could learn how to play. One way you could learn how to play is your dad takes it out and plugs it in and he boots up the game and then he plays for a few hours. And then you just watch him play. That's all you do. So he's demonstrating how to play. And then at the end of that, and he's not very nice, so he doesn't let you play.

13:16But then he goes and runs outside and does something else. And you sneak into his room, you plug it in and you try to play. How good are you going to be? Well, all you've done is tried to memorize what he's done. You haven't gotten to push any of the buttons yourself. You haven't gotten to interact with the game yourself. This is sometimes called expert demonstrations. And you're just trying to memorize what someone else is doing. The version of supervised learning, the supervision being like you just watch what he does and accept that that's the true way of doing the thing. Reinforcement learning would be your dad's like, here, why don't you play?

13:52Maybe he shows you once or maybe he doesn't even need to show you because the game is beautifully designed to sort of take you from not knowing anything to being able to play expertly. There's something called a curriculum. But you play. Maybe the first thing you do is you run, you hit the first bad guy, and probably this example is dated, but you lose a life. But then the second time, you press a button and you jump. And so you're taking actions. There's an environment that's giving you feedback. And there's this close connection between the environment, between actions that you can take and then the responses that you're getting.

14:26And then the final part is there's a reward. And the reward can be something that you get pretty often. For instance, every time you do something, there's some score that goes up. Or it could be just something that you get at the end. So you play a game of chess and at the very end, you get a reward, which is you won or you lost. But in the middle, you don't really know how you're doing until the very end. And so this is called sparse resistance rewards. But I think this is the basic idea in that, and there's obviously lots of variance here and ways to quibble with this. But it's this notion that you interact with an environment, you get a reward, and often it's in a way where you get this sort of feedback as opposed to just trying to learn from data that you don't get to interact with.

15:10And why does it work and why is RL so powerful? It works because of this ability to get feedback from the environment. You can go and learn, you know, if you're doing it right, you can figure out how to learn the things that you don't know. And I also think it's powerful because of this fact that it's much easier to learn when you're learning at the right level for you, right? So if you want to learn, you know, addition, you shouldn't read a calculus textbook. You want to learn by being able to practice and learn at the right level. I'm actually making the choices and learning from my own choices, whether they work or not, then I'm able to place it in a better context for the set of things that I understand.

15:58Great. And then conversely, what's the catch and how does RIL break? The setting where very difficult is the setting that I alluded to before where you don't get much feedback from the environment. You have to take many, many, many, many actions and then you get maybe, yes, that whole set of actions was good or no, it was bad. For instance, you're playing a game of chess and you don't know until you make all the moves. That has an opponent, so it's maybe complicated. Maybe it's you are trying to do a homework problem and it's a research level. Or someone gives you a well-defined problem, like we give our language models.

16:37And it's a problem that requires days and days of thinking. There's so many choices that you can make along the way. And at the end, if you don't get any feedback at all, if you're just hidden in the woods by yourself, scribbling in notebooks, it's very hard to make progress that way because you don't have any sense. If you get a yes at the end or you get a no at the end, you have no sense for which of the actions that you took, which of the things you did were good or bad. Okay, great. Now let's talk about how RL has been applied in the context of large language models. So was the first step historically RLHF?

17:16Yeah, I think that's probably fair, at least in a broad sense, that the first kind of RL that was done on language models was part of this post-training process to turn a model that just tries to predict the next word on the internet into either something that will follow your instructions, be nice to you, or fit the form of a chatbot. So do you want to define for people what RLHF is and sort of hide work quickly? The basic idea is that you could use, collect data from humans. That's so that RLHF is reinforcement learning from human feedback. So you collect data from humans and you train a value function.

18:01So you would show in the language model setting, say two different completions from a language model, ask them to say which is better. This sort of comparisons could be used to train a value function. And then you can use that as a reward for reinforcement learning process. Great. And you do that initially with humans, but then you build that into a reward model. Yeah. So you would train a model for this. And then now, because during the training process, you can't just pause your training run to ask some humans for input. Right. The feedback that would have way too much latency. So instead you need a proxy for what a human would say.

18:41So you train this model based on the human preference data, and then you can optimize against it, or at least a little bit. One of the famous things in the history of RL is Move 37. How do you train a model to encourage the model to do that kind of things and come up with brand new ways while being efficient and exploit known paths? Yeah, so the great thing about Go is that you can just train it. It's a zero-sum two-player game. You can train it in what's called self-play. it plays itself and it can go from playing randomly to expert play and it will find whatever the sort of best strategies are.

19:20So if that means exploring, great. That means exploiting. Actually, I have a funny story about this. So I met Noam Brown in grad school. He went to a different grad school than me, but he wanted to enter MIT's PokerBot competition. And he had a poker bot that was the best in the world, but it wasn't something that would compete against humans yet. He just won in this research competition. He collaborated with me and another friend to enter MIT's poker bot competition. This is great actually for me because I learned some really exciting work in AI and I got very excited about this while I was doing physics.

20:03We were playing essentially this kind of self-play equilibrium strategy. There's some nuances, but essentially we could not lose assuming we did not have any bugs in our code. The way this thing worked was that it was a tournament where you would be paired with, say, another person and play them. Depending on the amount of points you got in some sort of round-robin setup, they would eliminate the bottom half and they would keep going until you got to the final table, which would just be say you versus the other person and so the scores there was the award ceremony and we didn't know what what happened but the there was someone else who was um you know what what did the what was everyone's scores over time look like and there was you know say 64 i think there were 32 actually people playing so it was like around a 32 kind of tournament and 30 people over time you know their scores would were all very negative and going down and then there was one person whose score was like pretty much straight up and then there was another that was like pretty good but not like with a crazy slope and so do you want to guess which one we were?

21:14So we were the lower slope and then there was this other guy that had this crazy slope was just like completely crushing all the other players and then this happened for the round of 16, the round of 8, the round of 4 and then in the round of 2 it's heads up, us versus this guy who's like over the course of this tournament won way more than us uh overall like taking more money from from everyone else and then we crushed him because why because he was exploiting the weaknesses of of of everybody else right it's had some theory of mind to try to figure out oh this this guy you know does this when he bluffs and and so it was very i assume it was very good at like taking um you know exploiting everyone else but we were just playing the best possible thing that you could that you can do given you know so the the criteria was not maximize your your amount that you get from anyone else it was don't lose like put you know and and uh and you know so it's the best response to anyone's strategy and so at the end we had to win assuming we did it right and someone else playing the same strategy would would tie okay fascinating so just uh tying this back to the beginning of the conversation about the erdoge problem and and solving unsolved math problems.

22:26Presumably, the instinct would be that you need a lot of exploration, not exploitation. So how does that work in the context of novel scientific discovery? I think math research or scientific research in general has a lot of versions of both explore and exploit. To give the recent example, the OpenAI unit distance proof, I think, is very much in the explore setting where the model was happy to be contrarian and try to disprove this thing that everyone believed. And it was just looking for, it has this huge repository of understanding all of human math. And so it was spending a very long amount of time.

23:10I forget how many hours, but I think we published a rewritten version of this chain of thought, but like hours and hours trying different things. So it's clearly in the domain of exploration. a lot of times though you can ask these models to compute something that they understand very well and then that has a different structure and might look a lot like exploit. There's a paper that came out recently after the OpenAI result where the unrelated Erdős problem has something to do with if you have a set and you try to add the set to itself or you try to multiply the set with itself so take the elements and add them all together or take the element individualize or multiply them together and how many unique sums or products you get there's some conjecture around that and this this one was also disproved and and that was done by by humans and the core idea was um it's like a totally different problem but there there was inspiration from the from the unit distance one the idea that you you can sort of generalize uh from from the pick a certain type of numbers that had a certain property that the open AI model figured out and that they realized that like this applies in this setting so that's very much an exploit thing but and so so I think the process clearly is like this the actual discovery process I think normally when you talk about explore exploits maybe we're talking about when training reinforcement learning models how should we train them but I think there's this interesting point that in the scientific discovery process there's really this interplay between like exploring exploration and then exploitation in order to totally push the field forward switching to RL in modern LLM systems so there used to be a saying which I think comes from Yann LeChan that RL was the cherry on top of the cake but I think you have argued that things have switched now and that RL is the main part, the cake?

25:12Do you want to just walk us through what you were thinking? Yeah, I said that about a year and a half ago. I had to give a talk that was public and I couldn't say much. So I decided to invert this meme with this cake and the cherry. RL is really exciting. That's what I'm here talking about. And I think that when you have a lot of compute, you want to turn that compute into intelligence in a way that's useful. And RL is one way of doing it. And we just started doing it then. And we're going to do a lot more of it now. Why did RL start working well? It's not an entirely new concept. It's been tried for many years now.

25:56What is different now? Yeah, I'm not sure, to be honest, when people say it wasn't working, what that actually means. There was this 2016, 27, maybe even to 2018 before the transform period where DeepMind was all in on RL and OpenAI had Dota and Rubik's Cube and some other exciting results as well. But a lot of people were all in on RL and then there were language models. And the obvious thing to do was scale up the thing that worked, which was pre-training. and I don't know whether or what people tried for RL as you pointed out RLHF was a central thing that came pretty quickly originally was developed for in the in the context of game environments of like trying to prevent reward hacking by using I think the original paper was about using human feedback to like control like a character for what or something like that but there's an interesting thing to point out here, though, which is that there's this question of how do you get models to think in test time and reason?

27:07And there was a reasoning effort at OpenAI that was quite early and spent some time and came up with some algorithms. I think maybe the simple thing to say is that if you have a powerful enough pre-trained model, then it can start to do well at RL. It can start to think at use test time compute to, for instance, solve math problems that it wouldn't otherwise be able to do. There's a viral analysis from earlier this year, February, I think, that claims that RL produces less than one bit of useful information per 10 ,000 tokens. And then Carpathie called it sucking supervision through a straw. What is your take on this and the overall efficiency of RL?

27:52If you look at the DeepSeq algorithm, which is a public thing that we can talk about, then you train on sequences that are correct. So whether it's correct or not is maybe one bit of information. So I think you can see where that logic comes from. I think the question is, is this doing a kind of thing that you can't otherwise do? Maybe you would want to give more supervision, but how are you going to do that? But I think it's very clear that these methods have led to a bunch of breakthroughs in terms of the explosion of what the models can do, and both in coding and in science. I think broadly it's about getting models to think in test time, to use test time compute and do reasoning.

28:41And there's clearly a lot of the pieces of what the RL process is that's essential to make that work. What's your overall feeling in terms of how far we can go with that current sort of systems model where we have pre-training and then we have RL on top? Somewhat famously, last year, there was a conversation with Rich Sutton on the Dworkish podcast where his claim, in my best attempt to paraphrase it, was that LLMs were not really intelligence. And therefore, RL was the only way to do it. And pure RL, not LLM plus RLs. What is your take on this? I mean, obviously, you're on an RL team at a company that does both pre-training and RL combined.

29:30So what's your take? Let me tell another story. So before I did my PhD, I spent two years in the UK, and I was at Oxford for one of those years. And I was at a pub, as one does, and two of my close friends, one was a cognitive scientist, and one was a linguist. And so we had that sort of argument that you do in those situations when you're that age. and so something like physics is the most fundamental of all the sciences because it explains how the world works and everything is in the world i said this earlier my computer exists in the world i exist in the world we all follow the laws of physics and then the cognitive scientists said something like yes but then you have to you know you have to process it so there's all sorts of cognitive biases about that and and you know the way you collect data and learn something something but then the linguist was was like blah blah blah wichtenstein you know the everything goes through language.

30:24That's the method of communication. That's, you know, that's the way words mean things are the central thing. And when we want to talk about the laws of physics, we have to use language. And I sort of feel like he and Victor, like that, that was correct, right? That's what's what, or at least the path through AI suggests that that is a correct path. I'm conceding now to Kyle. And if he's, if he's listening, he's now a linguistics professor. This whole idea of reinforcement learning that kicked off the previous decades interested in AI, the sort of grounding, I think, that was needed was to make things really work, is through language, because everything goes through language, right?