Are We Misreading the AI Exponential? Julian Schrittwieser on Move 37 & Scaling RL (Anthropic)

32:43And also, we as a human, we don't do this, right? We just imagine in our head of, oh, I'm going to say this, then he's probably going to respond in that way. This meant that alpha zero, as it was, could not be applied to such problems because it required some way of, you know, simulating the game, scoring the outcomes. And the idea with mu zero was that, well we already have a deep neural network right these networks can learn a lot of things so why not let it why not teach it to predict the future of the environment the future of the world why not make the model be able to learn for itself what is going to happen after each action it takes after that you also applied this to code and math so that was alpha code and alpha tensor So zooming out a little bit, that evolution of reinforcement learning in games and then code and then math.

33:42What did you learn about the general power of search and learning that is today relevant in modern agentic AI systems? How did that whole body of work translate to what you are doing today? So games are a really good sandbox to learn very quickly about a lot of the reinforcement learning science. But, you know, the algorithms that work well, the kind of problems that we encounter, even from a technical point of view, how do we build a learning system that spans, you know, many data centers, uses tens of thousands of machines? Because games are very clean sandbox, very clean environments, so we can make many good experiments.

34:24And then now that we have a much more general model, right, the language models can do almost any task, but they're much more complicated and much slower to experiment with. We can apply those same lessons of, you know, we know how to build a really robust reinforcement learning infrastructure. And now we can build the same one for language models. You know, we know if you do this kind of RL, then the model will learn how to exploit the reward. and so we can apply the same lessons, the same mitigation techniques to the language models. If I understand correctly, I think MuZero had a learn world model.

35:06So basically it rehearsed the future for a better expression. So do modern LM agents have anything like that? Do they have an internal world model that lets them preview actions before they commit? So I think, yes, I would say that language models have not an explicit world model, but they do have an implicit model of the world. Because to be able to predict, you know, what is the next likely word in this sentence? How is this paragraph going to continue? They need to internally model, you know, what is the state of the world? that makes this person say that thing. And so it's actually somewhat similar to mu0 in the sense that mu0 also only had an implicit world model.

35:55It was never trained to predict what does the screen actually look like if you take an action. It was also only trained to implicitly predict if I take this action, what is the next action I should take? Or is it going to be good or bad for me? So in both those cases, you have an implicit representation of the world in your model that you can use to make predictions, but you're not actually reconstructing the full state of the world. Because reconstructing the full state of the world, that can be very expensive and complex. If you think about super high resolution video, audio signals, it's a very large amount of data that probably you don't actually need.

36:39If you think of human attention, we are only aware of a very small subset. of what's actually going on all around us all the time. Because that's the most relevant information that we actually need to make decisions. And that goes back to the prior discussion about pre-training. So the reason why pre-training and RL work well together is that you have that world model that's implicitly embedded into the corpus. Although the argument against it is that it's what humans think the world model is as embodied by language versus what the world model actually is. And that's my understanding of the debate.

37:24I mean, for the debate, I think different people have different points of view, so I don't want to speak for anybody. Yes, yes, yes. But yes, I think pre-training on this rich knowledge gives you some representation of the world already so that when you actually start to act and interact with the world, you can very quickly make meaningful decisions, meaningful actions. I like to think of it in a similar way if you look at many animals when they are born they very quickly know how to move how to run even if you look at gazelles for example in the savannah clearly they did not have time to really learn this from scratch a few minutes or hours in their cases they did not do pre-training but they have some evolutionary encoded structure in their brain because clearly it is very beneficial to have some sort of knowledge to make your learning more efficient yeah just rl in nature would uh would lead to not so good results like you're a gazelle and like you have to a b test whether to run towards the lion or away from the plane it's like the you know thousands of generations of gazelles acquired this knowledge over time yeah it was encoded in their genes and their brain structure in some way right and then you get to start on top of that.

38:41I think the main challenge or the main thing you need to watch out for is that you don't over-encode or you don't restrict your search space too much. If your pre-training, if your prior knowledge prevents you from exploring something that might be the correct course of action, that will be bad. So there is some danger there you have to be aware of. So this general idea of pre-training and RL work together in modern AI systems seems to be the big idea or topic of 2025, although, of course, I know it's been years in the making. Why did it take so long? It feels like RL, you know, progressed in its own direction and then pre-training worked in its own direction and those were slightly separate.

39:28Why did it take so long to put them together? Is it just purely practical and economic or anything else? scaling up the language models to the massive degree that we scaled them up took a lot of effort on its own and from a science point of view from an engineering point of view retraining and supervised training is more stable and sort of easier to debug because you don't have this feedback cycle you basically you know have a fixed target and we're trying to learn this target and so then you can focus on like you know is my training working and like is my infrastructure working and then you know this scales fall over versus if you compare to RL in RL you have this feedback cycle of oh I learned something and then I use that to generate my new training data and then I learn from that training data and now if you have you know something is not working it's very hard to figure out where in this cycle your problem is coming from you know maybe your training update was bad and that's why you suddenly started behaving badly.

40:31Or maybe the way you select actions to behave is not correct. And so you generate the bad training data. That's what messed up everything. So it's just much more complicated to get working correctly. And so I think it makes a lot of sense to first scale up the pre-training, the architectures, figure out something that works pretty well, especially if you can already get pretty far by some fine-tuning, some prompting. And then when it's clear that these models are really general, they are really useful, and we have them in a pretty stable state, then you can ramp up RL and take them even further.

41:08Even in our own work, if you look at AlphaGo, AlphaZero, we always follow the similar split as well, where we first set up the architecture of the network, the training using fixed supervised data. And only when we have that working really reliable, only then do we do the full RL loop and the full training. Just because debugging all of it at the same time, you're just setting yourself up for failure. It's really useful to be able to isolate the component and say, I have non-good data over here, I have a non-good target there. If the thing in between is not working, I can isolate it. And then we can isolate all parts of the system.

41:43How compute intensive is it to scale RL? and are there scaling laws for RL the same way you do in pre-training? There's less published literature about it. But I think if you look at all the RL literature over time, we see very similar returns on compute in pre-training and in RL where we can invest exponentially more compute in RL and keep getting benefits. There's going to be some interesting research to come to figure out what are the trade-offs between pre-training and RL compute. We know what should be the split for a big model, for example. It could be 50-50, it should be like 1-10, which way should it be 1-10?

42:31So I think that's going to be extremely interesting. But so far, yeah, we definitely see good returns on both. What's the latest state-of-the-art or thinking in the field of rewards? so in what you described for AlphaZero AlphaGo that was basically win-loss as a reward then it sort of feels like we went into kind of like fuzzy human matching this is good this is not good and now that we expand as per the above into more general fields where it's sort of unclear whether you win or lose how does that work? What parts of the evolution are you working on? Are you excited about? Personally, I don't work that much on reward modeling.

43:18I mostly work on sort of reasoning, planning, search compute, ways of making the model smarter by spending more computation. Yeah, thinking about rewards, I think the reinforcement learning process per se doesn't really care where the reward comes from. The algorithms are very happy to use any source of reward, whether it has a human feedback signal, It's like some automated signal from like, you know, winning, losing the game or passing a test. Whether it's something more model generated. For example, Anthropic, we have this paper about constitutional AI to, you know, have the model itself score whether you're following some guidelines.

44:02So it can be very flexible to what kind of reward you follow. The RL, VR, all the things, those are at this stage stuff that you see commonly used. Any thoughts? Yeah, I think we're seeing a huge mix of rewards and environments. And I think it's very much people are working very hard in figuring out what are the best reward sources and how do we scale it up and how do we get more rewards, more reliable rewards. That will be one of the key ingredients in scaling up RL further. And so switching from rewards, what is the latest thinking in terms of data, training data for RL? Again, following the evolution from like AlphaGo, it used to be human data and then like self-play.

44:55How does that work? Where does the data come from and what kind of data works best to train modern RL? Yeah, I guess a great thing about RL is that the data is generated by your model itself. So the smarter our models become, the better RL data we can generate, the more interesting and complex tasks they can solve, which then gives us more and more data that we can train on. Because the more complex a task, the longer it takes to solve the task, the more data it generates that we can then use for training. I think part of the challenge is to find tasks that are really representative of what people actually want to do with the model.

45:36Because now language models are so general, people are using them for so many different things. There's more and more of a challenge of, you know, we need to cover as many of those as possible in NARA to make sure that, you know, the model is actually able to do this diverse set of tasks. What matters more for training data? Is that quality? Is that quantity? Is that recency? I think that's a very interesting question that maybe doesn't have a super clear answer yet, or is maybe still interesting research to be done. I think we've seen papers arguing for different things, or we've seen different benefits, right?

46:11Clearly, we see improved training as we scale up the data, we can keep improving. But we've also seen very interesting fine-tuning results, papers published with a very small amount of examples. You can teach the model how to do an interesting skill. And I think we don't have any good scaling laws yet that tell us the trade-off, especially I think because it's very hard to measure what is the quality of a data point, right? Like how good is this example compared to this other example? Without being able to measure this, it's very hard to quantify the trade-off in any way. I think intuitively it's definitely true that if you have bad data, RL doesn't work that well.

46:51and if you have very high quality data it becomes much more stable for example I think that was like a it's very clear in like alpha zero days where you we spend alpha zero spends a lot of computation it does a lot of planning and search to decide which move to take and so that generates very high quality data to train on which then resulted in RL training that was incredibly stable so you can run it across continents take a long time to generate the data and then train on it. It's very robust, versus in modern RL with language models, the difference in how good is the model and what data it generates that we then train on is not so large because we more directly sample from the model and then train it, which then results in reinforcement learning that is less stable.

47:43As a one direction of scaling RL and making it more stable is by improving this, by, for example, putting more reasoning into your language model to generate much more high quality training data. That can then give us training that is much more stable and that we can scale up much more easily. I'd love to spend a little bit of time now on the general topic of RL and agents. So the famous agentic AI that everybody's been talking about, you know, breastlessly for the last year. so for people listening and you know as often in an effort to make this broadly accessible by you know a group of people in tech could you drive home the sort of intersection and overlap between RIL and agents does RIL power agents how does that work yeah so I guess like maybe first let's take a step on what do we actually mean by agent yes as compared to like a general language The second most debated question after a GIS is what is an agent?

48:47I guess, yeah, for our purposes, let's just say that an agent is an AI that can act on its own. Maybe take some actions on a computer, save some files, edit some files, send an email, whatever you want. But the main characteristic is that it doesn't have to interact with the user all the time. It can do things on its own. The reason why RL is very important for this actually connects back to pre-training because our pre-training data is not very agent-like. If you think of the pre-training data, there is websites and books and all kinds of recent text that has a lot of information, but it doesn't have a lot of actions.

49:33It doesn't really capture how do humans actually interact with the world. So if we take a raw pre-trained model, it's not a very good agent. Maybe you can prompt it a bit and sort of push it in the right direction, but it's not going to be very good at interacting and especially it's not going to be very good at correcting for its own errors because the pre-training data has no examples at all of how is our agent going to fail. And that's exactly where reinforcement learning comes in because in RL, we can take our agent, let it interact with the environment and then directly train on that interaction.

50:12So for example, if the agent did well we can reinforce those actions and if the agent did badly we can push it away from those actions and if the agent sort of did badly at the beginning but then recovered and managed to do well then we can also reinforce that recovery. And so that's super important because it allows the agent to actually learn from its own distribution of behavior. And that just makes it much more robust because now it doesn't have to generalize to something it has never seen before. It can actually learn on the actual problem that it's trying to solve. And that's why RL is really unlocking so much authentic capabilities now.

50:51If I'm an AI builder today, building an AI app, and I build it on top of Anthropic, Anthropic is whatever model is going to come with some of this sort of batteries included. But as a builder on top, do I need to do my own RL? There's this emerging space of like RL as a service where, you know, for this task or that task that I build on top of a general model that sort of like offers the ability to do RL or can I do a lot of damage just through prompt or like maybe like supervise fine tuning first? I think nowadays with the capabilities of top Anthropic Cloud models, top OpenAI GPT models, you don't need to do any fine tuning.

51:41You can take the model as is, ride your own tools, your own harness, and benefit from that Argentic training. Because doing good Argentic fine tuning is actually very hard. and so it's it's quite hard to do better than the top frontier models that you might get but on the contrary coming up with good tools and a good representation representation of your task makes a huge difference so like you know depending on how you express your problem for the model can make it way harder or way easier and so you can get a lot of mileage out of that What's currently missing to achieve the big dream of urgent AI?

52:24Is it model capabilities at the core or is it sort of like boring, quote, end of quote, engineering around reliability, tool use, safety? What needs to happen? I think we have sort of just basically improvements needed around the whole space. make the model better able to correct its own errors, make the model better able to continue going for long times without getting distracted, making the model just smarter in general, maybe making the model faster. There's basically a whole set of things that we know that we can improve. There's probably not one individual blocker, and that's why we will continue to see smooth incremental progress over model releases but sort of given how many things we know there are that we can do better on and improve yeah i'm quite excited about where models are going to end up i think that's actually you know one of the reasons why ai is a very fun field is that there are so many low-hanging fruits that you know you can do much better on but already the current models are so good that it's very fun to work on it it's like oh i can fix this thing it'll be even better versus you know if you're in a place where everything has already been solved and it's really hard to figure out how to make it better it's a very different story i spent a minute on on evals uh and um we we touch upon this a little bit but just to give it some some some proper space so there was um you know in your blog post that we talked to at the very beginning of this conversation this this this concept of external benchmark and then you quoted in your piece, Goodhart Law.

54:10So first of all, what is Goodhart Law? And then how should labs compare results so that it doesn't end up with this kind of leaderboard theater that we've seen a little bit in the last couple of years? Yeah, so Goodhart Law basically says that any measure that becomes a target stops being a good measure. And you can think of that intuitively that if you start paying, for example, programmers based on how many lines of code they write, while suddenly they will discover many ways to add more lines of comments, which is completely useless. And this is a very general effect that obviously, if you give people an incentive that they should optimize, they will try very hard.

54:54And we also see this with language model benchmarks. Of course, people want to get promoted, they want to launch their model. So any benchmark that is too easily measured or that has a lot of attention on it, people will optimize very hard for it, which means that probably the model will look very good at that benchmark, but if you then use it for your own task, you might get different performance. You asked about what do we do about this. It's very hard to prevent people from optimizing on the benchmark. One possibility is just periodically create completely new held-out benchmarks that nobody has seen before.

55:33and that gives you a fairly good estimate of model performance. I know, for example, a lot of researchers have their own toy problems that they use to test all the models precisely for that reason. So this is a set of problems that nobody has seen. You have a pretty good guess that it's going to give you an unbiased estimate. If you're an individual of your company trying to decide which model to use, it's probably something similar. Make your own internal benchmark that really represents what you care about and then measure on that. I think that's likely to be the most objective, most accurate way of measuring.

56:13Internally, what does that look like at a place like Anthropic or previously DeepMind? I know there are teams that are focused on evals. How do you think about what works, what doesn't in terms of internal evals? It definitely used to be easier to have good evals. Five years ago, the tasks we were doing, I think it was easier to measure model performance. I think nowadays it's much more difficult. And I think we try to not over-rely on evals so much because it's quite hard, for example, to measure how good is this model really at writing code. Yeah, I think it's one of the big unsolved or very important problems in the field of making really good evals that are both cheap to run, reliable, and accurate, because it's easy-ish to make an eval that ticks one of those, but to get all three is quite hard.

57:14For example, at the beginning, we were talking about OpenAI's GDP eval, no, GDP eval, and that one is very accurate and unbiased, but it's very expensive to run, because what it actually involves is taking human experts, having them do the task. And then compare the model task to the experts and rate it with multiple people. So it's very accurate, but it's extremely expensive to do. And related to that topic of evals, what's the latest in terms of our ability, or I should say your ability, to truly understand how models work, so the general field of mechanistic interpretability? You alluded to the fact earlier that RL, if I understood correctly, sometimes make it a bit harder because it does things occasionally in a more inscrutable way.

58:08My words, maybe not yours. So what is the latest? And indeed, does RL make things harder or easier? Oh, so what I meant before is that debugging RL in general, completely unrelated to interpretability, is harder because they're more moving parts. But it is also true that if you're not careful with RL, you can make interpretability harder. For example, one common thing with modern models is they do reasoning with the chain of thought. You could look at the chain of thoughts to see what are the model internal thoughts. And then you could also have a thought that, oh, maybe I should use that as a reward signal in RL and punish the model if it thinks the wrong thing.

58:47But then suddenly you completely destroyed your interpretability angle. So you sort of have to be careful that, yeah, you don't do RL on the signals that you actually want to use to interpret what the model is thinking of doing. That said, I think, yeah, there are some extremely exciting interpretability things happening, including mechanistic interpretability. I think actually last year, I think before John Anthropic, maybe even, there was a super cool golden gate cloud model where they found the neurons in cloud that were responsible for the golden gate concept. And then modified them to make a version of cloud that really loved the golden gate bridge in San Francisco.

59:24And so that's like a really vivid example of, oh, you know, we really understand what's happening in this model. And I can know what better way is there to verify that understanding than actually changing the behavior of the model. And so I think that's a super important direction for safety. As the models get smarter, we really need to be able to understand what is the model thinking internally? What is the values it has? Is it lying to us? Is it actually generally following the instructions? And so I think definitely an extremely important area to invest in and work in. Especially if people are interested in working AI or doing AI research, I think interpretability is a great area to get into.

1:00:03Yeah, perfect segue. For the last part of this conversation, I'd love to zoom out and talk about the impact of AI. So if we think that we are on the exponential and things are going to only accelerate from here, what does that mean? and certainly safety and alignment, which is a core value at Anthropic, hopefully in other parts of the field as well, but like Anthropic is particularly vocal about safety and alignment, let's say. How does that actually manifest? So we just talked about interpretability. What, for people who are concerned that this is going too fast and that we collectively are creating a monster, quote, end of quote, Can you give us a glimpse into the kind of work that is done for alignment and safety at a place like Anthropic?

1:00:56Yeah, I think the focus on safety alignment pervades all of Anthropic. And there's very rigorous processes where we train a model, whenever we want to release a model, both to analyze the capabilities of the model, verify the alignment of the model, ensure that it does not do harmful things on its own ensure that it does not enable malicious users to do harmful things and to the point where if we are unsure about the safety model we will delay the launch until we're sufficiently sure that it is actually harmless we will not launch and release a model which shows that people clearly take the safety much more seriously than any financial return or revenue.

1:01:49I think also in terms of research and resources, the teams working on safety and interpretability are a big focus of the company, which gives me a lot of confidence that we actually care about this and put a lot of effort into it. And at a more technical level, and to tie back an earlier part of the conversation around when we're discussing the pre-training and safety. So is safety and alignment an RRL problem? And by that, I mean the beauty of having pre-training is that you import that world model as we're discussing. but arguably you also import into your brain a lot of bad stuff if you collect data from the internet as we know there's good things but also a lot of toxic content so is alignment largely using rl to get rid of the bad stuff that is built into the pre-training we can definitely use rl to like shape the model behavior and ensure that for example given adversarial given bad input it sort of behaves safely or knows that it can refuse or is you know robust who attempts to prompt hack the model yeah I wouldn't view it alignment adjust like an REL problem I think it sort of it goes throughout the whole stack you might you know for example filter the pre-training data in some way you might after training you might have classifiers that you know look at the model monitor the model behavior to ensure that it is actually aligned you might when you write a system prompt for the model that you use you might put safety guidelines in there so i think safety alignment it really pervades the whole of research and the whole of you know product and deployment it's not just isolated into any one part and then another super interesting topic in the same vein of like the impact of ai is obviously the discussion around jobs so if as per the gdp of our discussion the agents are becoming just as good or better than humans obviously what does that mean for all of us in terms of our jobs what have you learned after the experience of alpha zero alpha go that that could give us a glimpse into what may happen once we all have super powerful agents do our jobs so i think the first thing that we didn't talk about yet so far is that artificial intelligence is quite, I mean, this may sound a bit simplistic, but it's quite different than human intelligence.

1:04:32So we can see that, right? The model may be much better at us on some tasks, like, you know, calculation, obviously, and like much worse than us at other tasks. So it is not, I don't think it is at all going to be any like one-for-one replacement. It's going to be much more complementary of a model is really good at something that maybe I really don't like doing or I'm not interested in or I'm very bad at. And then I'm much better than the model that's a mother part. And so I think it's going to be like a gradual process of we're all going to incrementally start using models more and more to improve our own productivity rather than have a model that one for one is able to do exactly the set of things we can do.

1:05:15For example, I use cloud all the time to, for example, refactor code or maybe write some front-end code that I don't want to write. At the same time, there's other parts where I'm clearly much better at putting than cloud still. So there is a synergy of use the best, most productive skills. I guess economists call it a comparative advantage. But there is this long process of will both improve our productivity incrementally. I think that process is going to give us some time of figure out politically and figure out economically, how do we want to benefit from this massive productivity increase?

1:05:55Even independently from AI, the promise of technology has long been that we're going to be all so productive, so wealthy, that we need to work much less. Yet mysteriously, we all have like 40 hours working week for decades. And so, you know, I think it's much more like a political, social problem of like figuring out how do we actually benefit from all these improvements and like, you know, bring the increases in wealth and productivity to everybody. And it's much less a technological problem. Which also means that we can't really solve it with technology. We have to solve it at like a sort of a democratic political level.

1:06:31How do we spread these benefits? Do you think that that increases inequality? quality so uh as you think about the impact of alpha go and and and mu zero what happened to the top go players and what happened to the top chess players did they did they disappear or did they get enhanced and better yeah i think at least in the case of chess and go there has been like more interest and it has become much easier for people to study how to play go how to play chess because now you don't need to find an expert tutor. Anybody can practice on their own, right? Spend a lot of time. I guess chess streamers are very popular on Twitch right here.

1:07:14And similarly, a lot of students are using language models to study. I think also for coding, right? Cloud code, these agents, they raise the bar of what anybody who has an idea can accomplish on their own. I think the larger picture, whether it increases or decreases inequality is quite hard to forecast. It both sort of raises the floor of what any person can accomplish, but it also gives very productive people an ability to be even more productive. It's possible that we see quite a difference between countries depending on the taxation, social redistributive system that they have in whether inequality increases or decreases, for example.

1:07:57Overall, I'm quite excited that it is very much non-zero-sum. It's very much, you know, increases the total wealth available in society. I think if you think about progress, if you think about prosperity, that is the most important thing. Like redistributing the pie is kind of a loser's game. To get more wealthy, we really need to grow the pie. You know, if you think of the agricultural revolution, the industrial revolution, the reason why we have much better lives nowadays is because we're so much more productive. We have so much more wealth. And so that's the key step we want to unlock. If you manage to make everybody in society 10 times more productive, what kind of abundance can we achieve?

1:08:44I think that's a good key question, right? What advances does that unlock in medicine? Curing diseases, halting aging, what does it unlock in terms of energy? Obviously, we have climate crisis. We need more energy to sustain our lifestyle. What advances in material science can we have? All of those are basically bottlenecked on how much intelligence we have access to and how can we apply it. So I'm incredibly optimistic about what will we be able to unlock in the next five years. I think we can go extremely far. Well, that feels like a wonderful place to live it. Thank you so much, Julian. This was absolutely fantastic.

1:09:30Thank you for spending time with us. Yeah, thank you for all the exciting questions and giving me the time. Hi, it's Matt Turk again. Thanks for listening to this episode of the Mad Podcast. If you enjoyed it, we'd be very grateful if you would consider subscribing if you haven't already or leaving a positive review or comment on whichever platform you're watching this or listening to this episode from. This really helps us build a podcast and get great guests. Thanks and see you at the next episode.