The RLVR Revolution — with Nathan Lambert (AI2, Interconnects.ai)

25:44I don't know if they did that much on top of O3. Every indication I've had from Over the Eye is that deep research is more or less a thin wrapper over just O3. Yeah, it's probably like one or two small things. They're like, oh, we can make deep research work by adding this small amount of data to the training thing. And then it just works. That would be how I describe it. I mean, what is it, Gwern, the anonymous person? He replied to my QStar post on Twitter the other day, and he was like, why was this all wrong? And it's obviously simple things don't scale. There's a lot of complexity because there's a lot of other exciting things in the AI field at the time, and OpenAI kind of sends out a lot of things that confuse people.

26:27But this would fit into that, which is deep research is a minor change from an existing RL trajectory of what was like 03, probably. They had already figured out that search was going to be better, and then we're like, okay, we can repackage this. And it's a simple thing that makes a big difference. And most of the things are like that once you have traction. I think once trying to get the initial takeoff on the sigmoid is the hard Q star thing. But then once it's like this, a lot of things in the middle feel obvious, which is why I would describe one of the things that we work on for Ulmo. It's like a lot of it is just having motivation to do things that feel somewhat obvious, but they're still hard.

27:04It's hard to get different recipes or it's hard to get a full reasoning recipe off the ground. It's just like a huge change because you have all this inertia on this eval suite. And then you have to figure out if you branch your recipe or do you start from like, do we just take like open reason or zero and start from scratch, which is like, oh, it's a whole other headache of things. It's just hard to move these projects that are anywhere above five to 10 people with inertia to get stuff done. But then once you're hill climbing, things can seem really obvious. Yeah. Okay. You covered a lot there.

27:36Before my next question, just to close the brave thing. Our friend Simon Willison wrote a post that Entropic added Brave Search as one of the subprocessor in their product. So that's where the thing came from. Now, to what extent it gets used, we don't know. We don't know. I would just kind of comment on a couple of things that he said and then we'll go on to your question. There's a very good post just on the retrospective of Qstar. There's a very good post that you had, which was, that I want to send people to which was 01SIOP. Right? That does imply the question of like if 01 was a PsyOp, what else could be PsyOps now?

28:12There's definitely PsyOps out there. I mean, the whole inference time scaling plot is such a PsyOp. You put these two things next to each other with an x-axis and it just looks like it's easy to control. Whenever you see an x-axis, you think it's easy to control it. Whereas for training, the left one was training. And training makes a lot of sense. Even if you go to really old RL papers, RL learning curves are a non-log x-axis usually and they look like this. They look like these, like whatever, like logarithm or exponential rise. And then if you take one of these and you make it a log X, it's a straight line.

28:45So like that side is like, oh, okay, we've seen this before with RL. But with inference time scaling, it being an X axis is why people are like, oh, there's a knob. I can turn search up a lot. Yeah. Which is like what breeds all these weird ideas. The core of that article is just they're taking points from within training or there's a natural variance and then you line them up. And if you line them up, then you get this nice inference time scaling behavior, which is now people, a lot of people have reproduced this plot on inference time scaling. And it's much clearer now, but at the time it's like, I see why I thought it was a knob.

29:18It's like, oh, they called it inference time scaling. You control it. I think the most interesting, well, you have a lot of interesting things in your blogs, but one that stood out was about RL and tool use. You said that it's easy in an RL experiment to tell the model to try searching, but then if it doesn't get results with the tool, is going to stop using the tool very rapidly. Can we impact that? So can there be a good tool that the model doesn't know how to use and then it kind of fails and then it stops using it? Can there be a bad tool that should be improved before giving up on it? How should people think about designing the tool, improving the model and kind of like where to intervene?

29:56This is definitely on the newer side for my things that I want to work on or have worked on. I think particularly in 2026, especially in the open side, all the infrastructure models will count up a lot where I want to go deeper on this in terms of deeper search style things or very inference-heavy multiple calls. And to answer your question, there definitely can be bad tools and there definitely can be the model just using them wrong. And something that I would want to see in a model is kind of not necessarily creativity, but an openness that it doesn't know exactly what it will get out of all of its tools and this uncertainty to just try a few different things, which almost seems classical RL behavior but if you think about what a language model does they're not necessarily confident but they have a path and a direction in their answer whereas that's a big change in these reasoning tokens is to have the notion of backtracking and things like that which is some sort of openness to the tools having things that are unknown in it seems like a really nice thing for the model to have which is like, oh, what if I try this?

31:00What does it get? But especially on the open model side, which is if this is going to work where people want to use open models with tools, it's going to be because people have private data stores and stuff. So if you were to train an open model that is going to be a good reasoner like 03, but on private records of some sort that will never get sent to the cloud, it needs to be thinking of I can try some things with this to get a sense for it before saying I have to give up. and if you look at tool like tool use right now seems much more similar to like code execution or it's just a part of a sequential path that you need to get to which is like I have a plan and if it fails at a certain step I might have a backup but it's not like this iterative of I need to fiddle with the environment in order to come up with my plan it's just that it's something that people probably are going to have to train into these models which is like you might just tell it, you don't know what is in this but your answer might be in it which is like a very odd prompt, but maybe it'll help.

32:02Yeah, when we had Eric Schlanz from Anthropic who worked on the Cloud Agent before Cloud Code, he mentioned they spent basically like majority of the time on like the tool design to give to the model. And then you just kind of learn how to do it. Are you usually, well, I don't know how much you've worked on actual this stuff, but are you putting the tools one by one in the RL process? Do you think that helps? Or do you usually give, is it better to give all the tools and let the model explore? I don't really know. We haven't gotten this to work. I would say it would probably depend on the model and your starting point.

32:31If your starting point is already good at tools, it can probably generalize more. But if you're doing this weird base model RL and you have to have this kind of curriculum, if you scale RL long enough, you're going to need a curriculum of things getting harder. That's pretty obvious. So in that case, it might be tools get added when things become too hard for it to solve certain questions, which sounds very intuitive, but also just really hard to manage. practice because what is your automated signal and your training run that is time to do that. That's why video games are so good because they're designed to unlock things as you progress but I think with things like search it's like you know if you're given access to a small data store or you're given access to all knowledge on the internet.

33:15Good feedback for the ArcGIS people for the V3 benchmark is like have things where the language model needs to learn to use new actuators in the world after a certain threshold. Wow. That would be RKGI 4 then. Yeah, probably. I don't know. They're cranking them out. They're cranking them out. They're actually doing a launch party, I think, in a couple weeks. It's fun to play RKGI. I don't know if you tried. Oh, I haven't. It's pretty fun. These are IQ tests. I used to be like, oh, they weren't that relevant. But actually, now that we have a gradient where LLMs are actually significantly climbing them, Now it's actually really more interesting to compare your own intelligence to the LMS.

33:58I'm with Noam on no harnesses. No harnesses, yeah. Yeah. I mean, harnesses are cool, but they're a handicap that's changing the learning dynamic substantially. So it's good demos, but I feel like the core thrust has to be no harnesses. I mean, it's always like, is it wrong to say that these are just inductive biases, right? Like, they're not in the model, sure. But anything where you're just looking at the results contaminates... This is just a different task. I think I talked with Greg about this at ArcGi, which I told him, do harness and no harness. You just have both different categories. You're trying to be transparent and build targets for Frontier Labs?

34:40Just do both. I don't think it dilutes that much. The no harness is going to obviously be harder, but then you just get more bang for your buck on your benchmark. Yeah, it's the same data set. Staying on the topic of tools, while we're at it, you had a really good summary of recent work in multi-tool RL, which had Loop and Retool and Toro and all these other things. And I think that this is just an area that's super rich for research right now. I just wanted to give you the space to highlight. What are your favorites? What do you think that people should explore? I could share what my moderate ambition, what would be fun research project things is, is you want to create some sort of competitive dynamic or a VAL, and it has to be so much narrower than what industry is doing.

35:25So I told you this at lunch, which is like deep research, but only archive papers. So you don't have to do a full index. You have a limited domain. You have to figure out how to measure it or something. or some, I think like it's good for academics to work on academic tools because they have very high domain expertise. They already know what's going. And just like figure out how to make that something that is either very useful to users, if it's going to be good enough for that or something you can't help climb on. And I don't know if this is like brainstorming on the fly of like take related works out of papers, just look at the text and break all the links and make an eval, which is filling in hundreds of related works with archive links.

36:03Like, that's a fun deep research style idea. See if you can do it with open models on a set data store with tools. AI2 has gone through a lot of discussions with this, which is if you're trying to have impact in AI right now, as an academic, you have to level up out of papers to artifacts, which is models, data sets, evals. Data sets and evals are easier for people to have impact on. And then the next thing is, like, what do people actually use? in AI too, especially in this like Semantic Scholar team that's now working on like information agents of different types. There's another thing that I'm like distancing, so I don't have all the names, but it's, can we make open models do that sort of thing better?

36:43It's like, can you make something that people actually care about? And then that's a whole level of impact that's much higher if you have actual users. It's hard for academics and small institutions to do that. But if you're working on agents, like dog feeding is viable. It's like, can we make ourselves a good Slack summary bot that we like or something. And just making these agents really tractable. I mean, that's one direction. Another direction is just he'll climb on humanity's last exam with tools. I just think it's kind of unlikely that we're going to win as an academic and a state-of-the-art number because they're going to start spending millions of tokens per query.

37:22And it's just a lot of compute burn. Beating that on the flop equivalents is going to be so hard. unstructured thoughts is something that I'm mostly like, okay, I'll get to this. Like I have more things to figure out on the modeling and what I call like skills level, which is just how do you do reasoning to induce inference time scaling and get high eval numbers? And once you know you can do that, you can take your knowledge with you to do it in more specific domains. There's skill and there's skill acquisition, right? I think the Arc AGI definition of AGI, I quoted it, what is it? It's like efficient, yeah, skill acquisition efficiency.

37:59Because I used to describe it as three words. Right. Yeah. Your emphasis on skills in your recent talks that you've done, do you want to sort of reiterate that thesis for people to pick up on? Yeah. So I've been thinking about mostly I'm trying to get ahead of what OpenAI, et cetera, are doing probably now if it's not in their models. And with all the agents, it seems that planning is a very critical task. So it's kind of how do you come up with the taxonomy for different types of things you need to train into reasoning models for when it'll be a bottleneck. And so I came up with four. And the foundational one was skills, which is what I would say that we have already done with O1 and R1, which is you do a lot of RL, you show the inference time scaling works, and you get really high benchmark numbers.

38:45And then the next three are kind of what comes next. And most of them are around planning. So what I had is three and four on my list were abstraction and strategy, which is trying to not use planning because planning is a word that people already use a lot. Their strategy would be the direction the model should go in. And technically, what are the steps of its plan? And then abstraction is how does it break it down into things it can actually solve? And then the fourth last thing is calibration, which is just not wasting compute and knowing when to give up and ask the user things. Because overthinking is obviously a problem.

39:19So it's easy to keep getting your VAL scores to go higher by using more inference time scaling. But eventually, that's not what people want in their models. They want a smarter training regime where the model is actually getting proportionately better for its training. There's a lot of papers on overthinking and stuff like this, which I think is like OpenAI wants it because they have to foot the GPU bill. If O3 just infinite loops itself for a bunch of people, that's not good. Does it actually? I don't know, but it might. I mean, like these reasoning methods definitely can make the models just kind of unstable and just, yeah.

39:57But it's also the GPT-5 idea, which is how do you get a model that just routes the question to the right? Maybe not necessarily a router, but just knows if it needs to do a plan or if it can just answer. If you look at DeepSeek R1 and you ask it a hard math question, it's not like, here's my plan of attack. It just starts. and having a model that knows when to be like, okay, here's my plan of attack. I might need to make myself a memory store. I might need to take like a cloud code approach for this query. I'm going to build a memory store and spin up some parallel searchers and then come back.

40:32Conceivably, this is all something you can train into a model because the searches or the parallel models could be like tools in that case. The simple way to describe it is we have something like thinking tokens and then answer tokens. And so the model should be able to optionally have planned tokens before thinking or before using tools. It's like, okay, here are the table stakes. I need to do these things. And these sorts of tasks will be harder versus easier. It seems more tractable than some far-out ideas for AI. It's like a language model can write a good plan. It just needs to be asked to do so, which I would bet that Cloud Code and Deep Research are doing this.

41:12You get a user prompt. And first the model is like, yeah, there's a plan tool in CloudCode. And first they break it down. And it's like, that is something they've trained into the models. I don't think DeepSeek doesn't have it built in, but it probably could do it. And just thinking about that interface between if the model needs it to be able to do the task end-to-end on its own. Can it do that sort of thing? I think that my challenge with this whole reconciling this approach with the no harnesses thing is that I think a lot of the way that people, especially engineers, want to model it is that the plans and the memories are tools.

41:50And there are no special plan tokens. There are no special memory tokens. It's just context or it's just, you know, whatever. Specifically for planning, because then you can do fan out to other agents for tool calls and stuff. so it doesn't have to be sequential. But I'm just like, is this a fork in the road? Or do we have to make a real choice here as to do we outsource things to tools or do we keep it native within the model's tokens? I don't think it's a subjective difference. I think mostly the planning idea is to make the point that people don't get things for free. And the planning improvements might be kind of mundane, which is like we were prompting Claude and its plans were bad in this way.

42:33let's give some data where its plans are more detailed or break things down into more steps so that it's easier for them to do it. Because it's in a black box, effectively. So if it hasn't been targeted, it's unclear of what the performance will be. Or on the open model side, it might just be the idea of having different models for different parts of it. Then you're really training a model to just be good at planning. And that's data that you need to come up with. I mean, you only use that model for that one part of it. Does it feel like plans are much more reusable and should maybe not be generated every time?

43:07I feel like especially in coding, for certain sets of tasks, you want to have similar types of plans. So maybe it's not the right way to ask the model to regenerate a plan every time. There should almost be like plan blueprints as like tools and then the model fills it in. Like, where do you think the balance should be? I think they're reasonable. A plan is obviously an intermediate goal. it just seems likely that there's failures on this kind of planning level I mean the same thing goes for these rubrics that are popular whereas a lot of the technique that is popular for so-called rubric things is you have a prompt and you have a language model generated a rubric for that prompt which is a few specific things it needs to get right and that's conceptually very similar to making a plan for every task I think whether or not it's like grading is you're going to have a different type of abstraction than executing.

44:01But I think what people are seeing is that it's cheaper relative to the effectiveness to just generate it. So plans are not super long and they're not that many tokens. So it's probably just kind of like, okay, we do this. Putting it in my taxonomy might be overselling it where it just needs to be a prompt and you just need to make sure that your model's not too weird at that prompting stage. I think your taxonomy is super useful, by the way. So skills calibration strategy abstraction. I feel like maybe abstraction might be the most underrated one or hardest to solve. The way that you introduced it was different than how you wrote in your blog post.

44:37You said it was basically not to overthink. That's calibration. Yeah. Abstraction is about breaking things down. Yeah. I think both of these strategy and abstraction make the most sense on the hardest tasks that we don't know if the model can do them. Right. So if you're assigning a task to a model that you don't know if it can implement it, the strategy is very important because it needs to be very specific and narrow. where if it's doing mundane code or deep research, the plan is actually not that interesting of a thing. But when you're at the frontier of if it can, I don't know, some GPU implementing thing, you could buy into the open AI and anthropic narrative, which is help me implement this research idea in our complex distributed GPU thing.

45:17My God. It's like, this is a task that's hard for a human. And for an AI to come up with the right plan to debug and do this is very narrow path. So therefore, the strategy is pretty important of does it start with certain tests and how does it actually build this out to complexity? It's obvious that I need to come up with more better examples for this. But I think as you push it, it's more natural to see that there's only a few plans that actually get it done. And then abstraction is just important as your task becomes so big. It's like a prompt engineering thing almost. Yeah. And it's like you only have 100K tokens you can generate.

45:52You need to make sure the model breaks it down so it's not just spawning a ton of infinite processes under itself, which I do agree that abstraction is an interesting one, especially when you start to think about these models that could call in other models to do subtasks for it or parts that can be parallelized with multiple searches or just more compute. I think that kind of folds into abstraction, which is just like, how do you approach a certain nugget of the problem? And I definitely say, like, I don't have experience building this. it just feels like if you're going to visualize AI doing the hardest software or other tasks, it's something that humans are very good about.

46:30So it's like, how do you come up with a research plan in 10 weeks? Like there's a lot of, how do you prioritize which experiments to do? It's like there's a lot of inductive biases that go into that, that I don't, like a language model would not do well at that right now. Probably memory would be helpful there. So you can just get like the way we do this in real life is we accumulate experience. Yeah. One thing I did want to dive in on was just parallelism in general. There's one case where with O1 and the sort of Qstar ideas, there was one case where it was sort of overhyped in some sense. But now it's coming back with O1 Pro and DeepThink.

47:09The theory is at least, correct me if I'm wrong, basically they run O1 eight times and then they have a reward model rated and then give you the best of the eight. Yeah. Something like that. Something like that. DeepThink, also the same. We don't know any details beyond that. I think there's a lot of people exploring that, at least on the info provider side of like, you know, how do we parallelize search and planning and all that. And I'm worried about getting too hyped about it. I think it makes a lot of logical sense. And this is one of those things where MCTS also made a lot of logical sense.

47:42And we were fooled. Well, I don't think we're using parallel compute in a way to search over like, low probability tokens. We're using it to get robustness. Like O1 Pro, it was so nice because it just had a very predictable depth to it, even on niche topics where like sometimes models just fail. Yeah, you had some numbers that went to like from like 10 to like 95 % or something. I don't remember the exact numbers, but that's what it feels like. It doesn't feel like you turn on O3 Pro to make it 10 times more likely to find some niche piece of information. Like maybe it'll be a bit more likely, but we're not getting that type of like searchy notion of getting more breadth or depth into our tree.

48:23So I think there's value to it where we want to use this parallelism on the, what are either like the most important tokens that we're generating or like, okay, I know this part is crucial. Let's just spend a bit more so that those tokens are better. But it's not a like transformative thing. the part that's potentially interesting on the transformative side is like if you can get much better verifiers so i think of verifiers of changing the slope of inference time scaling you spend more tokens at inference the better verifier you have if you're doing parallel it can extract a rare occurrence so like right now if our verifiers are only like they're good at like human preference it's like okay we don't need to we don't need to crank that up very much But if we are doing really diverse generations and your verifier is better, it'll do better.

49:13I think you could look at the extreme between a reward model and an oracle, where it's like the oracle is the more you search, eventually it works. So the slope is good. But a reward model is like there's really a capped signal out of it, at least if you're doing this preference type of thing. So the slope is pretty minor and it kind of has diminishing returns. So I do think that if you could fill that with more interesting verifiers, there's potentially more to get out of parallel compute. But I don't think it is as transformative right now on my outlook. It's more like parallel agents makes more sense.

49:48If you could break down abstraction nice, like as a throughput engine, if our tasks are taking a long time, rather than a peak performance engine. which kind of fits with the whole agent versus model thing where agents are much more about getting it done at all, like being robust and being fast where this model is one generation. It's like, can you get the answer right? Yeah. I will spend a little bit more time on this and I'm happy to move on. My pushback or counter to this is that it's a way to pull forward a hypothetical future model that you can then distill from. Yeah. Which is nice. Well, I bet people, I mean, they surely will use these for synthetic data.

50:26is just like the marginal gain on synthetic data is always very high. Or just like Amanda Askel will say, better prompting will effectively make it seem like you have the next generation model. Where most people don't put effort into their prompts. Or she had said something of those lines in one of her anthropic interviews, which is just like if you can really figure out how to kind of get into the certain states of the model. Yeah, yeah. Well, anyway, that's my pitch for why this is worth doing at all. I have a science fiction story that I want to write about quantum models. In a world where you could explore cheaply multiple universes, then pull forward the right one, that would work.

51:03This sounds too science fiction-y, but I feel like in a world where we could control quantum computing well enough to explore this and scale it up enough, it could be kind of cool. It also could be that parallel compute is grounds for interesting types of innovation. I don't know, what does it mean to have parallel compute with diffusion language models that generate all their tokens at once? Does that meaningfully change some sort of application? I don't really know. I think the diffusion language model would be fun if it works. You can have much more control over inference time scaling. I mean, Gemini has one.

51:38It's hard to suss out what it changes. But once we have all these knobs, I'm hopeful that it helps build some interesting types of innovation. Because the parallel stuff is new. Architectures can change. We'll see. I've been using the Codex best of end thing. And I feel like most of the generations are like, you know, 5 % different from each other. Because you use Ruby. No, no, no. I had a JavaScript one. I have a JavaScript one. So I should be good at that. I don't know if it's like just how the RL encoding works. One thing that I've noticed, these models always want to do if statements when there's like a missing M variable so that it doesn't fail when it runs.

52:20And I feel like that to me, that's just like a symptom of the URL. Yeah. The code is terrible. Like, you should not write code. It shouldn't silently fail if there's missing variable. It should just raise an error. Yeah. But I feel like the URL is like pushing the code in this direction. And then all the generation have the same pattern. You know, I generate four things. All of them use the if statement just in different pieces. Yeah, that is something I will definitely get over. That's just like the labs are trading off massive gains in performance or small detriments in usability. And it's like, do you ship that model?

52:53Yeah, you just ship it and deal with it later. But I'm sure that's a fixable thing. I think like to me, that's the question. It's like, you know, you talk about how you have gains in like pieces of the thing, but not in the full trajectory sometimes. Do you feel like these are examples of that? Or do you feel like as we get better, if we did a longer trajectory where instead of just writing this piece of code, you have to think about how you're going to maintain it later and like how it's going to run, that's going to fix it? Or it's hard for me to grasp. Yeah, the software stuff is not easy because it's almost like maintainability almost feels like a human preference type issue again, where somebody could look at it and be like, yeah, that's not as good.

53:35But adding the heuristic and trading seems very messy. Yeah. So maybe it is. I don't know. There's a lot more to dig into that, I mean, this is what Anthropic says they're doing, and just what are the actual frontiers in making, like they said, they're working on code only, and what does that actually mean? A bunch of it is going to be design trade-offs and how much autonomy the model has versus these potential side effects from training longer that we don't know how to get rid of. I mean, that definitely could be the sort of, a behavior like that is what I would say is like a simple thing to remove where it might just be obsessed with some code format that fails when you revisit it or something.

54:18Even if it's like everyone has seen it with just bypassing test cases. I think there'll be a bit more nuanced than that, but they could probably be super simple. This topic has a similar semantic content address for me as over-optimization, which is something that you've written about. It is over-optimization with a different reward function. I know. Okay. Well, I made that link. I want to verify that we are picking on the same wavelength. I just wanted to go over, again, specific topics on things that you've spent some time thinking about. You write that there are three types of over-optimization.

54:51First was RL for control. Second was RLHF. And third is RLVR. They always happen. Obviously, RL is no stranger to reward hacking. but like maybe like do you want to elaborate on how things are evolving in terms of how we're learning as an industry yeah so that three things breakdown is for people to put the pieces together for what has happened historically all of these over optimizations are a just the model optimizer is strong enough where it can manipulate the agent with respect to the environment or manipulate the environment in a way that's useful to its target signal. Also, for context, I think with what we're doing with language models in RL in general is that if there's something that can move its reward signal up, it'll move the easiest thing, the most direct things to move that signal up.

55:41So that's part of the story that I said on sycophancy, which is this reward model for user feedback was probably so obvious that humans just like to like stuff that is like people press that thumbs up. Long emoji filled bullet points. Yeah. Like all those things have just been really easy for the model to extract. So like once they added it, the model changed a lot and the score went up a lot. And it was easy for the RL to find that. In control, the oldest RL, the environment is normally a simulator that is fixed. There's no feedback. So the over-optimization looks like unphysical and nonsensical behaviors.

56:14There's the motorboat example going in circles. There's like an example is a project I was middle author on was like effectively over optimizing like half cheetah, which is this Majoko thing. Instead of running, it did car wheels off into the sunset and got like infinite numbers. It's like obviously not the intended purpose. It looks like a glitch. So it's just kind of manipulating the agent interface with the environment. RLHF is kind of a classic case where the model will just break down because the reward model is imperfect. So the environment is really imperfect in the RLHF case. It's so sparse.

56:48It's very artificial. Yeah, it's a very artificial environment. So it makes sense that these actions, which are generated tokens, will do things like reduce into just repeating one token over again. I think one of the early examples we had playing with this at Hugging Face was the model would just say JavaScript. It would be JavaScript, JavaScript, JavaScript. It was like some toy data set. And it's very obvious when you see it. It's probably harder to see when you're at the top and making decisions on when to stop training if you're doing a lot of RLHF. But that was kind of the phase that people have gone through.

57:18And now we're in the RLVR phase, which is we're giving the model reward when it does something quote unquote right. For math, it's a bit harder to over optimize, I think, unless you have tools and the model learns to search and cheat instead of learning math, which I'm sure somebody could see that out in the world, which is like, oh, I'll just find the, you're training. It's like the model's like, oh, you're training me on Stanford's problem set for CS, whatever, that it's seen a thousand times. So it's like, I'll just go get the solution manual, which I'm sure somebody can find an example where that has truly happened.

57:50But on code and maybe information retrieval, it's easier to fudge. So the code thing is like the easiest way to get a unit test to pass is just put a pass in it. That is not too surprising that a model can learn how to do that. and there for code you need more reward design which I think would be a nice for like a substantial academic work is like what is reward design in code for balancing this sort of like understanding this over optimization of test cases or avoiding failures or something like this. I'm sure there's it's not necessarily going to be a controlled environment because these models are complicated but I would guess you can reproduce that in some ways.

58:30Just to double click reward design means like for example giving credit partial credit for partially correct work. Yes, or like giving the model a slight penalty for doing the unit test thing, if you can detect it. Yeah, for cheating. Yeah. Which is, it adds a lot of complexity to training these models compared to math, which is just if the answer is right. I mean, you can look at the GRPO math and partial credit is weird in that because it's kind of normalized per batch. I don't know if I have a whole spiel ready on it for that, But it's also just, it becomes very complicated if you're mixing domains.

59:05And it's like, is partial credit in code better than partial credit in math or all of these things? It's like reward design becomes very complicated. And that's what you're incentivizing the models to do different things. Yeah. Is there any literature or hypotheses about mixing these things? So let's say you have the one for code, you have the one for math, you have whatever other verifiers you can come up with. And individually they work? Do they conflict? I think part of the intuition of RLVR is that the model is good at knowing which prompt area it is, which is why the models don't get worse on knowledge benchmarks if you're training on just math or precise instruction following.

59:45So the model just kind of develops an intuition for where the different prompts are in space. So the gradient updates will be different depending on your batches, which is partially why people will just say do big batches. So a lot of the model is activated and you have a less noisy signal with RL, but a lot of the intuition is that the model just kind of handles that. And there's interesting questions on sequencing. Like, do you do large-scale math and code RL to get the sequence length and then add in more general stuff? Yeah. Which DeepSeq mentioned, but that's one thing to go, the DeepSeq report is like math and code to more general RL.

1:00:21There's a question on where do you do tools if you're going to do like code execution and search within this. So I don't know if that's interweaved or if it's a second stage. Got it. Yeah. I don't have comments there. It's just like, it's surprising how much is not known. And you just need a lot of compute for ablations. The inference, high inference length generations definitely just kind of breaks all infrastructure. Because there's just so many tokens. There's more opportunity for out of memory or other things to go wrong. So it's like, just on a default, all of your training jobs need way more GPUs for the memory of inference.

1:00:55Sure. Or just like training. But it just makes it more of a pain. Yeah, that's a cost thing. You know, one of the maybe controversial takeaways from the GNOME prod, which you listen to, was that there's also just wall clock time of just getting feedback from the environment, whatever that is, especially if it's like a real world thing. And I'm just like, yeah, I mean, there's some point at which your trading runs cannot take longer than like a human life. Like, so to me, that was the wall. He disagreed with that. But like, that was what I meant by it. like at some point, long inference, you do want it to terminate within some reasonable amount of time, regardless, just as a user.

1:01:37Yeah. We have to find a way to accelerate internally within the training time faster than the passage of time in the actual universe. Yeah, I'm not worried about that problem, but I agree with you in principle. Right. So I'm stretching this out too far. I get it. As we kind of start wrapping up, what are other interesting ideas that people should pursue? Like in your AIE talk, you said, what I'm thinking about for scaling RL, you had big multi-domain datasets, difficulty filtering, long run times. Is there anything specific that if there's people out there that are either doing research or they want to do a company or whatever, these are like interesting things that you don't want to do that you want other people to explore?

1:02:19Most of them, I think, are not in the reasoning space, which like if their talks have been about reasoning. So I've been long talking about character training is something that I think is under-indexed on and been advising a student. Character level? Like personality training and how that, like different ways of changing the personality of the model from prompting activation or fine-tuning. Okay. Like data engineering. So stuff that Joanne Zhang does for OpenAI. So like how much does that matter? What are the fundamental research things? Hopefully I can share more than advising a student on that.

1:02:55So I've been saying that for a while. Just as a side note, do you like the model spec stuff that she's doing? Yeah. Okay, that trajectory. Yeah, so I've been an early fan of that. I mean, that's how she noticed me. It was like the only person that covered it when they first released it. I think it was like over a year ago. I liked it. Yeah, well, not many people did. Okay, all right, all right. You were first. I don't know. I don't know. But that's what she said to me. Well, we had a model spec talk, closed the whole conference, right? That was my sign of like, pay attention to this guy. But it's real because of what it sends to develop.

1:03:27It has the developer benefit of where your model's going. And then also just regulatory. I think it is very important to what is an intentional behavior versus just a training error. So I think for model transparency, it's really fantastic. And I've said that the model spec is much more useful than a constitution. Because a constitution is like an intermediate training artifact that you give to the training algorithm in order to get the model that you want. It is not necessarily like what model did we... We don't write down our goals of the model in a constitution form. By the way, have you looked at the constitution?

1:03:57Not recently. They talked about it. They put in Apple's design guidelines, but then also the UN declaration. So at this level, I've seen it. I don't know if they've updated it. That's very odd. I hope that Anthropik would write a model spec. I'm not too optimistic, but they're the next domino to fall. Well, so my take on that, actually, I pushed for this too late because it's opening. I already approved the talk and all that. But I was going to ask them to compare the OpenAI model spec to the Cloud4 system prompt, which is their closest thing to the model spec. The system prompt is incomplete because OpenAI has things in the model spec that their model doesn't currently do, or especially when they started.

1:04:34When they first released it, it was like, we want the model to be able to engage on sensitive subjects. And maybe even NSFW is in their model spec, which is they're just signaling of what they want it to do. and they say like, this is very hard to implement because there's all these obvious risks of doing this, but it's like in an ideal model where we can solve every problem, this is what we do, which I think is good, as I said, for many different stakeholders. So mostly my thing is like, there hasn't been a good like foundational research paper on that, but there's a lot to do. It also runs into personalization and personality are similar, which is like if open models are to win, part of it could be just like, everybody can have exactly the model they want.

1:05:14We're serving GPT 4.5. it's kind of its thing, you can prompt it. But if fine-tuning is more effective than prompting, everybody can have the model that they want. So it's a good, it's like an academic problem or an open ecosystem problem where people are fighting on the turf that it feels more likely to win, which is good. Is this somewhere where you, like as speaking as AI2OMO, you want to win? Or is this you're just advising a grad student on it? I don't think it's a differentiating factor yet, but I'm very open to working on it. I think open models have a strong role-play use case and character, personalization, all that stuff, right?

1:05:51Especially because people, they find their waifu, they want to keep their waifu. And that's the derogatory term for it. I would say that we've definitely discussed it. And I want to, part of Olmo should be that it is a base model that's easy to take in directions that you want. And we will have an opinion that is probably slightly conservative on personality. I mean, I've gone through the OpenA on model spec, and it's like most of these we agree with and be conservative on anthropomorphization. What do you disagree with? I don't remember. I did it a couple months ago. But a lot of it is like openness or transparency, which is like if we're training an open weight model personality, we're not going to withhold anything.

1:06:30And we have a different hierarchy. So most of them are on that type of information exchange rather than be kind. OpenAI's model stack is pretty agreeable if you read through it. And it's like, treat the user with respect. I'm raising kids that way. Just read the spec. Yeah. It sounds kind of stupid. But then the last thing is for people doing research, it's like wacky model routing things where you figure out a bunch of different models off hugging face to route to. Because an open model tool thing could use way more models more easily than any OpenAI product. because OpenAI is restricted to OpenAI's models where if like maybe, I don't know, OpenRouter's like, I'm going to make a product out of this, which is a router.

1:07:14Like OpenRouter actually does it. And they're like, our chat window knows the best model based on all this usage that we have for your query. There's people that started the other way, like Martian, not Diamonds. I don't know who else is. He would know. There's a bunch. There's a bunch. Yeah. So I don't know if that would work. HuggingPay should work on it. it's like it's a moonshot idea you don't know when given your Hugging Face background what is how does Hugging Face make money this is a very common meme question I think mostly like enterprise deals that's what they say which is like they're doing their thing they're supporting their people I mean they're great they're big they're profitable it's just not that obvious to most people I like the router idea for media models I feel like there's like so many there's like a long tail of like a background remover like a style applied higher like that is actually hard to find on the tech side i feel like just use the big model unless you're like under some like latency or price constraint you should just use the best model even when we're doing thumbnails i'm like okay i'm trying to remove a background of somebody and it's like i go on replicate and there's like 55 background remover yeah i just use adobe because it's a website well but that doesn't work like the photoshop model is bad on some things but again it's like or i want to generate a diagram to like mimic something and it's like well which model is better diagrams you know it's like those are not easy to find because none of the benchmarks part of the argument is that if distillation works really well we could just keep making the target for distillation smaller and smaller which is you have models that are very narrow right and they're mimicking these huge models on something that's like pretty i don't know like reformatting tables it's like can you do a table reformatter from markdown to latech in a hundred million parameter model like like if you get it small enough that is really economically feasible because it's effectively free at inference and instantaneous.

1:09:03My pushback on this is just if you're doing image editing, 4.0 should do all of it. Well, yeah, but I think it does. We're just not there yet. Give it five years, it'll do it. So why work on a router at all? You just scale up 4.0. I guess I think it's, yeah. Tell me where the logic is here. This is like a temporary thing. On device. On device. The local modeling community I think is much smaller than people give it credit for because most of the use for open models is still in APIs. It's like DeepSeek API. It's convenient. And it's like, if there aren't that many models, somebody is going to host it for cheaper than most people doing it themselves.

1:09:44That's pretty realistic. But there is a small community that needs local. The best outcome is if open models can compete on not just long tail things. But that takes the most transformation. Side note, so I resisted buying my own GPUs, building my own cluster. For this reason, I'm like, APIs will solve most of it. People are losing money to serve me models. Why am I having those? Except for the fact that 4090 prices have doubled in the last year. So actually you made money doing local models. How does that make you money? Because your investment goes up? Yeah, you can sell the card. So as you use 4090s goes up.

1:10:25Interesting. Should have bought a 4090. I got a 4070. Damn. What is this? Well, then it puts me on tail. Should I buy, you know, 1590 if it ever, you know, is widely available. Well, at GTC, they were doing the drops. Yeah, I know. It was crazy. We were like running to the camper to buy it. Any other topics before I give a closing question? Just generally your work, ROVR, like are topics of the day. I think companies should keep considering releasing open models, mostly for PR and onboarding. It seems like the way it's going if OpenAI is releasing it. Are you excited about that? Do you feel like it's like a psyopsis?

1:11:05The OpenAI model will be good. I expect it to be. No, they're pretty serious. It's not a psyopsis. It'll be best in class for some size category and some subset of tasks. That's like OpenAI only does things like that. You have to give them the respect they deserve. Yeah. That is a big, like open wins when more people are doing it. So that's a win. Yeah. Well, I mean, hopefully they are actually open about the techniques and not just the weights. Do we think the size of the open model tells us anything about the hardware that they're going to build? What? No. They're so secretive about this. That's why they haven't released GPT 3.5 or anything, because it's too revealing about internal stuff or plans.

1:11:45Oh, okay. No, so you're talking about Stargate or what kind of hardware? No, the Johnny Ive thing. No, I think that's a different fun factor. Yeah, yeah, yeah. I think that thing will run on the cloud. I don't think that'll run local anyways. Well, okay. We have to talk about it. Like it seems like every podcast we talk about it. So apparently the news from today, which I think you were looking at, was that it was like a ear device that they got sued over or whatever. But like, I think the ear form factor is pretty good. Like I actually did get there with B in terms of like, where does this ultimately go?

1:12:16Like you want something, you want the AI to hear what you hear. And where do you hear what you hear on the ear? Like that's pretty much it. I don't know if you guys have thoughts on wearables and where that goes. I try to be. I think it just knows too much. That's really my thing. But you want to give it context. Yeah, I have false privacy hopes. I think a lot of people, I mean, that's the whole thing. It's like people don't actually care about privacy. It's just note-taking, you know? It's just really good memory. I think the meta Ray-Ban form factor is good. I don't think it's as mass market.

1:12:49It's like if you get it in an AirPod-sized form factor, it's a way bigger market for obvious reasons. but the like sunglasses form factor is the thing that works, I think. Okay. I don't use them for AI, but they can fit the AI to work it. Like, yeah, empirically, yeah, it obviously works. Yeah. Cool. Well, the last question I was saving up was this whole, what is Meta doing? You know, you actually had a pretty interesting post back in, when was this? In April, you said, Lama 4, did Meta just push the panic button? I feel like back then it didn't actually push the panic button, but now they really push the panic button.

1:13:25That's fair. I think the panic button at the time was the whole LMSys model not being the model that they released thing along with a bunch of weirdities about the day of the week they released. But to be a model that claims to be open and then not release the model that is your leading claim is just like, that is like bad execution. Bad execution. Yeah, yeah, yeah. Which is fine. And then the recent stuff I think mostly can be boiled down to talent is cheaper than GPUs by a dramatic margin. And at the end of the day, it's like, okay, if we're spending this much, they go to the room and they stare in the mirror and they're like, wait, it might not actually be that ridiculous to spend this money on the top people.

1:14:04It's like, might as well try. They already spent it on VR. Somebody was bound to do this eventually. And it makes sense that it's like, if Apple in some way somehow decide like, we're going to do this, they're going to come in and do exactly what Meadow is doing. They need a founder mode CEO who's like, screw it, we'll take the L. The thought that occurred to me is, Meta, instead of spending on VR, they should spend on RLVR. Well, I think the question is, I think some researchers, most people will take the payday and happily move to Meta. Everybody has a bribe number. Right. It's just a number really big.

1:14:41Yeah. But I think some researchers are uncomfortable with the idea that this is sort of the great man theory of research that like you have to pay this much to get this level of talents and the talent is definitely distributed right a lot of the people that they would be paying this much have the confidence to redo things or to just do some of the same things and just like whether you call it feeling the agi or just drive to build things or like feeling the agi is not that different than a lot of things that have existed in silicon valley lore in the past, which is just people with the vision that are willing to execute on it and they see something coming.

1:15:19And those people make a big difference. I think you have those people and you remove bureaucracy. Getting technical, talented researchers is actually something that Meta has a lot of or has the ability to get a lot of. So it's a lot of recycling, which is very hard on individuals and morale of an organization. but that's like understand the approach yeah for sure um cool that's all um any parting thoughts on how you're gonna build the american deep seek that was a nice tweet yeah mostly if i have to look at like what my in the if you're asking me like what my 10-year goal is and it's like i only will have like a two to five year goal where i think as models are shifting more towards agents i think that like scaling is slowing it's like there's side of it of a fixed cost and a fixed path to getting towards something like American DeepSeq, or mostly just I would say it doesn't have to be American if it's fully open.

1:16:18You have everything and you commodify it, which is like, there's a few things that need to fall. A lot of it is just more resources, but it's like, like Olmo 32B is if you squint like original GPT-4 level and fully open. And it's like, there's a few levels that you need to go through. Like that's obviously a dense model. It needs to be taken to sparse MOE and you need to scale it. You need to have a lot more GPUs and then you need to do like large scale reasoning. It's like, that's the goal that I want to do. There's a lot, like that's what I want to do. There's a lot of complexity and navigating like how to work with AI.

1:16:56What does AI2 do to get there? It's very hard. I think that, I mean, it's a nonprofit. It's hard to get the resources and building a model is a lot of aligning a lot of different people. That's the DeepSeek story is they have great people. OpenAI has kept a lot of really good people for a long time. Anthropic has gotten a lot of good people right now. And it's like, it's a lot of incremental, hard technical problems that you need to stack up. That's what I would like to do and make work in the next couple of years, but it's not easy to get there. So that's the pitch is like AI2's best case scenario is AI2 is going to do other things.

1:17:31Like you can't just run a nonprofit or a company that says our goal is in three years to have an American deep seek. Like no one's going to keep paying the bills on that because you have to tell a better story. But that's like what I would like to do in that. And I'm sure AI2 will do many more interesting things along the way. Like product stuff. I don't think it's necessarily product, but like what are more like what are cutting edge things in AI that we can make a new architecture for certain things? Okay. Or like what are demos of open models working better, whether you have like private data or something?

1:18:01or just far out ideas that could take you off the transformer trajectory. I think that you still need to be doing these to kind of lead in AI. Thank you for working so hard on truly open source AI. Yeah, it's fun. I mean, it makes it easy to align values with what you're doing. It would be better for the world if more things were open and therefore a lot of it is just willing it into existence. and I think seeing what OpenAI does or is saying they're going to do as hopefully a win coming soon. DeepSeek was the most unexpected win that made some other dominoes fall. I think that is the path forward and see what it takes.

1:18:44Thank you so much. Thanks for coming on.

1:18:57you