Open Source AI Strikes Back — Inside Ai2’s OLMo 3 ‘Thinking"

26:45So a bunch of us got together, started planning, got in touch with a few companies who might be interested in supporting these initiatives. We got an initial grant from AMD at the time. There was about two million GPU hours. And so the idea that the researchers are interested, we had the compute. So we went to leadership at that time and sort of told them, hey, we're going to go do this thing. I hope you're OK with it. And one of the nice things about it, too, is like at heart, we are like a research lab. So everyone was like, sure, you figure everything out. Just have fun. Great. All right, Nathan, how about you?

27:32So you're a man of many talents. You do research. You write this very interesting blog slash newsletter called Interconnects. You do podcasts, you do a bunch of different things. So tell us about your journey. Yeah, I say I wear many hats to try to get the things I want to do done. I showed up to Berkeley as an EE, mostly PhD, admit in 2017. And then I saw that AI was happening and I decided that I want to try to do this, which started by going to all the names that people know, like Sergey Levin and Peter Abil, and asking to be in their group, and then they respectfully say no. And then starts the long process of learning how to actually do it without being directly embedded in these elite groups, which was a mix of robotics and reinforcement learning and finding my way there.

28:25So my PhD was in mostly model-based reinforcement learning, and then my one research avenue job was to go join Hugging Face when they said they were going to make an open source version of DeepMind and do a bunch of research. Realistically, my job was not that impactful or useful at Hugging Face until ChatGPT came out. And then I was like, oh, I should maybe just learn about RLHF. And that got very immediate traction as somebody trying to work in public with the team there. So like Louis Tunstall and other people at Hugging Face are still doing a great job on this. And we worked together for a while.

28:58And then mostly I was just like getting burnt out on remote work and met Luca in Hawaii at a fun conference. It was like, wow, I could have real life friends. And I joined the AI2 to work in person and tried to do the same thing, which kind of takes an evolution of the Olmo story, which is just like, I had a lot of motivated on trying to figure out these, what was mostly reinforced learning from human feedback at the time and make versions of these post-training techniques public. And then that kind of evolved through both Olmo and we have our post-training methods that was named Tulu, which is like we spent a long time to try to replicate what we thought was close to LAMA 3 post-training with multiple stages and optimizers, which is the project that came up with the name Reinforce Learning with Verifiable Rewards with a bunch of people.

29:44So it's kind of this evolving journey at AI2 in search for impact, which is what we think people are actually doing. And then largely the opportunity that Luca and I and others at AI2 fill is that there's so much money in AI and it only becomes increasingly so that the amount of people that can talk about these things in public and educate and get more people involved by spreading knowledge is ever smaller. So I describe my career journey as a lot of it is filling that vacuum and thinking about what's impactful there. So it kind of pulls you. When there's such a void, it has a sort of gravity to make it clear what you should be doing.

30:21You anticipated my question, which is sort of obvious, in a world where we see hundreds of millions of billion dollar packages offered by some commercial AI labs for people just like you. I was curious about your interesting motivation to join AI2, which is a nonprofit, but impact is the short answer, correct? Yeah, I mean, I've been here for two years and I wasn't famous when I joined. So let that be told to people looking for new jobs is that you want to find a job that you can grow into. And I think AI2 has been a really, really good place for that for many people. Because you have independence and are encouraged to go forth and do things and not be a cog in a broader, just like grind out language model machine, which is important.

31:08But it's harder to get visibility. So we alluded to some of it, but maybe a few words about AI2. So AI2 was started by Paul Allen. AI2 stands for Allen Institute for Artificial Intelligence. You mentioned some grants, Luca. And I think earlier in the conversation, we talked about a recent grant as well. I saw that it was$152 million from NSF and NVIDIA. So what is AI2? How did it start? Has it founded at a high level? AI2 was founded around 2014 by the league, Paul Allen. When initial AI2 was very focused on building machines that can do science, can understand science, solve science problems.

31:56That's when Semantic Scholar started as a repository of science paper. Slowly, one of the initiatives that started forming was more fundamental research around how language model works, how at the time it was called natural language processing was working. you had teams like LNLP doing great work since the very early since very early we worked on we always had this idea of like not just releasing artifacts or research but releasing the tool back in the day we had this very very widely used library called LNLP that would allow you to build and customize these models I'm going to jump in, it's cool because it's the namesake of our team name and has been for a long time at AI2.

32:46It was the main competitor to Hugging Face Transformers and they ultimately out-competed AI2 as the thing that people used for that because they had a very different model and amount of support. Luca can keep going. Luca is a little more. But we have been at it open sourcing for a while. I think it's something that folks here understood really early, this is before my time, that was important. both in like pushing science and also like unlocking, you know, commercial use cases that such a non-profit. Maybe we didn't anticipate, but folks, you know, you release a tool, people pick it up and do amazing things with it.

33:27Yeah. And, you know, we moved on language modeling more and more recently. Right now, AI2 has maybe like three main projects. One of them is Olmo model family. And there's variants of Olmo. There is some that I focus on, like the full pipeline, some that robotics, some that I focus more on processing images and video and audio. Is Olmo part of one of those variants? Yeah, you have Olmo as one of our projects that work on multimodal inputs. recently we released another one called Momo Act it's more focused towards robotics receives multi-model input and then can act in space and then we had the model that was able to do automatic speech recognition, another model focused more on document processing, could do OCR, so it's a nifty little family of models, we have a working group on agents for scientific tasks, arcing back to our roots.

34:37This is the ASTA family of initiatives. This is agents to help scientists do their work. And that just came out, right, like August of this year? Yep. The team has been cooking since middle of last year. But finally, we had our first release this year. There's actually two releases. There was the main ASTA release. And then recently we announced a partnership with Kaya, the Cancer AI Alliance, using some of the components in ASTA to help researchers make progress on cancer research. And then there is a third branch on AI for the environment, building models that can understand, so they can model Earth and can work with different signals to do prediction around the environment.

35:28and so on. I'm being a little bit vague on this one because I don't know if it has been announced yet. It's a preview right here. The Mad Podcast is making news. Okay, very cool. Awesome. All right, that's great background. So we got Olmo, we got Tulu, we got Asta. Just maybe one last question, like in terms of size, like what are we talking about? Like how many of you guys are there? 200 people between, you know, the research staff, engineering, cons, and other support roles. That's fantastic background. Thank you very much. All right. As previewed, let's switch tacks and go into Olmo 3 slash Olmo thinking slash Olmo reasoning, whatever you guys end up calling it.

36:13And I think it's a perfect opportunity to talk about how those rezoning models actually work. You know, in prior episodes of this podcast, we've had great conversation with folks like Anthropic or OpenAI. But not surprisingly, there's only so much they can talk about. and the beauty of what you guys do, which is the very essence of it all, is to make it open and accessible to everyone. So I'd love for us to talk about the whole pipeline from pre-training to post-training, the different parts, and make that super educational and explain in plain English what part does what. So could either of you start with just a high level of architecture of what the various subcategories of the pipeline are, and then we'll go into those one by one.

37:10Sure. I recently gave a talk on this at the Conference on Language Models, so I have them on the top of my head. I think I can provide a personal motivation for this, which I think as researchers, we are closely embedded in the community, and we see that there are a lot of people that are starting to do this reinforcement learning research after DeepSeek R1. I think most of this happens on the family of QN models, which is like QN 2.5 and QN 3 is between like 1 and 8B parameters. And I think something that like particularly motivated a lot of the fine grain details that we might not have time for in this podcast is that there's some questions hanging over the data used for QN when doing this RL research.

37:49Specifically, there's two papers. One is Spurious Rewards, Rethinking Training Signals in RLVR, which is one that I was on with a lot of people at UW in AI2. And then another one, which was Reasoning or Memorization, Unreliable Results of Reinforced learning due to data contamination. Yeah, actually, let's spend a couple of minutes on that. What does spurious rewards mean? I think the thing to know about this is that it's going to become you can trigger this rant on the technical side later. It's a lot of background on understanding what these algorithms are. But essentially, the question mark is like, did Quen include training data that is too close to the evaluation targets so that the research is picking up on weird behaviors within the model rather than the fundamentals on what this reinforcement learning is doing.

38:36So in other words, did they teach to the test versus enabling true thinking? Yeah, so I don't think Quen, like Quen didn't, it's a gray zone. Like, I mean, it's not, I think all the Frontier Labs will do this to some extent, which is how they're tasked is, you have a team member that's tasked with improving an evaluation. And then the easiest way to do this is to train on test. But they all have dignity as like, elite scientists where they won't do this. And the next closest thing is you do some sort of paraphrasing of the test set to create true data, new training data. So therefore you're not technically cheating, but you're potentially like, it's like where in the spectrum of you scrape GitHub for math problems versus you paraphrase the evaluation set, like where do you draw the line on like actually calling it cheating?

39:21And different people have different answers, But mostly, I think a goal that we kept coming back to, because we understand that ULMO is not like, you can look at the numbers, we're getting close to QN3 with reasoning or without, but this is not a 600 billion parameter model that people are going to immediately download and run ULMO code on or anything. But we want to make sure that our core audience could do the research that we want to do with confidence and debate it. So we want to give people access to every stage and you can then see how this impacts this new important area of research. So we're going to talk about six stages.

39:57One is like large scale pre-training, which is this training on all the Internet, predicting next tokens. Two is what we call mid-training, which is debatable whether or not it actually should exist. what technically it is, is you train on higher quality web data and with a change in the learning rate. Three is long context extension, which is absolutely essential for these reasoning models because they generate so many intermediate tokens before sharing an answer with you. And Luca has a lot of battle stories from that. And then we go into post-training, which in our case, like those three building blocks of pre-training are, I would say, more set and super essential.

40:35And then post-training, when you approach this, you have a bag of tools, which are optimizers, and you apply them in the order that suits your model, depending on size, capabilities you want. So we'll talk about things that we did, which is instruction tuning, preference tuning, and then we did some reinforcement learning with verifiable words again. But if we were to train a model that was 10 times as big, all this post-training stuff would change. But the pre-training and mid-training and long context, I think, would actually become looking pretty similar. So it's kind of a difference across the two phases of training where post-training is like a bit of an art and you have to do what is best for your specific use case and that'll change, but we'll kind of go, we can go through these too.

41:18Okay, great. And Luca, you're the pre-training guy and Nathan, you're the post-training guy, right? Is that fair? One of many, but yeah. One of many, but for purposes of this conversation. And before we dive in into each step, So this idea of pre-training plus RL seems to be the key idea in terms of progress in the last year or so. And I know the concept of it came up much before that, but in terms of implementation of it, what's the right way to think about it for somebody that's trying to learn about the space? Is one part better than the other? Or do they need to exist together? is is is is are currently delivering more game than pre-training what what's the overall kind of high level take i think the the way i like to think of it is um the pre-training phase um it's really like um a very expensive expensive initialization of the model um right you wanna like when i think of like oh what do we wanna um what is a good uh final set of weights that i can pass to Nathan and the rest of the post-training team is, well, I want a model that has great knowledge about the world.

42:38And it also can sort of, you can start seeing sparks of capabilities that you will want to model that then, you know, you want to chat with, have great capabilities. So it is a very expensive and very compute intensive way to like create an initial models out of like what is essentially like random parameters. But it's all about like, yeah, let's let's have this model have like a lot of knowledge of world facts and information. and let's have it so that it can start behaving a little bit like a chat model so that when we pass it to post-training and you have this reinforcement learning, there is some behavior to reinforce and to give rewards on so the model can pick it up.

43:32I would say that generally the reason why discussions are hard right now on whether or not people should care about pre-training or post-training is that we optimize pre-training for multiple years. and then there's a lot of kind of untapped potential on this type of RL where like a couple like what is said is that OpenAI figured out a whole bunch of tricks to get O1 to work and then it kind of showed that this area was possible and then this year has been a race to capture low hanging fruit on RL where I think that's kind of the biggest story is why we have all these crazy new models that appear like O3 with this thinking and tool use which are just downstream of like oh we could do very different things because we've done such a good platform as these malleable pre-trained models that we've been iterating on for a long time that this RL stuff, we just kind of could have tapped into it much earlier, but there's a lot of potential.

44:20So yes, the rate of improvement right now in RL is higher, but at the end of the day, it's going to be a dance between both of them where you need a better base model. It's said very commonly that a better base model and a bigger base model is much easier to improve with RL. So like if you take that as one of the core things of doing RL research, it's pretty obvious that pre-training is very important to enabling that. As a quick detour, there's been that podcast with Richard Sutton that was effectively saying that like RL was the way to go and that pre-training and LLMs was a little bit of a flawed premise because it was sort of an limitation of reality basically doing the way humans described reality as opposed to being confronted with the actual reality through RL.

45:08Do you guys have any quick take on that? My take is that a lot of people are being exposed to Rich Sutton for the first time, and Rich is a font of wonderful ideas, but often not ones that are going to be immediately practical. This is how you get things like creating reinforcement learning, but not necessarily things that are going to impact what GPT-6 is. So I've been on a critiquing rich life for many years before this in terms of making people try to interpret his ideas as realistic. I think the one from 2021 or 2022 is his reward is enough paper, which essentially is an argument that a reward function is sufficient to get any intelligent agent that you want.

45:47So I think that that's actually like rather than the technical debate as an entertainment of the whole community being nerd sniped for the first time by that is a distraction. the message is not that surprising like there is this fine line between the actual ideas and and there is the engineering around it a lot of making language of other works is engineering and like not in a denigratory way but in a way it's like we got to figure out like how to translate research at the end to practical things and so pre-training is just a good way to initialize one of these models. If a better idea comes out in the future, we can switch to that.

46:31No one is married to LM being the end-all solution. There's a big difference between just describing the system in theory and then actually getting them to work. Otherwise, there wouldn't be two and a half years between the original GPT-3 and GPT-3.5. Thanks for that. So let's take those six modules turn by turn. So let's talk about pre-training. What did you guys do specifically for this model? Pre-training is very interesting. The way we sort of plan... So a good background is to have is that pre-training, all that happens to be pre-training, we have to be very methodical in how we do it. Because first of all, it takes a long time to pre-train.

47:21I think it's standard practice among the frontier labs to try to cap your big final pre-training run to two months, not more than that. But to get to something that will not crush and burn in two months, during these two months, you have to do a lot of preparation around this. So we're really everyone who works in pre-training is fairly methodical. And just to sketch out how that works, it's usually you have a sense of, OK, the duration of this running is fixed. The number of GPUs I have available is will be fixed. And therefore, you know, you write the fastest possible code to train this model.

48:05You have this three. You can figure out, OK, how much data can I show my model? in our case the number was like six trillion tokens given that number then we go back and we figure out okay where what are the best six trillion tokens out there and the way you figure out is a combination of like what data you have access to you know we want to do this with we want to eventually release the data so we limit ourselves to data that is publicly available. So either internet text or PDF documents that you can find on the internet or code that you can find on the internet. And then among this pool, our initial pool was closer to 300 trillion tokens.

48:51You shrink it down till you reach your target number. And hopefully as you shrink, you only keep the best part of this. So you remove duplicates. You have a way to judge, is this document better than this other document? You know, we have a way to evaluate the capability of the model. So you pick, you know, if your evaluations want, I don't know, medical documents, because there's a medical test there. You figure out how do you pick documents that have good medical information. It may be to the expense of some other domains. But yes, it's a delicate balancing act to find this data. And after you commit to this initial run, you will do your training of this run.

49:40Or there's a lot of making sure that the way you design the model doesn't suddenly start forgetting what is learned. We call these spikes in the language model. But basically, you don't want this event that if they happen, you have to restart from scratch and you can't recover. So there's a lot of work on that. But after these months of training, you get to a final model. And then on this final model, it will still lack some capabilities that I know Nathan's team cares about. So these are things like long contacts or being able to solve some problem to start with. That's where things like long contact extension or mid-training happen.

50:26Let's get into that. So that phase two, so mid-training. So again, a term I personally hadn't heard of before. And Nathan, briefly describe what it is, but just double click on that. I've heard that some labs, instead of mid-training, they call it tail patching, which I think is a much better term. And, you know, the term is like the tail of training, a tail of pre-training. You patch the model so that the things that hasn't learned in pre-training, you will learn after. You learn at that phase. And of course, once you do, when you do that, you also need to make sure that the model doesn't forget stuff that's in pre-training.

51:07So that's why you mix some of the best data from pre-training, you do carry over. So you give it more like code data, for example, or math data, that kind of stuff? If the model maybe cannot reason about certain math problems, you do it. That's like when Nathan mentioned early, sometimes there is some leakage of things that look like the test during this phase. you know there is an uncharitable way to describe it just like oh they someone is trying to cheat there by adding this data but it's also like it's so easy for like accidentally leaking your test data in there we spend a lot of time making sure that doesn't happen because you want to add it's really tricky balance because you want the model to start being able to solve problems like the ones that you see during tests but you really don't want that test data to like accidentally leak there, otherwise you can't measure how well your model does.

52:06And then you mentioned long context, which is the third stage in the sixth stage pipeline. So why are the focus on long context? And I guess, what does long context mean in the first place? You want these models to be able to work with very long sequence of text, both as input. Imagine you want to give it, I don't know, a collection of documents. And you also want this model to be able to generate a lot of text in the output. especially now that you have this reasoning traces, right? This is thinking tokens. Why don't we train from the beginning the model to be able to do that? It's because the longer the input that a model is trained on, the slower it is.

52:47The rate at which it gets slower, it's higher than the length of a context. It's a quadratic slowdown. So we definitely don't want to do the entire pre training at this extremely long sequence. But at some point, we have to teach the model to actually work with these long sequences. And we save it for the very end so that we can do it in an efficient way. And I think you mentioned somewhere that data doesn't matter for long context. What do you mean by that? And then what does matter? Ah, this is getting a little bit in the weeds. No, Luca loves data. Luca likes to be in a dark room grinding out tokens to train the model.

53:27It's an emotional backdrop for those people. This is very technical stuff. Do I use QK norm? Do I use GQA? Doesn't really matter. But there are like technical decisions in how you set up your model that you can have the best data in the world and your model will not be able to reason over many, many tokens. So it doesn't matter in the sense that you can't train the model on bad data. You can have the best in the world. But if you set up your model wrong, you're never going to recover it. So sadly, I can't be the savior with the magic tokens to make the model good. We have to make the model with the right architecture.

54:14OK, so that's stage three, long context. Maybe just to bring this to life, what's the difference between before and after? Like if you have a 40 page PDF that you fit into the window, it will just get faster results or better results. What happens? At the beginning, you just can't do it. Like, you know, do you pre-train as something like four, eight thousand tokens? That's what we use for OMO3. That's what Lama is. that's about maybe eight pages if you use like you know double spacing new line kind of thing and after that we extend to about 65 in industry you have extension of a million token i think gemini recently announced like over a million token at that point a million token is like 10 books so you can work with extremely long amount of information it's nice you'll have to think about, you know, if you're building an application with this language model, you don't have to think about like, oh, of this amount of information, how the heck I'm going to extract the ones that I need to show the model.

55:22You can just give it all and the model will figure out. So it's really unlocks a lot of opportunities. All right. So that's the pre-training world between pre-training itself, mid-training and long context. So now let's switch to the post-training world, Nathan, if you will. So starting with SFT, which stands for supervised fine-tuning. Yeah, I think one of the things, especially for a model like Ulmo, where we're scrappy and putting everything together over time, is that one of the biggest changes is that when reasoning models become popular, the quote-unquote in vogue evaluation suite of the industry shifts to add a whole bunch more new things in.

56:04So one of the things that happens at every stage is, even if a lot of the data has overlap, is that you mix it in a different way. So I think Kyle Lowe and Mei Chen, that's another researcher and an intern, did this whole mixing procedure that we use across all these stages just to upweight the math code and reasoning stuff to make sure that what happens later in post-training is much more tractable and that all this stuff is set up. So that's the type of thing that we have to do that's kind of baked into everything. And then post-training, I think for this model, everything we're doing is operating in the assumption that this is about a 7B model.

56:41We're very narrowly focused, and therefore we're going to do what many people have done, which is called distilling from bigger teacher reasoning models. I think distillation is described as when you take the outputs for one model and then you fine-tune on it later. I think there have been a lot of broader discussions on this in the community. And then this supervised fine-tuning stage or SFT or instruction tuning is all about just getting the best traces from reasoning models out there or the community. And then just teaching your recently trained base model to behave really, really closely to what is going on there.

57:14So in our case, we took a mix of existing data sets like OpenThoughts 3 and modified it, which is from Bespoke AI Labs, a startup. And then we also generated a whole bunch of new data. So we ended up using a mix of teachers from like DeepSeq R1, DeepSeq R10528, which was their updated version. And then Quen's reasoning model, TWQ. It's like these tend to be pretty strong teachers. And then why is that? So you have a pre-trained model, but you for supervised fine tuning, you're still using a different model. Why is that in simple terms? Essentially because our small model is not going to be able to output as strong of text.

57:54So there's a kind of a fork in process where I'm talking about a small model. And if we had a bigger model, what we would do is do a lot of reinforcement learning to start. And the model then would take time to learn these interesting behaviors and have strong performance. But with a smaller model, the ceiling on that is fairly low. It just doesn't have the capacity to learn from these harder math problems. So what the common practice is, is you take the absolute best reasoning models you can get that are openly available with a good license, where you can just generate new data yourself and train on it and release it to the community, which is something we've been seeing a lot of this year.

58:29And then therefore the models that are closest to the frontier in performance with good license all happen to be Chinese models throughout the year for this case. And I think in our case, even if GPT-OSS had existed, I don't think we would have used it for synthetic data in this because that model is really designed for tool use, which is something that we did a bit of in this project, but not in the sense that that model is, which is like this many hop agentic reasoning with search and stuff. So the deep seeks and quen's of the world are just powerhouses at generating math and code answers and other things and being generally robust.

59:07So that's what we do is we have, I think, about 2.5 million reasoning traces, mostly on math and code and STEM, but also on chat and other general capabilities. And the model really absorbs a lot at this point. I think if you were to told us last year when working on ULMO 2, looking at it, that if we had a similarly sized ULMO model that gets like 95 on math and 70 on Amy on these crazy math evals, it would have been surprising. But this is just what you can get when you can extract data directly from these really powerful models and distill it down. And I think realistically, a lot of companies are going to want to do this because you can do this for your domain.

59:45We threw a blanket on, we want all of these evals from instruction following and make sure that you can actually talk to the model and not have it just become totally broken. But you can do this at any specific task you want if DeepSeek has coverage on it and it's very efficient. And then most of the process after that is, that is the foundation. If you're training a small reasoning model, you need to do this. And then the other things after are, how do you extract more performance? and they quickly become more technical or done because we want to do that, but maybe not efficient in our time. So 90 % of our time this summer is having great people battle reinforcement learning infrastructure.

1:00:26Because when Luka is hitting it, when you generate a lot of tokens, the time or compute increase and memory increase is quadratic. Therefore, you pretty much encounter every possible bug in your framework or every possible corner case that will make your job go to a halt. but like most of the performances through this SFT and the preference tuning that comes before it. But the RL is like, we need to do this in order to build the infrastructure for many of the future almost that we want to build later this year where they get bigger and they can do more interesting reasoning with tools and so on.

1:00:57So it's kind of like a nuanced point of the model. It's like, yeah, we'll show you that we got a couple of points out of doing RL at the end. But really the RL tooling is something that's so crucial to doing the next models that come from here. And thank you for that. And just to, again, in an effort to make this interesting to just a broad group of people who are curious to understand how AI works. So SFT is not RL yet, right? That's a supervised fine tuning. So that means that you basically show the model some like a golden, like a gold copy of what good looks like. And you train it based on that label data.

1:01:35Is that Yeah, so it's the same loss function as pre-training, which is you're predicting the next token. In this case, what it looks like is a question could be like, I don't know, like an Amy style, like a really hard math question. It would be like, list all the prime numbers within some constraint of X and K. And it's like this one sentence that is really hard. And then the model generates 30 ,000 tokens of let me think about this and do this. And to test this, I'll have to use this theorem and hypothesis, which is like the 30. We were talking about token intuitions for a bit, but 30 ,000 tokens to solve a math problem is pretty mind bending.

1:02:14So if I were to sit there and read this, it would be hours of me just trying to read this one math solution. So these models are very unintelligible in many ways. I think the reasoning model sometimes will go into a bout of guess and check for hundreds of attempts before realizing that they can no longer guess and check. I mean, this is our reasoning model. I think the frontier models could have probably done this and fixed this issue. But there's just really, really, really odd things in these tokens. But even with that, doing this next token prediction is an incredible foundation of performance that many people use.

1:02:53So it's not matching any sort of human reasoning or things that people might want it to be doing, but it is teaching the models kind of their own language of breaking down problems step by step in order to solve a goal. For this specific stage of SFT, do you want to talk about how you went about creating the data set for it? So precisely this representation of what good looks like for the model? Yeah, Luca, do you want to jump in? Do you have things to? The other thing I was going to mention is that sometimes in the big announcement of the frontier labs you don't see is what Nathan was describing around like having to do SFD to then do RL.

1:03:38It's very common. We're in a common situation where this is uncharted technology. Right. So you have nothing. You have to find ways to fix some components of your pipeline before you can build the rest of your pipeline and then go back fixing the first part. So for us is, OK, we want to do reinforcement learning on this larger model. OK, we need our reinforcement learning code to actually be super fast, super reliable and useful. If we need to iterate on that part, we want to iterate with smaller model first because we can iterate faster. They take less compute to work so we can do more things in parallel.

1:04:16OK, smaller models, they cannot do RL first. You got to create the data first. You have to go to the SFT. and then you know we're lucky enough that you know okay there are other great models that are open source that we can use to create this data so the alternative would be i don't know to spend it's not even the money to spend a lot of time instructing humans to create the same volume of data slow things down so it's a lot of this of like i don't you're building the tracks as the train is going down at incredible speeds and you have to figure out ways to like fix some parts of your pipeline so you can work on the rest.

1:04:53All right, so let's talk about the next stage in the pipeline, stage five, DPO and preference tuning. What does that do? Yeah, so this is one of the things that is thought of as like, hey, let's try this. We're not sure if it'll work kind of later in the process when you spend a lot of time on other things, and it works very well. I think DPO or direct preference optimization is not exactly new. I think it's a way of optimizing for preferences. It's related to this whole RLHF thing that we mentioned. Technically speaking, in one sentence, it's an analytically derived loss function that is essentially applying stochastic gradient descent to the RLHF objective.

1:05:36So it becomes much easier to implement than other things. And we used this in the past with OMO2, with 2Loot3, 2Loot2, other OMOs. and the question was like can we apply this out of the box on top of a reasoning model and we knew that it works in many different situations because we weren't sure what would happen with these long reasoning traces being included in the loss function and so on so then essentially like there's a student Scott that has been working on this what he calls the delta learning hypothesis which is like a intuition for understanding DPO as being more about the contrast between your chosen and rejected examples.

1:06:15So the core of preference learning is that you have kind of pairs or some grouping of completions to the same prompts. You have one question with multiple completions. And his intuition in work is showing that this contrast is more important than the exact magnitude of goodness of the answer. So what he did is he spent a lot of time in trying to come up with a good pairing of reasoning models, which is they're open source, or like they're open weights and they have a permissive license and they include the reasoning traces. because we kind of need this and you need them to be sufficiently well spread about.

1:06:46So we spent a bunch of time generating this data and doing some normal kind of like, let's fiddle with the learning rate in small things. And it's kind of just like, yes, this works. After we did it, we saw that Hugging Face did something similar with small LM. So they trained a fully open 3B model where they pre-trained it as well. And the funny thing is that we converged on using the same QEN32B and QN06B. So like the problem is that these small QN models and these small public reasoning models are actually so strong that getting a sufficient delta to another model to apply this preference learning technique was kind of hard.

1:07:25So like our past techniques, we kind of had groups of models we sampled from. But as these open models are getting better, these samples become too homogenous for the learning signal to exist. So it's a kind of cool experiment. It's a cool experiment because it validates this hypothesis of the changing tides where if you think about years ago with Alpaca and stuff, those models were so broken that having this group had enough variance and contrast in it where we could do a different type of preference learning. Where now it's really, you have to look really closely at the completions and make sure that there's a learning signal for the models.

1:08:00And we did this and it kind of gave us a boost across the board. I think it's like sometimes things look very easy when you've done careful data work and kind of set up to understanding your optimizers. I think Luca described pre-training as very scientific and post-training is like the wild west. There's many analogies. So it's like, we have me that made this SFT data set where I was like, we had a bunch of cloud credits and they were running out and were behind and I just generated as many completions as possible. So it was like a few billion completions from DeepSeq over the weekend. You're like, oh, we'll mix it and filter it later and I applied filtering and the answer was like, oh, we just include almost all of it.

1:08:41We would have liked to do more if we had resources for longer, but sometimes there's low-hanging fruit and doing the obvious thing yields a lot of results. It's like this SFT and DPO stage in a lot of sense are that. And then this RL stage is extremely hard technical grinding week in and week out to make the tools even run at all. and the disparity in post-training kind of tracks to me. You just have all these checkpoints flying around and it seems like chaos and then something that's extremely obvious gives you a massive gain. The DPO gains are the difference when being about this is not an apples-to-apples comparison but it would be the difference between being QN 2.5 level to almost QN 3 level.

1:09:25The thing that you apply to get there is sometimes really obvious And I think the frontier labs are much further down this path where they take these low hanging fruits so fast. But as a smaller team that's trying to map to what the changing priorities of the field is, sometimes it's just turning the crane on this really straightforward thing. I don't remember what I said, but Dario from Anthropics said very plainly, look, what works here is with 50 to 100 lines of code. He was saying it in the context of like espionage and him being scared about like some trade secret from from Anthropic being exported out.

1:10:06But the solutions at the end, everyone the industry favors are actually very simple. The problem is that there is a very large space of equally simple solution. And all the work goes in like, OK, how do you test these? How do you test them as fast as possible? how do you convince we have to be extremely skeptical in all the in any good results so how do you convince it that this results uh look good they're not just because oh sometimes there's a bug somewhere that caused it um some some something to be too good to be true um so yeah a lot of it is it's less about like the final solution what matters it's about like the speed at which you iterate and like how robust your tools is so like immediately after you see a good results and I know that this is a good result.

1:10:51And to complete the journey since we started talking about RL, so RL VR, reinforcement learning with verifiable rewards, let's spend a little bit of time on that sixth stage. In particular, Nathan, I understand that's your baby or you're one of the fathers of the baby. Do you want to walk us maybe a little bit through the history quickly? I think that I'm the person that got to bring it publicly to the world. It's well known that people across the industry have been doing this for years, and then the technique started to get far more impactful. It's broadly taking existing reinforcement learning algorithms or downstream evolution of proximal policy optimization, PPO, which is an evolution of reinforce.

1:11:39And then DeepSeek had their group relative policy optimization. I always try to say group robust. I think it's group relative. and like all these algorithms are really quite similar and it's you're you're training the models with whether or not they got the answers right or in the case of code whether or not the tests execute and don't fail i think one of the famous examples is that doing too much of this kind of or racing to get the low-hanging fruit from this rl approach is what makes all these code models do all these try except things to avoid errors because they accept all the errors I think that is just because the gains that you get in the model being useful is so much higher than the annoyance and the fact that it also does these stupid things and will fix the stupid things eventually.

1:12:24In the case of this ULMA model, it's not anything crazy. We cast a wide net on RL math problems. We do some data comparisons to see which data we think is the best for teaching these models. We do mixing with code and precise instruction following. And this mixing is effectively when you tune the big set that you have to what you've known from many experiments and to the specific model checkpoint that you're starting on. So if you have a really strong model and you show it really easy math problems, there's no learning signal. And if you have a really weak model and you show it really hard problems, it gets them all wrong.

1:13:02There's no learning signal. So the learning signal is all from the gradient of like you sometimes get it right and you sometimes get it wrong. Do you want to give the plain English definition of RLVR versus RLHF? RLVR and verifiable rewards is in the name. I think essentially the reward that you get from the quote unquote environment, which is like the completion or the greater, is whether or not you got the problem right. RLHF, the reward is essentially a reward model, which is rating the quality of the response based on a proxy to what humans would like. So it's described as being a much, like the RLVR reward is much easier to understand because these reward models tend to have a lot of problems and you can over optimize them much more easily because the reward models will pick up on features that are maybe emojis or something like this that you don't actually care about.

1:13:53Where RLVR is much better matched to performance characteristics rather than style. And what you tweeted, I think, or said somewhere that RL and long context reasoning distills is very hard. I don't know if that's RL in general, specifically this type of RL. What makes it super hard and very much the frontier of AI right now? So there's many ways that your tooling can fail. I think where most of these processes are set up right now is that you have a set of generation GPUs, which look like something like VLM, and you have a set of training GPUs, which is some distributed learning framework, which is where you actually have this RL update and loss function.

1:14:31And therefore you need to have some sort of system that orchestrates the two and passes information back and forth. And this kind of information passing back and forth is really annoying. It's a systems problem because you have like distributed error handling and things like this. So like a common case is when you have the most basic approach is that you'll have like one generation, this one math problem, The model is thinking and thinking and thinking and thinking. So you have all these GPUs working on one problem. So effectively, your whole system is somewhat idle waiting for the answer. And there's many other small things like this, which is this long context generation just uses so much memory that you then need to introduce different types of parallelism and stuff to do the generation effectively.

1:15:12And there's just a lot of subtle numerical issues. So I think it's just kind of stress testing a lot of the post-turning infrastructure that we have had by turning up a lot of different things that could go wrong to do the maximum. I think the things that the open community struggles with is that VLM and HuggingFace use different kernels to do the actual internal computation of the model. So these kernels are the things that make things like VLM really fast. But this then results in subtle numerical differences between the completions that you're generating from the model and then the log probs that the thing that's doing the loss function actually generates.

1:15:50And if you look at the math of these RL algorithms, it's assumed that those are from the same distribution. So therefore, this is a big root cause of a lot of numerical problems. And then if you look at what we're doing, a lot of labs have done throughout the years. They do different fixes to change these numerics. I like Thinking Machines had a famous blog of one of their first blog posts on like deterministic VLLM to make it exactly deterministic. And like that is really useful and people think it is key to their like Tinker API and doing other sorts of RL things where you just have complete control over sources of non-determinism.

1:16:24And that can just be like numerical lack of robustness in RL. And if we talk about trade secrets or whatever in RL, there's also these discussions on if the open labs have worse algorithms than the closed labs. And in reality, it seems like most people are using something like an evolved version of GRPO, which is a bit simpler than PPO. Some labs might be using a learned value function. It's not that important about the details. But what happens is that each lab finds the set of tweaks that they need to get really stable RL performance. And in the RL literature, historically, there's a pretty low bar on the amount of changes that are needed to call it a quote unquote new algorithm.

1:17:05But it's realistically like an implementation detail. So it's like a lot of everyone finds their stable configuration for operating and it's really dependent on the tool. So, yes, you could say that they have a different algorithm, but it's also not really something that you could easily exfiltrate from a lab because it's dependent on many layers of the stack. and maybe what chips they're operating on and all sorts of things. So it's just one of these things where training these models is complex and the kind of quick quips could never reflect that. The stack for post-training is also so new. Like software-wise, I feel like the big strides are happening in 2024 around like, oh, you do like both, you train the model, but also you run the model at the same time and that they have to happen at a certain cadence versus of pre-training the seeds of distributed pre-training.

1:18:03Like you had that in TensorFlow, which Google released in 2017, right? So there is a much more mature stock versus what you need on the post-training side. All right, so maybe as a last part to this conversation, it's been really fascinating and illuminating everything that you guys have described because in particular, it sort of highlights the complexity of the systems, like the multiple stages. And I love what you mentioned, Nathan, a few minutes ago when you said the pre-training is scientific and post-training. My words, not yours, but my interpretation of your words was like it's a lot of tinkering and putting things together in a way that you hope is going to work.

1:18:47And truly diving into how those models work on the one hand, But on the other hand, you know, each time you're like open a newspaper online or go on Twitter, like everybody's talking about AGI and how we're almost there and how it's going to change everything. There's a little bit of a cognitive dissonance between like the reality of trying to make those models work with all the unbelievable progress that we've seen, of course. But like, you know, that on the one hand and the discourse on the other hand. And Nathan, you've had a much more, I would say, tempered view of AI progress compared to some AI researchers.

1:19:30You had a great blog post very recently that you called Thought on the Curve. I'm curious what your latest thinking is. And, Luca, obviously, feel free to jump in any time. But in terms of what you described in that is saying in a prior one as complexity and complexity tax, which, again, like in view of the pipeline you just described, like it's one start to understand the level of sheer complexity that's involved in all of this. Yeah, so ultimately, I definitely describe myself as lightly AGI-pilled, and I think you have to be to appreciate the magnitude and gravity of the situation that we're in.

1:20:11But also, I think that I'm very far from believing in any sort of singularity being possible due to these things like complexity. On one hand, we talked about all these things which are low-hanging fruit to improve the models. And I don't doubt that it, I mean, like Schulte commented this on the pod and other places where at these labs, they still see low-hanging fruit in improving the models. In many ways, I don't think that their approach feels that different. They've just refined it relative to what we're doing. But at the same time, as things get complex with tools and adding more layers to the stack and you have to build a product to scaffold it, if the requirement to get the best out of Cloud is to use Cloud Code, which is some magic product and prompting relative to GitHub Copilot, this is one thing that you're going to need to get right in order to get AGI along with all these tool uses and stuff.

1:21:00So it's like as any system gets more complex, the pace of change is slower. I think any tech company has seen this. And then realistically, there's going to be physical constraints on the amount of infrastructure that we can build. So this belief is simultaneously giving us these new data centers. And I personally think hopefully a new power generation. But there's a cap. And for having a, in order to, like all these things are plateauing and then you 10x the compute and you get a big jump. Like you can't do this forever. So realistically, there's going to be some physical constraint that kicks in at some point.

1:21:33But balancing that with complex systems and the low hanging fruit results in like, I think these researchers are going to grind out improvements for multiple years, but never in a way that results in this kind of accelerating well that we get drawn into. So it's kind of, it's like, I don't know, in some ways it feels like I'm having my cake and eating it too, but it seems like the likely outcome if you look at other types of technology. Yeah, and that conversation was in a particular reaction to AI 2027, which is a really interesting conversation. I'll let you summarize the premise, but the short version is that AI sort of builds itself and therefore accelerates.

1:22:16Yeah, and I think they have these milestones like AI automates research engineering and then AI automates AI research and development. where it's like each of these are incredible jumps in performance. And I think what's more likely is this messy evolution. They deserve credit on their marketing and getting this impact for sure. But even them are now like, oh, maybe we should have called it AI 2028 or AI 2029. So I think like that is the reflection of like there are these real constraints, but the progress is also going to be incredible. It normally is the growth and capability of these models.

1:22:50But like having working on one, I think it's very unlikely that we will see like a discontinuity at any point. It has nothing to do with whether like we'll get to a definition of AGI or super intelligence that people are happy with. We will get there. It seems unlikely that the moment like it's going to be a looking back kind of kind of exercise of like, oh, these were the important milestone and this is what really worked. It's building this model is so much a collection of like refinements till to unlock the next stage that it's going to be this smooth trajectory. Whether it hits at some point where we don't have more capacity to keep improving on whether it forever accelerates.

1:23:38It's people are going to be disappointed if they want to see a moment where like one day they log into Twitter and AGI is there. Like it's messy and it's fun working on it because it's messy and it gives a lot of satisfaction. So to play it back, you're both saying yes to AGI, but no to discontinuity slash singularity. And one, is it fair? And two, if that's what you're saying, then for AGI using the current paradigm, basically what we just described in the last hour of pre-training plus RL gets us there? I think the AGI word is actually pretty not useful. I think that how I describe it is that big tech has all collectively realized that these language models plus scaffolding is going to unlock absolutely incredible value.

1:24:27And I have very high probability, barring extreme geopolitical situations, that big tech executes on this vision across the two to five years to build 95 to 98 percent of the way there of what you can do with our physical power constraints and what an LLM's ability is. And I think that that will be extreme, like the transformation from that by 2030 is going to be so powerful across society. There's a bunch of long tail, like there's going to be mass societal readjustment to what the internet and media and information means and like within five years. And that's mostly why I do this. And I think debating whether or not it's AGI is kind of secondary to the fact that this is coming.

1:25:12And we want people to study and understand what is happening. And to that last point, what does that mean? Study and prepare. What would you recommend people do? Although if people have made it all this way to this point of the podcast, they've already done a bunch of the work. So I think there's a lot of it. I mean, there's a lot of interest in AI across outside of the CS majors and on of the world. where it's informing policymakers. I think it still takes a long time for information to diffuse. And there's often not that many people that are engaging in this that are doing it just purely for this kind of, you can call it alignment and concern.

1:25:54There's just a lot of general noise. And I mean, I worry about concentration of power or all sorts of many things. And it's just trying to upskill people into understanding AI. so they can be engaged, like engage listeners and think about how it affects their domain. The other part, maybe I'm more positive to know, is like if the scaffolding is what really moves a lot of like from, you know, broad capability model to like something that actually has meaningful impact, that scaffolding is not just like, oh, only the labs of people with trained models can do it. Like the number of people can contribute to that, both in terms of people with tech expertise, and people with non-technic expertise, it's much larger.

1:26:37If the scaffolding is what really moves capabilities, what gets us to this incredible technology being realized, then the number of people can contribute to it is not just those who work at Frontier Labs. There's a tremendous amount of technical work to do, but also non-technical. As soon as you start integrating this technology in the life of real people, As soon as you start working on, you know, high stake medical application or other high stake domains, then a large amount of population can contribute in making this technology better and make it work for everyone. Just a base model. I feel like the number of people who can really can help making this technology really work for everyone.

1:27:21It's large. Everyone in society feels like it can contribute. All right. Well, that feels like a wonderful place to live it. Thank you so much, both, not just for this conversation, but for all the work that you're doing in open source frontier AI, which feels sorely needed and extremely important. So really appreciate the time and all the thoughts. Thank you so much. Hi, it's Matt Turk again. Thanks for listening to this episode of the Mad Podcast. If you enjoyed it, we'd be very grateful if you would consider subscribing if you haven't already or leaving a positive review or comment on whichever platform you're watching this or listening to this episode from.

1:28:05This really helps us build a podcast and get great guests. Thanks and see you at the next episode.