Asimov: Building An Omniscient RL Oracle with ReflectionAI’s Misha Laskin

27:56And the team does as well. Yeah, exactly. But I think this is also how it was very common at Google and I think other places as well for different parts of the code base to have owners. And so there are like these ownership files that we have as well. And basically, if you're on the ownership file, then the review has to go through you or it has to be approved by at least one of the members of the ownership file. And as people move around teams and so forth, the ownership files themselves get updated. So I think a pretty similar structure is probably going to hold here, but it's a lot more nuanced than building kind of an individual memory, which is just kind of personal to you and lives on your computer in your agents and B file or something.

28:38OK, if we zoom out and place like reflection overall in context a little bit and talk about the larger environment. Sounds good. Yeah. You know, coding as a as a root problem in this era of AI research is somewhat commonly held belief. Right. I think a criticism of companies that went after pre-training focused on coding was in reality, like you actually you needed language. You needed a lot of the capabilities, who can say exactly which, but the reasoning capabilities that could be elicited from large pre-trained models to do code anyway. And so you had to do all of the work without the general use.

29:18Is it specifically the availability of pre-trained models that are more capable and open source that made you feel like we can go after super intelligent, like autonomous systems in coding without spending the pre-training dollars up front as a new lab? Or help me think about that logic a little bit more. I think that's roughly correct for, you know, the sort of why you can get into the game sort of short term. And a bet that we made, you know, when we were starting a company a year and a half ago, was that there were pretty decent open-weight models out there. That pre-training, you know, we kind of saw it pre-training as starting to more or less converge on kind of a known paradigm.

30:02There's sort of a, there's a known big data set on the internet. Yes, there are going to be some algorithmic innovations, but you're basically extracting signal from an extremely noisy data set. And we felt like there's only so much signal that one would be able to extract without getting into just absurd dollars for scaling this in terms of what you're trying to get out of it. So what we thought would happen is that there'd be decent open-weight models. I think the quality of the open-weight frontier has surprised me. They're actually, the models are better than I thought they would be. And we thought that you can just focus on, you know, we're in this brief period in history right now where the RL flops are still manageable.

30:50Like you can really have a best in class product if you're focused. And yes, you'll need to put, you know, you still need a decent amount of GPUs. But from a flops perspective, it's nowhere near where pre-training is. Like two magnitudes off. Exactly. Right. So you can get into it and kind of build out a both kind of the product and a research arm. Our thought was that this was the time where you can actually start a, you know, a generational frontier lab that does not need to be coupled to a, you know, to a big cloud provider. because if you do it right, you'll actually be able to generate sufficient revenues to not have to be acquired or find some strange deal where the cloud provider kind of owns you.

31:42And that was kind of the model, I think, of a lot of what Frontier Labs look like for LLMs. I think we're already starting to see that, you know, this kind of more of a field-wide thing independently of reflection, right? When you look at how fast like Anthropics revenue is growing, I think, right, they're kind of in this spot where it's like a massive revenue generating business that's growing at an unprecedented rate. That is, but that was very much the ethos that we can come in, we don't need to pre-train. You can get by with, you know, two orders of magnitude less compute and really get something out there that's really good.

32:16I think that roughly speaking, you know, you won't need the amount of compute that I think Frontier Lab needs today as you're focused, But you'll still need kind of, you know, an order of magnitude less. So I think that the capitalization requirements are still high. There's no way of avoiding that. But I'd say they're and asymptotically, they're probably the same. But asymptotically, the idea is that at that point, you just have a generally generational business that can that can raise capital off of that. I guess part of my read at this point in time is and maybe it was always true, but especially now is your actual capabilities in terms of understanding what evals to go after, how to design reward models.

32:59There's perhaps like less understanding and more dispersion in the field in post-training strategies versus like, as you said, more maturity in pre-training right now. Because you can, if it was a simple question of scaling RL on language models, people would be doing it more aggressively right now. Right. And so actually, maybe that's a good question for you. Like, how would you describe the challenges in solving scaling here? Like why are we only able as a field to put like a much smaller amount of compute to work here and still get like best in class results versus pre-training scale GPUs right now?

33:39I'd say that there are two categories or one would think that things fall into. One is more around the problems, limitations of the problem structure. And the other one is, well, maybe the structure is fine, but you need algorithmic advances to really drive the next frontier forward. There's, you know, I'd say some mixture of both, but the biggest way I put is on the problem structure. So if you, the thing that I led for Gemini was reward models. I built out the reward models that were used to post-train Gemini 1 and 1.5. And I thought is that if you have a reward that accurately basically describes the outcome of any arbitrary task that you throw at it, then that's it.

34:25You know, at that point, it's just algorithmic advances. But even the very simple RL methods we have today will be able to get a lot out of this. Like they'll only be bound by their exploration abilities. That's the only thing, right? But if today, you know, we certainly are not in this world where we have clean rewards for every task we could imagine. And so we're kind of making as a field have to make sort of various shortcuts and compromises to that. So you'll have things like LLM is judged with different rubrics. And that works to some extent, but it inevitably, a noisy or like stochastic reward inevitably gets hacked.

35:03so you kind of need a lot of these and um you know and there's only so much you can extract out of them uh then you have sources that do have ground truth rewards but um there are not many of them and so you have to hope that by optimizing against those you'll get some generalization effects and so i think that the fundamental problem is like the reward problem you can either go in and say i'm just gonna all i'm gonna focus on is kind of rewards um or you can say i'm going to take things as they are and just be more creative in the methods that leverage the rewards that happen today. And so examples of that are basically every synthetic generation pipeline is some example of this.

35:46So it's a messy problem, but I think it's fundamentally like we're in a reward bound world. I don't think there's going to be any breakthrough that all of a sudden we go from we didn't have rewards for everything to we do because the reward problem in itself is at the time I called, I thought it was AGI complete. Now I'd say it's ASI complete. But by the time you have a neural network that can accurately verify any outcome, that is probably a super intelligence. And so then it goes back to, again, evaluations. What, if you're training your rewards, your reward models on something, like what are you evaluating against?

36:21What are the tasks that you want it to be good at? So that's kind of how I think about it. I think it's a fundamentally reward model or rewards bound field. And then there's also kind of algorithmic progress in terms of the RL methods we have today are quite bad, I would say, at exploration and credit assignment. Like they're sort of just like the fundamental algorithms are take the things that work and make them happen more frequently and the things that don't work and make them happen less frequently. But they don't discern at all along your, say, reasoning chain, which part of the reasoning was correct and which part was incorrect.

37:00And so that's why you get these reasoning chains that are kind of garden path meandering. Like they'll explore all sorts of things that are, you know, completely unnecessary and don't look at all like the kind of structured thinking that a person would have. That's how the algorithm works. It doesn't actually look at, there's no credit assignment step on any atomic level. And so that I would say falls into more algorithmic progress bottlenecks. Can I ask you for a few like hot takes quickly? Yeah, let's go for it. What do you think of all of these efforts, either in-house with, you know, labs and vendors or young companies just creating software environments that look like popular software to train agents in?

37:38All right. Copies of Airbnb or Amazon or Salesforce or Excel. Personally, maybe the take is not very hot. I'm very like bullish on it because how else are you going to? Maybe the hot take is that there's no such thing as generalization. There's just bringing the test distribution into train. Okay. That is an aggressive take. Wow. Yeah. So as long as like your, yeah, train distribution looks something like what you would actually want to evaluate for, um, then, you know, users will experience, experience it as generalization. I think, you know, I think there is some generalization that happens in these models, but, um, we probably as, as users overestimated because we don't actually see how they were made, but then, And, you know, yeah, if you saw, oh, the synthetic environment is actually very similar to the thing I was asking about.

38:22So it makes sense why the model would be would be good at that. Maybe six months ago, I think you you you said, like, I think it's possible we have my definition of ASI in a couple of years. Do you still believe that's true? I still do believe that's true. I think that where I think will be in a couple of years from now is that there will be kind of definitive super intelligence in some meaningful categories of work. And so, for example, when I say coding, I don't mean all of coding there, but there will be a super intelligence within some kind of slivers, some meaningful slivers of coding that are driving, I'd say, immense progress in the companies that can benefit from that.

39:03And the reason why I would say that the problem of ASI would have been solved by then is because you've kind of, at that point, it's just a matter of operationalizing what you know. It just so happened that these particular categories, you might have a super intelligent front-end developer because there's so much data distribution for that on the internet, and it's easier to make synthetic data for that. But at that point, you have the recipe, and it's just a matter of making economic decisions of is it worth sinking in X amount of dollars to get the data in this category to get kind of something close to super intelligence there.

39:41An example of that is what happened with reinforcement learning before language models. Effectively, the blueprint for building super intelligence systems was developed. It happened with the Atari games, AlphaGo. Then Dota 5 and AlphaStar were near super intelligent systems. And if OpenAI and DeepMind had sunk more compute into them, they would have definitely become super intelligent. It's just that at that point, it didn't really make sense. Economically, like, why would you do that? Then this is a definitional issue because I was going to ask, like, help me understand your view of like, I don't like one of the big criticisms of RL overall has been lack of generalization.

40:22That's been just kind of a general question for this direction. I do have friends at every large research lab that somewhat in a some I mean, tell me if you hear something of a different tenor or just believe differently. They believe we're going to have systems that are much more capable than humans and many types of knowledge work, but they believe less in generalization. And so in a resigned way, they're also, as you're saying, like, I guess we're just going to bring all of it under distribution one way or another. But that means like, you know, it's a little bit different than my my view of like it's at some point you're you're just, you know, you have enough capability that the rest you get for free.

41:02Right. The rest is sort of useful capability you get for free. I think I kind of have a similar viewpoint to to the people you describe. I think the generalization capabilities of these things has been weaker. First of all, it's all mind blowing if this exists. So we went from fundamental existential crises and generalization. Like this was the field of reinforcement learning before language models was we have these systems that we can make amazing, you know, at like very narrow tasks. We have absolutely no answer for generalization, like zero. And we went from that to things that, you know, feel like they're generalizing.

41:41They're certainly generalizing much better than anything we had before. But it's likely because the training distributions are so broad. So at least the way I think about it is more kind of output as a user. Is the system super intelligent in some meaningful categories of work? And then from a research perspective, is it obvious how to make it general for anything that you might care about? And at that point, again, it's just a matter of economics. Maybe there are some categories where collecting the data is so expensive and the return on investment is low, where effectively just better to have craftspeople than super intelligent AIs.

42:19So I think we're moving into this kind of world of jagged super intelligence where you have a handful of these super intelligences for categories that matter, maybe subsumed into one model at some point, but at first there'll probably be, again, I think there'll be a few companies that have kind of product model coupling that is super intelligent in different categories. I think an example of, again, starting to see the first glimpses of superintelligence, but in a way that hasn't really transferred to anything meaningful yet is, well, we have these like superintelligent test fakers now, like, you know, Amy, the Amy benchmark is completely saturated code forces and other competitive coding environments.

43:03The models are almost best in the world. And within the year, probably just the best in the world. And yet we have the, so we have the best competitive coding agents, then you go into a company and you ask them, have these things been helpful? And they say - It's uneven. Yeah. Yeah. Right. So in the parts of work that are really meaningful, you want to see these things driving meaningful kind of increase in GDP. And I think the only way you will see that is if you go into a company and there's kind of a universal understanding that, yeah, my engineers are double digit percentage points as a whole, every single one of them more productive, right?

43:43That's the kind of thing that if that starts happening across every field, then you'll see double digit increases in GDP. So I think that the kind of benchmark maxing that's, and it's a bit different than benchmark maxing used to be before, because you have benchmark maxing that is weakly correlated to customer outcomes, but it still looks very similar to taking a board game, training an RRL agent on it, getting kind of a landmark result in superintelligence, and then making a claim that, you know, superintelligence is solved. I think the reality is that deployment of it is half the problem, which goes back to kind of evaluating on customer problems and building product together with the models.

44:27So you must have seen the news of the Windsurf non-acquisition into either OpenAI, but non-acquisition into Google DeepMind. What do you make of it? We're seeing this verticalization basically happen across categories that are material to frontier intelligence. And one could argue that the first verticalized category was actually search, like through chat GPT. That's sort of a place where OpenAI verticalized first. And coding has obviously emerged as another kind of frontier level category that could, like all these companies have aspirations of - ASI. Yeah, ASI. And I think, you know, being basically trillion dollar companies or more, I don't think that it's really the economics that are the driving factor, but it's more that if you want to sustain frontier research, that's kind of what you have to become.

45:15And so coding has clearly become one of these categories where verticalization is extremely important. And I think that there are kind of two sides of the story, one on the Frontier Lab side and the other on the kind of more product side, like a startup that builds product but does not have its intelligence in-house. So I think on the Frontier Lab side, I think this is exactly kind of what Giannis and I noticed when we were working in Gemini is that your model is so far away from the product that oftentimes, even though you have the best model, does not at all mean that you have the best product.

45:53So there's a reason why basically startups are the places where kind of adoption of coding tools took off rather than the frontier labs. And so there's a verticalization happening there and some are going to do it successfully and some are not. I think that that's kind of we're already starting to see that with Cloud Code really being an example of a successful verticalization. I don't think it's guaranteed that a big lab can buy their way to the end user because the fundamental problems of your research team being far away from your product team will still be true and the company having a hundred different focus areas will still be true.

46:34So I don't think that acquiring an asset will change that fundamentally, but it does underscore the importance of verticalization. And then from the startup side, I think it actually puts companies that are in these kind of critical path categories like search and coding in a pretty existential place if they can't build their own frontier models. Not all frontier labs will be able to verticalize correctly, but some will, maybe one will. And that's going to be enough, I think, to kind of take the thunder out from a company that's built a great user experience on top of someone else's model. And I think some of those dynamics are probably starting to play out as well.

47:19I think that there are some question marks around if you're on this critical path category and you don't have your own intelligence, how do you compete when your competitor can you know just basically subsidize their product a lot more than you can um right because you're effectively as a as a startup that's building on top of these things to grow quickly you're subsidizing you know the margin that you know an anthropic or gemini or whatever is making and google and anthropic and opening eye can subsidize their products a lot more than you can uh so i think that companies that are don't own their intelligence or not kind of deeply integrated into a customer in some way that makes them hard to remove, find themselves in this pretty existential place as it becomes clear to the frontier labs that this is a category they need to verticalize around.

48:10I work with a few robotics companies. And so much of my lens on RL comes from that. And I think it is like far less clear in robotics that, you know, RL will be a dominant part of the training versus imitation learning. You'll actually appreciate this on imitation from humans using tools, right? Because we run this, I'm going to like describe this idea that is nuts, but I think it's just funny. We run this grant program twice a year for amazing people using ML in different fields. And it's called Embed. And one of the ideas I had as a joke recently was, well, like you just record everything, right?

48:56Like not obviously just the code base, but like your Slack and all your documentation and all your conversations because you are a software engineering team. And I'm 100 % sure that I can take that data set if you ship something into production to an end customer that has real issues at any scale and sell it to a friend who's a researcher at a lab working on this stuff. And so you have some floor value that is millions of dollars for your couple person company And like bonuses, like maybe the software company works, right? Obviously, this is like very noisy and I'm mostly joking, but I'm curious how you think about like exploring non-RL data sets that could be useful to you here.

49:40If that company existed, right, we would definitely pay for their data. There we go. See, it's not an idiot idea. Yeah, it's, yeah, especially if there's diversity. I think that'd be... I can sell the whole set. Yeah. So is the question around how do you leverage alternative sources of data? Yeah, the question is, I think there is like, I don't want to like over analogize to robotics, right? But within robotics, you have learning from world models, you have learning from sim, you have learning from embodied data of different types, right? Imitation, then you have RL. I think it's like much less clear that you can use RL for a lot of robotics today, especially some of the harder like manipulation problems.

50:29And I'm curious, just given, you know, your team has this enormous strength in RLs, like a starting premise, how you look at other types of data to create the, you know, coding agent experiences you want. So I was actually a robotics researcher for like in reinforcement learning that Peter Beale's lab is a robotics lab. And it was, you know, it was a mixture. Peter's lab was always around the intelligence problem and robotics as being a domain where you study it. And one of the, you know, the reason I came to lead reward models for Gemini was because that's the question I was studying with robotics.

51:07I was, you know, we had these RL algorithms for getting robots to do some very narrow tasks, like moving blocks and, you know, various kind of narrow tasks and simulation. and the question was well how do we get generalized um manipulators and um you know just how do we build this all into one system and it seemed like the rewards were bottlenecks so this a lot of what i was studying before uh starting you know getting into language models was how do we design reward functions or models for um for robotics or you know for 3d video games like minecraft or something like this that have, I think, similar challenges scientifically.

51:48The challenge is that if you think that language model rewards are hackable, vision language model rewards or other sensory signal rewards are infinitely more hackable. They're much more short-lived than rewards. Language is just a compressed representation of the world that we have that we magically have to start with. whereas if you're processing pixels or a sensory motor signal this is raw signal that has a lot more noise in it and so if you train a neural network that is sort of trying to detect whether this thing was manipulated correctly or this thing was you know moved correctly then that thing is just infinitely more hackable than anything you have in language models so the same problems blow up and become much larger.

52:38And so that's actually why I changed to language models, because I felt that this was a fundamental problem. But, you know, we now have these confounding factors of these noisy signals coming in. I think that in at least in a generalizable way, that's why it's really hard to get reinforcement learning to work with robotics. The one place where it really does work well is when you have a clean reward signal, which happens to be in these like locomotion like scenarios. So there's a lot of work on building very robust sim to real locomotion pipelines. And it's because it's kind of locomotion is just your body.

53:15Like you don't have to manipulate the world around you. And so you can actually build reward signals that are like, oh, you know, your quadruped is moving at this velocity without damaging its body kind of thing. Maybe it's a bit of a roundabout answer to the question, but it's that I think these two fields are very different in the data distribution that they support. And the kind of imitation learning data for language models is, of course, the internet. And it's, of course, people who've gathered all this data on how we write and so forth. And so aside from that, when we're generating synthetic data, the only scalable path is really reinforcement learning.

53:54The other thing that I'll say here is that when you're collecting data for robotics, you can do it in like this tele-op way. Like it's sort of, these are things like the things that we try to, are trying to train robots to do are very intuitive for humans as well. I mean, actually more intuitive for humans, right? People are master manipulators. So you can have a lot of kind of tele-op like data collection. The things that we want language models to do are sort of, you know, at the level of it's really hard to collect data of, you know, the chain of thought process that goes on in like a human's head when they're trying to solve some tasks.

54:32And that's kind of the data that you need. And so for that reason, I think language models favor this more like synthetic data, RL like approach where, well, it's easier for us to like verify whether the thing was done or not than it is to actually generate all that data from a person specifically. Maybe we just need like a network interface. Yeah. To get the chain of thought. Yeah, maybe. I mean, that's kind of actually when Janice and I were starting the company, we were thinking about, well, what like, you know, maybe we just like, yeah, somehow like have people speak into a microphone as they're doing tasks in order to capture that.

55:07Just stream it. Yeah. And it seemed, you know, logistically very hard to pull off. OK, one one final question about sort of reflections path from here. At what point do you this is a decision you get to make in the future, but at what point do you try to look at other problems beyond engineering and coding? Like, do you do you feel like there's a level of sufficient depth where you should just go attack different domains? The thing that makes coding as a category special is that it's not synonymous with software engineering. It's just kind of how we think about the market today. The reason code is special is if you believe that the way a language model will interact with almost any piece of software is through function calls and therefore code, then if you build very capable reasoners, coding reasoners that are sort of purpose built for organizations.

56:00So you've solved the kind of long context. How do I reason over a bunch of disparate source of information problem? And I can act on pieces of software through code. Then you've kind of built a system like the technology that will generalize at least operationally across other categories of work. And so the way I think about it is more first, just build, not trying to get too ahead of yourself, just first build the most depth-wise comprehension system for software engineers. This will naturally induce more reliable coding agents. You can plug that in as an MCP to your favorite IDE or coding agent, or use one of our own.

56:48You can kind of plug that into whatever surface area makes sense for the customer. And then sort of naturally start seeing where you're getting pulled from there. And the reason I think this will work is because we're, this is kind of what we're already seeing, right? In the, you know, how do you make the system useful for product managers or technical support people? And then, you know, I think moving on to things like sales or something like this, but there are already places where, you know, customers are pulling us in different directions. It's just kind of a matter of whether you engage on that today or not.

57:24And I think that the risk that a startup has is that, you know, you see a lot of shiny areas where you can go and you start kind of going diffuse before you've really nailed a category. So I think it's really important to be focused and not diffused in the short term. And that if you kind of build the right, as we kind of think about as a contextual core for an organization, in this case, an engineering organization, then you can naturally start expanding that into adjacent areas of work in that enterprise. Okay, last question, Misha, where would you characterize us as like being on the path toward deployment of these capabilities in different fields?

58:03I think we're a lot earlier than most people think that this is going to be one of those areas where the technological building blocks outpace their deployment. And so, yeah, within the next couple of years, the blueprint roughly for how to build ASIs will have been set more or less. Like maybe there are still some efficiency breakthroughs that need to happen, but more or less there will be a blueprint for how do you build a superintelligence in a particular category. Actually going in and deploying it and building it for specific categories of work, there are going to be a lot of product and kind of research innovation specific to those categories that will probably make this a multi-decade thing.

58:48So I don't think that it's a couple of years from now and GDP starts growing 10 % year over year globally. I think we're actually going to get there, but it's going to be a kind of multi-decade endeavor. I tend to kind of see a lot of patterns now in kind of real-world deployment with reinforcement learning research as it worked, again, before large language models. and before large language models it used to be you pick an environment like you pick go you pick starcraft you pick something else and you go and try to solve it with you know some combination of imitation learning and reinforcement learning and when you look at all those projects these were basically things that were called strikes within within deep mind and each strike within and outside of deep mind was a bit of a snowflake like the reinforcement learning methods and environment setup for Go was at a high level, conceptually similar, but in the detailed implementation level, very different from StarCraft, very different from Dota 5.

59:52And so I think that that's sort of, we're going into every big category having a different environment, right? And different kinds of agents with different tools. And that means that you'll need to, you'll have like general base models that you can start with, but you'll need to post-train things in specific ways for those categories. And we're starting to see that already in the sense that the model that powers OpenAI's codex is not the O series of models. It's a model called Codex, which was post-trained for that environment. The deep research models, like that's a specific environment. They're also post-trains for that environment.

1:00:28And I think we'll basically see more and more that any category that has a sufficiently large business around it that requires an intelligence core to power it, there will be all sorts of interesting design decisions at the research and product level of how do you actually gain the most performance out of this particular category. So I think we'll kind of see a lot more kind of depth first players emerge over the coming decade or so. I'm making a bet on it. And I also think that like part of to your point about choosing like the problem for the era, we don't get to choose at conviction a problem for 100 years.

1:01:06We do get to choose for like this decade or so. Right. And, you know, if you actually believe it's going to be a very long term endeavor to get to the sort of productivity and abundance you described, but we are going to get there, then, you know, the other thing you think about is like, like path to supporting the cost for bringing anything under distribution during a particular period. Right. And so I'd say like in the, you know, we've already backed companies in in some of these areas, but like, let's say in life sciences or material science, like it is more expensive to collect, you know, types of data you might need.

1:01:44And that might be a longer endeavor or one that you have to figure out how to fund, right? Or in robotics. And so I think it's a really interesting timing question of like any of these really big categories. But I believe coding is this era. I think coding is this era as well. This one, I think will take longer than people thought as well, because again, enterprise, there's organizational problems, just much different than the benchmarks that we have today. But I think it will be one of the faster ones. So I don't think that that's kind of a decade out. That's within the next, you know, say dozen, dozens of months kind of thing.

1:02:22So I think the next sort of generational companies in encoding are definitely being built today. Well, congratulations on the release, Misha. Thanks. Yeah. Thank you, Sarah.