Reflection AI’s Misha Laskin on the AlphaGo Moment for LLMs

28:36They're general in that you can take any task category, have a bunch of prompts for it for your training data and make a language model kind of iteratively capable, more capable on those things. I think someone needs to solve the kind of depth problem and the field as a whole, I think, or large labs have been really working on the breadth. That's amazing. and there's a big market for that and a lot of very useful things that get unlocked, but so one needs to solve the death problem too. I think that takes us really nicely into the unique insight that you and you on us have from working on AlphaGo, Alpha0, and on Gemini and the importance of post -training and data.

29:29Can you share a little bit more about how those experiences of shape the unique perspective that you have that gets us to the unlock with the gentic capabilities? One of the things I found very surprising about language models is how close they are to, you know, how often times, even if they're not working on something that you want them to, they're actually quite close. They feel like a nudge away. They feel like they need to be grounded in the thing a bit better. And that's, I think that was the insight that led them to be good and chat. like you could play with them and they're yeah a bit unreliable and they kind of go off the rail sometimes but they're almost Good chat companions.

30:09Yeah, and so Then the recipe there. There's a recipe for how do you take a pre -trained language model and make it a reliable chatbot? So by reliability there. It's just a The way you measure that is with human preferences do People interacting with this chatbot prefer over other chatbots or other versions of it the previous versions of itself so If the current version is much more preferred than the last few iterations ago, then you know you made progress. And that progress is made by collecting data for it. So it's collecting data for the kind of queries that users input into a chat box, the outputs that the models provide, and an effective ranking between those outputs so that you push the model to index over on the more preferred outputs.

31:04So when we say ranking, whereas that ranking comes from humans. So there are either human lablers or it's something that's embedded into the product. You sometimes might see thumbs up or thumbs down in chat GPT. It's harvesting your thumbs to know what your preferences. And that data is used to kind of align the model with the user preferences. That's a very general algorithm. That's a reinforcement learning algorithm. And that's why it's called reinforcement learning from human feedback or RLA chef. You're just up -playing the things that human feedback is pressing preferences for. There's no reason why the same approach would not be possible for enabling more reliable agency.

31:49There's a whole sequence of other problems you need to solve. I think the reason this is so hard is because as soon as you go into kind of agent territory, you have more than just the language outputs. You have the tools that they interact with. And the tools being supposed to send an email or work at an IDE or anything that an agent does, it does an environment and that requires tools. That requires the environment. And everyone who's deploying agents is deploying agents in different environments. And so there's a challenge of how do you integrate with the environment and how do you onboard agency onto them.

32:27So I think that that's why it's a bit of a schlep if you get into this kind of line of work and you have to be careful about the environments and kind of the way you structure because you don't want to overfit to some particular environment. But conceptually, it looks very similar to aligning a model for chat. There are just some more integration challenges that need to be solved along the way. Since you view AlphaGo as the pinnacle of building an agent that was truly capable, I imagine you're trying to usher in an AlphaGo moment with LLAMS. What do you think are the differences? Like to me, you know, with gameplay of a very clear reward function, you have the ability to self -play.

33:16Like is doing kind of the reinforcement running from human feedback? Do you think that's enough to kind of get us to an alpha -go moment in LLM's or like, well, I guess, how should I think of the differences here? I think what you said around not having a ground truth reward is, hey, key and maybe the key thing. But we learned from the previous era of reinforcement learning research that if you have a ground truth reward, your kind of guaranteed success, like that's kind of there've been so many very impressive projects that showed this at really unprecedented scale. I mean, you think aside from AlphaGo, there was opening eyes, Dota 5 or AlphaStar and let's say AlphaStar and Dota 5 are a bit more niche in the sense that you kind of have to play those games.

34:04to understand, but as a former Starcraft player, I was, I still am completely blown away by Alpha Star. Like, the strategies that the AI discovered were, it just looked like a very smart, like a smarter than us alien game, like upon the Earth decided to play this game and completely outcompeted us. So that's due to the existence of a number of things but a ground truth reward is really like is extremely important for tightening that behavior.

34:39Now, both with human preferences and for agency, we don't have ground, we, these are very general objects and we don't have ground truth rewards for whether something is accomplished or not. Like, even for a coding task, what's the ground truth of whether this was done the right way? Like it could pass some unit tests, but it can still be wrong. So it's a really hard problem. And I think it's the fundamental problem for agency. There are others as well, but this is kind of the big one. The way you get around this problem for chat is, again, through Arleigh Chath. Well, you train reward models.

35:18models. Her word models are, it's a language model that predicts whether something was done or not correctly. The challenge with that, so first it works well. The challenge with that is that when you don't, in the absence of a ground truth, when you have this kind of noisy thing, that it can be wrong, your policy or the agent quickly gets smart enough for it finds holes in their reward model and exploits them. To give a concrete example in chat, suppose you noticed that your chat bot was outputting was outputting some, let's say, harmful content or like there are some topics that you don't want it to talk about because they might be sensitive.

36:05And so you put in some data into your data mix saying like where it's examples of the chat bot kind of ignoring, or not ignoring but saying, I'm sorry, is a language model I cannot answer this. What can happen is that, okay, you now train a reward model against this, and suppose in your data mix, you really, you only put in data points that showed instances of this kind of, this happening, but not instances of a chatbot taking something like kind of sensitive and actually answering it. What that means is that the reward model could think that it's actually like a good thing when you just don't answer the user's query ever because I've only seen positive use cases of that.

36:48And when you train against that, the policy with language model will at some point get smart enough and discover that this reward model gives me high reward whenever I just don't answer whenever I punt it the question and it can collapse into a language model is just never answers your questions. And this is why it's very finicky and it's very difficult for this reason. I'm sure a lot of users who have interact with chat GPT or or Gemini or these kinds of models like probably through through interacting with them found sometimes that they kind of degrade and they all of a sudden like don't answer questions you know as often as they used to get slightly worse at something or are politically biased in some way.

37:38I think a lot of that is, well, it's artifacts of the data, but the artifacts in the data get amplified by bad reward functions. That is the hardest problem, I think. If I view the rough large model training pipeline or large AI system training pipeline as pre -training and post -training. I kind of think pre -training seems largely like a solved, like we're in the, you know, the techniques are solved and we're just kind of in the race to scale. Moments on the pre -training side, post -training still feels a little bit like in the kind of research phase of market where people are still figuring out what techniques will work in a general way.

38:21I'm curious to do you agree with that and in an ideal state, like what is pre -training responsible for doing, how should we, as laymen, think about it? And what is post training responsible for accomplishing and how should we think about that from the perspective of five -year -old? Yeah.

38:41I would generally agree with that statement that pre -training has become, there are a lot of details I need to get right and it's, by no means easy, so it's a very hard endeavor, but it's a better understood endeavor at this point. And one way that I think about pre -training is I actually think thinking about it through the lens of something like AlphaGo is quite simple and clear because it kind of, you know, rather than thinking about this massive internet thing, you just think about a very clear setting, which is clean setting, which is this game. You can think about the pre -AlphaGo as two phases.

39:24It has an imitation learning phase where a neural network imitates a bunch of expert amateur, like expert go players. And then it has a free -inforcing learning phase. And you can think about pre -training as the imitation learning phase of AlphaGo. You're just kind of acquiring the basic skill of learning to play the game. You're not maybe your neural network then is not the best in the world, but it's pretty good. it goes from zero to pretty good. And pre -training for a language model is going from zero to pretty good on everything, which is why it's so powerful. Post -training is, I think about it as hardening good behavior.

40:07What that means is, with AlphaGo, you start off with, you did imitation learning. You start off at a place where you have a neural network that can do something. It can, I mean, it can play a game pretty well. Then you apply this other recipe to it, which is reinforcement learning, which is then the network starts generating, you know, its own plans and kind of acting through the game, getting feedback, and that good and basically good actions get reinforced. That is, I would say that's post -training, and you can think about, from a chat perspective, you're hardening the model, like, the good behavior very along the chat axis.

40:46So it's actually quite interesting that the high level recipe for training alpha go and for training Gemini is actually the same. You have the simulation learning phase, and then you have a reinforcement learning phase. The reinforcement learning phase and alpha go is just much more sophisticated than what we have now. And the reason comes back to reward models. If you have a reward model that is fairly noisy and exploitable, and exploitable, then there's only so much work, there's only so much you can do before the policy gets smart and finds a way to trick it. And so even if you through the fanciest like RL algorithm at it, like Monte Carlo tree search with AlphaGo, it may not be that effective because it kind of, it collapses into this kind of degenerate state where the policies have their reward model before it could even do any interesting search.

41:43Like, suppose you're thinking about, like, if you're playing chess and you're thinking about what to do multiple moves ahead, but your kind of judgment is really bad at every move, then there's no point of planning 10 moves ahead. And I think that's where we are with Arleigh Chaff today. There's this wonderful paper that I think is very over or underrated, called Scaling laws for reward model over optimization. This is a paper from OpenAI studying this phenomenon. And what's interesting about, I mean, a number of things, but it's showed that this phenomenon happens at all scales and I mean, to try a couple of different RLHF algorithms and it happens for, at all scales for all algorithms that were tried in that paper.

42:31And I think that's interesting paper because it's the kind of fundamental problem of post -training. That's that paper. Yeah, just to put Paul in the thread a little bit. If you follow the results from alpha zero though, then we may not need pre -training at all. Is that a fair conclusion of what to make of this? I think that the at least my more from a practicality standpoint. When we went from or deep mind went from AlphaGo to AlphaStar, there was no Alpha0 of AlphaStar. There's no AlphaStar 0 that was, you know, released after that or anything like this. And AlphaStar had like a big part of it was imitation learning across a lot of games.

43:30I think with AlphaGo, it was like this kind of special place where you have, you don't only have a zero sum game, but you can get to the end of that game fairly quickly. And so it's again, you can get that feedback about whether what you did was right or not. Got it. Okay, so it's just way too unconstrained that we're problem to throw that out generally. Yeah, I think in practice, alpha zero would work generally for everything if we had Ground Truth or Word Functions or everything, But because we don't, you need to do the imitation running piece. This is just like a practical, we need to get it to the game somehow.

44:10You described earlier the importance from a technical perspective of having an agent in its environment. Also from a product distribution and getting the product in user's hands perspectives, it's important to think about what the right task categories are for users to first interact with the most powerful agents. What are some of the task categories that are on your mind? And what do you imagine are some of the possibilities that users could use these and their daily workflow? If you want to make progress along the depth axis, you could go for like, AlphaGo first, which is like a really hard thing, or you could kind of expand concentrically in the sort of complexity that have the task you're able to handle.

44:51And we are focused on kind of enabling depth, but in this sort of concentric way. And we care a lot about having a general recipe that is not, that does not kind of, you know, inherit heuristics that are special to some tasks. So from a research perspective, we're building general recipes for this. Now you have to ground those recipes in something to show progress. And at least for us, it's important to show diversity of environments. And so we're thinking about a number of different types of agents, web agents, coding agents, I have OS, computer agents. The important thing for us is to just show that you can have a general recipe for enabling agency.

45:51Switching gears a little bit, you've attracted a seller team already. Who else are you looking to recruit on your team? Yeah, we've been fortunate to be able to draw some talent from the top AI labs in the industry.

46:12And I think a lot of that has to do with both of the work that he honest and I did, but definitely, you know, I think a lot of credit goes to Janice and his reputation. You know that there is this, I was watching the Michael Jordan documentary and Michael Jordan was, well, one of the reasons he was so effective is because he was such an incredible kind of individual, like, basically contributor to the game, maybe be best, that he really inspired people on his team to get to his level, even if they couldn't get there. And he honest has this effect on people, like I worked very closely with him on Gemini, and he had that effect on me.

47:00I don't know if I ever got to the honest level, but I aspired to, and I definitely became a much better engineer and researcher through the process. And I think that's a lot of the draw, is that you get to learn a lot from him. We're primarily continuing to look for, so we're not hiring out quickly. We're hiring out, I think, more methodically. We're looking for, yeah, other, definitely interested in other researchers and engineers joining us on this mission. I'd say a commonality between everyone who's joined is we're all very hungry. Maybe that's how I put it. We could have, the honest and I could have stayed and tried pushing agents at DeepFind.

47:57And as I said, I think the reason we decided to do it in our own ways, because we think we can move quickly much faster against the school. And some of this urgency is driven by a real belief that we are three or so years away from something that resembles a digital age I. And by that, that's what I'd been referring to as Universal Agent. Something that has both this kind of breadth and depth of knowledge. And that means we're actually on a very accelerated timeline. Yeah. You're, you know, a few months in, you're kind of 5 % away from hitting that timeline. And maybe so this is also driven by how quickly AlphaGo went from Experts into field doubting this is possible.

48:56Yeah, it's kind of decades Human level or like expert human level go play was decades away and How effectively they were able to solve that problem within months? I Think we're seeing a similar kind of acceleration happening with language models. There's I think one viewpoint point you can have is that we've saturated a lot of what we can. We're on this sort of at the tail end of an escurve and we don't we don't view it that way. We think we're still we're still in an exponential. Part of the reason is that these things are so bulky and slow to train that there's no way collectively as a you know as a field of researchers and engineers that we've optimized it, yet, like, if it takes a few months to run, and a few months and a few billion dollars to run the biggest model, then how many experiments can you really run?

49:55So yeah, we kind of see things going at an accelerated pace, and we think solving the depth and reliability problem is something that is not getting the kind of attention it needs. Like there are groups that are following this as I would call it more like side quests within these big companies, but I think you need a player that is focused entirely on it to who solve this problem. Yeah. I love the framing of main quest versus side quest and I love the hunger and the zero complacency and impatience and a healthy way that you and the rest of the team have. And the only other thing I'd highlight is the revered reputation that you described for your honest with inspiring and motivating other people.

50:45I think is true for you and your honest from everyone we've known. We know it. So three years until I have an agent that will write with memos for me. Hopefully. I think the memos might be coming sooner. Yeah, that was one of my burning questions. Is this like decades away? Is this month's away? It sounds like you're closer to the, you know, months or small number of years away. Think small number of years. Wow. Yeah, it, yeah, it's honestly kind of alarming, I think the speed at which the field is moving. And part of, yeah, part of depth and reliability, it's also like, it is, I mean, reliability is safety.

51:30So you want these systems to be safe. I think that there's a lot of very interesting research in terms of... There's a recent paper from Anthropic on kind of mechanistic interpretability and that whole line of work is really interesting and I think starting to kind of get to the point where there's utility in it as well in terms of fighting like neurons in a model that are like, you know, lying neurons or, you know, that you can kind of suppress. But to me, safety is reliability. If the thing is kind of running around your computer, breaking all sorts of things, that's an unsafe system. Maybe it's like a utilitarian safety.

52:14Like you just want these things to work and do what you intended them, but what you ask them to do. So I have a few years to find another hobby other than my memo writing then. Yeah, well, or maybe you'll just have an army of AI interns that, you know, will do all the research work for you. Get wait. Rapping out our topic around reflection, if everything goes right, what is your dream for reflection? I think there are two angles of this question. One is we're working on this because this is the kind of scientific root node problem of our time. We're scientists. That's why we're so kind of interested in committing it.

53:00And it's really there's a world where you get to be part of one of the most exciting journeys in science ever made. And you've accomplished your goal of building universal agent. You have highly safe, reliable digital agents running around on your computer. Basically doing things that tedious work that you don't necessarily want to do. I think rather than people kind of going and spending less time working, I don't think the human need or like, and the human need to be productive and to contribute is going to change. I just think the capacity of each human's ability to produce and impact the world is going to dramatically increase.

53:51As in my line of work there is researcher, there are so many things that I spend time on that a smarter AI could help me out with to make faster progress towards our own goal. I mean, this is kind of a circular, but if we had something close to a digital AGI, we'd get much faster to solving a problem with digital AGI. That's one angle. I think the other angle is from, I guess we've kind of moved on to the other angle. It's from the user perspective of a lot of the things that we do on a computer are, Maybe you can think about computers like the first digital tool that we've been introduced to as people.

54:38In the same way that there were hammers and chisels and sickles that people used. I think we're moving towards the kind of layer beyond that where instead of you having to learn how to use all these tools with great precision and spend all your time on this, which actually is kind of time taken away from achieving whatever personal goals people have, that you kind of have these incredibly helpful AI agents that can help you bring kind of any goal that you have to fruition. And I think it's very exciting because I think the kind of ambition of our individual goals is going is it's already increasing, you know, in this local sense, like the software engineer can get a lot more done today with these tools, but this is just the beginning.

55:36And I think we'll, yeah, we'll be able to really We set dramatically more ambitious goals for ourselves and for kind of these sorts of things we want to achieve simply because we can offload a lot of the work that's needed to get there to these systems. So these are some things I'm really excited about. We'll close it out with a few questions that we like to ask everybody about this state of AI. First question, what are you most excited about in your field or more broadly in the next one year or five years and ten years? I think there are a number of things that the one, the local one that comes to mind is because the paper is fresh.

56:25Is this kind of work on mechanistic interpretability? And that I mean these models are largely black boxes and It's unclear It's really unclear how to study them as like what what's the neuroscience of language models like if you think about them as brains Yeah, and this seems like a really interesting line of work that Is now starting to see kind of signs of both science of it working beyond toy settings. So maybe like the sort of neuroscience of language models is I think kind of a really interesting. Awesome. Feel an AI to get into it. And more generally, if I was in academia, I'd probably be looking a lot at the science of AI.

57:14So this is the neuroscience of AI is one thing, but they're all sorts of things. all sorts of things one can investigate in terms of, well, what really determines the scaling laws that these models have, both from a theoretical perspective and from an empirical perspective how you change data mixes, maybe I've taken a step back.

57:40We're basically in the equivalent of what the eight late 1800s looked like for physics. This electricity was being discovered. No one knew why it worked or how. There was a lot of empirical results, but there was no theory behind it, which just meant that they were not very well understood. Then this very rich set of theoretical models were developed that were very simple to understand these phenomena and that gave rise to actually basically the next wave of empirical breakthroughs. And so I think the science of AIs kind of in that state right now, and I'm very excited to see where that goes. So interesting.

58:32Who do you admire most in the world of A .I.? I think most people, when getting this question, or some people might kind of put someone, yeah, maybe I'll take that back and say like, I want to emphasize people that I admire based on having worked with them and kind of see them operate because through my last kind I have a number of years in AI. I think they're a handful of people like this who've inspired me. And one of them is, so Peter Abiel is certainly one of them. He is, I think, I've never seen anyone, I think, almost operate as efficiently as Peter. That was something that, and to date, like since meeting him and to date, like I think there's sort of, you think a lot about research as a creative pursuit oftentimes, and I think what I learned from Peter is just sort of operational competence and efficiency around this.

59:50He's very creative as well, and like his lab does a lot of creative work, but I think it's brought, like these things are very hard and they need to be pushed hard and with great focus. And he ran his lab against a Titus trip that I've ever been on. And really helped focus all the projects. And so, yeah, I think I look up to him a lot both in terms of his. The work that he's done, obviously, it's remarkably cross -field. like he's both done well incredible break to work and reinforce some learning and on supervised learning, generative modeling and and a lot of it has been I think from kind of recognizing and enabling like talent.

1:00:41So there is this very much. It was like the group was it was a bunch of independent thinkers, students, PhDs, And people are kind of pursuing, like, what was interesting to them. But the way I saw Peter as a sort of like great amplifier. Yeah. Like, he kind of helped people amplify and focus on the thing that really mattered within their pursuit. I think a few other people that come to mind. So my manager deep -mind, his name is Vladimir. He's, yeah, I think also like a very an incredible, very creative scientist. He was first author of the Deep QN paper and then there's actually two papers at the time.

1:01:30There's like this H2C, A3C papers. These are basically the two algorithms that defined reinforcement learning and he, of a deep reinforcement learning and he kind of really pioneered both. Yeah, I think his strength is like he was Extremely kind and people oriented very humble despite his accomplishments Yannis as well. I mean definitely. I mean Yannis has them the Michael Jordan effect. I think he really Yeah, he just You just wanted to be the best you could be when you work with him and And our early chef team was quite small, and people pushed really hard in order to, like, largely I think inspired by him.

1:02:17Yeah, these are some people I really look up to. Thank you for sharing that. It's so interesting to hear you say about it everyone, and one comment on Peter Beale. I tell him all the time that he's also just creating a mafia of fenders in the last couple of years, and it's probably because he's taught them how to do many things and there's a self -selection that's naturally happening. The creative thinkers and the independent thinkers who come into Islam, but he's also taught them a lot about how to run a tight ship and how to focus incredibly well. So I'm sure that doesn't come with that intention on his part.

1:02:53Maybe last question, any advice you have for founders building an AI, you are, you're just starting your journey right now and I'm sure you've asked others for a lot of advice, what advice would you pass on to the next generation? I think one, I think I'll be in a better position to answer that question in a few years in a way that is much more meaningful. But I'll actually provide a piece of advice that I led through my previous startup, which has nothing to do with AI, to just work on things that are like internally, like really matter to you. In a way that is almost independent from what's happening around you.

1:03:33Like, in a way that when things go bad, like it's still interesting to you, like it's kind of, there's just some fundamental drive around this problem that independent of everything else that's happening is just really interesting to you. Maybe I say that about our eyes because there's this, it's such an interesting, highly capable, cool technology and so there's this sort of I think appeal of taking it and just kind of like, always just see what we can do. I think you never really find yourself in a hard place without having a very strong internal compass of independently of AI, that what it is it's important to you and what you want to do.

1:04:15And so having been in that position and previously that's kind of what I would have done differently and what I would advise to do. I really love that. The line that I like to think about is playing your own stadium and don't get distracted by the glitz and glamour of someone else's stadium. You need that internal drive and grit and obsession with the problem to get you through all of tough times. Yeah. And I think there are things that come with it. with like, if you really care about some problem, like you will care about the customers you're solving the problem for. Like having customers that you don't care about is like a terrible place to be.

1:04:57So I think yeah, it has to come. And it's not like, I think it's kind of actually hard to control like who you cared, who you don't care about. That's like a personal thing. So you can't, it's actually really hard to force yourself to like care about something out of necessity if it's kind of not aligned with something inside you already. So no more shopping and retailers for Misha? Yeah, so I was a building software for inventory prediction for retailers. And for some people, they'd really care about that problem. There's a reason they would. They've seen that problem. Maybe if you're a merchant at one of these retail companies, You've really felt it viscerally.

1:05:41And in my case, you know, I hadn't, who was like we were trying to make, you know, just trying to build a revenue -generating business almost kind of independent of like an internal compass. Be sure, thank you so much for joining us today. You are working on the most ambitious problem of our time. I love your framing of the root node problem of our time. And I think that is today agents. And it's very clear that both you and Yonis' experiences make you the very best team at what you do. Yonis, obviously, from a RLHF perspective and yours, from a reward model training perspective. And the insights and experiences that you've both had working on AlphaGo, AlphaZero, and Gemini were so excited for the future of reflection.

1:06:32Yeah, thank you for having me.