ReflectionAI Founder Ioannis Antonoglou: From AlphaGo to AGI

0:00Because it was a complex game, there was always a bit of worry about whether Africa was truly as good as we believed. So we actually had the conviction that the deep reinforced learning is the answer based on everything that we could measure and everything we could see. But that's the thing about this system is that there are not like classic computers where you just like know that they always produce the same answer. They're like stochastic, they're creative, and they have like some blinds, they hallucinate like similarly to how like model LLM's callousness. So you need to just like really push them and just like see exactly where the grape and the only way you could actually do that is by having like the best humans playing against them.

0:57Today we're excited to welcome Janis Antenoglo, a researcher and an engineer who has contributed to some of the most significant breakthroughs in AI. As a founding engineer at DeepMind, John has played a crucial role in developing AlphaGo, which made history by defeating Go World Champion, Leisidol. He later co -led the development of Museurio, which pushed the boundaries even further by mastering multiple games autonomously. Now, as he embarks in his latest venture with Reflection, he's focused on building the next generation of AI agents. We're excited to talk to Johnus about the breakthrough moments in AI history that he's witnessed firsthand.

1:38From AlphaGo's famous Move 37, his perspective today on what's next for the combination of reinforcement learning and large language models on the way to AGI. Janice, thank you so much for joining us today. Thank you so much for having me. Janice, you have an incredible background having worked at DeepMind as a founding engineer for over a decade, starting with some of the most notable projects that have really defined the industry. DeepMind quite notably created this notion of building AI within games to start. Can you share a little bit more about why DeepMind chose to start with games at the time?

2:18Yeah, so DeepMind was the first company to truly embrace the concept of artificial general intelligence or AGI. from the outset, they had ground ambitions, aiming to build systems that would match or exceed your intelligence. So the big question was, and still is, how do you build AGI? And more importantly, how do you measure intelligence in a way that allows for meaningful reasons and performance improvements? So the idea of choosing video games was that testing ground came naturally to the point founders. It was a Demisos Habits and Shane Lake. Because Demis had a background in the gaming industry and changed PhD thesis to find AGI as a system that could learn to complete any task.

2:56Video games provided a control yet complex environment where these ideas could be explored and tested. And to what extent he mentioned games are, they provide a very controlled environment. To what extent are games representative or not of the real world? Like if you have a result in games, do you think that generalizes naturally to the real world or not? So I mean, I guess games have indeed been viable for developing AI. And you actually have like a few examples of that. So you can see that PPO for example, which is currently being used in RLHF was developed using OpenAI, GIM and Michoko and Atari and similarly, you have like MCTS which was developed which stands for Monte Carlo 3SH and was developed through port games like Pac -Am and Go but at the same time games have like a number of limitations so the real world is and Messi is a bounded and it's a much tougher not to crack than even the most complex games.

3:51So even though they just give you an interesting aspect to develop new ideas, it's definitely limiting and it does really capture all the complexity of the real world. Okay, interesting though. So a lot of the techniques and algorithms that you've developed in the game environments, TPO, etc. These are used in the real world. Yeah, so PPO was actually exactly what just PD used for RLHF. And so MCTS, it's used in Yuzio and Yuzio has been used in like the real world in things like compression video compression for YouTube. In it was part of the cell driving system, Tesla at some time. And it was also like used for developing a pilot, like that was completely controlled by an AI.

4:43So yeah, I mean, you can see it methods like that being used in the world to solve their problems. So interesting. Yannis, I remember back in 2017, when AlphaGo, the movie, came out and it featured the incredible game of AlphaGo against LisaDoll. Can you take us back to that moment in time and maybe the year is leading up to it as you're building AlphaGo? How was AlphaGo specifically chosen as the game to focus on? So I think I'm like games you've always been a benchmark for AI research so like before Go you had chess and chess was like a major milestone with like IBM's deep blue defeating Gargaz Parof in the late 90s and And even though chess and go are completely different games and go with a definitely different beast, there is, like, games have always been acted as test beds for the development, especially board games for the development of new AI methods.

5:40Actually, even going back to the earliest days of AI research, Turing and Shannon, they both worked on their own versions of chessboards. So now the thing about like go is that it's a much harder problem than like chess. The reason for that is because it's almost closely possible to define an evaluation method, a heuristic. So in chess, you can just like take a look at the boards, you can count the number of pawns that like it side has, you can see what the ranks of these pawns are, and then you can just like make some, you can draw some conclusions of like who is winning and why. But I can go there's nothing like that.

6:18Like it's mostly human intuition. And if you ask a go professional player, like how they know whether a position is a good partner, but one they would say that after having playing the game for so long, they could just like fill it in their gut, but like this is a better position on the other one. So now there's actually a question of how do you encode the feeling in your gut into like an AI system, right? So this is exactly the reason why solving Go was considered the holy grail of air research for a long time. And it was the challenge that seemed almost impossible, but at the same time it was like within reach, people felt that they could actually get it cracked.

6:56And this exactly what Alphogo did back in 2016. And it kind of showed case two new methods, which is deep learning and reinforcement learning. Because back in 2015 and 2016, now we kind of think of deep learning and reinforcement learning as part of technologies, but back then we were kind of literally making their first steps and they were kind of like the new kid in the block and most people were kind of like really skeptical about them. That deep learning was another AI fad that would just like one last the test of time. So yeah, I mean, AlphaGo was chosen because it was like clear to show the that you actually have like the most the most performant agent in the world.

7:41You could actually evaluate it. You can have it play with other humans. And at the same time, it was within reach given like the latest developments in deep learning and reinforcement learning. I remember reading that. There's more configurations of the Go board than Adam's and the universe. By many letters and nine to them, that blew me away. I mean, I grew up playing Go and it felt like such a, you know, it's a very simple rule in terms of the rules. But I see why it was the Holy Grail. Maybe can you explain how AlphaGo worked technically maybe explains me like I'm a fifth grader because that is effectively my level of sophistication understanding these things.

8:18But how did it work? And you mentioned both reinforcement learning and deep learning were involved. But love to peel that back a little bit. Yeah, absolutely. So AlphaGo has two deep neural networks. So like a neural network is a function that like takes something as an input, input and produce something as an output. And it's literally like a black box. We don't really know exactly how it does it. Just like know that you can actually, if you train it on enough data, we'll just like learn the mapping. If we learn the function from input to the output space. So AlphaGo actually had access to two deep neural networks, the policy network and the value network.

8:53And the policy networks suggested the most promising move. So it will just take a look at a current port position. And we'll just like say, okay, you know, based on the current position, this is the list of moves that I would recommend you just consider playing. And it also had access to the volume network, we just take a look at a port position, and just give you a winning probability. What are your chances of actually winning the game starting from this position? This is exactly the gap feeling. It had its own gap feeling on whether the position is a good one or a bad one. So once you have access to these two networks, then you can actually play in your imagination a number of games.

9:30You can consider the most promising moves, then you can consider your opponents most promising moves And then you can just evaluate each move like the value network And then you can use a method called minmax. What that says is that I want to win the game But I also know that my opponent wants to win the game So I want just like pick a move that will maximize my chances of winning knowing that like my opponent will try to maximize their chance of winning. So if you actually like to do that and simulate a bunch of moves, then you can just like get the optimal action. And you know, the way to just like do this imagination, this planning, this search in the most efficient way is by using a tree sets method called Monte Carlo tree sets.

10:15So MCTS. So whenever people talk about MCTS, they literally just like mean this heuristic of how do I choose which features to consider so that I can make informed decisions. The role for reinforcement learning and deep learning and building AlphaGo was that AlphaGo first fall was a success of reinforcement learning and deep learning because this is exactly the two methods that powered AlphaGo. And the policy network was initially trained on a large a set of human games. So you had like many games played by human professionals and you just like consider every position and you consider the move they took at this position and then you have like a dipping in a network that tries to predict this move.

11:01Then once you have the policy network you need to somehow find a way to just like obtain a value network. So we did it in two ways. First we just took the policy network and we had it play against itself and we used reinforcement learning to improve the blank strength of the model. So we use a technique called policy gradient. So what policy gradient does is that it just looks at the game and then it looks at the outcome. This is the simplest version, like of policy gradient. It looks at the outcome of the game and for all the moves that led to a win, they'll just like say great, you know, just increase the probability of choosing this move.

11:38And for all the moves that led to a loss, it says great, now it's a decrease the probability of like this being selected in the future. And if you do that like, you know, for many games and for long enough, then you just like it and improve policy. Now, once you have this improved policy, you can just generate a new dataset of games where like the policy plays against itself and then you have like a huge amount of games where for its position, you know who the final winner was. So then you can take this network, you can take another network, a value network, and here it's predict the outcome of the game based on the current position.

12:12So what the network could learn is that if I start this position and I play under my current policy, on average this is the player who wins, like it's either a black player or the white player. So this is the first version of like a value network and you can just like give it within half a goal by combining it with the policy network. And what were some of the biggest challenges in building this and how did you overcome them? Yeah, so AlphaGo was not just a listed scientist but mostly I'd say an engineering marble. The early versions run on 1200 CPUs and 176 GPUs and the version that played against listed old used 48 GPUs.

12:52So like GPUs for like the first accelerator, custom accelerators. And these accelerators were like really primitive back then, because literally it was like the first version, right? Like now the later accelerators are much, much better and much, much more stable. So the system had to be highly optimized to minimize latency, maximize throughput. We had to build landscape infrastructure for training these networks, and it was a massive endeavor. It just required a lot of coordinated effort for many talented individuals working on different aspects of the project. But I just like walked you through a number of steps, just like obtained the Polish network and the value network.

13:32And each of these steps had to just be implemented at the limits of what was possible back then in terms of scale. And it had to be implemented in a way where people could just like think it with it, they could just like try the research ideas fast and get results fast. So yeah, lots of people scale in, you know, at levels that hadn't been implemented before. And it's kind of like working at the forefront of what was possible back then. I love your highlight of it being a research marvel and an engineering marvel. And I remember you sharing one time that part of the reason this project came about also was because Google had TPUs that they needed to, they needed a test customer for.

14:16and that was the spark. This album is AlphaGo project, so that's pretty incredible. How much conviction did the DeepMind team have that this is going to work? You mentioned that at the time, deep learning, reinforcement learning were still relatively novel, but DeepMind was very much founded with that belief. But did you guys think that you were going to be able to have these superhuman level results beating the top go player in the world? Was it a crazy idea and maybe they'll work or did the team have conviction like this is going to work? So I'd say I'd like to tell you the team had a cautious optimism.

14:49So one of AlphaGo's lead developers, Ajah Huang, he is a strong amateur go player and he had been working on Go for like a decade before AlphaGo happened. And we also had like a little report of computer games, a lot of computer players and you could say that AlphaGo was significantly stronger than anything that had come before. But Go is a complex game and there was always a bit of worry about whether AlphaGo was as true as good as we believed. So we actually had the conviction that the deep reinforcement learning is the answer based on everything that we could measure and everything we could see.

15:23But the thing about this system is that they're not like classic computers, where you just like know that they always produce the same answer. They're like stochastic. They're creative. So they all have like some blind spots. they hallucinate, like similarly to how like model LLM's hallucinate. So you need to just like really push them and just like see exactly where they break and the only way you could actually do that is by having like the best humans playing against them. Move 37, can you tell us what that was? It was such a monumental move and I think everyone watching it at the time it was and at least all maybe primarily was confused by that move.

16:08What was going on in your head when that happened? So yeah, I mean, move 37 in game 2 against Lise -Dol was literally just a spectacular moment in the sense that it kind of showed gaze to the world that AlphaGo has creativity and it demonstrated that AI could come up with shots that even top human players hadn't considered. So at first, like I still remember that, we thought that AlphaGo made an error, so that it actually hallucinated. It did something that it didn't mean to do. But then, turn out to be a brilliant conventional move that underscored that the system had a deep understanding of the game.

16:46That the system actually had creativity. It's good thing of things that people hadn't thought of before. I want to take us to another key move in the game. I think it was in game four. At this point, I was rooting for Lee because I was like a vortga, I used to win a game. I moved 78. I think AlphaGo made a mistake and Lisa don't know this is it. I guess what was the weakness there that Lee found during the game? Yeah exactly. So I mean Lisa Dole's victory in game four was literally a testament to human ingenuity. Like move, move 78 was unexpected and got off a go off card. Initially I have a goal like based on its evaluations, misinterpreted as a mistake and thought that it was actually like winning.

17:28So that's why it is to respond appropriately. And you know, this kind of highlighted the blind spot in the system. So the game shows that like while systems like Africa are extremely powerful at the same time, they still have vulnerabilities and there were like steel areas where it could further improve it. But how do you go about improving something like that? Do you need to show it a lot more data of, you know, kind of that type of human ingenuity move or how do you go about fixing and and patching those blind spots. So yeah, I mean, it's actually interesting that by the end of the games, like this, we just like put together a benchmark.

18:04We're just kind of like trying to quantify and just have a way of measuring the mistakes that like Africa makes. And you know, this is kind of blind spots, let's say. And then we just write an other approaches to just like the prove the algorithm so that we can solve these issues. And what happened is that actually the most effective way of getting rid of them was just like do what we were doing, just like at a higher scale and better. So we just like changed the architecture of the model, we just like switched to a deep rest net with two output heads. And we also like, we just had a bigger network trend or more data, then just like move 12 Alpha Zero and better algorithms and that kind of like made it so that we didn't have any hallucinations anymore.

18:55So in a way, which is like scale data, you know, things that are always kind of the well -known recipe in the field of AI. It's exactly what sold it in our days too. With scale and data, how much did higher quality data or maybe specifically data from great professional players, the best professional players make a meaningful difference? Or was it just any data? Now for us what matter was that we kind of like solved it using self -play. So we actually had access to the most competent go player in the world. And we just like used it to generate the best quality games and then we just trained on these games.

19:37So I guess like you know, we didn't need to have like human experts because you had like an expert in house. It wasn't human. Right. Huh. Interesting. Amazing. Well, I'd love to move on to the progression from AlphaGo to Alpha0. And you talked a little bit about this notion of self -play just now. Alpha0 was powerful because it learned how to play the game from scratch entirely from self -play without any human intervention. Can you share more about how that worked and why that was important? So Alpha0 was a game changer because it learned entirely from scratch through self -play without any human data.

20:13And this is like a major relief from AlphaGo, because AlphaGo, as I said, relied heavily on human expertise. So two things happened. First of all, Alpha0 managed to simplify the training process and also showed that AI will literally just get from zero to superhuman performance just purely by playing against itself. And that allowed it to just be applicable to a whole range of new domains that were out of reach, because there weren't enough human data for it. But the more important thing is that we just saw that Alpha Zero also solved all the issues that AlphaGo had in terms of hallucinations, in terms of plant spots and robustness.

20:58So like Alpha Zero was a better method, just full style. And you explained kind of how AlphaGo worked to a fifth grader. What would you tell the fifth grader to be the key difference technically that you've implemented with Alpha 0. So Alpha 0, just like Alpha Go, uses a policy network and a value network, along with motor carotry sets. So in that respect, it's exactly the same as Alpha Go. So the key difference is in training. Alpha 0 starts with random weights and lands by playing games against itself. And by playing games against itself, it iteratively improves its performance. But the main idea behind Alpha 0 is that whenever you take a set of weights, a set of policy and value networks.

21:41And then you just combine the moussech, then you just end up with a better playing better player, which is like increase your performance, you just become a stronger player. So what that meant is that we can actually use this mechanism to improve the model policy, the role policy. So this is what we call in reinforcement learning, a policy improvement operator. Whenever you can just take an existing policy and then do something, some magic, and then just like come up with like a better policy, and then you can just like take this policy and distill it back to the initial policy, and just repeat this process, then you have like a reinforcement learning algorithm.

22:21And I think it's like, you know, this is exactly what people are trying to do today, with like, you know, two -star or like, you know, synthetic data. This is exactly the idea of like, how can I take a policy, do something with it, planning, search, compute, whatever it is, and derive a better policy, which I can then imitate and just like kind of distill back to the original policy. So this is exactly what Alpha Zero is doing. It uses MCTSH to produce a better policy than it takes the trajectories, it trains this policy in value network on the new better trajectories and it repeats this process until it converges to an expert level go player.

23:02That's fascinating and counter -intuitive that kind of like starting without the weights that you would have from, you know, professional level players is actually a better starting place. The epitome of AI agents and games, it's cheap to think via MuZero, which the progression even from AlphaZero itself, and it's also where you became one of the co -leads or one of the leads of the game. AlphaZero was obviously impressive because of self -play, but it also needed to be told the environments dynamics or the rules of the game. And Muzero takes this to the next level without needing to be told the rules of the game and it mastered quite a few different games, Go Chas and many others.

23:46Can you share a little bit about how Muzero worked and why was this particularly meaningful? Absolutely. So Alpha Zero, as you said, was a massive success in games like chess, go, Shogi. So in games where we actually had access to the game rules, where we actually had access to a perfect simulator of the world. But like these two lands on the perfect simulator, made it challenging to apply to real world problems. And real world problems are often messy and they like the rules and truly hard just like write a perfect simulator of them. So that's exactly what Muzio tried to solve. So Muzio, masters the games, of course, like Go chess and Shogi.

24:26but also like matches more visually challenging games or games that are like a hotgoat like a Tari. And it does that without giving access to the simulator, just like lens how to beat an internal simulator of the world and then just use this internal simulator in the way similar to what Alpha Zero is doing. So it does that by using model based for enforcement learning, where what that means is that you can just take a number of trajectories generated by an agent and then try and, you know, So, Lennon model will end a prediction model of how the world works. So this is actually quite similar to what methods like Sora are trying to do now, where they just like take YouTube videos and they try to just like Lennon world model, but just trying to predict based on something from one frame what's going to happen in the future frames.

25:13So, Nusiel tries to do exactly that, but it does it in a way different from, you know, of the genetic models in the sense that they try to only model things that matter for solving the reinforcement learning program. So it tries to predict what the rewards are going to be in the future, what's the value of like future states, what's the policy for like future states. So only things that you need to thin your MCTS. But the fundamental is kind of like remain the same. So how do you just like learn a model based on trajectories and then once you have this model, you can just combine it with search and get super human performance.

25:51So of course, you can always decouple the two problems and have the model been trained separately from data out in the wild and then just like combine that with Museo. And we just found that back then, given the limitations of our models and the smaller sizes, is kind of like made more sense to just like keep those two together and only have the model predict things that matter for planning. So just like try to model everything because you're kind of hitting the limits of what the capacity of the model could take. So interesting. Is it right to assume that not only Sora, you know, takes the same approach, but maybe other world models or other robotics foundation models?

26:35Yeah. So anything that tries to just like build a model of how the world works and then just like use that for climbing. It's within you know, you zero like methods. So yeah, you can just like train it on YouTube videos. You can train it on like the inputs coming from like robots. You can train it on any environment. You can even think of like class language models as a form of models of like text. So like they they the model text. But the thing about text is that like the more than a little bit trivial, like you don't need to just, don't have many artifacts happening when you're trying to predict what the next world is going to be.

27:13Have you seen the ideas behind Museo kind of be used outside gameplay or in messy real -world environments?