Shane Legg (DeepMind Founder) — 2028 AGI, superhuman alignment, new architectures

0:00Okay, today I have the pleasure of interviewing Shane Legg, who is a founder and the chief AGI scientist of Google DeepMind. Shane, welcome to the podcast. Thank you, please be here. So first question, how do we measure progress towards AGI concretely? So we have these lost numbers and we can see how the loss improves, certain one models to another, but it's just a number. How do we interpret this? How do we see how much progress we're actually making? That's a hard question, actually. AGI, by its definition, is about generality. So it's not about doing a specific thing. It's much easier to meet your performance when you have a very specific thing in mind because you can construct a test around that.

0:44Well, maybe I should first explain what do I mean by AGI? Because there are a few different notions around. When I say AGI, I mean a machine that can do the sorts of cognitive things that people can typically do. possibly more, but that's to be an age I that's kind of the buy you need to meet. So if we want to test whether we're meeting the straight shoulder, we're getting closer to the straight shoulder, what we actually need then is a lot of different kinds of measurements and tests of all that spans the breadth of all the sorts of cognitive tasks that people can do and then to have a sense of what is human performance on these sorts of tasks and that then allows us to sort of judge whether or not we're there.

1:29It's difficult because you'll never have a complete set of everything that people can do because it's such a large set. But I think that if you ever get to the point where you have a pretty good range of tests of all sorts of different things that people do, cognitive things people can do, and you have an AI system which can meet human performance and all those things. And with some effort, you can't actually come up with new examples of cognitive tasks where the machine is below human performance, then at that point, it's conceptually possible that there is something that the machine can't do, that people can do, but if you can't find it with some effort, I think we'll practical purposes.

2:10You know, I have an AGO. So let's get more concrete. We measure the performance of these large English models on MMLE or something, and maybe you can explain what all these different and benchmarks are, but the ones we use right now that you might see in a paper, what are they missing? What aspect of human cognition do they not measure adequately? Ooh, yeah, another hard greeting. These are quite big areas. So they don't measure things like understanding streaming video, for example, because these are language models and people can do things like understanding streaming video. They don't do things, humans have what we call episodic memory.

2:52So we have a working memory which are things that have happened quite recently and we had sort of a cortical memory. So these are things that have sort of been in our cortex that have been. But there's also a system in between which is episodic memory which is the hippocampus. And so this is about learning specific things very, very rapidly. So some of the things I say to you today, if you remember them tomorrow, that will be your episodic memory hippocampus. Our models don't really have that kind of thing and we don't really test for that kind of thing. We just sort of try to make the context windows which is I think more like a working memory longer and longer to sort of compensate for this.

3:27But yeah, we don't really test for that kind of a thing. So there are there are also a bits and pieces but you know it is it is a difficult question because you really need to as I said intelligence, the generality of human intelligence is very broad. So you really have to start going into the weeds of trying to find you know there's specific types of things that are missing from existing benchmarks or different categories of benchmarks that don't currently exist or something. The thing you're referring to with a besiding memory, would it be fair to call that sample efficiency or is that a different?

4:00It's very much related to sample efficiency. It's one of the things that enables humans to be very sample efficient. Large language models have a certain kind of sample efficiency because when something's in their context window, they can then that sort of biases the distribution to behave in a different way. And so that's a very rapid kind of learning. So there are multiple kinds of learning and the existing systems have some of them but not others. So it's a little bit complicated. So this kind of memory or we call it sample efficiency, whatever, is it a fatal flaw of these deep learning models that it just takes trillions of tokens, far more, many orders of magnitude more than a human will see throughout their lifetime or is it something that just solved over time?

4:46So the models can learn things immediately when it's in a context window and then they have this sort of this longer process of when you actually train the base model and so on and that's they're learning over trillions of tokens but they sort of miss something in the middle. That's sort of what I'm getting out of here. I don't think it's a fundamental limitation. I think what's happened with large language models is something fundamental has changed. We know how to build models now that have some degree of, I would say, understanding of what's going on. And that did not exist in the past. And because we've got a scalable way to do this now, that unlocks lots and lots of new things.

5:28Now, we can then look at things which are missing, such as this sort of episodic memory returb thing and we can then start to imagine ways to address that. So my feeling is that there are kind of relatively clear paths forwards now to address most of the shortcomings we see in existing models, whether it's about delusions, factuality, the type of memory and learning that they have or understanding video or all sorts of things like that. So I'm not actually, I don't see there are big blockers here. I don't see big walls in front of us. I just see there's more research and work and these things will improve and probably be adequately solved.

6:09But going back to the original question, how do you measure when human level AI is arrived over beyond it? As you mentioned, there's these other sorts of benchmarks you can use and other sorts of traits. But concretely, what would it have to do for you to be like, okay, we've reached human level? Would it have to beat Minecraft from start to finish? Would it have to get 100 % on MMLU? What would it have to do? So there is no one thing that would do it because I think that's the nature of it. It's about general intelligence so it has to make sure it could do lots and lots of different things and it didn't have a gap.

6:42We already have systems that can do very impressive categories of things to human level or even beyond. So I would want a whole suite of tests that I felt was very comprehensive and then further the more when people come and say, okay, so it's passing a big suite of tests, let's try to find examples. Let's take an adversarial approach to this. Let's deliberately try to find examples where people can clearly typically do this with the machine fails. And when those people cannot succeed, I'll go, okay, we're probably there. A lot of your earlier research, at least on that exact I find, emphasized that AI should be able to manipulate and succeed in a variety of open -ended environments.

7:28It kind of sounds like a video game almost. Is that where our head is still at now, or do you think about it differently? Yeah, it's involved a bit. When I did my thesis work around universal intelligence and so on, I was trying to come up with a sort of extremely universal, general, mathematically clean framework for defining and measuring intelligence. And I think there were aspects of that that were successful. I think in my own mind it clarified the nature of intelligence, as being able to perform well in lots of different domains and different tasks and so on, it's about that sort of capability of performance and the breadth of performance.

8:13So I found that was quite helpful in lightening. There was always the issue of the reference machine because in the framework you have a waiting of things according to the complexity. It's like an ocum's razor type of thing where you wait tasks, environments, which are simpler more highly in this sort of, because you've got an infinite, it's a countable space of different computable environments. I've seen And that combo of complexity measure has something built into it, which is called a reference machine. And that's a free parameter. So that means that the intelligence measure has a free parameter in it.

9:01And as you change that free parameter, it changes the weighting and the distribution over the space of all the different tasks and environments. So this is sort of an unresolved part of the whole problem. So, what reference machine should we ideally use? There isn't really a, there's no universal, like one specific reference machine. People will usually put a universal chewing machine in there, but there are many kinds of universal chewing machines. So you have to put it, so a universal chewing machine, but there are many different ones. So I think, given that it's a free parameter, I think the most natural thing to do is say, okay, Let's think about what's meaningful to us in terms of intelligence.

9:44I think human intelligence has meaningful to us and the environment that we live in. We know what human intelligence is, we are human, we interact with other people of human intelligence. We know that human intelligence is possible, obviously, because it exists in the world. We know that human intelligence is very, very powerful because it has affected the world profoundly in countless ways. And we know if human level intelligence was achieved, that would be economically transformative because this type of cognitive task people do in the economy could be done by machines then. And it would be philosophically important because this is sort of how we often think about intelligence.

10:25And I think historically it would be a key point. So I think that human intelligence is actually quite in a human -like environment as quite a natural sort of reference point. So you could imagine sort of seating your reference machine to be such that it emphasizes the kinds of environments that we live in as opposed to some abstract mathematical environmental something like that. And so that's how I've kind of gone on this journey of let's try to define a completely universal, clean mathematical notion of intelligence to, well, Now it's got a free parameter. One way I think about it is say, okay, let's think more concretely now about human intelligence and can we build machines that can match human intelligence because we understand what that is and we know that that is a very powerful thing.

11:12It has economic, philosophical, historical kind of importance. So that's kind of the, and the other aspect of course is that, you know, in this pure formulation of combo -grow complexity, it's actually not computable. And I also knew that there was a limitation at the time, but it was an effort to say, okay, can we just even very theoretically come up with a clean definition? I think we can sort of get there. We have this issue of a reference machine, which is unspecified. So before we move on, I do want to ask on the original point you made about these machines, or these LLMs need an episodic memory.

11:51You said that these are problems that we can solve. These are not fundamental impediments. But when you say that, do you think they'll just be solved by scale or do each of these need a fine grain -specific solution that is architectural in nature? I think it'll be architectural in nature because the current architectures, they don't really have what you need to do this. they basically have a context window which is very, very fluid of course and they have the weights which things get baked into very slowly. So to my mind that feels like working memory which is like the activations in your brain and then the weights, the synapses and so on in your cortex.

12:31Now the brain separates these things out. It has a separate mechanism for rapidly learning specific information because it's a different type of optimization problem compared to slowly learning deep generalities. That's sort of there's a tension between the two, but you want to be able to do both. You want to be able to, I don't know, hear someone's name and remember it the next day, and you also want to be able to integrate information over a lifetime so you start to see deeper patterns in the world. These are quite different different optimization targets, different processes, but a comprehensive system should be able to do both.

13:12So I think it's conceivable you could build one system does both, but you can see because they're quite different things that it makes sense for them to be different. I think that's why the brain does it separately. I'm curious about how concretely you think that would be achieved. And I'm specifically curious, I was going to answer this as part of the answer. Deep mind has been working on these domain -specific reinforcement learning types setups, alpha -full, alpha -code, and so on. How does that fit into what you see as a path to AGI? Have these just been worth arguing all domain -specific models, or do they feed into the eventual AGI?

13:49Things like alpha -fold are not really feeding into AGI. We made learn things in the process that may end up being relevant, but I don't see them as being likely being on the path to AGI. But yeah, we're a big group. We've got hundreds of hundreds of PhDs. We're going to lots of different projects. So when we find what we see opportunities to do something significant like alpha -fold, we'll go and do it. It's not like we only do agi -type work. We work on fusion reactors and various things in sustainability, energy. We've got people looking at satellite images of deforestation. We have people looking at with the forecasting.

14:40We've got tons of people. We've got lots of things. On the point you made earlier about the reference class or the reference machine as human intelligence, it's interesting because in your 2008 thesis, one of the things you mentioned almost as a side note is, well, how would you measure intelligence? And you said, well, you could do a compression test. and you could see if it fills in words and a sample of text, and that could measure intelligence. And it's funnily enough that's basically how the ones are trained. At the time, did it stick out to you as especially fruitful thing to train for? Well, yeah, I mean, and it seems what's happened is actually very aligned with what I write about my thesis, which is the ideas from Marcus Hutter with AIXC, where you take salomon -off induction, which is this incomputable, but in a theoretically very elegant and extremely sample efficient prediction system.

15:36And then once you have that, you can build a general agent on top of it by basically adding search and reinforcement signal. That's what you do with AICC. But what that sort of tells you is that if you have a a fantastically good sequence predictor, some approximation of salonal conduction, then going from that to a very powerful, very general AI system is just sort of another step, you've actually solved a lot of the problem really. And I think that's what we're seeing today actually, that these incredibly powerful foundation models are incredibly good sequence predictors, they're compressing the world based on all this data, And then you will be able to extend these in different ways and build very, very powerful agents out of them.

16:25Okay, let me ask you more about that. So, Richard Sutton's bit or less an essay says that there's two things you can scale, search and learning. And I guess you could say that Elems are about the learning aspect. The search stuff which you've worked on throughout your career where you have an agent that is interacting with this environment and is that the direction that needs to be explored again, or is that something that needs to be out of tele -lums, where they can actually interact with their data or the world or in some way? Yeah, I think that's on the right track. I think there these foundation models are world models of a kind, and to do really creative problem solving, you need to start searching.

17:10So if I think about something like AlphaGo in the move 37, the famous come from all its data that it's seen of human games or something like that. No, it didn't. It came from it identifying a move as being quite unlikely, but you know, it's possible. And then via a process of search, coming to understand that the, that was actually a very, very good move. So you need to get real creativity. You need to search through spaces of possibilities and find these sort of hidden gems. That's what creativity is. I think current language models, they don't really do that kind of a thing. They really are mimicking the data, they are mimicking all the human ingenuity and everything which they have seen from all this data that's coming from the internet that's originally derived from humans.

18:02If you want a system that can go truly beyond that and not just generalize the novel ways so these models can blend things, they can do Harry Potter in the style of a Kanye a West Rapp or something, even though it's New Haven, because they can blend things together, but to do something that's truly creative, that is not just a blend in the existing things, that requires searching through a space of possibilities and finding these hidden gems that are sort of hidden away in the S &W. And that requires search. So I don't think we'll see systems that truly step beyond their training data until we have powerful search in the process.

18:42So there are rumors that Google DeepMind is training newer models and you don't have to comment on those specifically, but when you do that, if it's the case that search or something like that is required to go to the next level, are you training in a completely different way than CGPT4 or other Transformers are trained? I can't say much about how we're training. I think it's for you to say we're doing the sorts of scaling and training roughly that you see many people in the field doing. But we have our own take on it, our own different tricks and techniques. Okay, maybe we'll come back to it if we get another answer on that.

19:21But let's talk about alignment briefly. So what will it take to align human level and superhuman AIs? And it's interesting because the sorts of reinforcement learning and self -fleekeeping of setups that are popular now, like Constitution AI or RLHF, DeepMind obviously has expertise in it for more decades longer. So I'm curious what you think of the current landscape and how DeepMind pursues that problem of safety towards human level models. So do you want to know about what we're currently doing? Or do you want me to have a stable, what I think we need to be done. Needs to be done. Needs to be done.

19:57So I mean, what in two of what we're currently doing, we're doing lots of things. We're doing interpretability. We're doing our process supervision. You know, we're doing a red teaming, we're doing a valuation for dangerous capabilities, we're doing work on institutions and governance and tons of stuff, right? There's lots of different things. Anyway, what do I think needs to be done? So I think I think that powerful machine learning powerful AGI is coming in some time. Right. And if the system is really capable, really intelligent, really powerful, trying to somehow contain it or limit it is probably not a winning strategy because these systems ultimately will be very, very capable.

20:39So what you have to do is you have to align it. You have to get it so it's fundamentally a highly ethical, value aligned system from the get go, right? How do you do that? Well, I have a, maybe this is a slightly naive, but this is, this is my take on it. How do people do it? Right? If you have a really difficult ethical decision in front of you, what do you do? Right? Well, you don't just do the first thing that comes to mind, right? Because you know, there could be a lot of emotions involved in other things, right? There's a difficult problem. So what you have to do is you have to calm yourself down, you've got to sit down and you've got to think about it.

21:22You go think, well, okay, what could I do? I could do this, I could do this, I could do this. If I do each of these things, what will happen? And then you have to think about, so that requires a model of the world. And then you have to think about ethically, how do I view each of these different actions and the possibilities? And well, what may happen from it? What is the right thing to do? And as you think about all the different possibilities and your actions and what can follow from them and how it aligns with your values and your ethics, you can then come to some conclusion of what is really the best choice that you should be making if you want to be really ethical about this.

22:07I think AI systems need essentially do the same thing. So when you sample from a foundation model at the moment, it's like, it's bloating up the first thing. It's like system one, if you like, from psychology, from government, right? That's not good enough. And if we do RLHF, or what's it called, I can't remember. Anyway, it's the AI version without the human feedback. R, AIF, is that what it is? Oh gosh, I'm confusing myself. Anyway, constitutional AI tries to do this sort of thing. You're trying to fix the underlying system one in a sense, right? And that can shift the distribution, and that can be very helpful, but it's a very high dimensional distribution, and you're sort of poking it in a whole lot of points.

22:51And so it's not likely to be a very robust solution, right? It's like trying to train yourself out of a bad habit. You know, you can sort of do it eventually. Actually what you need to do is you need to have a system too. You need the system to not just sample from the model, you need the system to go, okay, I'm going to reason this through, I'm going to do step by step reasoning. What are the options in front of me? I'm going to use my world model now and I'm going to use a good world model to understand what's likely to happen from each of these options and then reason about each of these from an ethical perspective.

23:26So you need a system which has a deep understanding of the world, has a good world model. It is a good understanding of people, it has a good understanding of ethics, and it has robust and very reliable reasoning. And then you set it up in such a way that it applies this reasoning and this understanding of ethics to analyze the different options which are in front of it, and then execute on which is the method is a difficult way forwards. But I think when a lot of people think about the fundamental alignment problem, The worry is not that it's not going to have a world model necessary to understand its actions.

24:02I'm sorry to understand the effects of its actions. I guess it's one worry, but not the main worry. The main worry is that the effects it's cares about are not the ones we will care about. And so even if you improve its systems you're thinking and do better planning, the fundamental problem of we have this really nuanced values about what we want. How do we communicate those values and make sure they're reinforced in the AI. It needs not just a good model of the world, but it needs to say really good understanding of ethics. And we need to communicate to the system what ethics and values it should be following.

24:36And how do we do that in a way that's, we can be confident that a human level or eventually a super human level model will preserve those values or learn them in the first place. Well, it should preserve them because if it's making all its decisions based on a good understanding of ethics and values, and it's consistent in doing this, it shouldn't take actions which undermine that, there would be, there would be a consistent. Right, so then how do we get to the point where it's learned them in the first place? Yeah, that's the challenge. Yeah. We need to have systems. The way I think about it is this, to have a profoundly ethical AI system, it also has to be very, very capable.

25:12It needs a really good world model, a really good understanding of ethics, and it needs really good reasoning. Because if you don't have any of those things, how can you possibly be consistently profoundly ethical? You can't. So we actually need better reasoning, better understanding of the world, and better understanding of ethics in our systems. Right, so it tends to be the former two, which just come along for the right as these models get more powerful. Yeah, so that's a nice property because it's actually a capabilities thing to some extent. Right, but then if the third one is a bottleneck, or if the third one is the thing that doesn't to come along with the AI itself.

25:47What is the actual technique to make sure that happens? The third one, sorry, the ethical model. The ethical model. What are humans value? Well, we've got a couple of problems. First of all, we need to decide, we should train the system on ethics generally. There's a lot of lectures and papers and books and all sorts of things. So it understands human ethics well. And we need to make sure it understands humans' ethics well, because that's important, at least as well as a very good ethicist. And we then need to decide, okay, of this sort of general understanding of ethics, what do we want the system to actually value and what's sort of ethics do we want it to apply?

26:31Now that's not a technical problem. That's a problem for society and ethicists and so on to come up with. Now, you I'm not sure there's such a thing as true or correct, optimal ethics or something like that, but I'm pretty sure that it's possible to come up with a set of ethics, which is much better than the, what the so -called dooms are worried about in terms of the behavior of these AI systems. And then what you do is you engineer the system to actually follow these things. So every time it makes a decision, it does an analysis using a deep understanding of the world and of ethics and very robust and precise reasoning to do an ethical analysis of what it's doing.

27:20And of course, we'd want lots of other things. We'd want people checking these processes of reasoning, we'd want people verifying that it's behaving itself in terms of how it reaches these conclusions. But I still feel like I don't understand how that fundamental problem of making short follows that ethic because presumably you know it has mouths a little bit of books so understands Maoist ethics and understands all these other ethics you know. How do we make sure the ethic that we say this is the one we've decided at the distance study so on today that is the one it ends up following and not the other ones it understands.