Is Human Data Enough? With David Silver

0:03We're going to need our AIs to actually figure things out for themselves and to discover new things that humans don't know. And I think that's going to be a whole new era of AI that's going to be incredibly exciting and profound for society.

0:18Welcome back to Google DeepMind, the podcast. My guest today is the inimitable David Silver, an original deepminder, one of the key people behind the phenomenal success of AlphaGo, the first program to master the world's most complex board game and achieve superhuman performance. Now, at the end of today's podcast, we have a little extra treat for you, a conversation with David and Fan Hui, the first professional Go player to take on the AI. But now David has a bold idea about the direction that AI should go in next. After all of the current buzz and excitement and achievements of multimodal models, David has a plan for the path towards superhuman intelligence, a new phase which he calls the era of experience.

1:07This is a profound idea and not one without risks. David, welcome to the podcast. Hi, it's great to be here. Real pleasure. Thank you. Okay, so I have spent the weekend with a very enjoyable read of your position paper. And in it, you are talking about the era of experience. Summarise for us, what do you mean by that? Well, what I mean is that if you look at where AI has been for the last few years, it's been in what I call the era of human data, which is that all of these AI methods, they're based on one common idea, which is we extract every piece of knowledge that humans have and you kind of feed it into the machine.

1:48And that's one incredibly powerful way to do things. There's another way to do things, and this is what's going to lead us into the era of experience, which is where the machine actually interacts with the world itself, and it generates its own experience. It tries things out in the world, and it starts to build up its own experience. And if you think of that data as fueling the machine, then that will lead to this next generation of AI that we can think of as the era of experience. I guess this is in a way you're sort of thumping the table saying large language models are not the only AI, right?

2:25Like there are alternatives. There are different ways that we can approach this. That's right. I think we've really got a lot out in the field of AI of building large language models, harnessing the vast quantity of human, like particularly natural language data that's out there and kind of assimilating that all into a machine that knows everything that humans have ever written down. But at some point, we need to get past that. We want to go beyond that. We want to go beyond what humans know. And to do that, we're going to need a different type of method. And that type of method will require our AIs to actually figure things out for themselves and to discover new things that humans don't know.

3:01And I think that's going to be a whole new era of AI that's going to be incredibly exciting and profound for society. Well, OK, then let's talk about some other sort of famous AIs, famous algorithms that have employed different types of methods. most notably AlphaGo and AlphaZero, which of course notoriously beat the world's best Go players about a decade ago, right? Tell us about the techniques that we use in that and how they differ from large language models that we see today. So AlphaZero in particular is very different from the type of human data approaches that have been used recently because it literally uses no human data.

3:41That's the zero in AlphaZero. So there is literally zero human knowledge that's pre-programmed into this system. And so what's the alternative? How do you learn Go knowledge if you're not copying humans and you don't really know in advance the right way to play? Well, the way you go about it is through a form of trial and error learning where AlphaZero basically played itself millions of games of Go or chess or whatever the game is that it was wanting to play. and bit by bit it figured out oh if I play this move this kind of move in this kind of situation then I end up winning more games and then that's a piece of experience that is used to fuel it to become stronger and then it'll play a little bit more like that and the next time what will discover something new and say you know there'll be some new pattern it's like oh when I use this particular pattern I end up winning more games or losing more games and that feeds the next generation and so forth.

4:32And that learning from experience, this learning from the agent's self-generated experience is enough and was enough in AlphaZero to fuel its progress all the way from completely random behavior, all the way up to the strongest chess and Go playing programs that the world has ever known. They didn't start off just as like random empty boxes that kind of found how to play Go from nothing. I mean, initially when you were designing your Go algorithms, you'd worked out a way to encode Go games and then feed them in as a database, right? Yeah, that's right. So the original version of AlphaGo, the version which famously beat Lisa Dahl in 2016, this version of AlphaGo actually did use some human data to start it off.

5:14So we basically fed it a database of human professional moves, and it learned, it ingested those human moves, and that gave it its starting point. And then it learned for itself by experience from that point onwards. However, what we discovered a year later was that the human data wasn't necessary, that you could actually throw out the human moves altogether. And what we showed was that actually the resulting program, not only was it able to recover this level of performance, it actually worked better and was able to learn even faster than the original AlphaGo to achieve a much higher level of performance.

5:54That is such a strange idea. It's such a strange idea that you throw away the human data and you find not only was it not necessary, but it was actively limiting performance in a way. I think one of the hard lessons for people in AI, this is sometimes called the bitter lesson of AI, is that we really want to believe that all of the knowledge that we accumulated as humans is really important. We really want to believe that. And so we feed it into our systems. We build it into our algorithms. and what happens is that actually that makes us design the algorithms in a way which is maybe fitted to the human data and is less good at actually learning for itself and what happens is if you throw out the human data you actually spend more effort on learn how the system can learn for itself and that's the part which can then learn and learn and learn forever the bitter lesson i suppose in a way it's sort of saying accepting that it's possible that something could play Go better than humans can and sort of removing that ceiling in a way.

7:02That's right. That, you know, human data, it's really helpful to get you off the ground. But there is a ceiling to everything that humans have done. And, you know, we see that in Go. There was a maximum level of performance that humans have ever achieved. And we need to break through these ceilings. And in AlphaZero, we were able to break through that ceiling by building a system that learned for itself by self-play and got better and better and better until it blasted through that ceiling and went far beyond. And I think the idea of the year of experience is that we find the methods that allow us to break through that ceiling everywhere.

7:33We build AI systems that become superhuman in all of the capacities that humans seem so amazing, but we find the way to go beyond that. Let me just stick with Go for a second, okay, before we get onto the other ways that you can get rid of human data and thus improve on human ability. because it sort of sounds a little bit, when you say let's just get rid of all of the games of Go that humans have played and start with nothing, it sort of sounds like a magic trick. Just tell me a little bit about the techniques that you're really using there in order to get a machine to, as you say, chain together thousands and thousands of different ideas in order to be amazing at the game of Go.

8:17Well, the main idea is an approach that we call reinforcement learning. And the idea of reinforcement learning is you basically give the outcome of your game a number. And we say, you know, plus one if you win and minus one if you lose. One point. Exactly, exactly. And what we do then with reinforcement learning is we get the system to basically, we give it a reward each time it does something right. And we train the system to basically reinforce, that means do more of the things that gets more reward. And so in terms of, for example, if you've got a neural network, like we do in AlphaGo that's picking the moves, what you want to do is tweak the weights of your neural network a little bit in the direction that gives you more reward.

9:02And that's the main idea of reinforcement learning. But then, OK, I mean, a game of Go is quite long. How do you make it so that you do the right moves in the beginning so that you end up with the right outcome towards the end? How do you work out which bits of the game are important, I suppose? So this is a really important problem. It's called the credit assignment problem. And the idea is that if you've, yeah, you're absolutely right that you could have had, you know, 100 or 200 or 300 different moves. And then at the end, you just get one bit of information saying, you know, win or loss. And you somehow have to work out which of the moves in the game were responsible for winning and which of the moves in the game were responsible for losing.

9:44And there's lots of ways to do that. The simplest way is just to assume that everything that you've done contributes a little bit to that outcome at the end. And it sort of all comes out in the wash. One of the biggest moments of the AlphaGo story was Move 37 that everyone always references. Just tell me about that. So Move 37 was a move that happened in the second game of AlphaGo against Lee Sedol. and AlphaGo played a move that defied everyone's expectations. The traditional idea in the game of Go is that you play your moves typically on the third line or the fourth line of the board because this either gives you territory for the third line or influence on the fourth line and you never go below or above that.

10:29It just wouldn't make sense to humans. AlphaGo played on the fifth line and it somehow played this in a way that just made everything make sense in the board. It kind of connected everything together with this move on the fifth line. And it was so alien to humans that we estimated that only one in 10 ,000 probability that a human would ever think of playing this move. Humans were shocked by this move. And yet it helped win the game. And so it was a moment where humans said, look, here's something creative that happened. Something that a machine came up with that was different from the way humans traditionally thought about the game.

11:06that actually was a big piece of progress and took us beyond the kind of confines of human knowledge. And I guess if we really do want to advance AI, we sort of do want those alien ideas, as you put it. Do you think you've seen an equivalent of Move 37 with large language models? Move 37, in some ways, was special because it was the first moment. It was the first time that people had seen a big breakthrough like this. and because we've been in the era of human data we focused a huge amount on reproducing human capabilities and we've focused much less on going beyond them and I think until we really emphasize systems learning for themselves to go beyond human data we won't see huge breakthroughs the equivalent of Move 37 in the real world.

11:59It just, it seems unlikely to me. Because when you're angered in human data, you're only ever going to have human-like responses. That's right. And I think there are things you can do that allow you to maybe do things in the middle a little bit. So if you push me to say what's the greatest Move 37-like moment, I would probably pick out some work by scientists at MIT who discovered a new antibiotic that no human knew about. And I think that's an incredible discovery of massive importance to humanity. So in that sense, it goes way beyond Move 37. But what I like about Move 37 is that it's not just a single discovery.

12:38It's one of an infinite series of discoveries where the system can just keep on learning and learning and learning. And Move 37 is important to me because it represents just a single point in that infinite sequence of discoveries that can happen once you've got this kind of approach of learning from experience. Rather than actual result in and of its own right. Yeah, that's right. Give me a brief rundown of how AlphaZero worked. So AlphaZero is surprisingly simple. I mean, there's some very complicated algorithms out there in the world, but this one is really straightforward. So all you do is you start with a policy, a way to pick moves, and a value function, which is a way to evaluate moves and say whether they're good or bad.

13:22So you start with that, you run a search. And then what you do is you take the best move according to your search, and you train your policy to do more things like that, to do more of the good moves according to your search. And you train your value function based on how the game actually panned out when you played a game with this search. And that's it. You just iterate that millions of times and out pops a superhuman game player. It's like magic, basically. It does sometimes feel like magic. I remember the first time that really felt like magic to me was when we had just completed AlphaZero on chess.

13:59Someone had the idea of trying it on a different game. So we plugged it into a game that none of us could play, a game called Shogi, which is Japanese chess. And we had no idea how to play this game. What, you didn't even know the rules? So the system knew the rules. The agent, we taught it the rules. But none of us had the first clue of how to really, you know, strategy or tactics. It would have been like blunder after blunder if we'd been playing this game. And we just plugged it in. And it was literally the first ever time we ran AlphaZero on Shogi. We had no idea whether it was good or not.

14:33We couldn't evaluate it. But we sent it off to Demis, who's actually a reasonably strong player, of course. And he said, hmm, this looks quite good. I'm sending it to the world champion. And the world champion said, hmm, I think this is superhuman. And so it literally felt like magic because, you know, we just pressed go on this system and had no idea of the process and how it got there. But somehow out popped a superhuman shogi player. Can AI design its own reinforcement learning algorithms? Well, funnily enough, we have actually done some work in this area. It's work we actually did a few years ago, but it's coming out now.

15:12And what we did was actually to build a system that through trial and error, through reinforcement learning itself, figured out what algorithm was best at reinforcement learning. It literally went one level meta and it learned how to build its own reinforcement learning system. And incredibly, that reinforcement learning system actually outperformed all of the human reinforcement learning algorithms that we'd come up with ourselves over many, many years in the past. I mean, this is the same story over and over again. The more of a human you put into something, the worse it acts, the worse it performs.

15:51Take the human out, does better. Okay, if AlphaGo and AlphaZero then are really exceptional examples of reinforcement learning used to the best it can be, you still find reinforcement learning in the large language models that we have at the moment, right? Tell me about how they're integrated into these systems. So reinforcement learning is used in almost all large language model systems. And the main way it's used is by combining it with human data. So unlike the alpha zero approach, this means that the reinforcement learning is actually trained on human preferences. So the system is basically asked to produce outputs.

16:33And then a human says, this one is better than this other one. and the system becomes more like the one that the human prefers. And this is called reinforcement learning from human feedback, and it's been massively important in LLMs, and it's helped transform them from systems that just blindly mimic any kind of data that you see on the internet into systems that actually usefully produce answers to the kind of questions that people really want to see. And so it's an incredible advance. However, it feels like we've thrown out the baby with the bathwater. These reinforcement learning from human feedback systems or RLHF, they're very powerful, but they do not have the ability to go beyond human knowledge.

17:21Like if a human rater doesn't recognise some new idea and underappreciates that there is some series of actions that would actually end up being far better than some other series of actions, there is no way that the system will ever learn to find that sequence because the raters might not understand that better behaviour. That human feedback element, though, it does seem to give these models some sense of grounding. Like, I know the last time we spoke, grounding was like this really big topic, this idea that you want these algorithms to have a conceptual understanding almost of the world that we're living in.

18:00So if you take away or you remove that human feedback aspect, do you still end up with models that are grounded? I almost want to argue the opposite. I want to say that when we train a system from human feedback, that it is not grounded. And the reason is that we are basically, the way RLHF systems normally work is the system presents its response, its answer to a question, for example. And a rater says that's good or bad before the system actually does anything with that information. So it's like the human is prejudging the output of the system. So, for example, if you're asking for a cake recipe from an LLM, the human rater will look at the recipe that's output by the system and judge whether that recipe is good or bad before anyone has actually made the recipe and eaten the cake.

18:55And in that sense, it's ungrounded. Like a grounded outcome would be someone actually eats the cake and the cake is either delicious or disgusting. and then you've got grounded feedback that says you know this cake really was a good cake or this cake was a bad cake and it's that grounded feedback that allows the system to iterate and discover new things because it can try out new recipes that maybe you know expert chefs presume will be disgusting but actually turn out to be delicious yeah like a yeah monster munch muffin or whatever exactly yeah that's the most delicious food that ever existed okay that's interesting though because I have heard I mean even a conversation with Demis talking about how grounding gets into these models how they kind of have built this conceptual understanding of things and it sounds almost like what you're saying is that the grounding that they have is like a sort of superficial level of grounding maybe?

19:50I think human data is grounded in human experience so it's like the LLMs are sort of inheriting all of that information that humans may be figured out from their own experiments, for example, in science, a human might have tried to walk across water and discovered that they fell in, and then they might have created a boat and discovered that that floated. And all of that information can be inherited somewhat by the LLM. But if we want a system that actually makes discoveries and discovers some completely new form of propulsion across water or some completely new mathematical idea or some completely new way to...

20:28Part the seas. Yeah, new medicine and new approach to biology. The data just isn't there and the system needs to figure out for itself through its own kind of experimentation, its own trial and error and its own grounded feedback, whether that's a good idea or a bad idea. I got to talk to Aurel, Aurel Vinales, who really spoke about how we are running out of human data and that we are going to need to start creating synthetic data in order to fill that gap. I mean, this is related, right, to that idea. It's just rather than using LLMs to create more human dialogue data, you're going about the solution in a different way.

21:08That's right. So synthetic data can mean a lot of things. But, you know, normally it would mean that you've got some process where you kind of take your existing LLM and use it to generate some set of data. And I guess the argument is, similar to the ceiling that we have from human data, that however good that synthetic data is, they will reach a point where that synthetic data is no longer useful to the system becoming stronger. So the beauty of a self-learning system, where the fuel of the system is actually experienced, is that as the system starts to get stronger, it starts to encounter problems that are exactly appropriate to the level it's at.

21:47so it will always be generating experience that allows it to solve the next problem that it's it's encountering and so it can just get stronger and stronger and stronger forever there is no limit and that i think is what differentiates this particular approach of using self-generated experience from other forms of synthetic data just returning to your cake example though i mean if you kind of follow that through somebody eats the cake and says yes this was delicious You're using the human feedback then at the end of the process anyway. Are we talking about that or are we talking about maybe having systems that are completely untethered from humans and are embodied or in the physical world somehow so that they can get their feedback in that way?

22:28Look, I think the ideal is that like AlphaZero, we have systems which are able to generate vast volumes of self-generated data experience that they can then verify for themselves. And in many domains, that's going to be possible. And in many domains, it's not going to be possible. In the ones where it's not possible, we have to acknowledge that humans are a big part of the environment that we're in. We have to acknowledge that they're a part of the world that we want our agents to live in. And so it seems reasonable to think of humans as a part of that environment and to think of the way that they behave as part of the observation that the agent receives.

23:01I think the thing which I'm pushing back against and saying is not grounded is not that. it's the fact that the rewards that the agent learns from is coming from a human's judgment of like whether this sequence of actions is good or bad and the system is not judging for itself based on the consequence of those actions in the actual world and so you know one way to say it is that we shouldn't make you know human data a privileged part of the agent's experience it's just just just observations in the world and we should be able to learn from that like any other data. If we go back to that AlphaGo example earlier of assigning that reward, that one point that it gets at the end, is this almost like the way that we're handling AI at the moment is that the algorithm does its first 10 moves or 15 moves and then we insert a human in who says yes that's a good first 10 moves and doesn't allow the whole process to kind of execute fully before you input that little bit of feedback.

24:00That's exactly right. So imagine that we were training AlphaGo and after every single move, our best Go player comes in and says, oh, that move was amazing. Oh, no, no, that move was totally wrong. And then we get that feedback and we put it in and the system learns to pick the move that the human prefers. It would not end up discovering Move 37 because it would just end up playing like the human thinks is a good game of Go. And it would never discover the new ways to play Go that that human didn't know about. Okay, so I think the environment of Go, what you're saying makes a lot of sense in that environment.

24:37There are other environments too, where I think that this makes a lot of sense. I'm thinking here about the pinnacle of human thought of mathematics. Tell me what's been going on in that space. Like you say, it is an incredible human endeavour that's had millennia of human effort going into it. And so in many ways, it does represent like literally the limits of achievement by the human mind. And so naturally, we turn to it for AI to see, can we achieve those same levels of performance that humans have achieved over all of those years of endeavor? We recently put together what I think is a very exciting piece of work called Alpha Proof.

25:13It is a system that learns through experience how to correctly prove mathematical problems. So if you give it a theorem and you don't tell it anything about how to actually prove that theorem, it will go away and figure out for itself a perfect proof of that theorem and we can actually verify and guarantee that this proof is correct. One thing which is interesting about this is that it's the exact opposite of how LLMs normally work because if you ask LLMs to prove a mathematical problem at the moment, they will normally output some informal mathematics and say, just trust me this is this is correct and it might be correct but it might not be because we know that llms tend to hallucinate a lot they can they can make things up and the nice thing about alpha proof is that it will actually guaranteed produce the truth so let's think of an example here to kind of anchor this in people's minds let's say that uh prime numbers are something that can't be divided by anything but themselves and one and there are infinite number of them off you go prove it?

26:23Yeah. So the way alpha proof works is it's trained on millions of different examples of theorems, not just one. And what happens is it goes off and it trains on them. And to begin with, it can't solve the vast majority of them. 99.999 % of the theorems it just can't do. And these are theorems that humans have already proved. Are you feeding in? We feed into the system something like a million different theorems that humans have come up with themselves, but we don't provide the human proofs. We just provide the questions, but not the answers. So you're giving it stuff that you know is true, but you're just not telling it how to prove it.

27:02And sometimes we don't even know it's true, because what we actually do is we take the human theorem, the human question, and we actually turn it into a formal language. These aren't using language in the sense that language models are using, but they are using a form of language, like a mathematical language. That's right. So in fact, we do use a small large language model. And that large language model allows us to output programming languages. And in particular, we use a programming language that's called Lean that allows all of mathematics to be expressed. And so it's an amazing idea that mathematicians have come up with that you can actually formalize all of these kind of things that we normally talk about in English language or whatever language you happen to be speaking can be transformed into a perfectly clear, verifiable mathematical language that allows all of the ideas of maths to be expressed and also all of the ideas of mathematical proof to be expressed.

28:04So you can say, for example, that if A implies B and B implies C, then there's a way to go from that to A implies C. And that's the kind of thing that you can do in this mathematical programming language. You essentially write a program that takes you from one to the other and you have a proof of this statement. So we take our kind of million human problems and from that we generate 100 million formal problems. and some of those might actually not be possible or they might be incorrectly formulated or they might just be false. And it doesn't matter because all we do is we learn to prove those things and the ones which we can't prove become, we keep trying and keep trying.