Is Artificial Superintelligence Imminent? with Tim Rocktäschel - #706

0:00Really now, I think all these things are in place. We have powerful foundation models. We have really powerful mutation operators. We have powerful selection operators. We have models that can code quite well. And putting this all together means we'll have systems, as I said, that self-improve based on empirical evidence that they collect in a number of hard domains. And I think that's really what's going to drive further improvements in AI in the next few years.

0:42All right, everyone, welcome to another episode of the Twinwell AI podcast. I'm your host, Sam Charrington. Today, I'm joined by Tim Rocktushel. Tim is Senior Staff Research Scientist at Google DeepMind, Professor of Artificial Intelligence at University College London and author of the recently published Artificial Intelligence, 10 Things You Should Know. Before we get going, be sure to take a moment to hit that subscribe button wherever you're listening to today's show. Tim, welcome back to the podcast. Thanks. Great to be back. It's great to chat. We published that episode together a week ago, three years and a week ago, actually it was October 10th of uh 21 and we were talking about uh advancing deep reinforcement learning with nethack that was the episode title talking about that project uh what do you think has AI changed a bit since then I think not much happened I mean I mean to be to be absolutely honest um like there's not that much progress on that nethack benchmark right we have seen a lot of amazing progress in AI in terms of large language models, foundation models in general, but NetHack is still unsolved.

1:57So that grand challenge is still there. Yeah. Do you think LLMs are applicable there? Yeah, you can try them out. I think we'll have something to show in maybe a few weeks on that front. Oh, really? I think, yeah, I can already... I mean, you can try it yourself already. I think it's fair to say already like lms are not going to crack it right it's it's still a very hard hard exploration problem and even systems like o1 i think yeah won't won't get us all the way there or not even close let's put this way yeah we'll refer folks back to uh that episode to learn more about net hack and that challenge uh today i think we're going to focus a little bit on the book and more broadly your research into open-endedness and some other interesting questions raised by the book.

2:50Tell us a little bit about who the book is for. Yeah, so this is a popular science book. So this is new to me as well, right? Trying to write for a wider public audience. And to me, this was an exciting project because the book itself is part of a series, 10 Things You Should know. There are books about space and time and numbers and dinosaurs and trees. And now there's one on artificial intelligence. And I think it shows you a little bit what, I guess, important topic AI has become also for the public, right? And what I found intriguing about writing such book is that, I mean, on one hand, I think it's important to write in, I guess, simple terms about why AI has taken off so much in the last 10 years.

3:47From like AI for games, right? A lot of progress has been made with game environments like NetTag and before that Atari and Go and Chess. But why was there such a seismic shift in the last three years, right, with the advent of large language models? How do they work? And explaining that to someone who might not have any computer science or mathematical background I found important to do. But then, you know, I'm saying it's a popular science book, but I really couldn't resist to also talk about things that are really dear to me right now in my research. So topics around self-improvement, artificial superhuman intelligence, the automation of science and robotics.

4:27So it basically tries to do both a little bit. I pick people up that don't have really heard much about AI before, but then also getting very quickly to the frontier of what might be coming at them in the next few years in terms of AI progress. You mentioned artificial superhuman intelligence, and that was one thing that jumped out at me. I think your third chapter is artificial superhuman intelligence is achievable. And it's said very definitively. Talk a little bit about your beliefs around ASI and even how you define it. Is that differentiated from AGI to you? Yeah, so there was a great paper from Google DeepMind that was published this year at ICML called The Levels of AGI, where the authors try to basically define different kind of levels of capability that world get us from, I guess, things that are not really at the human level towards superhuman level.

5:29And the first distinction that these authors are making is between narrow systems and general systems. So within narrow systems, within narrow domains, we already have superhuman systems, right? We have superhuman Go playing AI, superhuman chess playing AI. And I mean, on the way there, you have things like calculators that are maybe really helpful in a certain way, but not superhuman in the math domain. So you can charter the way basically on narrow AI from things that are narrow and not superhuman or not even human level towards superhuman systems like AlphaGo. But then what we really do care about more and more nowadays is having general purpose systems, right?

6:12I think maybe at some point people still believe that you train a superhuman Go system and maybe that system would learn something so general about exploring hard domains that you could take something out of the domain of Go and apply it to the same AI and apply it somewhere else. Even if you remember back, I think with Watson AI, there was this belief that maybe this can be now applied to the medical domain. And it turns out that this is hard or even impossible, like training these systems in narrow domains and then hoping that they can generalize outside of these narrow domains is just never going to work.

6:51But at the same time, we have now seen systems like ChatGPT, Gemini, and so on, right, that suddenly are able to answer questions on an extremely broad range of topics. So that's where the authors of this Levels of EGI paper talk about, I guess, general AI and then the different levels of that. And the way they categorize things are basically what percentage of average human system can compete with, and then what percentage maybe of skilled humans it can compete with. And then at some point it can do better than any human on Earth on a certain topic. And for the general systems, the authors are arguing that with these large language models and chatbots, we are basically at this kind of emerging stage of AGI.

7:43And then if you go all the way up to level five, you would get to artificial superhuman intelligence. That means an AI that on basically any human task is better than any human on Earth. And in the book, I make the argument that that is feasible. And I guess the argument I'm making is that, first of all, we have an existing proof. It sounds almost a little contradictory in the sense that you kind of acknowledge that we've already achieved superhuman capability in narrow domain. So, you know, that must not be what we're talking about, but also that broad generality you think is impossible. Right?

8:30No, sorry. I'm not arguing that broad generality is impossible. I think we're seeing, as the authors of that paper argue as well, I think we're seeing signs of general purpose AI, which is what we see with chatbots nowadays. And the question is now, how do we get further on these levels of general purpose AI towards artificial superhuman intelligence? And there I'm arguing that that is possible. So your point earlier was more limited. I think the example you used was we're not going to teach something how to play Go, and then it's going to be generalizable. But that does not preclude us from taking some other path of teaching a generalized superhuman intelligence.

9:16Maybe let's be more concrete about what the difference here is. So in something like AlphaZero, what you have is you have a narrow domain like Go. And within that narrow domain, you can simulate lots and lots of games. And you can even have this AI play against itself, right? So you can have this AI improve itself by playing against itself, against your previous version of itself. And from that, you get a bootstrapping that gets you basically to a system that's in an open-ended way and just further and further improving itself in the domain of Go or Chess to the point where it's explored the space of strategies in this game so much that it's better than any human in the game.

10:01And that gets you, again, superhuman artificial intelligence for a narrow domain like Go or Chess. But what happened with large language models and why we are now starting to talk more and more about AGI and also more and more about artificial superhuman intelligence is that the paradigm shifted. We are not looking at one specific narrow domain, and we also don't train these models in just one narrow domain. Instead, we are taking internet scale, textual data and image data and train a foundation model that, as we can see empirically, after it's pre-trained and then fine-tuned and then LHF to be more helpful and harmless.

10:44And we see basically these models be able to pass different kinds of, for example, university entry exams. We see these models to be able to, for a very right range of questions, provide sensible answers with some caveats, right? Philocination and so on. We also see these models be able to code to some extent, right? So that's a certain generality that we've never seen with RL and self-play systems before. And the question now, again, is it feasible to think that these systems can further and further improve to the point where they become superhuman on many and at some point all of these tasks?

11:26And to be more specific about the paper or to kind of jump to its results, they have, I guess, as you mentioned, five different levels for each of narrow and general. and the levels here are non-AGI, emerging AGI, competent, expert, virtuoso, and superhuman or ASI. And then essentially it states that we've achieved, well, general non-AI is not AI. We've achieved emerging AI only, basically, and competent AGI, expert AGI virtually. And all of those higher tiers are not yet achieved. And so with that as a foundation, let's go back to your argument that ASI is achievable. What tells you that? The fact that we got from non-AI to emerging AGI?

12:34What tells me that is, well, a few things. So first of all, we do have an existence proof of an intelligence that is general in the sense that it can, for any kind of problem domain, set their mind to it and do really well. And that's humans, right? That's people. And on top of that, we have now systems that have been consuming internet skill data of texts and images on the web that gathers foundation models that are only at the level of emerging AGI, but that do encompass notions of what humans find interesting. There's a great paper called Omni and OmniEpic from Jeff Klun's lab showing that basically these foundation models, they learn basically what humans generally find interesting.

13:28So when you have a massive exploration space in which you want to throw an AI at, then now we have a way basically to massively prune down this search space. So that's one thing. The other observation is that these foundation models are really good at generating variations of data. So you can give it a few examples, some few short, basically prompt examples, and it can generate variations of these examples. Now, when you take these two things together, what you get is basically something that can generate data or generate variations and something that can select for what is interesting. And putting these things together, you, I believe, can build very powerful open-ended self-improvement systems.

14:17It requires a third thing as well, which is connecting this kind of process back to some kind of source of empirical evidence. but then you have basically the ingredients for technological evolution which generates variations, tries them out empirically and then selects or first selects which ones are most interesting and then tries them out empirically and then further selects which one to further vary. And indeed by now there's multiple papers, some of them also from my group, on building such self-improvement systems based on this kind of recipe. And then with that in mind, I think it's not an extreme stretch to say that we, I think, will see more of these kind of self-improving systems that obviously use foundation models as the foundation, but then actually become more like an agent that collects empirical evidence by themselves and try out things.

15:18Let me restate that argument. So essentially humans are the existing existence proof of intelligence that was created through an evolutionary process. And in language models, we see kind of this ability to evolve and iterate. and so therefore if we project you know if that evolutionary and iterative process is a path to or the path to intelligence LLM should be able to get there is kind of the crux of the argument is that fair I think I wouldn't call it LMS anymore at that point I think it's building agentic systems on top of these foundation models but yes I guess that's another question yeah sorry go ahead no go ahead i was gonna ask you what you think asi looks like meaning is it llms is it transformers is it um you know or you know do those things there are many who believe that those things you know won't and can't get us to asi um but it sounds like you're saying you believe that those things can get us to ASI in concert perhaps with other things?

16:44What are the things that you think it requires, I guess? Yeah. So we had a position paper at ICML called Open-end-ness is Essential for Artificial Superhuman Intelligence. So it is building systems that are not directly optimizing for a specific task or for a specific output or outcome, but that are instead by themselves autonomously are exploring problem domains, finding interesting artifacts, stepping stones that then become enablers basically for future progress that you then can run for longer and longer and longer, and they would just accumulate more and more interesting artifacts. And the things that you mentioned, LMs, foundation models, transformers, they might be the kind of core models that drive maybe some of that exploration.

17:35But it is, I think, the main bulk of the work that you'll see in the next few years, and I think one is maybe already one example from OpenAI, is that it's the kind of systems you build around such LMs, right? The systems around them that, again, can, for example, go out there and try to collect empirical evidence for certain domains that can, based on the feedback that they get empirically, self-improve. And that can, at some point, I think, be let loose, right, to generate an increasing wealth of interesting artifacts, whether that's, for example, research insights and explanations, or whether that's maybe certain data that's interesting for certain scientific domains.

18:21What do you think let loose means in this context? Do these agents need to be, you know, given free range or, you know, given free range, both in terms of freedom, but, but, but also in terms of, you know, some kind of objective that is incenting, you know, exploration, do they need to be just kind of turned out to explore without any constraint? Or will we have to give them direction in order to guide them to the kind of intelligence gains you're suggesting? I think we will be providing direction because of a few reasons. One is, I guess, an AI safety aspect. I don't think we want to... I mean, when I said let loose, I think that was maybe very...

19:18Well, let loose is a spectrum, right? So I'm trying to get a sense for where on the spectrum we are. No, I think we would want to direct them in the sense that, I think, first of all, from an AI safety perspective. Secondly, compute, while it's still increasing, I think we're never going to have in the foreseeable future the kind of endless compute that we would want, right? to really just let these kind of self-improving systems explore all kinds of domains equally, right? We will, as humans, have to make decisions what we find most interesting as domains to apply such processes. But within a certain domain or question that we might have, right, I think within that we will be giving these systems a lot of freedom.

20:09It's not going to be what people believe for a long while. We have these domains where we can define a reward function and we can just reinforcement learn our way towards general capable AI. I don't think that's true. I think for most interesting domains, we just don't have that reward function that goes back to NetEq as well. NetEq is a great example. We don't have a reward function that tells us how to win in this game. You just have to have a system be curious and start exploring by itself and see what it discovers. And I think that's actually true, I think, for most of the domains that we find interesting.

20:44It strikes me that there's some idea of a generalized reward function that isn't specific in that it's telling the agent what to do, or I guess the reward function is not telling the agent what to do. It seems like there's a meta reward function that needs to exist, or we might benefit from it existing. But I'm wondering what the research landscape looks like around that. I mean, so there was work on trying to come up with intrinsic reward functions, right? Things like empowerment or being able to exert control in the environment or learning generally about environment dynamics and basically steering and reinforcing agents towards regions in the environment where maybe it has a lot of uncertainty of how the environment or the dynamics work.

21:37So there are a lot of attempts of building such intrinsic reward functions that don't rely on some kind of gold standard ground truth reward signal that you can just directly optimize for. But I don't think that's what I'm talking about here. I think what I'm talking about here is when you talk about reward functions, you automatically talk about reinforcement learning. So you will use that reward function to reinforce a certain behavior. But what I want to see, and I think, again, we're getting there with everything that has happened in the last two years, again, models that have ways to create variations of what might be interesting to explore and then shrink down the space of things they could explore to those things that are interesting.

22:26and interesting here, again, as measured by what humans generally would find interesting, and that can be done via proxy using a foundation model. And then being able, as I said, going out there and collect empirical evidence to refine maybe some of the ideas that this model has. And this isn't really reinforcement learning, right? This is, I think, really an open-ended search method. And maybe it might use, again, reinforcement learning in certain bits, right? Like if it learns that or if it sees for empirical evidence that a certain behavior leads to interesting empirical results, then maybe I want to reinforce that behavior.

23:05But reinforcement will just be one component in this bigger system, and it's not going to be the main one, I think. One of the areas that I think we look at ASI and, you know, with some hope is in producing advances in science and medicine and fields like that. You cover that a bit in the book as well? Yeah, I think, I mean, first of all, AI has been already used a lot to make really fantastic scientific discoveries. I mean, literally last week, the Nobel Prize was awarded to protein folding, right? It's a fantastic example. But what I think the future holds is basically a time at some point where these systems can even more autonomously make scientific discoveries.

23:59And I think that is going to be really interesting, partly because if you apply that to a domain like AI itself, right? It becomes self-referential and it becomes, you know, not just improving, you know, certain question or like a certain metric that you as a human define, but also improving the way it's improving along the way. And we had a research prototype for the domain of prompt engineering research that we published this year at ICML called Prompt Reader. and the idea is that these large language models, a lot of the behavior of these models you can shape by the way you prompt them. That's well known, right?

24:42You have prompt strategies like chain of thought prompting that have dramatic effects on the capabilities of these LMs. So if you change the prompt strategy, I mean, not just the question you give to the model, but the kind of prompt you give before, right, in terms of how it should solve this or decompose this question, then these models can get massively better in terms of mathematical reasoning or coding and so on. So wouldn't it be nice if you could automate this process? So could you have the LM basically talk to itself, suggest to itself variations of existing prompt strategies, and then empirically try them out and see if they lead to better reasoning capabilities, and then evolve those further that work best so far?

25:24And that's indeed what we found with PromptReader. and that's basically showing you some kind of automation of science in a quite narrow domain of prompt engineering but i think it's it's no stretch to to yeah think about where that might lead to if you scale it up further and if it becomes more general that sounds like a cross between a prompt optimization tool like dspy and an evolutionary strategy is that the general idea Yeah, it's an evolutionary strategy. But I think the thing that's so interesting at this point is that we see evolutionary approaches being applied directly to language itself, right?

26:07So that's why I said if it's talking to itself, like it's suggesting to itself variations of prompt strategies. And the way it's doing that is by being prompted by us to do it, right? So that means the entire mechanism that defines this evolutionary search is again a prompt mechanism. And because that's true, it can be applied to itself. So it can try to improve the prompts it's using to improve, to mutate prompts and to do this evolutionary search. That's why you can make it self-referential. And evolutionary usually implies that given two or some number of choices, there's a choice that is better and that propagates.

26:48Is that the LLM making a judgment as to which of it, which of the prompts, you know, it prefers or is it evaluated against some task or criteria? I'm wondering if, you know, there's some implication that, you know, we still need some kind of objective function somewhere, even if we push it onto the LLM or, you know, push it deeper into the system. Yeah, so there is indeed an objective function in that sense in that the fitness function is measuring, given an evolved prompt strategy, how well does that evolved prompt strategy do across a range of reasoning tasks? So it basically gathers empirical evidence through that way.

27:37And it's grounded in that way as well. So I'm not saying that there's not going to be, there's going to be no empirical evidence that guides basically this kind of search. I think it needs to be grounded. Otherwise, this system can just hallucinate to itself that it's found something really cool. And it's not helpful for us. Right, right. But I think it's very powerful once you have open-ended domains like computer science, maths, or AI research. Once you have these domains, there's a lot of things you can explore. And indeed, there was papers like this already, like FunSearch, for example, again, from Google DeepMind shows that you can basically evolve computer programs and empirically show that they work better than maybe some of the had-crafted computer programs that people have been developing before.

28:33before you know and thinking about the application of agents and these kinds of you know eventually superhuman ai to science and medicine the recent ai scientist paper comes to mind that was david ha and others

28:55does the the work you just described and your work in the area kind of compare to you know how might it compare to that work or if you've looked at that work like what are your thoughts on that? Yeah, so I think it's a fantastic existence proof. So basically, the different ingredients I was mentioning in terms of creating variations of data and then empirically testing them out and then having some kind of loop that can improve over time, they have all of these things together in this AI scientist paper. So I think it's really, really great in that regard that it basically demonstrates that this is possible.

29:34And if we were to, I think, say something along the lines of levels of AI science, I think it's still very early, right? So I think we can make a similar, I guess, categorization of the kind of levels you would have to go for. And maybe what I would say is, you know, there's maybe a kind of bachelor student level where you as a supervisor, you have to double check every experiment that maybe the student is running to make sure that it's actually implemented correctly and that the data is interpreted correctly. And then you could go up the levels right to master's, student, PhD student, postdoc, maybe at some point professor that kind of supervises multiple AI scientists.

30:21But I think the work that we're seeing in the space right now, I think it's still very early days. So it's, I think, at the level of maybe producing, and I think they say this in the paper as well, producing maybe workshop paper, maybe level quality. And I think we'll see, I guess, ways in the future to evaluate this more thoroughly, right? I mean, I think at the end of the day, we have to, um, we have to qualitatively look at the generated papers and see if they actually produce interesting, valuable insights and, and maybe at some point we'll get there, but I think still a lot of stuff to do.

30:58And do you think that the, or an evolutionary approach is required to get to a higher level or is it, you know, just an interesting option to explore among, you know, many possible others? No, I think it has to be evolutionary. I mean, science is an evolutionary search. We have certain explanations right now of how the world works, and then people come up with variations of these or new ones, and then they have to live up to empirical scrutiny, and we keep those explanations that can best explain empirical evidence and that are also hard to vary according to, for example, David Deutsch, that I'm citing quite a lot in the book.

31:41So in a way, you know, science, the way humans do it is already an evolutionary search, right, for better explanations and for better insights. And I don't see any other way of how an automated scientific process can work differently. It has to be evolutionary as well. Is there a distinction between evolutionary in the sense you're just using it and evolutionary algorithms and like more, you know, system level? You can still apply the general idea that you were describing, picking the best of two options and using that as a mechanism for advancing and not be an evolutionary system in the technical or computer science sense.