#128 Yoshua Bengio: Dissecting The Extinction Threat of AI

0:00It's very hard in terms of your ego and how you can feel good about what you do to accept the idea that what you've been working on for decades might be actually very dangerous for humanity. In a 1955 paper titled The Computer in the Brain, John von Neumann wrote that the possibility of replacing the human brain by a machine, which will be superior to it in any or all respects, is not excluded by any natural law that we know. It's therefore possible that the human race may be extinguished by machines. Von Neumann's warning has since been echoed by many other AI experts, from Stephen Hawking to Elon Musk.

0:50The concern captured the public imagination earlier this year when the Future of Life Institute issued a letter calling for a six-month moratorium on large language model development, saying that the development of artificial general intelligence could lead to human extinction. Then in May, the Center for AI Safety issued a statement warning that the development of artificial general intelligence could pose an existential risk to humanity. Many who had not signed the Future of Life letter signed this one, including Jeff Hinton and OpenAI CEO Sam Altman. Yoshua Bengio, who signed the Future of Life letter, has spoken on this podcast before about the threat, but I spoke to him again specifically about the extinction threat, whether it's a useful characterization or simply adding anxiety to a generation already troubled by looming existential threats such as climate change.

2:00Sir. Just a moment. Sir. Yes. I apologize for interrupting you, but I wanted to remind you that Oracle is offering a full implementation of NetSuite. Thanks for reminding me. Please continue. Okay, you can go back to your station. Thanks. Yeah. Beauregard reminds me that the first time in NetSuite's 22 years as the number one cloud financial system, You can defer payments on a full implementation for six months. That's no payment, no interest for six months. To take advantage of this unprecedented offer, go to netsuite.com backslash ionai. AI, that's www.netsuite.com backslash I on AI, E-Y-E-O-N-A-I, all run together.

3:01Now, here's Yashua. Hey, Yashua. I wanted to talk to you about your essay on how an AI could become rogue. And since then, this extinction debate has ramped up. On my Twitter feed, there was a long thread by somebody with a blue checkmark. That doesn't mean much anymore, but basically bemoaning the generation we've already destroyed because extinction is inevitable now that we've developed these language models. And that just, I mean, it just, to me, that debate, which is bleeding into the general public discourse, has gotten out of hand. But, you know, your essay was really the first concrete response I've seen on how things could go wrong.

4:11So I wanted first for you to just present that scenario and then I'll ask some questions. um i don't know if i told you last time but i'm also writing another blog post which um lists pretty much all the um pros and cons of that discussion that i've heard so yeah let me summarize the debate in the way that max stegmark did it near the end um so he said um yoshua now we we are on a river and you know going down the river and on our collective boat and yoshua hears uh some waterfalls downstream um and melanie says there is no waterfall or you know if there is one it's going to be maybe in 300 years so we shouldn't worry and jan says um yeah of course there's a waterfall we will agree on it's about you know uh a few years to a few decades away uh but we'll figure it out when we get there so that's like a high level kind of description of each one's positions um max also talked about some existential risks that i haven't been like really covering myself or uh talking about in in what i wrote uh that has to do with a slow um disempowerment as we get more and more dependent on the eye and the eye builds up its power at some point um it becomes very very hard to pull the plug even if we could um because we're completely dependent on it and it you know the society would break down and and yeah so so that's another kind of scenario which is a sort of uh interesting to think of and some people in the ai safety have been also thinking about but i've been mostly talking about the simpler scenarios that i talked about in the rogue ai essay so the very very simplest one uh doesn't even require like some very fancy knowledge of machine learning or anything or reinforcement learning or ai safety it's just that But it just assumes that in the near future, which as I said, there's a lot of uncertainty, like is it years or decades?

6:59We know how to build a superhuman AI. and if that recipe is publicly available and even worse if the weights of a trained AI are available then it becomes very easy for a huge number of people to issue instructions that if they were executed by the AI could lead, you know, terrible catastrophes. And in the worst case, could lead extinction. And, you know, I can explain maybe with some scenarios what that means. Yeah. Yeah, so that's, I feel like I haven't heard any like serious criticism of that. I mean, maybe the strongest counter argument is Jan saying, don't worry, we'll build, we'll use the progress in AI in order to build counter, you know, measures AI that are going to help us fight off the rogue eyes.

8:16and because there will be more good AIs than bad AIs, the good guys will win. Maybe, and I think we should definitely do that. In fact, I'd like to work on something like this, but I don't think it's a silver bullet. We know in military situations or conflicts that sometimes the attacker has an advantage because the attacker can do things silently and sometimes there is not a lot of time to react or to prevent uh you know some large amount of damage then I talked about the what I called uh what I now call the Frankenstein scenario so so there is the sort of naive scenario where somebody naive but but but also but plausible where somebody just out of anger or uh you know conspiracy theory you know crazy beliefs or for military reasons issues these highly destructive instructions um a slightly um kind of less obvious but but but also fairly likely scenario and uh that enlarges the set of people who would do this is someone issues instructions or you know designs the ai so that it would have its own self-preservation instinct in other words one of its main goals is to survive just like us and every other living being which essentially means we end up with a new species on earth a new species that would be smarter than us.

10:01And, you know, the past of more powerful or smarter species acting in ways that have driven to extinction, other weaker, less smart species isn't very reassuring. Just in the last few hundred years, humans have driven almost a thousand species to extinction. That's the ones that are you know that we know about and that's not to mention our cousins hominids uh that have all disappeared we don't know why but you know uh be surprising if if homo sapiens didn't have a role in this and and when we drive a species to extinction it's not necessarily that we want to destroy them it's just that they are like a their destruction or their disempowerment is a side effect of us trying to achieve our goal.

11:01You know, we want more land, more profit, more food. And that's mostly the way that we've driven a lot of species to extinction. Or some species still exist, but they're completely disempowered. So gorillas are still there. We're probably going to keep them around. But how much agency do they have, you know, over their environment? It's all controlled by us. I don't think humans want something like this for homo sapiens. Yeah. Can I just interrupt for a minute? Because certainly these scenarios have been in the public space and in science fiction novels and movies for decades. um what happened that that uh and and certainly you've acknowledged uh sort of distant risks in the past people have talked about it but the consensus by you and jeff hinton and others was that that is so remote we don't have to worry about it right now there was this dramatic uh step function, you know, where things progressed very quickly with the transformer algorithm.

12:24But why now? I mean, that generative AI, certainly there are abuses in the current current level of technology that are possible through different disinformation and that sort of thing. Or if they're hooked up to the internet, you know, you could construct scenarios where they could do damage to things that are available through the internet. But why did you suddenly decide that this threat is near-term enough that we need to act on it?

13:10In a few words, we've reached the stage of essentially passing the Turing test, meaning that it's hard to distinguish if you're talking to a human or a machine, if you're exchanging with gpt4 i mean and right now if you ask it it will tell you that it is a machine but but it could easily be the case that it doesn't do that so that was of course thought by alan turing himself many many decades ago at the beginning of computer science that is a way to evaluate whether we've breach human level intelligence. Now, I don't think we have, because if you take much more time to kind of try to trap GPT-4, you can find some issues.

14:07But there are like, I don't know, hundreds of billions of dollars invested to fix those issues because GPT-4 and, you know, chat GPT have been working so well. uh and in fact the work that i've been doing for many years is all about fixing those issues it's adding the system 2 capabilities to deep learning right now deep learning is very good at system one capabilities which roughly correspond to intuitive immediate reaction you know behavior that you do without taking the time to think about it and that require a lot of practice in order to hone in. System 2 allows us to do things where we don't even need a lot of data.

14:52We can just think through, reason our way to find solutions to problems. And we have a sense of whether some statement is true or not. And we can reason with causality and things like this. But there's been progress on System 2 and other issues that people are working on. it's hard to say if we, you know, maybe we're close to a breakthrough on these aspects, maybe just two, three years, or maybe it's 20 years. There may be other obstacles we don't see right now, but from my vantage point, I'm very concerned that we're not very far. And I also know that society takes a lot of time to adapt, to come up with countermeasures, regulation international treaties everything that i need i think will be necessary in order to reduce those risks and finally you might have you know you asked me like why now why didn't i like think about it before why didn't jeff think about it before well so we thought it was so far away we didn't need to worry um but also there i believe there's a psychological effect which might be still at play for a lot of people it's very hard um in terms of your um feeling of uh you know your ego and and and how you can feel good about what you do to accept the idea that what you've been working on for decades might be actually very dangerous for humanity because there are people who've been working on ai safety for you know at least a decade and others longer but mostly it's been in the last decade a lot of work has been done i mean it's very very marginal in the machine learning community but still there are people who have been worried for for for some times and they've been writing papers to try to think about different scenarios and countermeasures and so on um people like stuart russell for example who wrote a very nice book, Human Compatible, about that and how we might potentially build safe AI systems.

17:12So I think that I didn't want to think too much about it. And it's probably the case for others. You were saying that you could walk us through the extinction event, and it's really the word extinction that that bothers me as i said on twitter they're they're they're apparently credible people who are already sort of bemoaning uh the extinction of the human race because it's too late uh to stop this development uh and and when that gets into the public uh it it It just spreads fear. And, you know, there are a lot of wonderful things. This is bad, for sure. Yeah. And there are a lot of wonderful things that AI can do.

18:06I mean, in terms of extinction events, you and I have talked about this. We are facing an extinction event with climate change. Very little doubt. Those, the systems that contribute to climate change are so complex. We need the pattern recognition capabilities of AI. We need the reasoning capabilities of generative AI to help solve those problems. So from my point of view, AI is probably our best hope to avoid extinction. But when you tie AI to extinction, you know, yes, it seems to have woken the regulators up, but it creates a very strong headwind for the positive uses of AI. Yeah, yeah. I'm totally with you on this.

19:08and i would just like us to be wise and walk this fine line where we can get the benefits or at least a large number of them but avoid the catastrophic outcomes so nothing is going to be easy and just like uh being submerged by fear isn't going to help either um here's the thing about desperation i don't think that we're cooked um let me take the the climate change analogy um climate activists could feel desperate and just drop it drop the ball but they don't we should have acted 20 or 30 years ago we we did it but But now we are today. What can we do to move things in the right direction? And there's until we have agency, which we do right now, it is possible to reduce those risks.

20:26That's what I'm trying to do. And yeah, we have to be careful not to create a panic, but trying to steer something positive that is going to help on both sides. There's this concern about superintelligence. right now what we're dealing with is a system that by ingesting massive amounts of data has learned to predict the next token in a series essentially when applied to language that it predicts the next word there's there's no uh deeper intelligence uh there uh there may be the illusion of uh of even sentience but you're involved in building these systems uh and you know that there there is no uh uh, uh, what's that?

21:35Yeah. There's no ghost in the machine. Thank you. Uh, and the idea that, that we will arrive at the point where we can build that, uh, no one knows how to do that. Uh, there's certainly, uh, people trying, it just seems the scaling up of transformer models was surprised everybody at what the system could do. But that's still a huge leap from building a machine that has an independent intelligence and agency. So no, actually not. So, you know,

22:22AutoGPT showed you can take ChatGPT and create a thin layer around it, a wrapper, that provides it with agency, for example, to act on the internet. And you say that, you know, there's a huge gap. Well, how do you know? I don't see that. I see that it could be a short while. I don't know for sure. And because I'm not sure, I don't want to take the risk that, oh, let's ignore the potential problem.

23:03Also, there is no ghost in the machine in your brain either, as far as I know. Yeah, but there are many, many, many more structures in my brain and in your brain than a neural net. Yes, that's true. It's true. But it looks like, so the kind of intelligence we're gradually building is not a copy of our brain. You know, evolution has come up with our design through many torturous turns. It's not clear that it's the only solution to intelligence. Personally, I think that we've solved a large part of the problem. System two is something that came fairly late. the scale of evolution. We're not sure, of course, but it looks like something that might not require that many bits of information that's been added on top of all the things in evolution I've found before.

24:24So that means we may not be that far, and we have already some plans to deal with that. What in your research, I saw recently you said that you're going to shift your research to safety. What in your research are you abandoning that would lead toward more powerful AI? What I would like to avoid right now is to put out information that's publicly available that would help to bridge that gap that remains too quickly. I will continue my projects in AI for social goods uh which is about very specialized systems that that you know don't understand how society and humans work but maybe some very small part of it um and and like i need to take the time to think about more precisely what is my best course of action the uh you you you talked in the the uh paper uh on on how a rogue ai could arise about open source and the danger of open source uh and and you talked about uh you know genocidal humans getting a hold of an ai and and doing right something destructive with it uh on in in both of those cases even with open source code you need at least today a tremendous uh financial resources uh to marshal the hardware well you don't need to marshal the hardware yourself you're right it's available through the cloud but you still need to pay for it no no if if you if the if the weights are shared then it's very cheap to um you know fine tune it put the layers you want around it so it's going to be specialized through the task you want which may be some some malicious objective or some nefarious objective so you're saying if there's an open source pre-trained model yes that you can copy the weights uh that's already the case so i mean but not not with you know uh superhuman ai but it's already what what essentially facebook is doing with with uh open sourcing uh models yes including the weights yeah and then on the agency side uh you have these wrappers like auto gpt that that do give uh these large language models access to the internet.

27:30However, and goals, right? They turn them into goal-directed AI systems. Yeah. Although I've played around with AutoGPT. We talked about that last time. It's not as straightforward. That's not the point. The point is that, I mean, people who know about reinforcement learning know that it's easy to design something like this. now that the reason it's not working that great is because auto gpt itself i mean chat gpt itself is not that smart but what about two three five ten years from now yeah uh the uh you also uh in your paper and i appreciated this talking about uh corporations as a form of artificial intelligence that do a lot of bad things in the interest of a goal that's not aligned with society.

28:37So a lot of this, when you say work on safety, are you talking about working on the so-called alignment problem, how you instill human-aligned values and goals and prevent? That's one way. So the other way that I've been talking about is to try to design AI systems that don't have agency like instruction

29:10and are trained not to please us, like current chat gpt um but to be truthful in a probabilistic sense uh to understand how the world works and then to answer our questions based on this knowledge um now these could still be turned into agents in the same way that you know auto gpt works so you need some other defenses and and i think with a combination of both like technical aspects and uh societal aspects like you know who has access what kind of governance you put around these things um who decides what we do with this and so on. I think we can considerably reduce the risks. The EU has passed its, though not passed, but the Parliament has submitted its first draft of its AI Act.

30:20Canada, I'm not sure where they are in the process, but are close to doing the same uh do you think these regulatory efforts answer your concerns they're stepping in the right direction but they haven't been designed for catastrophic risks so can you walk us through uh an extension extinction scenario i mean that's what everyone's a little alarmed about, a little, when people like you talk about extinction. So if you could just give kind of one scenario of how that could happen and understanding that, as you called it, there's still a gap between the AI that we have today and the AI that would pose that sort of a challenge.

31:16One scenario that I think about is this Frankenstein scenario where we have a tendency to want to build AI at our image. And in fact, a lot of people today think this is the right thing to do. Problem is, if we do that, it means we are giving these machines the same kind of self-preservation instinct that other you know that every animal has and with that come as a important sub-goal if you want to survive first of all, you don't want to be turned off. That's the first thing. So if we ever have the intention of turning one of these systems off and it understands that, it's going to do its best to prevent us.

32:19So you can see already a conflict here. Also, the best way to survive is to figure out how to control your environment. I mean, all animals do that. to the extent it can. But now we're talking about an entity that is smarter than us. So we are part of its environment. That means it wants to control us. Yeah, but we're not talking about embodied intelligence. There's no physical manifestation. That comes easily. So first, the kind of systems we're building now, and we know how to build probably for the next few years, you're right, don't have a body. They have actions, though, in the real world, which are, for example, the dialogues or the images or the videos that they can produce.

33:18And potentially, if somebody connects them to the Internet, all the things you could do on the Internet. um so how could they um you know do something in the in the in the non-virtual world well they can convince they can manipulate people they can pay them for doing some things you know this is you go on some websites and you ask people to do things legally or you can pay them to do illegal things like organized crime is happy to do things without asking where the money comes from and these different people can be contributing to a you know a bigger project and looking at their piece doesn't seem so terrible right at some point you could imagine that these systems will be able to design robots that are you know better than the ones we and then have their own body as well i mean it's not the same thing as our body because we you know if our body goes away we are we're finished but but these systems can reproduce themselves they you know if if one body dies it's all right they're you know copies of the code in many places um in a way they are immortal unlike us you know our body is going to eventually uh not be functional anymore um they can copy themselves as as many times as necessary um so the the the yeah that's the kind of scenario by which they can take control now how could that lead to what kind of actions could they do people have been talking about two main vectors and maybe they'll think of something else one is uh bioweapons and the other is cyber security attacks and by the way with cyber security you could gain control of uh things that that have an importance in the real world like uh your energy infrastructure or weapons bringing down our communications infrastructure, you know, combined with other things could be terrible.

35:48So, for example, if you combine that sort of thing with the release of a bioweapon, in other words, new pathogen, that would be more virulent and viral than anything that we already know. It's quite conceivable. There's already research going on to use AI to better understand how our cells work. And I think this could be an extremely positive revolution, but it also means that we might reach a point where AI systems could design new living beings, starting with viruses, that could either help us a lot or harm us a lot. Yeah. So it's really an issue of agency, of whether or not we allow these systems to have agency.

36:39As you pointed out, there are already these wrappers on GitHub that are giving GPT-4 agency. although as i said it really doesn't work because gbd4 hallucinates and auto gpt just compounds the hallucination and next thing you know you're stuck in some endless loop where the thing crashes but as you said that you know it's early days those those problems will be addressed So would this mean, on the regulatory side, outlawing, allowing these systems to have a live connection to the internet? at i mean uh maintaining them as air-gapped systems is that the solution or or do you think it's more about uh just regulating uh the the systems themselves i mean how large they can be or how capable they can be so first of all you know i've just started thinking about these questions i i don't feel like i have all the answers at all i think we need as a priority to spend more of our brain cycles to study these questions um both on the side of ai safety like the computer science side and on the policy side but it seems reasonable to consider some forms of open source and sharing of models to be something we want to limit when these systems are dangerous or potentially dangerous of course but most of what we do in AI right now is it's not dangerous at all right so we're talking about systems that don't exist yet, but we don't want them to be too easily accessible.

38:51Although the flip side of that has, I mean, you've been through all these arguments endless times at this point, but the flip side of that is you have a small group of people holding tremendous power. And that was the argument behind setting up OpenAI in the first place, ironically, was because Google had this proprietary tech that was extremely powerful and there was a thought that it should be open sourced. But of course, OpenAI then came up with extremely powerful tech and closed it as well. So, yeah, what do you say to that argument that you're then concentrating me? who gets to decide who controls it it's just the whoever has the most money to hire the most engineer well not necessarily um we could set up governance systems by which these organizations work for the public um so concentration of power is something that's already happening and that we want to avoid.

40:13Absolutely, I agree. So we have to find ways to be safe and preserve democracy because democracy is all about not concentration of power. How can we do that? So one thing I've thought about, and several people have talked about a CERN of AI. So the idea is, actually, I think there should be several, but to have organizations that are non-profit and under very strict governance to make sure that we avoid the arms race between companies or between countries that is going to happen if we do nothing. So it's not just concentration of power, it's also the arms race. In other words, companies or countries competing with each other and cutting corners, which could be dangerous from a safety point of view.

41:17Now, you know, nothing is easy and, you know, I'm not sure if governments will be eventually convinced that this is the right path, but that's my opinion. So what I mean by this kind of governance is that to avoid the arms race, it would be something in which as large a coalition of countries as possible, and especially those that are opposed to each other, like the US, China and Russia, are part of that governance. so that what is going to be done with that AI is going to be managed so that it's not targeted at, you know, hurting some other country or something like that.

42:16Yeah, and I think that if we allow these kinds of super powerful AI to go on in companies, they would also have to obey very severe protocols of safety.

42:33Yeah, with people like you and Jeff peeling off from the basic research, do you think on the one hand that progress toward AGI or toward this super intelligence that that we're concerned about will slow or do you think that there there'll be a parallel track where uh we're working on safety and guardrails and policy and regulation and on the other track there's you know yen lacun is not slowing down on his world model efforts right i think it's going to be parallel tracks for a while

43:26I think that even if Jeff and I didn't contribute to the public advances in AI, there are so many people working on this, so much money invested that, you know, we're going to continue to move forward. but and that's exactly the reason why i think we need these tightly governed organizations that will build hopefully the first superhuman ai but in a safe way that can help us mitigate potential attacks from rogue eyes coming from the uncontrolled efforts

44:14But really, these are all speculations. I think it's very early right now. I'm just drawing these as potential scenarios. I think more people need to think about it and exchange, and these scenarios need to be evaluated and discussed across many stakeholders. Yeah, I saw Andrew had a conversation with Jeff Hinton. And one of the things they said is that right now the AI community is divided on this question. And in certain quarters, as I referenced on Twitter, that debate is becoming pretty heated, pretty angry. and what needs to happen is the two sides within the research community need to come up with a consensus view because otherwise it leaves the public and presumably regulators confused.

45:24You know, who do they listen to? Yeah, it would be nice. I don't know if it's going to happen. there's always been issues that divide scientists on which policy have to take decisions. Think about climate, tobacco, for a while at least.

45:54The way that I think about this, if I was in government, is that there are scientists who think this is a very dangerous issue. There are scientists who think we should not worry about it. We don't know who's right. But the stakes are high if we ignore the problem. And it turns out, you know, the concerned one, we're right. So I think we need to apply the cautionary principle here. as a society. Right now, that means regulation. Are you working with the Canadian regulators or with any of the EU or the US, for that matter, on advising on regulations or legislation? Well, I've mostly been working with the Canadian government, but I've been talking to senators in the US as well yeah and uh so you're just to sum up uh you know we don't know how distant this threat is but uh the threshold of of capability has been reached where it's time you think to start working seriously on these problems uh but without scaring the public about imminent extinction which i think is what's creeping into the public discourse yeah it's not clear how to do that yeah i mean the strategy um the way i try to present things is not to focus on extinction risk, but just catastrophic risk, which is already bad enough.

47:51Think about nuclear weapons. I mean, the most part they're used will survive, but very badly. Climate is likely like this as well. We'll probably adapt, but society could be badly destroyed. Many people could suffer and die. That's it for this episode. I want to thank Yahshua for his time. If you want to read a transcript of this conversation, you can find one, as always, on our website. That's www.ey-on.ai. And remember, I don't think this singularity is near. But AI is changing your world, so you better pay attention. you