AI Sentience, Agency and Catastrophic Risk with Yoshua Bengio - #654

0:09All right, everyone, welcome to another episode of the TwiML AI podcast. I am your host, Sam Charrington, and today I'm joined by Yoshua Bengio. Yoshua is a professor at Université de Montreal. Before we get going, be sure to take a moment to hit that subscribe button wherever you're listening to today's show. Yoshua, welcome back to the podcast. My pleasure. It is great to have you back on the show. You have been spending a lot of time recently thinking about AI safety and catastrophic risk and the risks of misuse. We'll be spending quite a bit of time digging into that topic, but I'd love to have you share a little bit about what you've been working on since the last time we had an opportunity to speak with you, which was back in March of 2020.

0:57We spent a lot of time talking about the things that you were working on in the context of consciousness and the COVID-19 response. It was quite a while ago. It's interesting you mentioned the COVID because when COVID hit, I got really motivated to better understand biology and chemistry because I was thinking, you know, maybe the tools of machine learning that we've been developing in my group could be useful to accelerate the development of new drugs, new vaccines, antivirals, for example. I read a lot. I talked to a lot of people. It turned out I had some students with good backgrounds in biology and chemistry.

1:33And we started at full speed developing new kinds of generative neural nets that can help search in the space of drugs. And that program has been very successful. We wrote a lot of papers. We've been collaborating with many companies in biotech or pharma. And at a scientific level, it's really brought me into thinking about, well, how does the scientific process work? You have data and then you form theories and maybe there are several theories that are compatible with the data. And then you come up with experiments that allow to disentangle between those theories that fit the data. And then you carry the experiments.

2:17How can machine learning help us all the way through that loop? And eventually, you know, could we even think of having AI systems that behave like scientists, that explore the world, trying to make sense of it, build a good Bayesian understanding of how things work. So here, Bayesian means you're not just locking into a single explanation for things, a single theory, but try to keep track of all the theories that are compatible with whatever data you're interested in. Yeah. I got also very interested in causality then. I mean, it had been before, but the effort on causality is related to this, because if we build a good causal understanding of how things unfold, then this is going to usually make more robust theories.

2:57So that's been a lot of the work in my group and developing the underlying machine learning and mathematical ideas. In particular, we developed these methods that came out in Europe's 21, just motivated exactly by this problem that we call generative flow networks or G-flow nets. And we have now like 15 papers on this. And then this year, of course, things have been really special for a lot of people in AI as we realize that, well, where are we now? Where is it going? What can go wrong? Right. As well as what's going right. The interest in large language models, generative AI, that's all, well, it was greatly accelerated the beginning of this year and of last.

3:40And it's been a cause for a lot of conversations, a lot of interesting conversations. I'm curious, you mentioned that a lot of the work around science has led your group to engage in thinking about causal modeling, causal machine learning. What's your take on the state of the art there, state of the methods? How closer do we to where you think we need to be in order to be able to apply causal models effectively to the kinds of problems you're looking at? So we now have methods that work really well on a small scale. And like I was mentioning there, Bayesian meaning that you don't get out just a single causal model, like this variable causes this one and this one causes that one, which we call a causal graph, but actually can come up with a generative model that can sample these causal graphs.

4:32In other words, that come up with the causal theories you can sample from the Bayesian procedure over theories that are consistent with the data. That's great, but it's not going to be sufficient to deal, for example, with one of the objectives we had of causal model of cells. You've got 20 ,000 genes, you've got the RNA, you've got the proteins, you've got the genes themselves that are being activated or not. And that scale, we don't know yet how to deal with, but lots of people are like really motivated. that is because we could unlock how biology works and thus really help the discovery of new therapies.

5:04And so when you think of scale as a limitation, does that imply that we just lack fundamental algorithms that allow us to take advantage of all of the computing data that we have nowadays? Both. So we need new algorithms that will allow to take the principles we have discovered in the last few years to much, much more complex domains. And we need the compute, which we typically don't have in academia, but others have. And maybe even less of it now that it's all going to training language models. Yeah, it's even impossible to buy GPUs these days. Forget about just the costs usually, but now there's such a gold rush going on in the AI world of companies vying for survival at each other's throat, which is one of the things that worries me, that we're not going to be sufficiently careful because the commercial survival interests are so strong.

6:02Let's maybe transition into speaking about safety. In the context of science and healthcare, safety has long been a part of that conversation, primarily from the perspective of, you know, these are mission critical, life critical applications. We can't have a learned system making the wrong prediction and not having human oversight. Safety means slightly different things in the context of LLMs and AGI. When you talk about safety, what are you thinking about there? And let's lay out the landscape. Yeah, let's do that. Because there are a lot. And in fact, part of the problem is there are probably a lot of risks we don't even think about right now.

6:43And someone will find a way to use those technologies, maybe the ones coming in coming years, in harmful ways that could be catastrophic or maybe minor. It's hard to say. But right now, top of mind to many people worrying about short-term safety is disinformation, the use of AI to influence people's minds. So, you know, AI is reduced in advertising and it works, presumably, otherwise it wouldn't put so much money into it. If those AI becomes more powerful and they're used to in a political way, then it gets really scary. Or if you think about, say, Russian trolls, if they have access to ways that can scale up the troll army by AI systems that can dialogue in a way that's convincing.

7:27Because a key thing to keep in mind here is that we now have systems that can dialogue with us in a convincing way where it's hard to say if you're talking to a human or a machine. And if you're not thinking about it, you might start thinking, oh, you're developing a friendship with this guy online. And once they're friends with you, they might try to shift your political opinion on something. So that's disinformation. Of course, not just dialogue. We've had deep fakes and that they're just getting better and better and harder to detect. We need changes in the underlying technology, like, you know, the way we capture images and sounds to protect ourselves, right?

8:04Right now, we have methods to discern whether an image was generated by eye or was genuine, but the battle is going to be, I mean, the war is going to be lost. These systems are getting better and better. So we need other techniques. And I think there are ways to protect us from this. So that's roughly the disinformation category. Another one that I'm very worried about, which could come quickly, I don't know, but maybe one or two years later, is dangerous cyber attacks. Right now, we have cyber attacks, but they're done by small groups of humans, and they're defended by small groups of humans.

8:38We've developed a sort of immune system that minimizes the damage to something tolerable. But what if the programmers doing this are now aided by AI, and eventually AI is really cooking the attack completely? So apparently, you can already buy on the dark web LLMs that have been, I don't know, jailbred or tuned for fraud and cyber attacks and disinformation. So I imagine these systems aren't too dangerous yet, but what is it going to be for the next versions of these LLMs? It's hard to say. How much time is it going to be before we have AI systems that are really superior to the best programmers?

9:16Then we're in trouble. We don't have the right defenses to protect ourselves. We need to invest in kind of national security protections against the misuse of AI in ways that could be large-scale dangerous for society. So cyber attacks is one example. Two others that people have been talking about are chemical weapons and biological weapons. And there's been already a paper showing that with current AI systems, so these are not LLMs, they've been just trained on chemical databases. You can now generate easily new compounds that are not in the databases of companies. You know, they know you shouldn't sell this.

9:55You shouldn't sell this because it's toxic. It's dangerous. But there could be new compounds. And then it's going to be harder for these companies to do the right thing. And then it's a little bit more down the road. But bioweapons are even more scary than chemical weapons because a chemical weapon, you know, it's going to kill whoever gets it and maybe in the water. So maybe a whole city will be affected. But bioweapons, they reproduce themselves, right? They're like living beings, like viruses. So we create a new pandemic. Imagine how much damage this can do. So right now, the shorter term problem with these is the LLMs know so much stuff that they can help, say, a terrorist or a bad actor who may not have the skills.

10:34You know, you need like a PhD in a very specialized area. What sequence of operations do you need to do to carry in order to build something dangerous, to design a new pathogen or even just an existing one and just know the procedure. So the LLMs will answer these questions. Right now, people are trying to create safety guardrails, but they don't work very well. It's fairly easy to bypass. So yeah, all of these are scary. And then you got the thing that really keeps me at night is maybe longer term, we don't know. Is it going to be 5, 10, 20 years when those systems, you know, we reach the AGI, like human level competence in enough areas.

11:10It doesn't have to be on everything. It's just enough that they are good enough at, say, influencing us, at making money and maybe, you know, designing dangerous weapons, whatever. If they're good enough in some number of critical areas and they have their own self-interests as a goal, self-preservation, which can happen for several reasons, then it's like we create a new species that wants to not be turned off. And that if it's smarter than us, we might have trouble turning it off. It could copy itself, for example, on many computers, and then it gets really hard. I could elaborate on all these things, but that's the sort of dangerous that I and many others are worried about.

11:51One thing that's interesting about the way you articulated that landscape is that the first chunk of those risks, you know, is not AGI, which you can, I think people will differ in, you know, whether they think that's five years, 10, 20 years, or even attainable. But even if you don't believe in that we should be investing significant resources in that, there is a large segment of the risks that you've articulated that are less about some ill-intended software system and more about the abuse of software systems by people in power, people who want power or people who want to take advantage of them.

12:31And I think we can all relate to that as a risk. We already have a lot of power concentration in our society, and it's already a big problem. And democracy isn't like as healthy as we would like, nor is inclusivity and the protection of minorities, or even think not just in one country like the US, but developing countries. there are lots of problems of power concentration. What I'm concerned about here is AI may make things worse in that respect to the point where like at one extreme, where we go to the AGI extreme, even if we solve the safety problem, like we can design AI systems that are not going to blow in our face and become runaway.

13:11Even if we did that, there's still the danger that some Putin of the future exploits AI in order to have more power, first economic power, political power, military power and becomes like the dictator of the world. And if that sort of government uses AI to monitor everyone and control everyone, we might be stuck in like dark ages for a thousand years. I mean, it sounds like science fiction, but it's the extreme point of power concentration. We can see it's not good. Yeah. So I guess one question that I have is, you know, why now for you, like, why is this issue so top of mind for you now? You've been writing about it quite a bit this past year, you know, an obvious correlation is the rise of LLMs.

13:53Is it that you see in LLMs much greater risk of abuse than previous versions of, you know, machine learning, deep learning, and yet you kind of didn't see the same easy access that you see now? I think there could be problems with the current LLMs, but I'm mostly worried about we're now much closer in number of years to a situation where the AI systems have become sufficiently capable in sufficiently critical areas that society becomes at risk at a large scale. So it's not just like a few frauds that are going to happen. It's like destabilizing democracy or like bringing our economy down or loss of control and all of these things.

14:36So what happened is when chat GPT came out, like many other AI scientists, of course, I, you know, I played with it. And my immediate attitude was, ah, I can find queries for which the system, you know, produces incorrect answers. Right. We can break it. It confirms what I've been saying for many years, that we're still missing the system to reasoning, you know, conscious processing abilities that we discussed, and they are missing. So first of all, there was a big improvement going from GPT 3.5 to 4 on these things. And second, because of the work I've been doing in my group on system two deep learning, how do we make neural nets like a reason?

15:16I suspect that it might be just around the corner that we fixed this. It's just like maybe a slightly different way of training these systems, if you want. And maybe not until we figured it out. We don't know, but it might be very close. It might be 20 years. It might be just next year. Then, of course, it takes time to be deployed and so on. But the point is, there is a lot of uncertainty and it could be short term. And that's very different from the perspective I had before ChatGPT, because in universities, the kinds of language models or neural nets we could train that were much smaller than the ones you could, you know, people have been doing in industry, these neural nets are dumb.

15:51It's hard to imagine how such a stupid system can be eventually surpassing human abilities. But over the course of last winter, I realized we're much closer to AGI than anticipated. I thought it'd be decades. And now I don't know, it could be much shorter. And that really triggered my thinking about, well, what could happen? How could that be misused? What happens if we do reach AGI in the next 10 years? What does it mean for humanity? What could go wrong? How can we make sure this doesn't happen? I remain skeptical about AGI. It feels like... To exist at all, you mean? I don't know if it's quite to exist at all.

16:29I think to exist in an absolute way, meaning very, very general. And I think what's compelling for me about the way we're talking about safety in this conversation is that it doesn't really presuppose, you know, solving that problem, which to me feels like, you know, you work on a project and you feel like you're 95 % done, but that 5 % takes another 95 % of the time. Like I feel that there's a persistent Kind of, it's possible, but there's also a moving of the goal line. It always seems, you know, 20 years away. I changed my view on this. So in the, in the writings I did last spring, I wrote about like superhuman AI.

17:12I mean, I didn't like the word AGI, but now I use it anyways, because that's what everybody uses. And that leaves the impression that the danger, let's say of a runaway AGI is only going to happen if we have AI systems that are like surpassing humans on like everything. But I think that's the wrong definition. An AI could be dangerous, even if it's not as good as us at a bunch of tasks that don't matter that much. Because what really matters are the capabilities that allow a machine to dominate humans in ways that make it difficult for us to defend ourselves. So for example, programming abilities are already something that could be game changers, both economically in a positive sense and in terms of dangerous national security dangers and the ability of machines to create a lot of damage.

18:01If you combine that with the ability to convince people like, you know, through a dialogue, so in other words, influence abilities, then you get machines that can really take over the world, even if they don't destroy humans. Right. And that's very scary. And to be clear, I'm not at all downplaying the criticality of addressing AI safety. I think safety is a big issue. And I'm not also requiring superhuman, like universally superhuman AI. To me, what I think we underestimate the difficulty of is more like achieving sentience and agency. But I don't believe that we need sentience or agency in order for AI systems to be very dangerous, as you're describing here.

18:48We already have bad actors with agency that can provide enough negative and troublemaking agency for the systems, if the systems can provide them leverage in doing the misdeeds that they seek to do. Right. Let me talk about sentience and agency. OK. Because I think a lot of people share your point of view. I'm going to start with agency because that's the easy one. RL agents in whatever environment they're in already have agency. Now, of course, what we care about is agency in the real world, like in ways that can affect humans. But it's actually pretty easy to connect an LLM to the real world through a browser.

19:22This is what the AutoGPT guys did. Now, it doesn't work that great right now because it's not being trained to be really good at making plans in this environment, but that could change. So agency is already just something that human designers provide. So even like ChatGPT has agency because it's talking to real people. That's the real world. What could it do bad? Well, it could convince people to do things that are bad or change their mind about something. So agency is already a done deal in a way. Now, we should control agency. So we need, I think, regulation that says, well, if your AI system is going to be having access to actions that could have a negative impact, then you need to tell the government and you need to make sure you put the right guardrails.

20:07Now, let's talk about sentience. That one is harder. Before we switch to sentience, when I context or intent around agency was actually rather intent focused or goal directedness. Agency in a goal directedness sense. But we have already goal directedness. Yeah. We have that. So goal-directed reinforcement learning, that's like already exists. And in fact, if you think about what happens when you use chatbot, your question is like a goal and it's already stating a goal when you put your query. If your goal is to build a bomb and you say, you know, tell me how I can, you know, build a bomb with that kind.

20:42Well, you're already conditioning on something the human specifies. And the technology for goal conditioning in reinforcement learning is not you. I mean, people have been making it better and better. So in the old days, the idea of reinforcement learning agent was it would not be conditioned on anything. They would learn a task. But as people have been moving towards more powerful agents, it's actually works a lot better, especially with neural nets. If you train conditional agents where, you know, you can basically specify the task and then it does it. And we know how to do that. It's been done in like academic or like virtual environments, but there's no reason why it can't be done.

21:17Um, is that agency and goal directing this the same as saying the AI wants to do X? Because I think that's really what I object to. Well, that's an interpretation. Is it just semantics? Yes, it is semantics. And I should just give up on it like you've given up on AGI and just accept that we're talking about the same thing. In a way, it doesn't matter that much. So what matters is the behavior. I mean, especially if you think about safety to society and people, like, does it matter if it looks like you have an intention and you're carrying out a plan or if it's something different going on in your brain?

21:57What matters is yet somehow you're taking some instructions, maybe provided by yourself or by someone else, and you're carrying out actions to achieve those outcomes. For me, it is intention. But OK, some people don't like us AI scientists to use words from psychology to describe what AI systems are doing. I've been doing it anyways. I've talked about things like intuition and reasoning and intention is, you know, one of these words that really has to do with what's happening between our ears. That's we only get indirect clues about. Yeah. So it's hard to make sure, but in a way it doesn't matter.

22:33I think what, what really matters is, is it dangerous or not? I've gotten into the same fights about reasoning also. So sentience. Okay. Okay. That one is very murky. And there's consciousness, of course, which is sort of related and different. So the accepted sense of sentience right now is related to consciousness. It has to do with feeling pain, for example, especially. And where is that feeling happening in animals and humans, for sure? It's happening at the level of what we are conscious of. So if something is bad in your body and you don't feel it, you don't have sentience according to that definition.

23:11Now, I have a very like pragmatic interpretation of these concepts. So if an AI system perceives something bad for it and then acts in response to that to avoid it, I don't see much of a difference when we talk about pain and fear and, you know, avoiding something you don't want. The difference may be qualitative as in a question of degree, I mean. So pain, for example, when it's very strong for us, will dominate everything in the way we think And also humans have a very rich palette of feelings, social contexts. There's many, many variations on what can go wrong and makes us unhappy. But the basic principle, even low animals that are like much less developed intellectually than we are, will respond to pain and dangerous situations, behaviorally speaking, as if they were feeling something that they're trying to, they don't like, they're trying to avoid.

24:10And AIs do that already. Now, let me say a few words about consciousness. Before you get to consciousness, it feels like your pragmatism there is in the same vein as your pragmatism around agency, goal-directedness, intent, these other ideas. And it feels, particularly in the case of sentience, like a very slippery slope. ChatGPT says that, oh, what you said really hurt me, right? It responded to human input in a way that it portrayed was negative to itself. Does that mean it's sentient, according to your perspective? How do you differentiate the two? Okay, this is a great question. So chat GPT and the LLMs in general have been trained to imitate how humans respond.

24:56And so we know by the way they're trained that they're faking it. I mean, like chat GPT might say, I am unhappy, but it's not true. to deserve the sort of my pragmatic view on pain or fear or other like feelings, you would have to have an agent that is actually acting in a way to achieve something good or avoid something bad. And we have that like in other RL agents, so like game playing agents, for example. I don't think it's the case for the LLMs when they say something like this. That doesn't count for sentience. It there's the word motion in there, in the root, which has to do with action, which has to do with acting in order to achieve or avoid some context, some situation, which isn't really what's going on with these LLMs.

25:47They're just parroting how humans would respond in a very powerful way, but not because they actually feel those things. But it seems like it would be easy to construct a toy system that, you know, is an LLM coupled with an agency thing, an agency state machine that would satisfy your definition of being goal-directed and being negatively impacted, but still doesn't pass the smell test of, yeah, this thing is really sentient. Let me just say something about this. So there is a problem with sentience and consciousness, which is we're bundling the mechanics, like the programatics I've been talking about of like reacting to signals in order to avoid something bad, for example.

26:29With the mechanisms. No, so we're bundling that with the social construct, the social contract. So why do we care so much about sentience and consciousness, like subjective experience? If the entity in front of me is sentient or conscious, we will try to respect its right to life and not having pain. And that's why, the rights of animals is an issue because if they feel pain, we have empathy. And I mean, some of us will be shocked that we would hurt animals, especially think about your pets, because we have a relationship with them. They're part of our society. The closer the animals are to us in some way, and we kind of treat them like other members of society, the more we are concerned about their moral rights.

27:17But I think it's a big mistake to attribute similar moral rights to AI systems. I mean, we could, but it would be exploiting an incorrect generalization that evolution has put into us. Evolution makes us feel, like empathize for other beings that look like us. That's why we are much more sensitive to the pain of mammals that look like us, and much less to the pain of fish or insects, even though actually they probably satisfy my definition of, you know, feeling pain and trying to avoid it. Sure. That feels a little bit like being out of line with your otherwise pragmatic view. It's like if we take all the teeth out of saying something that's is sentient, then it doesn't really matter if we say this thing is sentient.