Yoshua Bengio: Pausing More Powerful AI Models and His Work on World Models

0:00Hi, I'm Craig Smith, and this is Eye on AI. Hi. This week, I talked to Yoshua Bengio, one of the founders of Deep Learning, about augmenting large language models with world models and inference machines that would give AI systems the ability to reason on reality. We also talked about the famous pause letter, of which Yoshua was perhaps the most prominent signer. We are sponsored this week by NetSuite, Oracle's cloud-based enterprise resource planning software to help any business manage their financials, operations, and customer relationships in a single platform. For the first time in NetSuite's 22 years as the number one cloud financial system, you can defer payments on a full implementation of NetSuite for six months.

0:59No payment, no interest for six months. Take advantage of this special financing offer at www.netsuite.com slash IonAI, that's E-Y-E-O-N-A-I, all run together. It's important that you add the IonAI so that NetSuite knows that its sponsorship of the podcast is worth it. That's www.netsuite.com slash E-Y-E-O-N-A-I. It's an unprecedented offer. Hey, how are you? Very good. I know you don't have much time. I'm going to jump right into it. first of all, on the letter, specifically, the letter has triggered this FTC complaint. It was made by the Center for Artificial Intelligence and Digital Policy.

2:06I don't know much about them, but it was a complaint to the United States FTC asking them to stop OpenAI from releasing any more powerful model. It follows on the letter. The first question I have is your name, to me at least, stood out among the signatories because you are known for being very measured in your comments and very reasoned in your view of the risks of AI and your skepticism about the advent of AGI or human level intelligence anytime in the near future. CHET, GPT and GPT-4's capabilities took even people within the community by surprise, but certainly the general public. So why did you sign the letter?

3:10many of your colleagues, Jeff Hinton, Ian Lacoon, I could go on and on, did not sign it. The people who did sign it, the prominent names who signed it, Gary Marcus, Max Tegmark, these guys are known to be very vocal in their concerns, much more so than you. So that's the first question. I've been speaking for many years about the long-term danger to society of having very powerful tools at our disposal.

3:51It's like any technology in a sense. The more powerful they are, the more useful they can be, but also the more dangerous they can be if they're misused. And I had the impression for many years that the way our society is organized in each country, but globally in general, it's not adequate to face the challenges that very powerful technologies bring. And in particular, AI, but I'm thinking of biotechnology in the future, which is something great that I think could be incredibly useful for humanity as well. So, but the more powerful it is, the more wisdom we need in the way we deal with this. And right now, for example, the system of competition between companies, as we are seeing now accelerating with these large language models, has benefits.

4:50potentially this has driven innovation in some ways in the last century. But also it means that companies are a bit in the haste and may not take the precautions that would otherwise be warranted. So in the short term, I think we need to accelerate the countermeasures to reduce risks. And that's regulation. Very simple. We do that for many other areas. Almost every industrial area is highly regulated when it touches the public. Whether it's airplanes or chemistry or drugs or food, they're all heavily regulated. Not so much computing and AI within that. and because these things are becoming very powerful, I think it's urgent.

5:55We kind of really change gear here. Now, why did I sign, like why now? Maybe last year I wouldn't have signed this letter. It's because we now reach the threshold. The threshold is a Turing test, meaning that we have systems that we can dialogue with and we can't be sure if this is coming from a machine or human. And that could be exploited in highly dangerous ways that can threaten democracy. And I care about democracy. Yeah. An interesting aside is the potential that AI has for enhancing democracy. Absolutely. Like any other thing that AI can do, It could be very bad or very good. But right now, where are the investments going?

6:47How much investment is there in like AI tools for democracy? Not so much because it's not clear where the profit is going to be. And it's true for other areas of social importance. In healthcare, there is, of course, quite a bit, but it's minuscule compared to what is going on in what is invested by large companies who care about advertising, recommendations, search engines, and so on. You said that there's a need to accelerate regulation that I think everybody agrees with, but the letter didn't call simply for accelerated regulation. It called for this pause. Was that meant, as many people believe, more to get attention because the reality, although i saw chelsea finn on twitter saying she was going to stop work on anything uh that pointed towards toward agi uh but is that uh is that pause was it meant to be realistic or is it simply a way to emphasize uh the urgency the chances that uh a request like this you know be followed by by these companies was small and I don't have high hopes.

8:11But the important thing indeed is to say that we need to coordinate. And the short-term coordination that can happen is between companies. Maybe a few companies decide, well, let's take a pause to improve our testing, documentation,

8:32ethics, studies around what we're doing. but really, at the end of the day, it has to be in the hands of governments. That's going to make sure that every company is going to do it. And even more difficult, but essential, is that at the end of the day, it has to be international. It has to be international treaties where the main countries that can do these things agree. Just like we've done in the past for many international treaties where there are risks that could be global. technology. And then on this FTC complaint, and I know that you haven't read it, so I can't really expect you to comment on it in detail, but since the letter was released, is there a plan for presenting governments with legislative language or regulatory language from Future of Life Institute or the like I'm not representing the future of life Institute yeah no I understand and I didn't even write that letter I I was sent it and asked if I would I would sign it and I said yes I proposed some changes it didn't happen but I decided to sign anyways well let me put it this way is there a movement among people that sign the letter support the letter I mean particularly the the most prominent people to pursue regulation with their independent governments?

10:10Is there any coordination at all? There is. Not necessarily among the people who signed. I don't think that's organized that way. But there are lots of groups of people who have been working on this in the last few years. For your information, the Canadian government is moving forward a legislation that's probably going to be the first in the world to be voted around AI, probably this spring. Europe has been working on this for several years. It should come in 2023. So loads and loads of scholars, scientists, and government policy, public policy people have been working on this for years. I've been involved in the creation and for two years as the co-chair of a working group of the global partnership on AI that's a new organization has been created by France and Canada which now has about has about 30 countries it has connections to OECD and it it's it's all about international coordination around AI.

11:21And that working group was the working group on responsible AI. And a lot of the things we discussed for a few years now are precisely these things. And in fact, I just witnessed an exchange about very kind of precise suggestions regarding following up on the ladder. So regarding watermarking, for example, and mandatory display of the origin of the content. Is it human? Or if it's images, was it really recorded? Or is it generated by machines? So these are two different things. And they're connected. it. So first of all, watermarking just means that the way that the words are generated, or the pixels are generated, is not going to look any different to a human.

12:18But a machine with the right program that could be provided by the company that does these things, like OpenAI, for example, will be able to very, very easily say this with very high confidence, like 99.999%. Not some like guess, but I really be sure this was generated by the, you know, GPT 4.0 or something like that. Yeah. Let me just jump in though. I'm fairly familiar with the regulatory efforts going on, but the letter didn't call, the headline wasn't, you know, speed up regulation, it was to pause development. No, but okay, don't read the headline, read the letter. Well, this is the problem.

13:06This is my problem with the letter as someone who's peripherally involved, is that I read the letter. My brother, who immediately sent me a text, of course, didn't read the letter. He read the headline. And the FTC complaint is not about speeding up regulation. It's about preventing open AI from releasing more powerful models for the time being. So, I mean, do you support that kind of action that is coming out of the letter? I agree with the pause. However, I don't think it should be just open AI. And I don't think it should be only the United States. So it's going to take more time. But I really think that it has to move to the international arena as quickly as possible.

14:04And by the way, China is also like probably ahead of the US in terms of regulation of AI for different reasons. but in a sense they are as concerned by the destabilizing effects that these systems can have on public opinion as democracies are for different reasons. They don't want to lose power. We want to make sure that the democratic debate takes place as it should. And so maybe there's an opportunity for an agreement, even though it comes from different angles. Yeah. Okay. I don't want to get too hung up on the letter, but I prefer talking about the research and the developments. I mean, it has been impressive what's come out of the scaling of transformer models.

15:02The only constraint at this point seems to be money and electric power. Some expertise. And expertise, that's right. But there are things missing in the model. And that's what I've been focused on in talking to people. That's what I focused my research on. Yes. So that's what I wanted to talk about. So on the drawbacks, there's, of course, the hallucination problem that OpenAI is working on reinforcement learning with human feedback. I had a very interesting conversation with Jan LeCun about his view that you have to build a world model. I agree on that. Yeah. Because these large language models, their grounding in reality is what's contained in text.

16:00That's not the only reason why we need a world model. There are other reasons. And one reason I focus on is because humans separate knowledge of the world from how to take decisions. And in fact, scientists do that as well. And engineers do that. Just to understand that separation, consider a goal-playing system like AlphaGo. So the knowledge that's needed here really is the rules of the game, like how you make points and so on, and the fact that you want to have as many points. I mean, you want to win and a person who has more points wins. Once you have that, you don't need to interact with the rest of the world.

16:54So the world model here would be easy to get because it's very small. but the machinery to take that and turn that into good decisions, which we call inference in machine learning, that's very complex. In fact, doing exact inference here, like finding the exact right move, the best move is intractable. It's exponentially hard. And that's why we need really, really large neural nets to do the inference. So AlphaGo is just doing inference. I mean, it's learning to do inference by checking itself with the world model. It's checking itself because it's playing against itself and obeying the rules in a sense.

17:35And right now, if you look at something like large language models, there is no separation between the knowledge we're extracting from the world, like the rules of Go, and how we should answer questions. It's the same system that somehow implicitly has the knowledge and is doing the actions, like answering questions. and let me give you another examples where i'm going to illustrate why it could be useful to to do that to have that separation i i you know i had only one car accident in my life and i never drove down a cliff or something probably would have died instead i had lots of thoughts imagining what would happen if i you know drive over a cliff or what would happen if I suddenly break on the highway and the person behind me hits me but I don't need to try it because I have a world model where I can like simulate these kinds of things at a very abstract level and so there are lots of good reasons why you want that separation like the policy of how I drive and my knowledge of how things like causal knowledge of what happens if I do this?

19:01What's going to be the cause and effect change that can happen? And science is all like this. People, scientists come up with theories. They're like world models. And once they kind of want to test a theory or believe a theory, then you do engineering. Okay, so now let's solve for the bridge that's not going to fall, which is not the same thing as like the laws of physics and mechanics that are involved in the world model right so there's a difference between the two um but right now in our large neural nets trained end-to-end which i contributed to in many ways we're not doing that we have a single big neural net that embodies everything somehow and that's detrimental in many ways um you're i think it's one of the main causes of overfitting uh for example because there is no place for like a real world model it's there's there is no explicit place for reasoning reasoning causally or reasoning you know more generally um of course if you look at gpt it seems to kind of reason but if you if it has to reason over too many steps it tends to get it wrong.

20:20It's not just hallucinating. It's like getting it wrong. I did a little experiment. I asked it to do additions or multiplications of numbers. And, you know, if it's just like one digit numbers, it works very well. If you have three digit numbers, it can't do it usually. I mean, it claims it does it and it gives you wrong answers or wrong explanations,

20:49which is interesting because to do this you have many steps and you can tell it oh do it step by step show me how you do it right like people have been doing and it still gets it wrong so it's very weak in terms of reasoning so what reasoning is like really really like you separate the knowledge from how you use it that reasoning is using the knowledge combining pieces of knowledge in sort of a rational coherent way in order to go from a to b that's reasoning like planning planning is a special case of reasoning so in your research how do you what what very high level how do you build a world model i mean i'm sure uh you know yans uh joint embedding arc uh predictive yeah or yeah so we've uh come up with a new framework for training inference machines um and they and it can also be used to give to help train the world model um i don't know what kind of people you have in your audience i don't know how much technical i can go here?

22:07Go technical because people their eyes glaze over it's too if it's too technical but I'd rather have it in for those that follow it. So we've come up in the last couple of years with something called generative flow networks or g-flow nets and what they do is they learn to probabilistic inference so they're neural nets which could be very big potentially that learn to do reasoning essentially through a sequence of steps and after that sequence of steps they get the kind of reward so they're connected to reinforcement learning if the steps are coherent with the world knowledge so the world knowledge is given by some other you know net or some other piece of code, if you want, that says whether the result makes sense.

23:01So you have two things to train. You've got the world model that embodies knowledge and can check whether things are coherent. Let me give you an example from, say, mathematics that people can understand. You know more or less what a theorem and a proof is about, right? So the world model here is just checking that the proof is correct in the sense that each step is logical and coherent and it uses things facts that are known like you know if you add x plus y is equal to y plus x so you're allowed to do that transformation and the inference machine is proposing solutions to a problem like prove this theorem and it needs to do it usually through a sequence of steps where it uses knowledge And if it messes up, it hallucinates things along the way, like GPT does, the reward is going to be very bad.

24:05And so it learns to be coherent.

24:12And that's not something that explicitly ingrained in current large language models. They get it because they've seen so much text and humans tend to be coherent, but not always. and there are lots of subtleties here. For example, who's speaking? Maybe I'm speaking not the truth because I'm trying to convince you. And so there are lots of subtleties in language that means that it's not always like the truth that you're seeing, right? And so we need machines that understand the notion of truth in a more fundamental way, I think. By the way, this is something that we're doing with neural nets, but the idea behind reasoning and logic as a building blocks of AI are from the early days of AI.

24:59Like classical AI, symbolic AI, that's what Gary Marcus has been saying. I think that the way that these guys have been proposing it is not going to work. But what can work is to train a neural net to behave in a coherent way, to behave rationally, as in manipulate pieces of truth together to come up with proposals for things that make sense. I understand there's the inference model, the world model, and then the language model. Yes, exactly. The language model is yet another thing. In the brain, the language part is quite distinct from the reasoning part and the math part and the knowledge about action and so on.

25:47Right. And that's kind of the problem right now. Now, the language model, it produces coherent language. It doesn't really know whether something is true or not. Yes. In your structure, the world model would provide that ground truth, and the inference model would execute reasoning, and then the language model would express it. it would do more than execute reasoning because reasoning you have to understand reasoning is very hard because there are many paths you could follow think about proving a theorem is it easy no because how you know which pieces of knowledge do i combine in what order there's an exponential number of possibilities that's why you need jointed models so you need actually models that are very similar to what we use for like large language models but instead of generating words and imitating what humans have said, the general sequence of pieces of, I mean, they pick pieces of knowledge or they pick pieces of solutions and then they get rewarded if they do it right.

26:58And by the way, the inference machine is also important to train the model itself. So if there are things that are not given an input, so let me give you an example. You see an image and you're trying to come up with an explanation for what's in the image. And maybe the image doesn't come with labels, but you're trying to do supervised learning. Like humans look at images and they make sense of them. So there are well-known principles in machine learning to deal with that situation where you don't observe the full story. You have to, they're called latent variable, like Jeff Hinton calls them hidden variables.

27:41The model needs to come up with like a story that explains the data. And the inference machine is the thing that comes up with the story. And then the world model checks whether the story is consistent with the image. And that's how the inference machine gets better. It's trying to make stories that are consistent with the model. But when it does that, it helps to train the world model as well, because now that you have a story and an image, you can see, you can build a model that makes the right links between them. So a lot of the work that Jeff and others and I did in the last few, in the early decades, especially of deep learning with probabilistic models like Boltzmann machines and so on, were based on these ideas.

28:25But I think that now we can use these large neural nets, like the large language models, but not for language or reasoning. On the world model side, though, it sounds, when you say hearkening back to the early days of AI, it sounds like a rule-based system where you load it up with... No, because we're going to learn the rules as well. And we're going to learn them in a probabilistic way. So here's another aspect that's missing from classical AI. Given any finite amount of data, you can't be 100 % sure of what is the right world model that explains the data. Because there may be multiple interpretations.

29:07It's like in science, you have usually different theories that are compatible with the data. And what scientists do is try to come up with different theories, diverse set of theories. That's important. Because if we just go for the first theory we find, it might be completely wrong, even though it fits the data. But what if we know that we have two competing theories? We can be safer. Maybe sometimes they agree and then we should go that way. and sometimes they disagree, and then we should be careful. So one of the problems with things like chat GPT right now, and many of these models, not just this one, is that they are often confidently wrong.

29:49It's not just that they hallucinate, that they're sure of what they're saying. And of course, that's bad, right? That can lead to catastrophic outcomes. But I guess there are some humans who are like that too. But I guess for humans, maybe a lot is tied to their ego. But in good circumstances, people have a good sense that some belief that comes to them isn't completely sure. And they're actually good when you ask them to bet. If they have to bet money on something, somehow they'll exploit the fact that they're not sure to bet more or less money. So our brain is doing that calculation. It's not something we verbalize, but we kind of sense it and then we act accordingly.

30:35and that is very important because coming back the connection to the world model is that really what we need to do is not to like have one world model but to have a distribution over world models so it's not like we have a neural net that is the world model I think that's the wrong picture the right picture is we have a neural net that generates the theory that corresponds to the world model Just again, the same kind of neural net I've been talking about. But now instead of coming up with explanation for an image, it comes up with a general theory that explains a lot of data. And the theory could be something that can be verbalized the same way that scientists come up with theories.

31:19So you see, we are talking about things that are related to classical AI, but in classical AI, not only there was no uncertainty, but it was all handcraft. Like the theory is a set of rules and facts that somebody puts down. and you know maybe that can be part of a solution of something but really many of these statements they're not like 100 % sure and so people starting to play with probabilities and so on but it we never got it quite right and the reason we never got it quite right is that getting it right is competitionally it looks competitionally intractable but it turns out it can be approximated by these very large neural nets.

31:57They do a good job at that. So we can have large neural nets that just like we do spit out theories or pieces of theories that are relevant to what we're currently seeing. Rather than have one theory that's hard coded either by humans or even by a machine and that's like one list that we don't change and then we trust thereon, That doesn't make sense. We keep revising our views and our views are uncertain. So we have a distribution of our theories and we don't say, oh, there are these 10 theories and this one is 0.8 and this one is 0.6. That's not how it works. And the reason is there's an exponential number of theories.

32:40So we don't explicitly enumerate them. Instead, we generate pieces of theories according to their Bayesian probability. And there's a lot of evidence from cognitive science that people do that. It's happening unconsciously because that jointed model is something that's system one. It's behind the scenes. You don't control it. You just see your thoughts coming out. I can imagine the inference model. I mean, there's a lot of inference models out there now. How do you train the world model? Okay, well, that's the cool thing. If you have the inference machine. Oh, sorry. It just finished the question because there is a certain amount of reasoning in large language models, as you found in your own research.

33:32And so and that's contained in language. But if you want to build a real world model, you want to get beyond language. So do you do it using video? I mean, how do you do that? right uh the the world knowledge i mean the world model could be about all kinds of things images um concepts in the world i i like to focus on the abstractions of the kind that we can verbalize but you can also and yon is sort of more focused on the uh perception and and action part. But the principle, I think, could be applied to both. And the principle is that there is no world model. You only have an inference machine.

34:20So now what I'm going to say is a little bit mind-twisting. You only have an inference machine. And what that inference machine does is it can generate the world model or the piece that you need at the moment. So And when you have such a piece, you can also evaluate how your regular inference is going. Am I answering the questions right? So if I see an image about dogs chasing cats, a piece of my theory of how the world works may come to me about things I know about, things I think I know, I think are true about cats and dogs. and that will serve to provide a reward for having decided that i need to run out quickly to stop them because maybe the cat is in danger or maybe the dog's in danger

35:22so what i'm trying to say is we don't actually need a separate world i mean i'm saying the opposite of what i said earlier that's why i'm saying it's mind twisting so now that we've established we want a world model and an inference machine, I'm going to say that instead of a world model thinking of like a separate neural net, because there are many possibilities of what the right world model should be, you have actually just another joint of neural net that generates just the pieces of world model that you need on the fly. And it's probabilistic. So maybe one day you view things one way, and the other day you view things a different way, because really there is uncertainty.