AI Godfather: Should We Stop AI? | Yoshua Bengio

31:23Right. A lot of the labs have a fiduciary duty to shareholders, legally speaking. Yes, yes. Isn't it in everyone's best interest to have a reliable AI? Or in a world where you have a scientist AI and then eventually a longer term, something that's technically sound, even if it's super intelligent, is there anything that we would lose? Because you hear, okay, we need to solve all these scientific challenges. And sometimes we get these incredible breakthroughs. In a world where you have a guardrail, do you lose some of that? Because I'm still not fully seeing what the downside would be of making these systems more alive.

31:52Or do we just need the public to be more aware of a technical solution to start advocating for? So some of your audience may know about the tragedy of the commons and the prisoner's dilemma. They are well-studied scenarios, theoretical scenarios, but that reflect a reality we already see in many ways in our world. think about how nations are not doing the right thing for climate. You might say, well, shouldn't it be in their self-interest to figure out a way that the world is not going to break down, the climate is not going to break down? Well, yes and no. So in this competition, the self-interest actually leads everyone in a bad place.

32:37In the tragedy of the commons, it's in the interest of each farmer to bring their cow to the commons because so long as there is grass, they're going to have an advantage if they do it rather than not doing it. It's like pollution, right? If it's cheaper to pollute, then the self-interest of a company is to continue doing it. And how do you escape such a scenario? Well, individually, the companies can't. The only way to escape the scenario is to change the rules of the game. And who changes the rules of the game? It's government. Now, it's tricky here because even a single government, like the U.S.

33:14government, could change the rules of the game for their company, but they can't change the rules of the game for the Chinese companies and maybe other countries. And so the only way to change the rules of the game is at international level and multilateral coordination. Right. And let's talk about that because I think it's something important to point out because even some of the movements that are local or regional, to say we need to do AI this way or stop AI until we get things together, even just in New York or even in this one city, this is a technology that you actually have to think about it globally.

33:45There's no other choice. You can stop it in one state. It really does nothing for the safety of humanity. So when you think about a global treaty or something of cooperation, what would have to be included for you to feel like this is going to work out? Because at the end of the day, countries do engage in things like intelligence gathering, espionage. They hack one another's critical infrastructure. So even if we had, let's say, the technical solution to make AI safe and we installed all the guardrails, countries may not always want to use them unless there's something that everyone's come to the table and signed.

34:22What would you say has to be in it for it to work? So I'm going to start with the endpoint. The endpoint is a planet where most of the countries, especially those that have the power to build powerful and potentially dangerous AIs, agree on three principles and then have the technological tools to make sure it's not just words on paper. So the three principles are first, safety. In other words, everyone who builds a powerful AI needs to make sure it's done in a way that is not going to cause severe harm. So we're seeing right now that the most advanced AIs can be used as a weapon through cyber attacks.

35:10So that's not good. We need to find ways, which may not be just in the AI itself. It might be how we might have to change the internet. We have to maybe change the rules, all kinds of things. So that's safety. And of course, we don't want to build like rogue AIs or stuff like that if we get to superintelligence. So that's one thing. Everyone in the world has an interest in making sure the people building AI will evaluate the risks and monitor them and give us guarantees that nothing bad is going to happen with their work. The second thing is very important. And it's a commitment that the power of the AIs that are being built, because intelligence gives power, is not going to become a tool of domination.

35:58By any one country. By any one country or any one company. Because domination can be economic domination. like one company basically owning half of the world's economy. This is a dream of the investors who are putting trillions of dollars. They want to become the companies that are driving everything on this planet. It's not good. It's not good for capitalism. It's not good for democracy. It's dangerous from a geopolitical point of view.

36:28People will resist that in violent ways potentially. The third one is sort of the flip side of non-domination, which is kind of benevolence. You want the benefits of AI to be shared. I was at the UN yesterday, and the member states expressed in their vast majority the concern that they're going to be left behind, that the inequalities that exist at a geopolitical level right now are going to be amplified by AI. And this is plausible. I don't have a crystal ball, but we need to make sure it doesn't happen. We need to make sure that the benefits, whether they are in health, education, or productivity, or whatever that AI can bring, are shared.

37:18For example, if AI allows to save money through automation in one country, the profits should remain in that country so that it can help the people lose their job. But right now, with the laws that exist in the world, there's a good chance that those profits will go back to the companies in a different country who build those AIs. So that's the third principle. Okay, so it all sounds like, oh, you know, it's an application of the general principles of like the UN Charter, for example, with human rights to the situation of, oh, there will be very powerful AIs in the world. Well, how do we get there?

38:09Because, you know, we're very far from that world. and I'm going to mention two aspects that I think are encouraging. One is the motivation for the US and China to negotiate. So because of mythos, and mythos is not an isolated thing, there will be more and more AIs that can be weaponized. Right now it's for cyber, eventually it could be for biological weapons or whatever else we haven't thought of. because knowledge gives power, right? And even weak actors, terrorists, cults, could use that power in destructive ways just by having an internet connection. And it's in the interest of the U.S. to make sure that Chinese companies are not creating something that can be used by those weak actors and vice versa.

39:04The Chinese government also doesn't want the American models to be used by third parties against China. You have to realize that a cyber attack against our critical infrastructure could cripple our economy. Like imagine we don't have access to our money in the banks for a week or that our transportation supply chain breaks down because of these cyber attacks. So it's pretty serious. But the bottom line is you can see that as AI becomes more powerful, there's an incentive for countries to sit at the table to negotiate something to make sure the safety part that I was talking about is going to be handled.

39:46The second thing is, even though it seems not in good shape right now, multilateralism still exists. And in fact, the vast majority of countries in the world want a world in which there are rules so that you can't have a country invading another one just because they want and they have the power to do it. And that applies to AI because AI is going to be very powerful and it can be weaponized in a military sense or even in a political sense because AI, I just read of a recent study showing we've reached the point where AI systems are significantly stronger than humans at persuasion. They can make people change their mind through a dialogue.

40:35So imagine the political power this gives if it's not controlled, if there are no guardrails to change public opinion in another country or even your country because you want to win the elections. So we need to have global agreements about how AI is developed and used to make sure these bad things don't happen. And now you can see that there's the two things I've talked about interact in a positive way. So it will be in the interest of the leaders, say US and China, to make sure that in all of the countries there are rules that mitigate those risks. even the countries that don't build the AIs but when they need to maybe have rules in how people interact on social media or have access to and use AI systems or whatever.

41:38So the trade-off here is yes, we want to benefit from these AIs but we don't want other parties to do something that will hurt us and so we have to agree to common rules. So I think there is a path. Another reason that's related to this multilateralism desire in most of the countries in the world is the fear that, as I mentioned earlier, the fear that they will be left out, that they will be dominated economically. And so you had Canadian Prime Minister Marconi, for example, at Davos saying, if you're not at the table, you are on the menu speaking about countries. and then saying that the only way for middle powers like Canada, but think about European countries and pretty much every other country, the only way to be at the table is to form coalitions, to form a union of countries who think we need rules and we need to agree globally.

42:41And maybe it's going to start with a few countries and grow because you want to be in the club where you know that, at least within the club, AI is not going to be used against you, that the, I don't know, medical advances thanks to AI are going to be shared and all of these good things, and no one is going to build an AI that's going to be a danger to you. So there's an incentive for the runner-ups and the middle powers to move towards such a global coalition, And there is also an incentive for the leaders like China and the U.S. eventually to be part of something like this. Right. It's in every country's best interest to not have the stock market crash because that boomerangs throughout the world.

43:28It's in everyone's best interest to not have a cyber weapon that boomerangs around the world. And we've seen how that goes with stucknecks and different weapons. How does any country that's not the U.S. or China actually build negotiating power and leverage? because we even have seen in the last few weeks with Mithos one of the most powerful items that we know to date, when it didn't go as planned and the U.S. government had asked Anthropic to change something within the model. And there was some disagreement. I don't think we all know exactly what happened. But the U.S. government said essentially no foreign national can have access to this technology, even if they work at Anthropic.

44:03So you could be an allied country that was using Mithos to understand your vulnerabilities in your critical infrastructure. And that model was yanked. So if that is just a preview to how the two most powerful countries when it comes to AI may act if things don't go their way or they feel like their best interest is threatened, how is it that you are Canada, Ireland, Zimbabwe, Tanzania, anybody else that you're going to actually be able to have any power in these negotiations? Yeah. This is an important question. Thanks for asking it. And again, I'm only like suggesting and hypothesizing. I wrote a paper about this on the need for middle powers, but more broadly, all the countries except the US and China to get together, not just to negotiate a treaty, but to build up the cards to be at the table.

45:03So actually, just by forming a coalition, these countries already have carts. They have rare earths. They have lithography machines that are used. Technical talent. Technical talent. They have energy. They have fabrication plants that build memory. They have different pieces of the puzzle. That make them indispensable. And individually, each of these pieces is not sufficient to be at the table. But once they form a coalition, it's a different game. So that's one aspect. The other aspect is they can take a chance because we don't know what is the timeline for super powerful AIs, if ever. They can take a chance to try to build their own models.

45:51And in my opinion, they don't have to do it in the same way as the Americans and the Chinese. They can build models that are going to, in a way, be complementary to what the Americans and the Chinese are building, like these safeguards, models that bring something that actually is useful for commercial deployment, like reliability. Like, people don't, like, companies don't want an AI to start doing, like, crazy stuff on their networks and their databases and with their customers, as we are starting to see. So there's a real demand for this. And if those countries work on research projects that could end up being something useful globally to the leading companies, I think they have a greater chance of being at the table.

46:42Right. They have their own strategic leverage. Exactly. And so even beyond, I think we use the idea of sovereignty, but it's much more about strategic indispensability. Yes. Right. Not every country has everything. And to make that really powerful AI, there are things from other countries you're going to need. And so that becomes non-negotiable. There are other things they need to do, which is accelerate the development of data centers and also do it in a way that they will keep some sovereign control over these data centers. So if you're a government of, say, France or Germany, you don't want your government's data, information, interactions with civil servants and politicians that are happening for a chatbot to at any moment become accessible to the U.S.

47:35government. So these governments are really worried. It's not just yanking. It's also having access to all that information. and so I've talked to many of many governments around the world and they're really concerned about these issues so they want to own not just the models but the enough of the infrastructure that they know that their data is going to remain private as it should unfortunately there is a you know patriot act which allows the u.s government to access all that information even in a different country if it is in the hands of an American company. And so from the point of view of these governments, this is unacceptable.

48:20And so there's the mythos events where the models could be pulled, but even if they're not pulled, there is still the concern that you lose your private national security information and even citizens' private data could be used against yourself. We need a world where this is not possible. And if the U.S. doesn't come up with legal ways to constrain itself so that other countries will trust the deals that are made about privacy, for example, then the countries will try to find alternative solutions. And that's what they're struggling to do right now. So if a country doesn't have its own data centers, it's possible that they're making all sorts of advancements in health care or you're using your AI systems for your banking.

49:15But if that's an American AI system and it's going back to an American cloud or American data center, you don't actually have control over your citizens' data, which is a nightmare. And I think a lot of people maybe don't understand that part of it. Yeah. I think for many governments, it's more of a national security issue. I think in Europe they care a lot about privacy, you know, citizens' private data, more for ethical reasons. But it does matter. But as I said, even if the data center is in Europe, let's say, but it's an American company that builds it, the data can still be accessed by the U.S.

49:55governor. As I understand the rules, I'm not like a legal expert, but that's what I understand. And so from the U.S. point of view and the American companies, I think there should be an incentive to negotiate rules where everybody wins, where people feel like they can trust the American companies, the American models. So how do they trust the American models? Well, they need to make sure those models are not going to be behind the scenes oriented towards goals that are not good for them. Because we've seen examples of AI models that were biased politically, for example. again if you know your population is acquiring information through all these AI models and it's going to like in a subtle way change political opinion that's not acceptable either so it's also a threat to democracy to not have those levers to not even know if there's you know So for sure, no scheme to exploit the power that AI will have over people through all these interactions.

51:10Right. The world that AI is going to start to generate for people is going to really shape their perception. And that, I think, is actually one of the most under-discussed soft powers of the future. Whichever country, whichever AI systems your citizens tend to use the most, that's going to deeply shape how they see the world. and you could do it really subtly, even if it's a benign use, a writer building a movie and they ask for some feedback on their script, you could slowly nudge that writer, that director towards one worldview over another. And the same kid that's going to generate a story, the person writing an essay, all of these subtle ways to shape perception.

51:47And you had mentioned superintelligence as a potential future that we move towards. Do you think that that is probably an inevitability eventually, but we can also, there's a way to build safe superintelligence, but we're probably going to head there at some point? Clearly, it's not inevitable. I think some people would like us to believe that it is inevitable. Yes, that's true. So is it feasible? And even that we're not sure of. But let's say that if you look at the data on increasing capabilities of AI and how many lines of code are being written by AI, anentropic, for example, and it's growing very fast.

52:28we can't deny that it's a reasonable possibility, that we are on track apparently to build machines that will be smarter than us in many different ways. In a way, that scales. So you can have a million GPUs, and then you can do what a million people would do or even more. So we should plan for that possibility to be real. It's also possible that there will be a scientific obstacle Many researchers think, oh, this is never going to work. This approach is like, oh, LLMs can't possibly be at human level. Honestly, I don't know. But I see the data. And I see that we're going in that direction. We can also, so that's feasibility, right?

53:14But then I think the more important question is who decides according to what goals? And for me, that's a democratic question. It's not because we can build a virus that would kill everyone that we should do it, right? Right, exactly. And I mean, so what can we do? We have a really interesting, and I do want to touch on jobs because that is a through line on this show. But I also want to touch on this community because we have a really interesting broad audience that listens to the show. On the one hand, you have people that are living their life and trying to understand what's coming next. So they can make a difference or adjust, adapt in their own corner of the world.

53:58Then we have world leaders. We have people at NATO, DARPA, all sorts of different people that listen. What can we do tomorrow that can help, that can push this towards some of the futures that you're trying to build, some of the things you're raising awareness about? What can we do to make a difference here? So we need more people to understand the stakes. that depending on the choices that we are collectively making through our leaders, our companies, individually as consumers, as voters, we are shaping the world. We are choosing a future. And if we do it without understanding what that future looks like, it could be a very bad future.

54:43And I think right now most people completely underestimate how transformative the advances in AI are likely to be if we continue on the current path of growing AI capabilities. So awareness is like the primary thing. Once you understand that, hey, there's a fire coming to your house, you do something. We've seen that to some extent with climate activism, but we don't have such a thing for AI right now. It's changing. I see the polls. I see like 90 % of Americans are concerned. They're mostly concerned about short-term effects, and that's important. But I think if more people understand the longer-term gravity of the situation and the fact that governments are the only ones, whether internally or through international coordination, that can really steer the world in the right direction.

55:42I say the only ones because the companies, as I said, they're stuck in the forces of competition. Then it might become a political issue at a level that matters for elections. And I think there's a good chance we will go there. But we need to do it faster. And so thank you for your work because we need to have democratic discussions, debates. What do we want? I mean, the real question is, what kind of future do we want? Yeah. Yeah. And I think it's, that's a question that we don't ask enough. I think we know the kind of futures we maybe don't want, but what kind of future are we also fighting for?

56:28And I have found in my work in Foresight, if people don't know, aren't shown any of the visions of the futures that could be possible, for instance, there could be a world in which everybody is rooting for AI in healthcare and AI medical breakthroughs. I think we're all rooting for AlphaFold, right? Everybody wants those types of scenarios, but at what cost? And so if we're not aware that there are people that are working on the technical solution, so it's not zero sum, then the whole thing feels hopeless. And I think that that's my worst fear is that people check out because they feel like there's no path to root for when there are several paths and there are people, we don't do the best draw, but uplifting the voices that are fighting for something.

57:09But I think we need to make that more known, that there are solutions and people are building them. Exactly. So we can build tools based on AI that will be extremely useful, and especially in scientific research. But that is very different from the current path in which we're trying to build new entities that look like people, that people become friends with, Is that necessary? I mean, people, of course, are attracted to that, just like they're attracted to interact through social media. It feels good. But is that really good for us? Studies suggest no. And we need more scientific understanding.

57:52But we should have the data to be informed of those possibilities. we should encourage explorations, research into AI that will be beneficial and not dangerous. And we need to see that it's, I think the most important thing I realized is we need to make these discussions very concrete. Because when it comes to political views and changing political views or, you know, having an issue raised in the minds of people, if it's very abstract about some future, you know, it just doesn't work. So we need to find a way to communicate that speaks to people. So currently we're seeing on the political scene the issues of AI with children, with people harming themselves or others with the help of AI as an example of something that touches us because we care for our children.

59:01Like the reason I'm in this both at a technical level and, you know, on the policy side of things is because of my children. I didn't need to do that. Like I'm at the end of my career. I'm the most cited according to Google Scholar. I don't need any of this. I could take it easy. But I couldn't live with myself thinking that we're apparently going towards this potentially bad future and not doing anything about it. And every one of us can do something, just like every one of us can do something for any political cause. And it is political. It is political because the key decisions are going to be the ones taken by governments.

59:47Right. And this is where democracy is so key. It needs to be something that's on the ballot. And I think, again, bringing it down for people in a way that people can understand and in a way that they don't feel paralyzed by the information. There are people working on solutions. There are options. And I think that is the whole thing with expanding the decision space for people. There are options. Companies are also deciding to build AI in certain ways or not build AI in other ways. So people are making choices. And anywhere there is a choice, there's a different future to move towards. I've talked about the children issues, but let me talk about the labor issues.

1:00:26Yes. If the only forces at play are the competition between companies, then all the jobs that can be automated or simply made more efficient with less people and more machines, they're going to be automated. Because if you're a company and you don't do it and your competitors do it, you lose. So again, it's this tragedy of the commons issue. Even though maybe it's not good for our society to have such a rapid transition where lots of people are in the street and have no revenue.

1:01:04So, you know, there's also benefits to making things more efficiently. But we have to be in control of those changes so that it's done in a human-centric way. For example, maybe there are jobs that make sense to automate because actually people don't really want to do those jobs. And maybe others, we really want to value them more. And maybe there are ways for governments to steer in a way that those transitions will be at a pace that people can adapt. maybe there'll be more demand for jobs that require human-to-human scales. I expect something like this many people do, but people will need time to retrain.

1:01:55And that raises the other issue with the labor transformation that's plausible again. It hasn't happened at that scale yet, which is how do we make sure that the profits made from the automation end up helping the people who lose their job? That's fundamental. And the current government programs and even at the international level don't really have good answers to this.

1:02:31I don't think we are currently in a political culture where it'd be easy to tax those companies heavily. But what else can we do? I mean, the money is going to go in one place, but the need is in a different place. So we should think of what are the options? And people have proposed options like giving shares of these companies to individuals or to governments. And I don't know what is right. I'm not a political scientist. I'm not an economist. But I'm just raising that this is a question that needs to be discussed and that we need to have a democratic discussion about this. And then there is the international aspect of this, right?

1:03:12Because if people are losing their job in country A or the profits are in country B, how do the workers in country A survive? We saw this with globalization too. So we need a way to make sure all boats are lifted. And we don't, because it's also bad for country B, because if people in country A are so angry and revolt, there's going to be violence, right? And there's going to be terrorism. And, you know, it's not good for us. And we do talk about jobs a lot on this podcast. And we've had people that you also know, Ajay Agrawal, Avi Goldfarb. So we have a lot of economists here to discuss it. And, of course, the jury is still out.

1:03:50Nobody can predict how the economy is going to reconfigure. It will. And there will probably be new strange things that people do the way we're here in the strange podcast room and made all this up. But the transition, I think, could be really rough. It doesn't have to be, right? And this is one of those futures that we can see that things are going to change, regardless if you believe all new jobs are coming, they're going to be incredible, or maybe not as many jobs, or maybe we have a two-day work week, whatever you think is coming. I'm worried about this transition period. And I actually don't even think it should just be people who lose their job because everybody's data has been a part of the making of these AI systems.

1:04:27None of these systems would be as successful or as good as they are without your data, my data, anyone who's ever posted anything anywhere, written anything. Yes, including in the last few centuries. Completely, right? So humanity's cultural heritage is currently being exploited to build those systems. So who owns this? Why would one group make so much money out of all that heritage? Everybody should get a part of it. And I also don't think we should wait until chaos happens or wait until when people are disempowered. If there's an elevator going up to prosperity on the back of these technologies that everybody helped build, the entire society should be on that elevator.

1:05:08Yeah, I completely agree. And for some of these things to work, like some of the shares of those companies somehow end up helping people. Well, we better do it now before those shares go through the roof for this plan to work. Yeah. And it's not even just UBI. I think that can also be a scary future for people. We don't necessarily want to be in our pools floating around getting a check for OpenAI. There can be all different innovative ways that we could structure this that could also lead to people making competitors to these companies. If you give people the strength now, there'll be much more agency to take shape or to take part in what's coming.

1:05:53But I do think that that is a conversation that needs to happen now and not wait for the disempowerment. The other reason it needs to happen now is that democracy is slow. Yeah, very slow. A debate takes time and people like reading and hearing about different views takes time to change their minds, to understand what's going on. And then bureaucracies are slow. You know, governments, it takes years for a law to go from conception to being applied. But yours might be the scale at which this transformation is happening. Not months, I bet not decades. Yeah. Yeah. And I mean, even if it all works out, we should all still get a say just to keep power in check as well.

1:06:39So in this moment, I know you said your grandson, I've heard you talk about your one grandchild. Yes. And that he was quite a big catalyst for your pivot and the research that you're doing today. As you said, I mean, compared to many of us, you're quite successful. If anybody could kind of hang it up, it would be you and you're continuing to go down this path. what would you hope his future looked like, right? If you were thinking about what he may be studying or how he might spend his time, what do you envision in that future? Wow. I've been focusing more on the futures that I don't want him to be part of.

1:07:23But I think there's a really incredible positive potential Now, there's the usual ways that people talk about health, education, but there is also a world where we have no choice but peace. so see we are building these machines that could become extremely powerful and could become weaponized and you can think of a little bit like what happened with nuclear weapons and if we don't find a way to make sure they don't they're not weaponized we might all lose I heard you speak about you know biotech and I heard about from a UN committee mirror life. In other words, building, say, bacteria that would be not visible to our bodies and that could be designed in the coming few years or decade.

1:08:38And AI could help to do that. And it could wipe out all of life on this planet. So why I'm saying this is we may not have a choice. Either we find a way to make all of humanity benefit or we may all lose.

1:09:07I don't know what's going to happen, but there's an incentive. Right. Right. It's in everyone's best interest and truly everyone's to get this right. And we may end up, to go back to your question about my grandson, we may end up in a world that's actually much better from a human point of view in terms of peace, in terms of respect for each other, in terms of diversity, in terms of, of course, well-being in a material sense, in an educational sense, in a medical sense. compared to the world today where if you think about most people on Earth, including in the United States, there is so much stress, so much uncertainty about the future, so much fear of losing your job, so much concern of not being able to buy food next week or pay your rent.

1:10:01It doesn't have to be that way. and technology, if we can govern it right, can bring all this. But there's this big if that we have to take seriously. And I think humanity has been in really tough situations before and governance has happened, treaties have happened, negotiations have happened. And of course, this is a different time than, let's say, some of the nuclear arms races where there were active standoffs and people still came to the table. So it is possible, right? Is it probable? How likely? How quickly? But if something's possible, it's, I think, at least worth us trying. And we don't really have a choice, right?

1:10:43That's exactly my philosophy. We don't know if we're going to be successful in bringing this beautiful world or at least avoiding terrible futures. But it is plausible. It is something that's worth a shot, that's worth fighting for, for our children, for our grandchildren, for ourselves. And there are so many people, I know many listening today, that that's what they're willing to wake up and fight for, and people like yourself. Professor Bengio, it's been a pleasure. Thank you so much. Thank you.