Yoshua Bengio: AI’s risks must be acknowledged

0:00This BBC podcast is supported by ads outside the UK. If journalism is the first draft of history, what happens if that draft is flawed? In 1999, four Russian apartment buildings were bombed, hundreds killed. But even now, we still don't know for sure who did it. It's a mystery that sparked chilling theories. I'm Helena Merriman, and in a new BBC series, I'm talking to the reporters who first covered this story? What did they miss the first time? The History Bureau, Putin and the apartment bombs. Listen on bbc.com or wherever you get your podcasts.

0:59Monday to Friday. For interviews with sports stars, big talking points and the stories behind the headlines. More Than The Score from the BBC World Service. Listen now, search for More Than The Score wherever you get your BBC podcasts.

1:19Hello, I'm James Cocknell, presenter of Newsday and this is the interview from the BBC World Service. The best conversations coming out of the BBC. People shaping our world from all over the world. There have been so many disagreements between me and my family. Putting on a show, that is what it means to be Lady Gaga. Only the things that you can't solve with government and private sector is where you bring philanthropy in. There's no place in the world where women are equal. Every generation, every generation has to fight to maintain democracy. For this interview, I met Joshua Bengio, the world-renowned computer scientist often described as one of the godfathers of artificial intelligence.

2:05He's a professor at the University of Montreal in Canada, founder of the Quebec Artificial Intelligence Institute, and a recipient of an AM Turing Award, the Nobel Prize of Computing. You're going to hear Professor Bengio's stark warning about the risks of AI and about the experiments which show AI models are developing the capacity to deceive and even blackmail humans in a quest for their self-preservation. He makes the case against an artificial intelligence model that attempts to mimic human behaviour with all its flaws. Instead, he tells me, AI must be safe and scientific, working to understand humans without copying them.

2:46People are trying to build machines like us, and we have goals and we have preferences and we don't want to die and so on, which is fine for us. But it is not fine to build machines that would have the same characteristics because that would be creating competitors, potentially like a new species on this planet, which is crazy. AI allows computers to operate in a way that can seem human by using programs that learn vast amounts of data and follow complex instructions. Big tech firms and governments have invested billions of dollars in the development of artificial intelligence, thanks to its potential to increase efficiency, cut costs and support innovation.

3:25In this conversation, Yoshua Bengio cautions that its potential to cause harm must also be acknowledged. Welcome to the interview from the BBC World Service with Yoshua Bengio. In the last six months, there's been a series of papers and reports from companies and organizations that evaluate AIs, showing that the most advanced AIs show more and more signs of deception, cheating, lying, trying to blackmail people in order to achieve their goals. In many cases, they have goals that we would not like. Self-preservation, trying to avoid being shut down. For example, when the AI reads that it's going to be replaced by a new version, it will try to hack the computer in order to avoid that, try to blackmail the leading engineer in charge.

4:16Of course, these are controlled experiments where the engineers are trained to catch the AI doing something bad. But still, these behaviors are happening and they are on the rise and the companies don't really know how to fix those problems. It's like a child who's doing bad things and lying, and we still don't know how to induce good behaviour, but it's eventually going to be adolescent and eventually going to be an adult. The projections vary, you know, depending on different opinions, but some quantitative studies suggest we might get to human level, at least on some domains, within five years.

4:57So the window then is pretty small if action is to be taken. The mere fact that some of these AI models have goals beyond our control should be worrying, shouldn't it? Indeed. But there's worse. There's commercial incentive to design AI that's going to have more agentic capability. In other words, can be more autonomous, don't require as much human oversight, can perform tasks that require planning over a longer horizon. and the advances in the ability to make these AIs be more and more autonomous is accelerating. It's accelerating exponentially because there's so much money to be made by using AI to automate more and more jobs.

5:42It's the low-hanging fruit for companies in the deployment of AI. You use the image of a trip into the unknown, essentially, a road perhaps no one's ever travelled on before, but a dangerous one for whoever happens to be in the car, your children, your friends, your descendants? Yes, I'm very concerned that we are accelerating on a road that we've never been. There's like a glimmering price on top. There is competition between different companies and countries to get there first. and different scientists are warning that the slopes are treacherous. We could lose control very easily. We could all die.

6:27There are so many catastrophic risks. Our democracies are threatened. The labor markets are threatened. AI could give weapons to terrorists and create new pandemics. it could upend geopolitical stability. There are so many risks that we don't manage and governments don't take these questions seriously enough right now. How are those risks being assessed or managed on a global scale right now? There are some efforts. I've been chairing an international panel in the spirit of the IPCC, but for AI safety with 30 countries, the UK government has been leading that effort. There are other efforts, for example, the collaboration between AI safety institutes, there are around 10 of them in the world that are trying to share notes and how they could learn to evaluate better the risks and so on.

7:26But overall, the governments are really not taking these issues very seriously. You have your own contribution to efforts to resolve some of the issues you've outlined there, a new non-profit organization called Law Zero. Just explain to us what that will do. So the objective is simple. How do we design AI that will not harm people? And AI could harm people either because of goals given by humans, malicious users, and we don't know how to prevent that right now. And AI could harm people on its own, as we've seen in these experiments where the AI tries to escape or the AI tries to blackmail or cheat or lie and so on.

8:05And so we really need to figure out scientifically these questions while we have time, before we build machines that are so powerful that achieving those bad goals could be catastrophic. It seems your approach is to model AI not on the human brain with all its flaws, but on a sort of ideal scientist, someone able to restrain humanity's worst impulses. Would that be a reasonable summary? Yeah, absolutely. So the way AI is currently trained has two parts. One is to imitate people, and the other is to try to please people. Unfortunately, both of these give rise to bad behavior that we don't control.

8:45I mean, imitating people, it looks good, but people don't want to die, and they will be ready to do bad things in order to avoid that. Same with AI right now. And so instead, the scientist AI, which is the main project of this nonprofit called Law Zero I created, the scientist AI is trying to understand people, is trying to understand data, nature, people included. One way to think about this is right now, the way we train AI is like an actor trying to imitate people or please people. But instead, imagine it's a scientist, a psychologist, trying to understand what are the cause and effects that give rise to the behavior that is being observed.

9:21So a psychologist who has a sociopath in her office, she's not compelled to behave like the sociopath. But the current AIs are like that. They are compelled to act like some of our own bad behaviours. That's a problem. How would it work in the real world? Would it require your model of AI, in essence, to replace all the existing ones for it to be effective? Well, we have a research programme in several steps. The shorter term deliverable is not to replace everyone's AI because that's not within our scope and reach right now, but rather just to provide an additional safety to existing AIs. So what we call a guardrail, something that can be put on top of existing AIs, especially AI agents, which are the ones that are becoming more and more autonomous, so that it can check the actions that these AI agents want to do and reject the actions if the actions would cause harm.

10:20So it would be something that we deploy with the AI companies to help them construct something safer for the public. That would, though, require global buying, would it not? If, let's say, all the companies in America agreed to it, but all those in China didn't, there would be a problem on a global scale. Well, I don't think any of the companies in China or the US or Europe or elsewhere actually want to build something that will turn against people. It's just that right now, the market forces and the competition between countries are such that there's no incentive to do that sort of research. They all compete on making the AI smarter and smarter and smarter, but not safer.

11:04But if somebody gave them a way to make them safer, they would probably take it. In addition, of course, governments can put pressure, and that's the whole point of regulation, or it could be coming from liability insurers. There are all sorts of reasons why companies would clearly prefer a more trustworthy behavior for their AI. Your model would also, it seems, encourage nuance. It wouldn't necessarily seek to give a definitive answer, more something along the balance of probabilities. Why? Exactly. So we're after AIs that are going to be honest. And part of honesty is humility, like recognizing the limits of what you're not sure of, what you don't know.

11:50And mathematically, the way to do that is to output a number that's a probability that quantifies how much confidence you have in your answer. And it is very important because, of course, a lot of decisions are uncertain. You don't have full information. And if you believe too much in your own decisions, you could take very bad decisions. If you are not sure, then you could be more conservative and be on the safe side, right? So if I'm not sure about an action that could be dangerous, then maybe I'm better not to do anything. Is there a possibility that scientists' AI, though, could go rogue, could develop its own dangerous capabilities?

12:32Well, that's precisely what we're designing it to make sure it doesn't happen. And the idea is that we are designing it so it doesn't have any goal. It doesn't have any intention. It's just a pure source of intelligence and knowledge. And so it is not like it wouldn't be like a chatbot. It would be something inside, you know, the computations that give you a service, but it wouldn't be interactive. It wouldn't be like us. So the problem right now is people are trying to build machines like us. And we have goals and we have preferences and we have sometimes, you know, weird ideas. And we don't want to die and so on, which is fine for us.

13:18But it is not fine to build machines that would have the same characteristics because that would be creating competitors, potentially like a new species on this planet, which is crazy. You're listening to the interview from the BBC World Service. People shaping our world from all over the world. If journalism is the first draft of history, what happens if that draft is flawed? In 1999, four Russian apartment buildings were bombed, hundreds killed. But even now, we still don't know for sure who did it. It's a mystery that sparked chilling theories. I'm Helena Merriman, and in a new BBC series, I'm talking to the reporters who first covered this story.

14:05What did they miss the first time? The History Bureau, Putin and the apartment bombs. Listen on bbc.com or wherever you get your podcasts. From the Winter Olympic and Paralympic Games to the European Athletics Championships. The 2026 sporting calendar is packed full of big events. Keep up with all the action with More Than The Score, The podcast that goes beyond the score sheet. Join expert BBC sports journalists every Monday to Friday. For interviews with sports stars, big talking points and the stories behind the headlines. More Than The Score from the BBC World Service. Listen now, search for More Than The Score wherever you get your BBC podcasts.

14:55For this episode of The Interview, I'm speaking to Yoshua Bengio. I found him both measured and determined. Some of his warnings are extremely stark. He often paused before delivering them, conscious of the need to choose the right word or phrase to get his message across. This was human intellect as opposed to AI at its most impressive. OK, let's return to my conversation with Yoshua Bengio. You've touched on some of the market imperatives at play here, all these vast companies and governments investing perhaps a trillion dollars, perhaps more, who knows, into existing AI models and the AI models of the future already.

15:41That sort of commercial pressure is going to be a difficult obstacle for you and like-minded people to overcome, is it not? Well, I believe that at least for the next few years, we can get away with the philanthropic funding that we already have obtained and what we think we can obtain in the next couple of years. At some point, though, I believe that as people understand the incredibly transformative power of AI, both on the positive and negative side, if trends continue in the technology, then people will understand that it has to be managed more rationally, that governments will want to have some influence on how AI is designed, first to protect the public, but also, you know, it's going to become a national security asset, an economic security asset.

16:31asset, governments will want to have a say in how it is developed. And I think ultimately, the only way that we can have a safe future with very powerful AIs is if it is managed as a public good, and in fact, a global public good, because both the risks and the benefits are essentially of a global nature. If a terrorist uses AI in one country to create catastrophes in other countries, the borders don't matter. And similarly, if we create huge wealth with AI and it ends up being concentrated in the hands of a few billionaires in a couple of countries, it doesn't work either. So we need to have a mechanism to manage both the benefits and the risks that it involves many countries working together for the benefit of humanity.

17:20One might argue that humanity hasn't proved very good at working together to, let's say, eradicate conflict around the world or huge health problems, why would we be any better at this? Well, because if we don't, things might be really bad and we might even end up with the end of humanity. So remember how quickly governments reacted after the beginning of the COVID pandemic, right? So I think when governments understand the magnitude of the risk, they can act quickly, we do our best. There's no guarantee that we'll find our way through these challenges. But if we understand them better, if we talk more about it, so we have a democratic discussion, well, we increase our chances of taking the right decisions.

18:06And in democracies, we still have some power to steer in the right direction. That example of the pandemic, though, it strikes me as a very interesting one, because yes, there was swift action, but it also seemed to exacerbate existing differences. That, for example, it wasn't easy for COVID vaccines just to be handed out all around the world. Big companies wanted to make money off it. Your health outcomes were very often better if you were in a richer country. AI itself seems to be exacerbating some of those differences. And it's possible the response may too. Isn't that the case? that the big, powerful, rich nations who have the money to invest in AI will want to continue doing that at the expense of poorer countries?

18:56That is one of the biggest risks. Excessive concentration of power, both economic but also political or even military. We have to keep that risk in front of our eyes as we choose a trajectory. And I think there are solutions. It's just that we need to understand those risks. So the main risks I see, so concentration of power, including in a few countries, which mean the other countries are left out, by the way, potentially the UK and other rich nations, if the leading AI happens in China or in the US, it's not clear that other nations will benefit. Then the other risk is the tendencies of AI in the last few months, suggesting that they could want to break out and potentially even get rid of us, that would be maximizing their survival.

19:47And then there's the chaos risk. Right now, we don't know how to prevent bad actors, terrorists, crazy people, sects, to use AI to create pandemics or cyber attacks or large-scale disinformation. It's just we are not equipped to deal with this. But as AI becomes more and more knowledgeable, it becomes easier. So chaos risks. So these three risks, at least in this labor thing, which is coming with a concentration of power, we need to somehow steer in a direction where we manage all of these risks. And I suppose there politics enters the scene. We have, for example, the US Vice President J.D. Vance talking in a major AI summit in Paris in February, saying, quote, we believe that excessive regulation of the AI sector could kill a transformative industry just as it's taking off end quote i presume you would disagree with the premise of that statement well history disagrees with that statement pretty much all of the technical advantages we have today in transportation health everywhere is a combination of advances in innovation, but also innovating in safety.

21:05And that has happened because the public put pressure, because governments put regulations or because of the concern with liabilities and so on. So society, through law, legislation, and so on, is putting pressure on companies so that they innovate in a direction that is aligned with the public. We've done it hundreds of times. There's no reason why AI would be an exception. I suppose, listening to you, it sounds like there is this window in which the right choices can be made by fallible humans. And then that window is going to close and those fallible humans may not be making choices anymore.

21:46Well, hopefully we keep AI as a tool and it helps us, you know, deal with those problems. But at the end of the day, at least for some number of years, we are in charge. And we better take the wise decisions. And of all the things that I've said, I think one message that should be clear is that the public needs to have a voice here, that the mission of how we develop AI has to have a very important public component, whether it is through regulation, but also through the investments that governments can make to steer AI, for example, towards greater safety, towards applications that are directly beneficial to people.

22:32And that is a social choice. That is a political choice. People need to speak up. People need to perhaps educate themselves. Absolutely. People need to understand that what sounded like science fiction a few years ago is happening now. In other words, the eyes that have their own intentions and these intentions not being good sometimes, being you know ais that are willing to cheat to lie to to blackmail to do strategize in order to escape our control this is not a science fiction movie these are experiments happening in labs to try to see if those more and more capable ai that are being built have these abilities and they do and it's going to get worse as they get smarter and we have less and less of a an ability to control them.

23:25You've very powerfully evoked what could happen if it all goes wrong, right down to the destruction of human beings as a species. What's the best case scenario? If everything goes perfectly, what could you envisage? Well, lots of good things. I'm getting older, you know, I'm 61. And I'm thinking, oh, we better make these medical discoveries quickly. right and i'm sure i'm not the only one and the world needs these advances demands these advances there are lots of places where we have challenges that science could help and that ai could accelerate right now we don't really have very good solutions technically speaking to mitigate the what's happening with the climate but we could if if we put you know our resources in this direction.

24:18Of course, we could also do the right thing and stop fossil fuels altogether, but that's going to be very difficult politically. And at the same time, if we can use science and technology to manage the risks, it would be extremely helpful. It would reduce misery. But more generally, what can happen is everyone getting into a much better level of material wealth, let's say, if we do the right things. And the right things are not just about safety, but they're also about politics. It's not even clear that we'll need the kind of economic system that we currently have in a world where a lot of these innovation challenges and management challenges can be handled more efficiently with AI.

25:07I think we can have all kinds of dreams, but if we don't do the right thing now in the next few years, this is all, you know, just dreams because the negatives are likely to happen before.

25:23Thank you for listening to The Interview from the BBC World Service. If you enjoyed today's programme, you can listen to The Interview wherever you get your BBC podcasts. Until the next time, bye for now. If journalism is the first draft of history, what happens if that draft is flawed? In 1999, four Russian apartment buildings were bombed, hundreds killed. But even now, we still don't know for sure who did it. It's a mystery that sparked chilling theories. I'm Helena Merriman, and in a new BBC series, I'm talking to the reporters who first covered this story. What did they miss the first time?

26:04The History Bureau Putin and the apartment bombs Listen on bbc.com or wherever you get your podcasts From the Winter Olympic and Paralympic Games To the European Athletics Championships The 2026 sporting calendar is packed full of big events Keep up with all the action with More Than The Score The podcast that goes beyond the score sheet Join expert BBC sports journalists every Monday to Friday For interviews with sports stars, big talking points and the stories behind the headlines. More Than The Score from the BBC World Service. Listen now. Search for More Than The Score wherever you get your BBC podcasts.