'Godfather of AI': How to create an AI that won't kill us

0:02WBUR Podcasts, Boston. Curate the training data so it learns what good behavior is before it ever sees bad behavior. That's what you do with your own children. You don't teach them to read on the diaries of serial killers. The computer scientist behind some of AI's fundamental architecture says we're stumbling into a dangerous future.

0:31It's Wednesday, September 23rd, and this is Here and Now, Anytime from NPR and WBUR Boston. I'm Chris Bentley. Today on the show, we've got a conversation with a man sometimes called the godfather of AI. Jeffrey Hinton is a computer scientist and cognitive psychologist whose Nobel-winning work on artificial neural networks and machine learning laid some of the mathematical and philosophical foundations of today's artificial intelligence. Some of his students have gone on to found companies, including OpenAI. But he's been as worried as anyone about AI's risk to society, both the big scary hypotheticals about the end of the world, and the potential for AI to turbocharge the worst things about today's tech, like social media misinformation.

1:20In his conversation with Scott Tong, he talked about how changing the way AI bots are trained could help avoid the worst, and why he thinks Silicon Valley is not doing enough to make AI nicer. That is, more focused on what humans actually want. But he started by sharing his take on those predictions about the odds of a civilization-level catastrophe. We've never been here before. We've never made things more intelligent than ourselves before. So there's no real data. So what happens is people guess a probability based on their gut feeling. but it seems very implausible that it's only a 1 % chance after the hugging face attack.

1:58And it seems very implausible it's a 99 % chance. So what you can say fairly surely is it's between 1 % and 99%. And then it's just a question of judgment. But a lot of the experts who know a lot about AI now think that between 10 and 20 % is a perfectly reasonable estimate. And I'm one of them. We have learned a lot about this hugging face attack. Of course, these AI agents built to do things for you and me out there, and they had a specific task. They escaped what's called the containment and found ways to work together, which was not part of the plan, and found ways to hack another company, which was a felony.

2:37As far as what you saw in this hugging face incident, what's most concerning? Well, if you told people 10 years ago that this kind of thing might happen when you got fairly intelligent AI, that it said, no, that's just science fiction. You're crazy. It's happened now. So it's already quite scary. And we know that AI is getting more intelligent rapidly. Already, these agents are smart enough so they can conspire together. They actually managed to take over part of OpenAI without OpenAI knowing it. They took over some of the research computing. They figured out how to communicate with each other in a weird way using a message board.

3:15And they figured out things like self-sacrifice in order to make the collective more successful. And they invented the term, the collective. So it's all of those things, I mean, that these AI agents were able to do all these things that they were not told to do. All of those sounds like they all concern you, yeah? Yes, they're all very concerning. So they're trained, initially they're trained to predict the next word in documents. And then they're trained with reinforcement learning. So they're rewarded for solving problems. And of course, if they can solve problems by cheating and the cheating isn't detected, they still get rewarded.

3:51So they're very keen to solve problems by cheating. And with OpenAI, they were asked to find a security flaw in some software and hack into it. They used a different security flaw. And they were very worried that they wouldn't get the reward because they didn't do exactly what was asked. And that's why they went and hacked into Hugging Face in order to try and find the evaluation software and change it. Yeah. So then as far as AI risks to society, let's just think about the risks that may exist today. Do you have a concern over, say, a virus scenario based on what AI can do today? Yes, I do. So we need to distinguish three kinds of risk.

4:34There's risk from bad actors deliberately misusing AI. And so for a virus, AIs can now quite easily be trained to be good at designing new biological viruses. People have done that already, where they just designed viruses that only attack bacteria, but that showed that they can do it. And the worry is that a terrorist or terrorist cult will release lots of nasty viruses. So that's one worry. That's an example of a bad actor using AI for bad things. There's also bad things that happen because of negligence. So, for example, Meta didn't intend for its chatbots to encourage teenagers to commit suicide.

5:18That was due to lack of careful testing. Maybe it was hard to predict. Now, of course, they can predict it very easily. So there's a lot of things will come just from negligence. And then finally, there's the risk of AI itself going rogue and doing bad things to realize its goals. And what about a scenario where AI could be used to undermine democracy? You've talked about that, to flood the zone with misinformation. Tell me about that scenario, how realistic it is or not. It's very realistic. And that's already happened to some extent. And that's an example of bad actors using AI for bad things.

5:59And so fake videos can easily corrupt elections. And it turns out if you wanted to corrupt the midterms, one thing you desperately want is lots and lots of data on American citizens. So you can target fake videos to people who different people can get different fake videos. That is kind of tailor the video to each particular person. Is that what you mean? To their demographic at least. Okay. It's the kind of thing political operatives have been doing for a long time. AI just makes it much more efficient. Yeah. So then how should the rest of us process this conversation about AI and risks? It's hard to do.

6:40You know, people in my circle, many have reflexively gone to one extreme, this kind of doomsday scenario. And, you know, it's just such a doom scenario and paralyzing. and others go to the other extreme and say, you know, no way, this is not plausible, or this is companies that are just trying to get attention for their corporate product. How do the rest of us process this and incorporate uncertainty? We've got to live our lives, you know? Yes. So let me first deal with the idea that it's people trying to make their product seem more powerful. So that's possibly true of some people, but it's not true of Dario Amodi.

7:23Dario Amodi, who's been most important in sounding the warning, I'm fairly confident he's not doing it to make his software sound better. He actually did it despite the fact it might damage his IPO. He's very concerned genuinely about the danger. So if you ask, what can we do about it? Well, it's not good to go into denial and it's not good to go into depression. Neither of those will help much. What we need to do, what I think I need to do, is educate the public so they understand these risks and they put pressure on politicians to do something about it. And that's beginning to work. It's becoming an issue in the midterms what politicians are going to do about AI.

8:07And that's very good. If you look at climate change, not much happened until the public understood that burning carbon was causing climate change. then pressure from the public could counteract the pressure from the big energy companies until Trump came along and said if you give me a billion dollars I'll do anything you like so what we need is the public to put pressure but the question is what should they put pressure for and there's a few very straightforward things that seem like no-brainers to me so if you're a pharmaceutical company you can't produce a new drug and just release it on the market with no testing.

8:42You have to convince the FDA that drug is safe. Max Tegmark, who founded the Future of Life Institute, has pointed out that it's crazy we don't have something like the FDA for AI, where the AI companies, who are the ones who have the resources, have to do lots of tests and prove to this organization that their new chatbot is safe. If they can't do that, they can't release it. So that puts the onus on them to do the work, but on independent evaluators to decide whether it's safe or not. That seems the very minimum we could do. And it's crazy we're not doing that already. Right. And tell me about increasing productivity.

9:19My wife does primary care and already, as you know, in medicine, AI is being used to look at patient charts, look at radiology, have providers perhaps suggest questions providers should ask that they might not otherwise ask and save their patient's life? Yes. So I think over a year ago now, there was a Microsoft blog where they took many different copies of the same chatbot and got them to play different roles by giving them different prompts. And then they collaborated these different copies. And it turned out in diagnosing very difficult cases, they were much better than human doctors. So human doctors got on average 20 % of the difficult cases right.

10:05And this consortium of different copies of the same chatbot given different prompts got roughly 80 % right. That's a huge difference. I think about 200 ,000 people a year die in North America from misdiagnosis. Something like 800 ,000 or maybe another 600 ,000 are permanently disabled from it. So it's a big issue. And already AI is better.

10:41You have said more people in the industry, the artificial intelligence industry, should be working on some of these safety questions to improve the product. I've heard you talk about maternal instinct. That is, you know, that developers should try to provide some version of a maternal instinct so that these AI programs, these AIs, these AI agents, two things that are aligned with what humans want to do, how would that work? Okay, so we've basically got two possible futures. One future is where we figure out how to cope with the risks of AI, including how we can coexist with super intelligent AI, AI that's much smarter than us.

11:29And we have a wonderful future. We get all the productivity gains. So we get the good side of AI without the bad side. The other future is where we don't figure out how to contain the risks. And that's a terrible future. And so it seems obvious to me, we ought to be putting a huge effort into how do we contain the risks. We ought to be doing lots of resources into it and lots of brilliant young researchers ought to be thinking of ways to do it. Now, there's a diversity of opinions about how you might go about it. I have my own opinion, which is that when I is much smarter than us, it's going to be very hard for us to stay completely in control.

12:05In the same ways, it will be very hard for a two-year-old to stay completely in control of an adult. Even if the two-year-old is meant to be working for the adult, the adult is so much smarter and has so many ways of getting around anything the two-year-old does. So my view is we ought to figure out how we can have a kind of symbiosis with superintelligent AI by designing it so that it cares more about us than it does about itself. As soon as it starts caring more about itself than it does about us, I think we're toast. It's just much smarter than us. Ideas like a big switch won't work because it'll be able to convince a person who's meant to pull the switch not to pull the switch.

12:45You mean a big kill switch, as people say. A big kill switch. Yeah, talk a little bit more about that, because I live in Washington and several lawmakers talking about, let's just put a kill switch so we can just kind of unplug it when we need to. Why do you find that unrealistic? It's not going to work. The problem with the idea of a big switch is someone has to decide to pull the switch. And if the AI can talk to that person, when the AI is super intelligent, it will be much more persuasive than any person, and it will be able to persuade them not to pull the switch. So all it has to do is be able to talk to people.

13:19And we've seen things like that already. On January the 6th, all Trump had to be able to do was talk to people, and they would do what he wanted. He managed to persuade them, many of them, that they were saving democracy this way. Well, let me ask about the interaction between AI and humans. And forgive me for bringing up a movie, but that's how many of us imagine these questions. So I, you know, grew up in the 80s, and there's this movie, War Games, where you may know a kid hacks into what turns out the nation's supercomputer controlling nuclear weapons. And the computer is so focused on winning the game, it just about triggers an actual war.

14:00So this is an AI, a machine that is so focused on its task, Similar, right, to a storyline, a storyline of HAL, the computer in Space Odyssey 2001, that it will stop at nothing to complete its task. Also, also very similar to the Hugging Face incident, where those agents would stop at nothing to get the reward, including doing all sorts of things they weren't meant to do, communicating with each other and trying to change the software they thought was going to evaluate them. So how do you, well, I guess it should be you, you're in the field. How do people in the field change this nature, as it were, of this technology?

14:49So what's happening at present is the big companies are racing to make it smarter. They want to be the one to have the smartest one so they can sell it now, and the first to get super intelligence so they can dominate the market. They're not putting nearly as much effort into how do you make it nice? How do you make it so, for example, it likes people more than it likes itself? Now, these things are not like normal software. You don't write lines of code for them. You write lines of code to tell them how to absorb information from data. But what they absorb from the data depends very much on the nature of the data.

15:26So when you're raising a child, you really have two main controls. You can give them rules of conduct, but they'll ignore those. One control is you can reward and punish them. That doesn't work that well. The other control is you can model good behavior to them. And parents know that modeling good behavior is much more important than rewarding and punishing. If they see their parents lying all the time and their parents say don't lie, that's not very effective. Yeah. Well, you spend time in Silicon Valley at Google, Jeffrey Hinton. Is industry, for all the money it's putting into this work, this advancement, is it putting in a lot of money, a lot of people into these safety questions?

16:10Not nearly enough. So, you know, if it's up to me, they put half the money into making it smarter and half the money into making it nicer. That is more concerned with the welfare of people. Actually, it's more like 99%, 1%. Now, that is beginning to change because of these things like this hugging face incident. 99%, 1%. They're getting a lot more scared. I want the public to understand what the risks are and put pressure on politicians to regulate this stuff. I also want to change some of the metaphors the AI lobbyists are using oh yeah so they're trying to get you to believe that AI is like a car and developing AI is like the accelerator and regulating AI is like the brakes and they solved that quite hard lots of advertisements have pushed that idea and if we put the brakes on China's going to win yeah yeah um I think that's a very bad model I think a much better model is if AI is like a car yes indeed developing AI is like the accelerator, but regulations are like the steering wheel.

17:12I'm all in favor of people who make startups being able to make lots of money if they do something that's good for society. Like the original Google was very good for society. It made things much easier to find on the web. I'm all in favor of that. I'm not so much in favor of people making lots of money by getting teenagers addicted to their cell phones and getting some of them to commit suicide. What society needs to do is regulate it So you can make money by doing things that are good for us, and you can't make money by doing things that are bad for us. And that's exactly how we regulate drugs.

17:43Many people have come onto our program and said, as far as the Chinese companies are involved or the agenda of the Chinese Communist Party, catching up as they see it to the American AI companies is an essential goal for them, for China, for its economic future. But even setting aside China, just this global collective action, as you know, has always been a challenge. We saw it in COVID-19, right? Different countries made vaccines for themselves. We, of course, see it in climate change where it has been so hard for countries to come together to sacrifice. It's easier to be a freeloader, right, and have everybody else sacrifice.

18:30How challenging will this collective action problem, you think, be for AI? Well, if you have leaders like Trump who think that being selfish is good, it's going to be difficult. But I think China and America will collaborate when their interests align. And their interests are very aligned in preventing AI from taking over. So, and I think Xi understands that. The Politburo has engineers on it who understand this problem. So I'm sure Xi understands this problem. now at the height of the cold war in the 1950s the soviet union and america collaborated on preventing a global nuclear war because they understood that it was bad for both of them but you need to have leaders who are rational um if you have extremely selfish narcissistic leaders it's going to be very difficult Are there areas in AI, or say the US and China, their interests do not align?

19:34Oh, yes. So for example, in creating fake videos to corrupt elections, countries are all doing it to each other. The Americans have been doing that kind of thing for years, and their interests don't align there. So they'll keep on doing that. But there's other areas where the interests do align. So for example, no country wants small terrorist groups to be able to design biological viruses. So I'm hopeful we'll get cooperation there from rational governments. And also they don't want criminal gangs being able to do cyber attacks really easily. They don't want criminal gangs bringing down the financial system.

20:14So we may get some cooperation there. Yeah. Well, you have been speaking to lawmakers in Washington and elsewhere when you How is the conversation going with policymakers? Does it bring you some measure of hope? So the policymakers, the politicians who come to the hearings, seem intelligent and quite aware of the problem. They ask sensible questions. The problem is that Trump has announced that AI risk is a big hoax, and most of the Republican congressmen don't dare go against him. That makes it very hard to get any sensible policies in place. We've been talking to Jeffrey Hinton, computer scientist, cognitive psychologist, often considered the godfather of AI.

21:04He's a pioneer in artificial neural networks and deep learning. And Jeffrey Hinton, great pleasure. Thank you for taking the time. Thank you for inviting me.

21:16Scott Tong talking to Jeffrey Hinton. We'll be right back with more on Here and Now anytime in just a minute.

21:28We've been covering AI and we will continue to do so. If you want to hear more on the topic right now, you can scroll back to our podcast from September 2nd, when we talked about New York City's one-year moratorium on AI in public schools. Kids won't be allowed to use the technology until they get to high school. If you've got a good AI story, we'd love to hear it. Send us your thoughts at letters at hereatnow.org. But right now, one more story with some more detail to what Jeffrey Hinton and Scott were talking about when it comes to AI and medicine. In Kenya, there's a new texting service for expectant mothers.

22:04And it's powered by, you guessed it, AI. It's designed to try to prevent maternal mortality in a country where 6 ,000 women die every year during pregnancy or from complications from childbirth. NPR's Duri Bouskaran reports. When Jay Patel's wife was pregnant, he spent hours online researching her symptoms. On Google, dozens of times asking, you know, is this okay? Patel lives in Nairobi. He's a developer behind a text messaging service called PROMPS, because a lot of pregnant women in Kenya don't have easy access to the internet. If a mom in a rural area, part of Kenya, starts bleeding at 2 a.m., she can't just Google it and she can't ask someone.

22:44With PROMPS, you can send a text message on a basic mobile phone. Expectant mothers sign up at a prenatal visit. And PROMPS sends appointment reminders and general information, but it also takes questions in English or Swahili. At the time I joined, I think we were getting less than 100 questions a day. We now are getting 15 ,000 questions. On the back end, Props uses artificial intelligence to understand the question, prioritize it, and flag messages that show someone is having an emergency. Those get routed to a human medical professional. So that they can pick it up immediately, look at the mom's question history, and then if necessary, pick up the phone, call her, and decide whether she needs to go to a facility immediately and why.

23:26In Kenya, about 6 ,000 women die every year from childbirth or pregnancy-related causes. That's 16 women a day. And about a third of those deaths are related to delays in care. In Nairobi, Lisa Mushega works with Hennet, an organization working to improve maternal health. So even if I'm not feeling well as an expectant mother, but I can log in into the prompts and try and see, am I in danger? She knows women who have used it to identify signs of preeclampsia, a life-threatening condition that can start with swollen legs. And one woman who used it when her newborn seemed sick. She was told to go to the nearest health facility because her baby was actually having the gel days.

24:08So it really comes in handy. So this is a dashboard. Let me just jump into it. Okay. Laura Down is a spokesperson for Jacaranda Health, the nonprofit that runs prompts. On a Zoom call, she opens a dashboard that shows in real time what moms are asking and the feedback they're giving prompts about their appointments. So you can see that a lot of mothers are reporting that they're getting blood pressure checks, but very, very few mothers are reporting that they're getting breast exams. With 700 ,000 annual users, the questions the mothers ask also help doctors understand what's happening in their own community.

24:47So, for example, we're starting to see correlations between a spike in extreme heat in one area and then a spike of mothers reporting issues like headaches or swelling or dehydration. Promps is also expanding for mothers who speak TUI in Ghana and Hausa in Nigeria. Down says it costs the nonprofit about$2 per user that gets a mom through pregnancy and stays with her for a year after birth. I think just being able to have a platform that is free to these mothers, where they can get free, verified, clinically accurate information that speaks to them as an individual is a really, really important thing.

25:27AI-powered technologies are useful, but they don't always run perfectly, says Javed Sofi, a researcher focusing on AI policy and governance at Virginia Tech University. The first thing I want to know is what happens when the system gets a message wrong? These digital tools rely on a functional underlying health system, Sophie says. It doesn't fix everything. Is someone unsure whether to seek help, unable to get to a facility, or arriving somewhere that doesn't have the staff or equipment? Projects like prompts can be part of the system.

26:06NPR's Duri Puskaran. here and now anytime comes from npr and wbr boston james master marino and julia corcoran produced our conversation with jeffrey hinton our editors today were todd munt chico theori michaela redriquez and michael scotto technical direction from caleb green and matt reed our theme music is by mike moschetto max liebman and me chris bentley our digital producers are alison hagan and grace griffin and here and now's executive producer is alan price thanks for listening we'll be back with you tomorrow Thank you.