Black Box: episode 6 – Shut it down?

0:00This is The Guardian.

0:07Hey, this is Michael Safi, host of The Guardian Investigates, and I have some news. We've made a brand new series of Black Box, our chart-topping podcast about the way AI is reshaping our lives. This time, it's about this sort of extraordinary change that's happened right under our noses. More than a billion people suddenly talking to machines, sharing with them their deepest secrets. Maybe you are too. Black Box, the chatbots, is coming very soon on the Guardian Investigates feed. And because Today in Focus is off this week, we're bringing you the first season of Black Box instead. These episodes are exactly as we broadcast them back in 2024.

0:50And just to note, some of the laws and regulations around AI that we talk about in the episode have been amended since then. Black Box is next.

1:04This is a story about the end of the world. And it starts with a joke.

1:13About two years ago, the scientist Geoffrey Hinton, one of the godfathers of AI, was working at Google. Tinkering with a program most people haven't heard of. There was a modelling Google called Palm, P-A-L-M. It was basically a Google version of ChatGPT, never released to the public. It was just research. And Hinton was checking it out when it did something that terrified him. Palm couldn't explain why jokes were funny. Not all jokes, but quite a few of them. Here's one of the jokes it could explain. It's someone saying, I was going to fly in to visit my family on April 6th. And my mum said, oh great, your stepdad's poetry reading is that night.

1:54So now, I'm flying in April 7th. I know, not a very good joke. But the point is, when you asked Palm 2 why this joke was funny, it could tell you. The joke is that the speaker's mother is trying to get them to go to their stepdad's poetry reading. But the speaker doesn't want to go, so they are changing their flight to the day after the poetry reading. Hilarious. To Hinton, though, this was really frightening. He has a really good sense of humour. And to him, that's something innately human. To get a joke, you need to understand so many nuances of language, human nature, how people think. It told him these things he had helped to create, they had crossed a barrier.

2:39And I'd always sort of, just for myself, use that as a criterion of whether these things were really getting to human levels of understanding. that they didn't just understand language literally. They understood why a joke was funny. So maybe we're much closer to human level intelligence than I thought. Maybe we're not 30 to 100 years away. Maybe we're only 5 to 20 years away. I was always worried about things like fake news, corrupting elections, things like surveillance, things like battle robots, things like AI replacing jobs. But that's when I got really scared about the existential threat.

3:18In that moment, Hinton realised the technology he'd helped bring to life was becoming a catastrophic success, getting smarter quicker than he ever expected. And that was going to be a problem. That they would get a lot smarter than us and they would take over.

3:42But Geoffrey Hinton was late to this party. Someone had got there much earlier, two decades earlier, and tried to warn the world.

3:54The first episode of Black Box was about the godfather of AI, the man who spent his career trying to bring machines to life. This final episode is about Hinton's shadow, the godfather of AI doom, A guy who spent his entire career trying to kill this technology before he says it kills us all. And who now thinks it might just be too late. From The Guardian, I'm Michael Safi. This is Black Box. Episode 6, Shut It Down. By the year 2030, there'll be computers that can carry out the same amount of functions as an actual human brain. But humanity is accelerating into a future that few can predict. Well, is artificial intelligence out of control?

4:49Open the pod bay door, please, pal. You've spoken out saying that AI could figure out a way to kill humans? How could it kill humans? I'm sorry, Dave. I'm afraid I can't do that. We've come quite far. Stop. We need to rethink. We need to rethink. We need to rethink.

5:13Eliezer Yudkowsky grew up in an Orthodox Jewish household in California. He never went to school. He was educated at home. I remember being pretty studious, much more wrapped up in my parents' book collection than the rest of life, really. But the kinds of things Yudkowsky was reading about as a young kid were, I would guess, pretty different from the kinds of things you and I were reading at that age. Nuclear power, space travel, asteroid mining, the idea that the solar system could support a lot more than the number of people currently on Earth. How old were you when you were getting into that book?

5:54I think nine years old. Okay, that's pretty young to be getting into those sorts of topics. I mean, I think I turned out okay. By 11, he'd moved on to even more serious stuff. Ideas like, can you scan an entire human brain and move the person into a computer? Can you build artificial intelligence? And this, he realised, this is what he was into. So from 16, I knew that was what I was going to be spending my life doing. Yudkowsky was coming of age in the late 90s, just as the internet was taking off, and spending most of his time arguing with people on online forums and email threads. And that's when he first started to get noticed.

6:33He mostly emerged from the internet, right? From the very early internet. That's how I think of him. Tom Chivers is a journalist. He's written a book about Yudkowsky and his ideas. And there was this one essay Yudkowsky wrote when he was just 17 years old that caught a lot of attention. It was full of teenage angst about the state of the world. I've got the quote here. I have had it. I've had it with crack houses, dictatorships, torture chambers, disease, old age, spinal paralysis and world hunger. I've had it with a planetary death rate of 150 ,000 sentient beings per day. I've had it with this planet.

7:06I've had it with mortality. None of this is necessary. We can end this. And he said there was a solution. One day soon, humanity would create super intelligent AI.

7:22His point is, if you build something really, really clever, then it can solve these problems. The problems we have are the problems of humans not being clever enough to solve their own problems. That was what I hoped for. Everyone lives happily ever after, but literally ever after. Up until the universe ends, and I wasn't too sure about us not figuring out a way to get out of that. Super intelligent AI isn't chat GPT. It's not AI that can recognise a face or see cancer. It's something that can do everything we can. Think faster than us, knows more, knows everything every human who's ever lived has known.

8:01It would literally be beyond our comprehension, like an ant trying to understand an iPhone. A human being thinks for 100 years and then dies. What if you have thought for 10 ,000 years? What if you have read everything that's ever been written? And then some. This is the notion of superintelligence.

8:25Yudkowsky, again just 17 years old, was writing these essays and discussing them online with philosophers, professors, big-time thinkers, and holding his own. He started gaining a following and persuaded donors to give him money to start an institute dedicated to solving all the world's problems by bringing superintelligence to life. Over the next six years, he's doing this work and begins to engage with another school of thought. One that says, actually, superintelligence wouldn't save humanity. It wouldn't usher in some utopia. It would be dangerous. And these arguments that at first he dismissed, now they start to nag it him.

9:07As a young lad, I thought that if you built something very smart, it would automatically be very wise and then very moral. But what if superintelligence isn't automatically nice? And as a result of staring at that issue, my entire false belief that very smart and capable systems are automatically nice started to fall apart. In other words, the AI he thought would save us all was much more likely, in fact, logically certain, to kill everybody instead. So in 2003, Yudkowsky decides he's going to turn his life's work around. He's no longer going to focus on trying to bring super intelligent AI into existence.

9:55Now he's going to try to convince the world this stuff shouldn't exist. And he writes this collection of essays and blog posts that are collectively longer than war and peace. They're just short of the length of the Bible. And they come to be called the Sequences. The core of Yudkowsky's message wasn't just that super intelligent AI won't be nice. It won't be nice, but it won't be cruel or evil either. Oh God, there's some machines that are starting to take over. He says, you need to forget everything you think you know about killer AI. Forget about the Terminator. Why are they killing everyone?

10:37Destroy any possible threat to Skynet. This won't be a scenario where the AI becomes conscious, decides it doesn't want to be ruled over by humans, and launch a war against us. Great movies, but apparently, very inaccurate depiction of the AI-pocalypse. If you're going to use a film for the analogy, the film you should use is Disney's Fantasia. The Mickey Mouse movie? Yeah, the Mickey Mouse movie.

11:06If you haven't seen Fantasia, it's this old Disney movie.

11:13Mickey, as in the mouse, is some kind of trainee wizard, and one of his jobs is to fill a cauldron with water. But that's hard work and it's boring and takes a long time. So Mickey enchants a broom and asks it to take some buckets and fill up the cauldron instead. And then the broom grows little arms. It grabs a bucket in each arm and waddles off to the well. A few hours later, the castle is completely flooded. The broom had followed Mickey's instructions perfectly, filled up the cauldron. But once it was full, it just kept going.

11:53For Yudkowsky, the key thing here is that the broom wasn't evil. The broom had no opinions. It was just doing what Mickey asked. What Mickey asked was the problem. When he said, fill up the cauldron, he expected the broom to understand all of these unspoken nuances. Like, at some point, you'll need to stop. If the room starts flooding, there's a problem. them. But why would a broom know that? The broom just does exactly what you tell it. And so would an AI. Just seems mad. But it's because we have human brains with human values and human philosophies and all these things, which it simply doesn't have.

12:35So it's sort of, our model of human wisdom does not apply to it. The sort of fundamental idea of AI, of AI risk is that the AI will do exactly what you tell it to do. And that it can only do exactly what you tell it to do, but what you tell it to do is not what you want it to do. The point is, you can't just tell an AI what you want it to do. You need to tell it all the things it shouldn't do to achieve that goal. And things can very quickly get out of control. Let's take another domestic chore. Say you build a super smart, all capable AI robot and say, clean my house. Simple. And at first it does.

13:14It sweeps up, it mops the floors, and then it decides an even better way to achieve this goal would be to have thousands of versions of itself spread out across the whole house, cleaning all day, every day. So much cleaning that eventually you can't even live in your house anymore because every surface is being cleaned all the time. You just have to leave, lock the door and find somewhere else to live. But that's not the end of it. At some point, this robot realises, at this rate, the world is going to run out of cleaning products. And if that happens, it can't achieve its goal. So it goes and strips the planet's resources and puts them all into creating a nearly endless supply of Mr Muscle.

13:58And the last evidence of human civilisation on this Earth, the last thing standing, is your astonishingly clean house.

14:15I know, this all sounds a bit mad, but the idea here is, it's really hard to align what you want with what AI actually does. And once you create something incredibly smart, incredibly powerful, you'd only get one go at getting it right. So maybe you're thinking, okay, that's not great, but why don't we just give it an off switch? The AI starts to get a bit crazy, pull the plug, turn it off. But Eliezer and others say, no, that wouldn't work. It is very hard to fulfill your goals if you're switched off. So if the AI has reason to think that you will be coming to switch it off, then it will take steps to avoid that.

14:59If I decided that you were dangerous and I should switch you off, you would try and resist my doing that, I imagine. And for the same reasons, an AI would try and resist me switching it off if I thought it was going to be dangerous and it'd be better at doing it than you are. The point is here, if you're in a position where this thing's trying to outsmart you, it's going to outsmart you. I I mean, by definition, it's super intelligent. Yeah, yeah, that's exactly it. And maybe its next step is to make copies of itself around the internet. If it's really, really powerful and has control of nuclear codes, maybe its next step is to nuke everybody.

15:33Okay, so you can't count on being able to turn it off. But everything Tom just said assumes that the AI is somehow connected to the real world. It wouldn't be able to copy itself to the internet or access the nuclear codes if you just kind of stick it in a box, put this oracle-like superintelligence on a single computer, lock it in a room, and disconnect it from the internet. And if you want its help in solving a problem, you just go into the room and have a conversation with it. Sorted, right? Apparently not. There's a danger in just talking to a thing that understands you far better than you understand yourself.

16:12His argument here boils down to one question. Do you really think the smartest entity that's ever existed couldn't convince you, manipulate you into setting it free in the world? Really? It's not like there's a sort of mechanical lifeless mind trapped in a computer. It's like there's an alien civilization that thinks a thousand times faster than Earth trapped inside a computer. If you were in those shoes, how would you get into the real world? In any case, Eliezer says, the box thing is probably a moot point because that's not how things are going. OpenAI does not keep GPT-4 in a box unconnected to the internet.

16:55They trained it connected to the internet from the very start because the computers that train things are connected to the internet. And then what was profitable was to connect ChatGPT to the internet and sell it for$20 a month to everyone. You're describing a world where humanity can somehow just decide en masse, this technology is so dangerous that it needs to be kept in a tiny box. We do not have that capability at the moment. We should, but we don't. I don't know. Maybe all of this sounds a bit far-fetched. But Eliezer says, if you sit down and think about it, you'll see. It's all rational.

17:33And over the years, a lot of people have taken Elieiz's ideas about how to think rationally about AI, how to think rationally about every problem in the world, very seriously. They spawned a whole movement, people who call themselves rationalists, devoted to the ideas laid out on his website. It was read by people very much in Silicon Valley or in computer science names who would later be involved in DeepMind or who wrote AI textbooks and things like that. It was an influential thing. One thing that I find amazing about Eliezer's story is that when he was outlining these ideas, warning the world about AI, AI itself was a bit of a joke, barely able to tell the difference between a dog and a cat.

18:22But he was absolutely convinced and worried that much smarter AI was coming. And on that at least, he turned out to be exactly right. Yes, there's no person who made these captions. This was machine made. Yes, it's true. Gradually. And then rapidly. A computer program has just beaten a nine-day professional. With Yudkowsky warning at every step, slow down. This is bad news. It's that you can actually screw up and the entire human species stands at risk. Geoffrey Hinton's vision came to life. Users can now have spoken conversations and have been translated between two languages almost instantly.

19:08Neural networks started to work. Alexa, what do you do? I can play music, answer questions, get the news on weather. Powerful AI systems, capable of deep learning and like the human brain. Too complex for us to understand. Black boxes. While Eliezer watched on in horror.

19:35Elias, in the years that you were writing about these topics, gaining followers and influence, we also started to see significant progress when it came to AI, especially with neural networks. What was it like for you to watch the field start to progress? Pretty horrible. Deep learning is pretty much a worst case scenario from the perspective of being able to align or even understand AI systems, that was a death sentence. Back when I was getting into this field, nobody knew that it was going to be giant inscrutable matrices that were going to win all the prizes. But there was still a hope back then, that by the time you got to something like ChatGPT, you'd have some idea what it was thinking in there.

20:18And that is not what we have. What we have is AI that is barely a step above how natural selection built humans, completely blind. We do not have AIs that are built, we have AIs that are grown. And then at that point it's hard to control because it's hard to understand. A bit over a year ago, all of a sudden, the world started talking about AI. And right now, ChatGPT is rattling the AI world. ChatGPT arrived. Get ready to laugh, cry, maybe even throw your computer at the window, because the latest AI technology, chat, GBT, is here to shake things up. AI that could have a conversation, pass the bar exam, discuss the plot of any book.

21:03Artificial intelligence that actually seemed intelligent. It is writing better than most of my students write at this point, you know, college freshmen. And as people started to understand, we had entered a new age. Yudkowsky kept trying to raise the alarm. now to much bigger audiences. I tried to get this very important project started early so we'd be in less of a drastic rush later. I consider myself to have failed. Nobody understands how modern... Audiences who didn't always know what to make of his apocalyptic message. Why would they want to go in that direction? Like, AIs don't have feelings of sort of envy and jealousy and anger.

21:48But he had one high-profile person who, last May, announced he had come over to Yudkowsky's way of thinking. Geoffrey Hinton says he wants to be free to talk about the possible dangers of technology that he helped to develop. Not about everything, but about the bigger point, that AI could be, and in fact was likely to be, extremely dangerous. Given the rate of progress, we expect things to get better quite fast. So we need to worry about that. And in the months since then, fears around AI have become just as prominent as hopes of what it could one day do. We face a genuine inflection point in history.

22:32One of those moments where the decisions we make in the very near term are going to set the course for the next decades. The Biden White House in October issued a major executive order that will force the biggest AI companies to share safety data with the government before they release their models to the public. A few months ago, Rishi Sunak gathered global leaders at a summit dedicated to the existential risks of AI. Get this wrong, an AI could make it easier to build chemical or biological weapons. Terrorist groups could use AI to spread fear and destruction on an even greater scale. Criminals could exploit AI for cyber attacks, disinformation, fraud, or even child sexual abuse.