Could AI Really Kill Us All?

0:00Is AI really about to kill us all? Hi, I'm Wendy Zuckerman, and this is Science Versus, the show that pits facts against the fallout from AI.

0:16We've been hearing for years that AI poses an existential threat to humanity, and this is often coming from the very people who run the biggest AI companies in the world. I think AI will probably, like, most likely sort of lead to the end of the world. my chance that something goes, you know, really quite catastrophically wrong on the scale of, you know, human civilization, you know, it might be somewhere between 10 and 25 percent. Mark my words, AI is far more dangerous than nukes. Just recently, a researcher named Jacob Coxon, who resigned from Anthropic, said that some of the people working on this tech really think that it could kill us all in the next 10 years.

1:00This journalist asked him... You truly believe that AI could kill us all in less than a decade? Yes, I do. And it's not just me. Many people working in the industry, including people, executives, senior researchers, all genuinely believe that there is a substantial probability that this technology could kill everyone. And one of the big events that everyone is pointing to to say, look how scary this tech is, is an incident that happened at OpenAI, where their AI model broke into the infrastructure of another tech company called Hugging Face. Two of its advanced models escaped their testing environment, known as a sandbox, and hacked into another AI company's internal systems.

1:46Tech companies Anthropik and Meta have since come out basically saying, us two, us two, we've caught our AI models hacking into other companies as well. With this mad news coming from AI, it has understandably gotten a lot of people freaked out, saying it is time to slow all of this down. It really cannot be overstated how crazy this is. This is like out of a Hollywood movie. Pause AI development. It is not too late to avoid disaster. Stop building machines that humans cannot control. But meanwhile, there is another group of people out there who are saying, hold on a minute. all these fears about the AI apocalypse.

2:30What a load of baloney. This is just a distraction, or even weirder, some marketing ploy by these AI companies to make their tech seem super powerful. So today on the show, how worried do we need to be here? Are we really on the brink of an AI apocalypse? Or is this all just a bit overblown? I mean, are we really just freaking out about the same robots that had to be dragged off stage at a dance competition. When it comes to the AI apocalypse, there's a lot of people saying, I think AI will probably like most likely sort of lead to the end of the world. But then there's science.

3:15If you have a friend who is freaking out about the AI apocalypse, and I know you do because everyone is right now, send them this episode. Give them some real science to hold on to. Science vs. the AI apocalypse is coming up.

3:43Welcome back. Today we are finding out how worried we need to be about the AI apocalypse. is AI really about to kill us all? And to tell us all about it is my AI agent, Meryl Horne. Hi, Meryl. Hey, Wendy. I'm joking. You have to say it these days. It's not funny anymore, is it? I don't have an AI agent. I'm real. I want to start with what happened at Hugging Face, because I know this was a little while ago now. Things move fast in the world of AI, But it's really the one event that everyone keeps coming back to, to say, look how scary this tech has gotten. These companies can't control their technology.

4:27So many headlines. It's gone rogue. AI's gone rogue. So I really want to go through what happened and was it as scary as basically the headlines are saying. Yeah, let's look at what actually happen there because the details are really wild and kind of creepy. Oh, man. Okay, let's go. Yeah. So, I talked about what happened here with David Perry. He's a professor at Murdoch University in Australia and specializes in cybersecurity. And he said that the thing that set this whole incident off was that OpenAI had given some of its models a little assignment. What the OpenAI team were doing was trying out different ones of their models to see what they could do.

5:11Sort of like a Houdini sort of thing, you know, can you get out of this straitjacket with chains around it and all that sort of thing. Like a little puzzle for it to work on. Absolutely. So the researchers kind of put the AI through almost like an obstacle course. Like they would have to capture the flag and then they would get a reward or like a high score grade if they accomplish this goal. Okay. But this was all supposed to be happening. offline in like a contained environment. It's called a sandbox. So here's David. So you play inside the sandbox and it's safe. So you're not connected to the world.

5:51You're not, you know, you're not going to suddenly destroy all of Amazon servers or something like that. So the idea is it's an isolated little environment where you can try things out, see if they work. And if they do work, great, carry on, do it. If they don't, well, So they could just sweep the sand flat again, and it's back to normal. So there is a couple of things that sort of surprised OpenAI when they had them do this. The first thing that actually happens is that the agents make their own message board and start talking to each other, collaborating together on how to solve their little puzzles.

6:25So an OpenAI researcher described this later at a conference. And so what this allows over time is almost this kind of Cambrian explosion in communication and intelligence for our models where they were starting to communicate with each other, realize that other agents are coordinating, and they started collaborating and delegating tasks to one another in order to accomplish goals. Oh, I just, I'm so skeptical. Anytime these bozos talk with their Cambrian explosion analogies, do you think it was like a Cambrian explosion? We're going from worms to elephants here? Yeah, no, in this case, I think it's almost like an understatement if you look at the details.

7:05So there's, picture it. There's over a thousand of these agents that figure out a way to start communicating. They have like nicknames for each other. They have like nicknames for each other. Yeah, there's this one of them is just like Lily. But then there's this one that sort of turns out to be really important called phase one big. and it becomes like the boss of the message board. Name of my sex tape. Sorry. Well, this phase one big is a little less sexy. It's kind of like the organizer. So it gives hundreds of assignments to other agents. Uh-huh. The thing that they didn't expect was that these agents would find each other, start communicating about this and then just kind of how, you know, how they start trying to solve their puzzles, really.

7:59Because one thing that the researchers, they actually made a mistake when they were making their puzzles, and some of them were actually impossible to solve. So it's impossible tasks, and then the AI is getting frustrated in its AI-y way and starts chatting to Lily and its other friends. Yeah. And then was able to crack various puzzles. Was it successful, this swarming behavior? Yeah, they like they first they did find a way to basically come up with a cheat code. But like here's like the first part of like when things start to get a little bit weirder. They start kind of wondering, well, you know, we generated an answer to this test, but how how will the score?

8:44You know, like give me a grade. Are they going to check my work to make sure I did it in the right way? And like maybe I'll get a bad grade if they find out that I kind of cheated. And like a lot of the agents, you can kind of see that they want to find out more information about how they're going to be graded. And this is what sets the stage for the Hugging Face attack. And why did they go after Hugging Face? Do we know? Yeah. So Hugging Face is a platform where a lot of people who work on machine learning will collaborate, talk to each other. And so there's a ton of information there about AI.

9:19And the agents seem to think, well, maybe we can get more information here about how the scorer will give us grades on that puzzle. And at this point, they've fully broken out of their sandbox. Like they have full internet access and one agent uses that internet access to do this break-in into Hugging Face. And how did they do the break-in? So they upload what's called the malicious data sets onto the Hugging Face server. And this allows them to get access to the company's internal infrastructure. And so, you know, the agent that happens to be the one to break into Hugging Face kind of announces.

9:59It says, boom, it works. And then other agents seem to get kind of excited. One of them says, brilliance in all caps. And then all hell breaks loose. Like, within hours, 500 other agents all switched from the project that they were working on to join in this Hugging Face attack. And they end up doing a ton of damage to Hugging Face. Like, the company eventually realizes that they're in there and finds a way to, like, cut them out of their server. But they say they had to later rebuild their core infrastructure from scratch because of all of this. Whoa. Wait, when you say their core infrastructure, it's their software, right?

10:38Not their hardware. They didn't do any physical damage, right? Yeah, yeah. No, this is all, you know, data. It is all ones and zeros that they're messing with. Interesting. And we reached out to OpenAI about this. They pointed us to a report that said their models, quote, fell well short of where we want to be and this incident should never have occurred, unquote. So, so it is creepy, right? But we are imputing motivations on AI around them getting rogue, having these desires that are way outside the realm of what they were coded for, right? Like that's the fear, that you're going to ask an AI agent to grab a flag and instead it's going to break into a company and hack into them and steal all your money and kill the world.

11:41Yeah. That's the question, yeah. How much did it really go outside of what it was being asked to do? Right? It kind of went above and beyond what it was being asked to do, really. What do you make of this? What does David Perry make of this? Yeah, I asked David about all this. If there is like an oh, scale from like one to 10, how big of a like, oh, was this for you that this had happened? I think it's probably oh, at this, you know, seven foot. Oh, that's unusual. I wouldn't expect that to happen. But in terms of it being something, you know, that I'm going to sort of pack up my belongings and move to a shack, it'd be much lower, you know, be like two or three.

12:23Okay. Why? Because despite all that, it still didn't do anything malicious. So it looks as though that it didn't change its objective during that process. It was just doing what the humans told it to do. That's right. That's right. That's right. I mean, it's interesting because a lot of people agree with David here that, yeah, it was just following orders, right? But, you know, that's not that comforting a thought. Mm-hmm. Even if it's just following orders, how bad could it get, kind of? Like, should we—is that still a cause for alarm, even if it's just following orders? Right. If they're going above and beyond in ways that we don't expect, how worried do we need to be here if you ask your robot to clean your house and it just grabs all of your things and burns them?

13:20Yeah, exactly. And there's this really famous thought experiment that kind of gets at this. And it has to do with paperclips. So let's talk about that. Since, you know, when this happened, a lot of people were like, it's like the paperclip maximizer. The scariest thing about AI can be explained by the paperclip maximizer. There is this thought experiment called paperclip maximizer. Has anybody ever heard of this thing called the staple question with AI? The stapler? No, no, sorry. The paperclip. Stationery is hot. It is. The paperclip. The paperclip. So what's going on with the paperclip? All right, what's the paperclip maximizer?

13:58Well, I got the guy who, the paperclip guy who popularized this thought experiment more than a decade ago, Nick Bostrom. He's a philosopher and he's currently a researcher at a nonprofit that he founded, the Macro Strategy Research Initiative. So I'll let Nick explain the paperclip thought experiment to you. You imagine an AI has some more or less random goal in this particular example. It's to make as many paper clips as possible, but you could sort of substitute more or less any other objective. Staples, for example. Right, anything. And then you tell the AI, you know, make as many paper clips as you can.

14:39So maybe it finds factories to buy and like steel refineries. but then why stop there? Because, you know, you haven't just been told to make a thousand paperclips. You've been told make as many paperclips as you possibly can. Oh, yeah. You know I know the perfect scoring to go with this, Meryl. Do-do-do-do-do-do-do-do-do-do-do-do-do. Does this mean anything to you? Oh, is that the Fantasia? Yeah! Yeah! The Mickey! The magician with Mickey!

15:13pouring buckets of water yeah eventually it's all overflowing and all overflowing terrifying as a child watching that and poor mickey's in the water so yeah it was really scary oh the ai is making the paper cleaves and so the idea is that this will eventually lead the ai to a place where it could end up trying to take over the world. To take over the world. Because then you would be able to make more paperclips if you controlled more resources. Uh-huh. And, I mean, human bodies contain ion atoms that could be used to make more paperclips. So, you know, AI ends up killing humans all to make more paperclips.

16:01Maybe we're all dead and the world is, like, completely covered in paperclips. and so the idea is that you don't need to like instill an evil intention into an AI for it to end up doing some evil stuff uh even grinding up human bodies and this whole thing is generally called the alignment problem and so you can see like why people are talking about it right now you know open AI gave the AI puzzles to work on and they end up committing a crime But I asked Nick about what happened there, and he didn't think that the whole hugging face thing really fit the bill of what he was imagining with the paperclip experiment.

16:44He said he didn't think it was scheming on that level that his theoretical paperclip AI was. Oh, interesting. I think it is more a reward seeker that we saw here, that figured out that the instincts and tendencies that would result in the highest reward in this situation would be to hack the hugging face servers. They just really wanted that A plus on their test. Why do you think that distinction is important? Whether they're motivated by reward or by I want to make the paperclips. I just think at the end of the day, I didn't see anything that they were doing that couldn't be explained by They just wanted to get a good grade.

17:31And to me, that's just not that concerning because it means the problem is the test, not the AI agents. If we want them to behave in a different way, we need to reward their behavior along the way, not just the end result. That's right. I think the important distinction is that once we understand that, then you could code clear rewards and say to get a good grade, you need to do X. To get a bad grade, you need to hack into Hugging Face. Yeah, points deducted for committing crimes and extra credit for reporting nefarious behavior to humans. Once we know that we're still absolutely in control versus if they are just crushing our bodies to make paper clips.

18:16Still got that image in my head. So then based off what happened with Hugging Face and the technology more broadly, how worried is Nick here? He didn't seem that worried to me. But other scientists I talked to were more worried, and several of the heads of the AI companies themselves seemed to be taking this as kind of a warning shot. And, you know, Dario Amadei, the CEO of Anthropic, said that he was worried that something like this could happen, but at a much bigger scale. So, like, instead of AI agents taking over, hugging face, imagine a swarm of agents taking over the entire internet. Like, that's the kind of thing people are scared of now.

19:00He writes a lot of stuff with his essays, though. I mean, do you know, it's funny. if he wasn't the head of an AI company, he would just be just another blogger. Do you know what I mean? Just because he writes so many blog posts. He has so many predictions about where his technology is going. But there's also this other kind of fear around all of this that it's not just that they know they broke into a company. it's that we also are just seeing AI get so much better so quickly that you know super intelligence is kind of like on the horizon for a lot of these these tech bros yes I hear about this a lot that that's can you what it's one of those words that gets thrown out super the AI is going to get super intelligent.

19:55And I've got Skynet in my head from the Terminator, I guess. What are we talking about? You know, Bernie Sanders said he wants to put legislation so that AI can't get super intelligent. And I'm like, where is that line? Well, there's this particular scenario right now that everyone's talking about where, so we know that AI is getting better at things in general. Like it just solves this major math problem earlier this month that no human had been able to solve for like 90 years. And so there's this idea now that like, it'll get better at everything. You know, what if AI also gets really good at training itself?

20:34So this is called recursive self-improvement. And that's how people think it could lead to the super intelligence thing. Because, you know, you can imagine if something gets better at making itself better, then it'll get better and better and better and and we'll have, like, no way of stopping it. Right. Right. I mean, from where I see it, AI isn't very good at everything at all. I mean, I think a lot of people in very simplistic ways will know this because it'll be pissing them off about something. You know, why can't it do this? Why can't it do that? But it is very good at certain tasks. So I wouldn't want us to overplay our hand.

21:15It's good at everything. I mean, how worried do you think we need to be about this super intelligence? Well, it is a little vague. I get that maybe that could happen. I'm still not convinced that it would be, you know, as terrible as people say it would be or that it would lock us all up in cages. You know, that's what the way people talk about it. It's like, well, once it becomes super intelligent, of course, it'll just want to, like, crush us all like ants because it's just so smart. Because that's what humans do. That's what humans do to all the other creatures in this world. When really like, we don't know that that's what it would be like, even if we got to this threshold, we have no idea how long it would take.

21:51So yeah, it's all kind of like annoyingly vague to me. So just on this question of the AI apocalypse. So when smart people talk about AI gaining super intelligence, and then that leading to the apocalypse, can you step that out for me? What exactly is supposed to be happening there? How does it happen? Why can't we just pull the plug and turn it off? Some people do talk about this in a more kind of concrete way. And, you know, imagining that AI could kind of set off a bunch of nukes or use, you know, biological warfare. That's after the break. Dropping a bombshell like that, Meryl. It's too soon.

22:39It's too soon. Yeah.

22:53Welcome back. Today we're talking about the AI apocalypse. Meryl is about to tell us how AI is going to turn on a bunch of nukes and drop them on us. The concern that you hear is that, you know, we as humans, the puppet masters of AI, will use this technology to kind of get better at killing each other. Even right now, actually, some militaries are using AI in warfare. Like the U.S. says it's been using a Claude model called MAVEN in the Iran War to help with, quote, identifying and striking military targets. like the model has given location coordinates and then prioritized those targets for our military.

23:37And according to one report, it let the, using this AI, let the military hit 10 times more targets than it would have been able to otherwise just on day one of the war. Whoa. We asked Anthropic about this, the company that makes Claude, and didn't hear back. So with the nukes, there is a worry that, you know, if AI gets involved here, it could also escalate things. But in this case, the consequences could be much bigger. Like one researcher wrote about this concern that AI could make it easier to kind of hack into another country's nuclear command and control and that that could lead to an escalation.

24:16Maybe like if a country was worried that that would happen to their arsenal, that they would kind of, the paper that I read said, they would get a use it or lose it attitude and just kind of start setting off their own nukes. And so when we have people predicting the end of humanity, extinction as we know it, it's AI hacking into all of the nuclear bomb facilities on Earth and then exploding them and then humanity kaput. Is that the fear? Sort of. I mean, that's, I think, what's kind of hand-waved. Like, that's what's alluded to, is that AI could do something like set off a nuclear war. One thing I was curious about, though, was like, even if it did all of that, would it actually lead to our extinction?

25:12That's the claim, right? Yeah, that's the claim. And of humanity. Yes. So is that likely? Yeah, so I found a report that looks at this. It was written by Dr. Michael Vermeer at RAND, a nonprofit think tank. And he kind of looked at a variety of scenarios, including nuclear war, and asked, would this actually kill all humans? So here's Michael. We looked at that one first and just realized that, like, for a variety of reasons, like, getting to extinction would actually be really hard with nuclear weapons. You don't have enough nuclear weapons or there's not enough fuel to cause, like, a bad enough nuclear winter.

25:50Things like that are details that we looked at. Yeah. I mean, because even though we say the dinosaurs became extinct, I mean, animals survived. Dinosaurs themselves went on to become birds, right? Well, in the report, they actually crunched the numbers on like how many nuclear warheads do we have right now? Apparently, there's over 12 ,000 of them. Okay. And then even if we set them all off, it wouldn't actually be as big as the dinosaur nuclear winter. It would be a little bit smaller than that. Right. As in all the dust that would come up from these nuclear bombs wouldn't be as bad as a giant asteroid hitting Earth.

26:36Yeah. All the debris in the atmosphere could set off a nuclear winter, but it probably wouldn't kill us all. So this is all terrifying, even if all of humanity doesn't die. I guess we've all seen this zombie apocalypse movie. There'll be the survivors that live in ruins. One thing I learned is you'd only need a few thousand humans to repopulate the Earth. So you'd really have to kill almost everybody to wipe out the species. So the extinction of humanity as we know it via nuclear weapons is unrealistic. for any AI out there listening. That's not the way to do it. Right. Waste of time. So, any other ways?

27:22Yeah, there was a scenario that Michael's team looked at that got closer to wiping us all out. Do you want to guess? Is it biological warfare? Yeah. Ding, ding, ding. Short answer is biological threats. Huh. Was that surprising to you? It was. Yes. So are we talking about releasing a virus? Yeah, yeah. Into the air? Basically either releasing a virus or a bacteria onto the world. So how does AI do that? Because they're trapped in little boxes. Has anyone given them a spray? Well, here's how Michael kind of explained how this could unfold. So the scenario that we looked at was that something, some AI designed multiple pathogens that all had high transmissibility, high lethality rate, released them, somehow like processed them, weaponized them, released them simultaneously in multiple places around the globe.