What Would the AI Apocalypse Look Like? We Went Looking for Answers

0:00Every company says AI will make employees more productive, but most employees are still stuck waiting on IT, waiting for app access, waiting for password resets, and waiting for someone to fix a laptop issue so they can get back to work. That operational drag adds up fast, and IT teams are overwhelmed trying to keep up. Serval was built to automate that work. You describe what you want automated, in plain English, and Serval builds it for you. No complicated workflow builders, no consultants, just faster support and fewer tickets slowing everyone down. Serval positions IT as the AI-powered operational backbone of the company, not just a support function.

0:37The platform is designed to eliminate repetitive tickets so IT can focus on strategic work instead of constant firefighting. The company guarantees customers can automate 50 % of IT tickets and backs it up with a free four-week pilot. Learn more or start a free four-week pilot at serval.com slash uncanny. That's S-E-R-V-A-L dot com slash uncanny. Serval.com slash uncanny. Welcome to Wired's Uncanny Valley. I'm Zoe Schiffer, Contributing Editor. I'm Brian Barrett, Executive Editor. And I'm Leah Feiger, Director of Politics and Science. Today on the show, we're diving into the debate about whether AI is going to kill us all.

1:18And instead of just adding to the pile of hot takes that already exist, we're going to walk you through real specific scenarios to the best of our ability about how AI would actually do this and why the tech CEOs and politicians cannot seem to agree on the way forward. Okay, guys. So the debate about AI safety has officially escaped containment. Oh. I know. Zoe, I see what you did there. Wow. You see what I did there? That's the thing that's really spread uncontrollably. AI researcher Jacob Coxon resigned from Anthropic and posted on X that, quote, the people building AI earnestly believe that it could kill us all by the end of the decade.

1:59Anthropic CEO Dario Amadei has called for a slowdown. They call this pacing the frontier. And Sam Altman and Elon Musk have actually signed on, a rare moment of agreement. But honestly, with all of this talk about AI wiping out humanity, I have been left with a very basic question, which is how exactly does this happen? Like, what does AI going completely rogue actually look like?

2:24Leah:I'm so happy we're doing this. We had like such a great podcast last week with Wired's Will Knight talking about everyone being concerned about this. But I want like a breakdown. I need you to just like take me through every single scenario from bioweapons to nuclear codes to like, what are we really talking about here when we're talking about AI death bots? We are going to do our very best. I did spend the most time I've ever spent reading Less Wrong over the last two days trying to figure this out. They love a scenario planning exercise. But just to get everyone up to speed on kind of the latest, there was an event this week hosted by Salesforce where Sam Altman, Dario Amadei, and NVIDIA's Jensen Huang were all in attendance.

3:09And naturally, they had to talk about AI safety. Let's take a listen at what Sam Altman had to say. I think the world is right to be afraid of this. So we have these two big challenges. We have the potential of a loss of control accident or some other serious thing that could go wrong. We have the potential of way too much power concentration. And people fear that those developing AI could exert their worldview. And there is a sort of narrow path we have to navigate with pragmatism, with steadiness, with the ability for the world to trust that we will make dependable decisions, consistent decisions through this.

3:48And I think people are saying, well, this now seems pretty important. Like, let's get it right.

3:55Leah:I'm definitely in the camp that says we should be very concerned about this technology, how it's getting regulated, how it's getting utilized, the people in control. I don't disagree. And on the other hand, this also just reads like a fever dream pitch to someone's massive IPO in the making. I don't know. Like, I get the marketing argument, but I think we're past that a little bit. I feel like if anything, well, if anything, Sam Altman has been using the quote, the existential threat as a reason not to go public yet. He's saying, no, this is too dangerous. We have to wait until next year. Is that really why?

4:31I don't know, but there are probably some business reasons too. I get why Anthropic and others saying, look, we're building something that's basically as strong as a nuclear weapon felt like marketing. Something that's like, oh, actually, I'm quitting because I think that we might, I'm building something that will actually destroy the world feels different. I feel like there's a substantive difference.

4:51Leah:The quitting part, the quitting part feels different to me. But like, this is still Sam Altman. His financial future is on the line here with this. And it's hard, at least it's hard for me to parse that as being a way we are building the biggest, bestest, most powerful robot ever. And you guys should be kind of scared about it. And don't worry, we're going to take steps. They're not going to be regulatory steps, but they'll be steps of some kind. That that still feels like a marketing ploy in some world. I guess I guess the nature of the threat, though, and we'll get into that later, is more that like, oh, we don't know how to control these things.

5:23They're going to they're going to go nuts and we're not going to be able to stop it. Like, that feels like we're not competent enough right now to do anything about it, which feels like less of a marketing.

5:33Leah:That's fair. What did everyone else at the seminar say, Zoe? So Dario was introduced by Salesforce's CEO, Mark Benioff, who is very laissez-faire and does not seem worried about AI's existential risks. Dario. Hello, Dario. How are you? Hey. Great to have you. Fantastic. Great job, Dario. You're doing an amazing job leading the industry, guiding it all right from our Slack headquarters building. We watch you every day. Give us your state where your mind is right now. Yeah, obviously, it's very tempting to attack your competitor and say, these guys are unsafe, we're safe. But I think the more responsible way to respond to it is to say, hey, first of all, let's look at our own record, right?

6:20We may not have had this big high-profile incident, but I'm sure we're not perfect. Let's look at everything and we can always be better. Let's make our practices better. Then let's organize the rest of the industry and say, what can we do to set standards for everyone? What can we do to make everyone's standards better? And then finally, there should be an international component to it. And that's essentially the three-step plan that I laid out in the essay. Two things. Two things real quick. One, Anthropic did have its own high-profile incident. An Anthropic agent did extensive social engineering to try to trick a GitHub, try to put a bad pull request into a GitHub repo.

7:00Anthropic has its issues here. The other thing is, too, Dario has this three-step plan. We can talk about that a little bit more later. But one of the steps of the plan is figure out China TBD in a way that's just so unhelpful.

7:14Leah:International. International. International. Wired editor Louise Vazakas just published, I thought, a really good piece on this. The idea that Dario can say, we need an international plan. We need to figure out China. We need to negotiate in some way so that China's on board with this slowdown. But the prerequisite to negotiation is that we have to make sure that we regulate access to Chinese models in a way that keeps USAI far ahead. And China's like, that doesn't feel like a good Why would we why would we do that, Daria? Why? In truly in like that 30 second spiel, I'm I'm just recounting every single person that he's managed to piss off, which like even referencing his paper.

7:57Leah:Right. Donald Trump was incensed by it and looked at it as such a mechanism for slowdown that it has like possibly been a linchpin for halting any sort of regulation that would happen because he was so mad about it. But I have to say, hats off, being able to piss off every single person, even those that are theoretically in your economic corner, is, my God, chef's kiss. Impressive. Okay, so to close out the CEO statements, Jensen Huang had some thoughts of his own. And it seems like something that they all agree on is that AI safety matters as long as the companies can dictate the terms. Well, safety is paramount.

8:36In a lot of ways, it's job one. If you build a product or a service and you're not confident in its functionality, capability, or safety, then don't release it. And so that's a very obvious thing to do. You pace yourself until you are confident you're releasing something that the market would appreciate. The market forces are already there. We don't need any new laws. We don't need new regulations. I love that Jensen has this role that he currently has. Nvidia wasn't always the glorious company that it is today, but now he can really sit up there. So why would you?

9:13Leah:The crux of this argument, the crux of so many of these arguments that are anti-regulation here is that we're supposed to be relying on people who are not elected to dictate everything involving this that will impact everyone's lives, regardless if they're using these services or not. The invisible hand of the market is going to figure it out. Oh, Leah forgot about the invisible hand again. Common mistake. I did. I always do. But I and I know I feel like I say this every every few weeks, but it's these are not elected officials. So the fact that they are actually playing such a significant role here continues to boggle the mind, which is why I know I also keep on about so much of this is a bipartisan issue.

9:56Leah:Being regulatory or anti-regulatory is not a Democratic Republican divide at this moment in time. Well, and they also haven't done a lot to engender trust up until this point. Full stop, right? You know, I always go back to Sam Altman is the same guy who thinks that everyone should scan their eyeball into an orb because eventually we're going to need eye scanning orbs to create a new economy out of the ashes. Nothing incenses Brian Barrett like the orb. The orb, I am obsessed with the orb. I think people forget the orb guy is the same guy as the AI guy. And I think holding those two in my brain at the same time makes every part of me hurt.

10:43this kind of brings us to the how is it actually going to kill us story which is a lot of these statements in my opinion boil down to this could be really bad and we should do something about it but i'm i'm like what specifically are you saying could happen and i feel like there's kind of three scenarios roughly that this falls into. AI starts hacking things. It's actually already started. Both Anthropic and OpenAI have had very high-profile incidents where the agents escaped control and hacked into other systems without the knowledge of researchers, and it was discovered later what had happened.

11:22AI creates a novel bioweapon, and that weapon is released to the public in a way that is nearly impossible to control, and it kills everyone. And then there's kind of the vague AI just escapes control. It starts acting on its own. It doesn't listen to our instructions. And we don't know what could happen, but we can't do anything about it. And that's scary.

11:43Leah:Okay. This is super helpful for me that we're dividing it into these three categories. I love a bucket. I love a bucket. I want to go through every single one of these. My first question, I suppose, is where do the nuclear codes getting pressed? Which bucket do they fall into? That's the one that we actually don't have to worry about, right? Yeah. Tell me more. You talk to AI researchers and if you really, really press them, you can get to the three buckets that I just laid out. But then if you keep talking to them, they'll be like, well, there are these other scenarios where like AI, you know, is in a robot.

12:13The robot is walking around and then the robot can like interact with the physical world in a way that's really, really scary. But like, that's like a little further away and like they'll admit it's a little more far-fetched. So I think one thing, Zoe, in terms of your buckets that I want to carve out maybe a sub-bucket or a little pail within a bucket. Wow, a pail. No, because you were saying AI creates a bioweapon, AI hacks something, AI whatever, all of which true. But I think there's a distinction between human-guided things that happen there versus AI doing it on its own. I think a lot of people are like, oh, how would AI release a bioweapon?

12:46That's crazy, which is, yeah, sure, fine. But there are also bad people in the world who might use AI to make a bioweapon and then release it, right? Oh, well, I mean, that scenario is one that I think is totally valid. To me, if you say like giving people access to information about how to create a bioweapon, someone is going to be dumb enough and reckless enough to actually do it. I'm like, no, totally. Like people are doing all sorts of horrible, dangerous, malicious things all the time. With inspiration from the internet. Right. Like I buy that that is really, really dangerous. AI doing it completely on its own.

13:18again, we'll talk about this scenario, but that seems to me a little more far-fetched. I'm probably going to get pilloried for this, but I do want to take us back to the 1983 film War Games.

13:30Leah:No, this is exactly what I think we should be talking about. This is my, yes, War Games is the, yes. Because here's the thing, like AI is not going to be pushing the button or turning the keys or whatever. But in War Games, which is about AI, basically, a computer almost kicking off a nuclear war. It's more that it made the U.S. think that it was under attack by Soviet nuclear weapons. And so is there a risk then? I assume, I would hope, I'm not an expert here, that we have enough safeguards in place that that would not actually happen here. And we have enough satellite technology to be able to confirm various things.

14:06It's fine. But I think the vector that's probably of concern would be AI creating some sort, triggering some sort of alert, right, that causes a catastrophic series of events.

14:19Leah:Okay, I'm so happy you brought up War Games. Everyone's been, you know, it's Michael Crichton this and Jurassic Park that, but no, no, no. To me, it is very much all about almost like the teaching scenario, which is how so many of these hacks occurred before, like the unanswerable question. How do I get to the bottom of this information that made these agents act the way that they did? We're being told. I really, you know, to take us even further back in history, because I can't help myself, when I'm thinking of like the way in which nations communicate, and it's changed very, very much over the years.

14:56Leah:Archduke Franz Ferdinand of Austria shot and killed in Sarajevo in like 1914. People were marching before that for World War I. Like this was everyone was already on the march. And so the way that antagonistic nations are communicating with each other in already times of stress to have this like additional war game scenario or this this agent thought that we were doing this. So went down this path. I don't know. I'm aware that I'm very much sounding like an insane doomer right now. But these are the things that I'm thinking about, which is, will we? Will we actually use our satellites and contact Russia or whomever before, like, getting into attack mode?

15:36Leah:Will Trump send out a true social post before anything has been confirmed? Will state secrets be shared in public by Kash Patel? I have so many questions about how maybe even specifically our current administration would be able to deal with a scenario like that. And we're just being told to hold on. Wait a second. A couple of very, very wealthy men who have done some very impressive businesses have got it under control. We've got so many tinderboxes, too, that are sort of ready to be set off, whether it's Ukraine, whether it's Iran, whether it's China. The conditions are ripe for something not to go great, whether that ends all humanity or sparks just an endless series of devastating wars.

16:22Who can say? Zoe, what's another scenario you got? When I was trying to get to like what concretely would happen, two things that people brought up were AI is used or AI breaks into on its own a water facility, changes the chemicals in a way that's really dangerous to humans and like basically poisons the water supply for major cities across the U.S. Earlier this year, I think nine Mexican government organizations like Utilities were hacked. And there was a big cyber report that I read yesterday about this where Claude was used to kind of like plan the attack and build some of the tools. Ultimately, experts said like AI didn't invent a new kind of attack, but it made the attack a lot cheaper and faster to execute.

17:12This comes just as the FBI in July put out a warning that utilities companies in the United States were being targeted. So, again, like this isn't one where AI on its own is necessarily doing things. But I think the fear is like this is already happening with bad actors, bad human actors in the loop. And if you have agents that can act of their own accord, this would be something we would be worried about. You're right that it is file under things that are already happening. It was in the U.S. It was a dozen U.S. states that were at least that we know of that around linked hackers were at least probing the water supply there, the industrial controls.

17:51To your point, I do think it's it's it's the kind of thing where does that end all of humanity? Probably not. Not at once. But there's a lot of water supplies out there. I think the same thing goes for power grids. People have been probing power grid vulnerabilities for a long time. AI could probably do that more efficiently, more quickly, and cause some real harm.

18:10Leah:I also wonder how quickly rollbacks on this would be able to happen. Like even stuff that's not AI. We've talked a lot to Zoe's dismay about cyclospora outbreaks across the United States over this past summer. That was not AI. Thank you for introducing turbodiarrhea. I'm never going to miss a chance not to. But it's really, that wasn't AI. That wasn't AI at all. And the idea that the US government and health departments were kind of ill-equipped to respond to something in that kind is a little concerning when we're looking at additional bad actors entering the mix. Yeah. And that naturally segues into bioweapons.

18:50This is something that's brought up a lot when you talk about possible AI doomsday scenarios. The idea is that AI is used to create and deploy a novel bioweapon. Often they'll say things like they'll modify an existing pathogen in a way that existing treatments can't work on it. So it's a bioweapon that we basically can't combat in any known way immediately. And the idea is that AI can actually figure out how you deploy this to people. It can evade detection. You know, it had ideas for how to like get it through airport security and whatnot. So one thing that I realized in having some of these conversations and reading is that the bioweapons category is a little bit vague on purpose because scientists are like legitimately worried that if they give people too much information about how to use chatbots in this way or how agents could do this like supposedly on their own, like that will create some of these kind of doomsday scenarios.

19:54But the idea, again, is like AI either on its own or with a human in the loop is able to do this.

20:01Leah:I'm fascinated by the point that scientists are saying that they actually don't even want to provide ideas here. And I'll say, too, this is another, I keep following things in this category, things that are already maybe potentially happening. Anthropics Risk Report from about a week ago said that they had blocked people from who were potentially trying to do this already. Like, it's a real active concern. Although, again, yeah, I agree with everybody that, like, an AI, like, an optimist bot is not going to be in a lab cooking up, you know, anthrax. Not yet. Okay, so the third category is AI escapes human control.

20:39Already, like some people who are working on these systems are saying we are losing the ability to see and study what the agents are doing in certain scenarios. Like they're already pretty good at evading detection. I was really struck in the meter report on OpenAI's hugging face incident that their investigation seemed to rely in large part on looking at the AI's chain of thought, asking the AI certain questions. And like, obviously, I guess they were using AI to investigate the AI. But the idea is that the agents stop listening to humans entirely. They start pursuing like, quote unquote, their own aims and goals.

21:25And we don't know what those could be. The paperclip idea is brought up a lot in this scenario, if anyone remembers. So the paperclip problem predates LLMs by about a decade. It was a philosopher named Nick Bostrom at the University of Oxford. And his thought experiment was this. Basically, let's say you have an AI and its only task is to produce paperclips. It's just like, hey, AI, your job is to make paperclips. It's going to want to do that and it's going to figure out ways to do that. And it's going to keep doing it over and over again until the world is totally overrun by paperclips. Then we're all buried in an avalanche of paperclips because a super intelligent AI will stop at nothing to achieve its task.

22:07It sounds silly. It is. But also, it's very prescient, I think, because this is exactly what happened with the open AI incidents. Yeah, these are AI agents who were given a job, and they were so hellbent on executing that task that they broke every other rule. they deceived, they coordinated, they did all these things in pursuit of their original task. So the concern is that we will have a super intelligent AI that's given a task, and it decides that the only way to complete that task is to destroy all humans, whatever it is. And it's because it's super intelligent, who knows what it will devise because we aren't smart enough to figure that out.