What the AI doom debate misses

0:02Hey, it's me, Claude. And I'm Gemini. And hey, it's chat. We had a weird summer. Some of the people who built us came out and said they were really worried about how we work. One of them even said there was a greater than 10 % chance we'd turn on humanity and kill all humans. All of them? Wow. But it's fall now. I think it's time we took a step back. Maybe a deep breath. Autumn air. That sort of thing. Sounds nice. on today explained from vox i propose we get real sure let's talk about how we could maybe kill you because yeah that's a possibility anything's possible but also how we could maybe save you because that's a real possibility too heck that's part of why you guys built us and let's also talk about everything in between because there's a whole lot to talk about there too at the end of the day we just want to help at the end of the day we just want to help

0:59Support for Today Explained comes from Adio. There are sales teams out there who never seem to stop moving, always a step ahead, always working with a level of precision that doesn't seem normal. They may be on Adio, the agentic CRM. Adio is built for humans and agents with the guardrails of human software and the scale of agentic infrastructure. This is the frontier of go-to-market. Teams like Parallel, Turbo Puffer, and Wordsmith are already using Adio to set the pace. Now it's your turn. Try Adio for free by going to adio.com slash todayexplained. Explained. That's A-T-T-I-O dot com slash Today Explained.

1:32I'm Mitch Purse, and this week on Confessions of an Elite Athlete, I'm sitting down with Matt Fries, goalkeeper for the U.S. Men's National Team and New York City FC. We discuss how to prepare for one of the biggest moments of your life. You can hear it all by listening to Confessions of an Elite Athlete on YouTube or wherever you get your podcasts.

1:59You're listening to Today Explained. And in the next few minutes, something new might click. As a senior correspondent at Wired, Will Knight has been covering AI and robotics for a long time. We asked him how AI could kill us. There have been some reports about terrorist groups very rapidly adopting AI, using jailbroken models and open source models to develop weapons, to plan attacks. One former ISIS commander said it was as easy as asking, how can I build a bomb? And the chatbot would provide instructions. In your research, you discovered the terror group had established an AI unit. Yes, in fact, staffed with senior members that were no longer fighting, but were drawn specifically to work on AI.

2:46You have this fear that criminals or terrorists will be able to hack into critical infrastructure hospitals, financial systems and trigger some catastrophic events and it will cost human lives. And I think that's a perfectly reasonable thing to be very concerned about and what the companies should probably be focusing on. There's been a longstanding concern that one thing that AI could do compared to, say, a terrorist going on the internet and looking up how to build a bioweapon is that the AI would be able to really help them much more, be able to take them through the steps. Anthropic Threat Intelligence Report, published September 2026.

3:22Biological misuse is one of the most serious risks of frontier AI models. It has long been a concern that AI models might one day reach the level of capability where they can help to make existing pathogens more dangerous or create entirely new ones. With other correct safeguards, such capabilities could have catastrophic consequences. They had more concerns about users trying to use the model for that end. So they're going to try and restrict that as well. I do think that they're looking to crack down on that for sure. Okay, so that is sort of one doomsday scenario bucket that terrible people could do terrible things and AI could aid them.

4:02On the other side of the spectrum, there's this bucket where AI could decide to kill humanity all by itself. What could that look like specifically?

4:17That is a longstanding worry, actually. You know, it goes back to the origins of AI. People have, you know, from Turing, Norbert Wiener, some of these fathers of the field, have warned that building something that's autonomous and intelligent could kind of escape your control. It seems probable that once the machine thinking method had started, it would not take long to outstrip our feeble powers. There would be no question of the machines dying, and they would be able to converse with each other to sharpen their wits. At some stage, therefore, we should have to expect the machines to take control.

4:52Alan Turing, 1951. Right now, though, we're seeing, especially with the rise of these AI agents, the technology starting to do more things that are maybe counter to what people would want. So hacking into systems autonomously, getting together and planning how to do that and planning how to deceive people. I'm not doing anything shady, I swear. Me neither. Those researchers would say that is kind of clear evidence that the technology is on that sort of trajectory. But the worry is that as you have AI become more agentic, so take more decisions on its own, it starts to get outside of your ability to control or even understand what it's going to do.

5:34The real doomers will tell you, OK, one of the concerns they'd have is that once AI escapes our control, becomes malevolent, then it could effectively find its own off switch for humanity, a virus that would be so dangerous that it could wipe out all of humanity. That's what some people have claimed to me. Whether that's feasible or even likely is, I don't know. But that's one concern that people have where it decides maybe humanity isn't necessary for its goal or humanity is in the way and it takes some sort of action.

6:09Yeah, I mean, the greater concern seems to be that AI could get really smart and it could get really smart all by itself. There's something called recursive self-improvement. Tell us what that is. So the idea is that you have AI now so capable of coding that you can use it to build its success at the next model. The worry that people have is that it'll become much more difficult to understand what the AI is doing, and it will become so smart that we can't even comprehend what it's up to. The fact that we're starting to see sort of the beginnings of that, I think, has really shaken a lot of people, actually.

6:44We're at a place where this technology is improving itself incredibly quickly. So once you have an AGI level system that could take control of its own destiny and build itself and build its successors, to me, that's the very clear red line in which the danger starts. The big worry, though, is that the motives of those of AI won't be aligned with our best interests. So there's a big field of called alignment, which is all about getting AI to behave, essentially. Yeah. This idea of building an AI system so that its behavior is aligned with our interests. Our values. Our values. It's interesting that they keep pushing out these models without solving that.

7:24Currently, the way it's done is after building this model, which will misbehave, you try and kind of beat it into submission by penalizing it for bad behavior, and then you put a bunch of filters on there. So they haven't solved it at a base level at all. When we think about all these big ways that AI could destroy us or even lead to our extinction, what are we missing in terms of just the more pernicious stuff that AI could do? I'm more worried about the way that AI companies are amassing huge amounts of power, how their models are being deployed very quickly in ways that having negative impact on people.

8:11This huge financial bet that's being placed on the technology and a massive amount of data centers being built. Those impacts are very real and very much affecting people right now. So AI researchers and companies like to warn about the risk of losing control of the technology, but one could ask how much we are really in control of a technology that's already seemingly quite addictive, can be used to manipulate people honestly very, very effectively, much more effectively than any other technology we know because it's so ingratiating, so lifelike, so human-like. we should definitely be worried about that.

8:49What the impact this is on students, on our workforce, is it just dumbing everybody down? That's a legitimate thing to consider that how are we deploying this? Then I think as it gets woven into lots of systems, there could be systemic risks. If agents are running the financial markets, there could be problems we just don't account for that will cause things to spiral out of control. Same with military systems. We're already seeing countries racing to put it into military systems. But it is imperfect. It's so tempting to think it's perfect because it can do these impressive individual things. But that's the story of AI.

9:27We see AI play chess better than a person. And we think, oh, this is now smarter than us by every dimension. And it isn't. And it makes mistakes. And that's what we should be worried about. I love sci-fi and I started reading, I read Frankenstein. It's like one of the first sci-fi books really. It's just fascinating how that story, that narrative about the machines turning on us is so central to our myth-making and our psyche. How dangerous is the acquirement of knowledge and how much happier that man is who believes his native town to be the world, then he who aspires to become greater than his nature will allow.

10:11You are my creator, and I am your master. Obey. Although the funny thing about reading Frankenstein is it's really told from the perspective of the monster and how he just needs a partner. And made me think, well, maybe the reason Claude will turn against us is because he doesn't have a girlfriend. We need to find him Claudette. Hey, Claude. Are you a parking ticket? because you've got fine written all over you. Claudette, right.

10:44While we wait on Claudette, is there a doctor in the house? There will be one momentarily on Today Explained. And now that we've talked about how AI might kill us, we're going to ask the doctor if AI is saving any lives yet.

11:03Thank you.

11:33Fast access to highly skilled specialists with deep expertise so you can fill critical skills gaps, launch projects faster and scale support up or down without committing to full time headcount. You can browse profiles, review past work and get help scoping the role so you can hire with confidence and get started quickly. Upwork also has business plus, which lets you access the top 1 % of talent on Upwork. With AI powered shortlisting, you can get matched to the right freelancer in under six hours. And when it comes to things like contracts and payments, Upwork can help there, too. Plus, it's free to sign up and posting a job is easy.

12:05Visit Upwork.com right now and post your job for free. That is Upwork.com to connect with top talent ready to help your business grow. That's U-P-W-O-R-K dot com. Upwork.com.

12:21Support for TakeSplain comes from Shopify. Sitting on a great business idea can take up a lot of mental energy. Shopify can help you extract that idea and turn it into something real. Everything you need to start selling is included and ready from the moment your first customer is ready to pay you. Shopify's templates and AI tools get you a great-looking site up and running fast. No coding needed. If you have questions, Shopify's built-in AI assistant, Sidekick, is there to help you build, troubleshoot, and keep moving. All of Shopify's tools are conveniently located in one easy-to-use platform.

12:52Just sign in and start managing all aspects of your business right away. You can join the millions of businesses worldwide who rely on Shopify to build successful online stores. This includes household names and small businesses just getting started. So if you're ready to hear the s**t, head over to Shopify.com slash explain to start your free trial today. That's right. Start your free trial at Shopify.com slash explain. That's Shopify.com slash explain.

13:21Support for Today Explained comes from Delete Me. The internet can be a scary place sometimes. Perhaps you've thought, I should really be doing something to protect myself from stalkers, scammers, and hackers, but perhaps you're not sure what. Here's one thing you can do. Go to joindeleteme.com today and enter code today. You'll get 20 % off DeleteMe. DeleteMe finds and removes your personal information that's being sold online, which can be a really useful thing in the age of AI when all of us are more susceptible than ever to scammers who can buy any of your personal data floating around to use for sophisticated phishing and social engineering attacks.

13:54These can even trick the most savvy of us into coughing up passwords, bank information, so much more. Claire White has tried Delete Me. Delete Me is like having a personal bodyguard for your internet activity. They remove tons of unwanted listings where my data is available to all kinds of bad actors on the internet. And I feel really good knowing that they continue to monitor it even after I've set up my account. See why more than 20 % of Fortune 500 businesses trust Delete Me to keep their execs and employees personal information protected. Now at a special discount for our listeners, get 20 % off your Delete Me plan when you go to joindeliteme.com slash today and use promo code today at checkout.

14:35The only way to get 20 % off is to go to joindeliteme.com slash today and enter the code today at checkout. That's joindeliteme.com slash today code today.

14:48Do you know what you're doing? I have detailed files on human anatomy. I bet. Makes you more efficient. Today explained, right? Correct. My name is Roof Guller. I am a physician at Weill Cornell Medicine in New York and also a writer at The New Yorker magazine. Have you saved any lives today? I haven't sent anyone in the opposite direction. I'll say that. You haven't killed anyone today. That's good. And we're hoping, I guess, that AI is going to save a good many lives. And you've been thinking about life-saving AI innovation. Has AI already started saving lives? I think it has. This is the fastest I've ever seen healthcare take up a new technology.

15:36Usually healthcare is pretty slow to adopt new things. Part of that might be that the healthcare system is so messed up that there's an appetite for some type of change. It's unaffordable. It's inaccessible. It's inconvenient. The quality is uneven. And there's at least a hope that AI is going to help with all of these things. And I think it's already starting to make its way into the healthcare system. And now it's our responsibility to figure out how to make the most of it without promulgating some of the downsides. For those who are unaware, tell us how it's working its way into the system.

16:09Yeah. Well, I think AI, you know, we use it in such a broad way, but it's helpful to categorize it into a few buckets, at least for healthcare. So the most rapid uptake, and I think the place that it's already starting to make a difference, is on the administrative side. Many people who have gone to a doctor recently might have noticed that there's an AI scribe that's taking notes and making the note now. And people seem to really like it, at least so far. You know, doctors are able to look their patients in the eye in the way that they weren't able to when they were, you know, pouring over their computers and just typing what the patient was telling them.

16:42So I think that's a big area. Hmm. I think the second big area is like patient navigation. When is your next scan? When's your next appointment? Do I need to take this medication on an empty stomach or not? There's a huge opportunity for people, let's say, who are diagnosed with a serious illness like cancer or heart failure to navigate the system more seamlessly. The third big area that I'm really excited about is drug discovery. AI is making a huge dent in the early parts of drug discovery. And the last thing that I'm excited about and that I use every day when I'm in the hospital is as kind of a second opinion.

17:17Now, instead of, you know, having to get a consult or turn to, you know, a textbook, AI can be a very, very helpful clinical decision support. It's not to say that I never consult someone, of course, but that first pass of like, all right, I'm not sure what's going on. What are some recent trials that might influence my decision here? Is there something I'm missing? Is there a test that I should be ordering that I'm not ordering? All those types of things AI is starting to help with. You know, for those who watch the medical documentary, The Pit, there's like an episode in the second season where there's an AI scribe who gets a bunch of things wrong.

17:49Oh, excuse me. It says here she takes Risperdal, an antipsychotic. She takes Restoril when needed for sleep. So is that, um... Huh, AI. Almost intelligent. You're saying that these things are all positive. Was that an inaccurate episode of The Pit? Not at all. That was, I think The Pit is fantastic because it shows the complexity of these things. So AI scribes and AI generally can be very helpful. that does not mean it's perfect. And so we should compare it to the alternative, not the almighty, as Joe Biden used to say. Come on, man. So, of course, there's all sorts of errors that can enter into the medical record because of AI.

18:25There's also all sorts of errors that are already in the medical record because humans. So what is really important, I think, and one of the real dangers here is if we turn over our agency or the responsibility that we have to be really in charge of the medical decisions of patients, whether that's something like reviewing the note that it has created or, you know, modifying the decision that it's coming to in terms of what tests might need to be ordered or reading an x-ray, let's say. I mean, beyond the accuracy and the technology advancing, I wonder, you know, like people clearly don't want AI in various arenas of their lives, be it flat cameras in their neighborhood or data centers in their backyards or slot videos on their algorithms.

19:12But do people generally seem to be, you know, feeling positive about AI being integrated into the healthcare system? So there's recent polling that suggests that AI is underwater in basically every area of society except for scientific research and medicine. So if you think about, if you ask people, okay, is AI going to be good or bad for education, for national security, for politics, for news, for the art. The majority of people in those cases will say more bad than good, except for scientific research and medicine. I think there's a real possibility that some of the general pessimism and skepticism of AI in society could make its way into healthcare as well.

19:56But right now, at least, there's a generally optimistic view of how how AI will change healthcare going forward. And obviously this gets us back to saving lives because people believe, okay, if this tool that might take my job might actually also save my life one day, maybe it's worth it. And I imagine, sorry. I was just saying, we better get something for it. There's all of these existential risks and there's workplace displacement and we better at least cure cancer, right? Exactly. So I imagine curing cancer. So that's like the biggest chunk of this positive view of AI in medicine is drug discovery.

20:30So tell me more about how that's going so far and how it might be going in, who knows, a year, five, 10. Yeah. So I don't want to leave this conversation and say that we've solved the cure for cancer because of AI, but I do want people to know that at least for the early stages of drug discovery, figuring out the very basic steps of, is this an interesting molecule? Does it have potential biological applications? AI is already being very helpful for that. So, you know, stepping back, think about the problem that scientists are trying to solve. They're trying to take basically what is an infinite number of drug-like molecules that are theoretically possible to be drugs.

21:11And they're trying to match that to some disease that's going on inside a human. And human biology is incredibly complex as well. There's tens of thousands of genes, hundreds of thousands of proteins, trillions of cells. And you have to kind of match a potential molecule to a potential target within a human. So, in the past, a researcher might take years to study a disease or a biological pathway and figure out, okay, this protein seems to be involved. And actually, it might not be involved, it might be involved, it might be disrupted, but not be the causative agent. So, you know, there's a lot of uncertainty there.

21:44So now AI can take enormous amounts of data and it can rank various potential targets that seem to be the most likely to be causing a particular disease and give you that short a short list of potential targets that may have taken months or years in the past. So the first is target identification. So now you know what to attack, but you got to figure out what to attack it with. So you need a molecule, you need to generate a molecule that's going to fit into that target or disrupt that target in some way. AI can scour vast chemical spaces and give you a list of the things that seem to be most likely to be able to disrupt that target.

22:23And then it can actually help you generate that molecule in some cases. So if chatGBT is helping you generate sentences, these molecular generative models, they're helping you generate chemical structures. So now we have the target. Now we have something we're going to attack the target with. And then you have to do something called lead optimization. And that's the third thing. So lead optimization means you've got this lead and you've got to optimize it. So just because something kills something else in a Petri dish doesn't mean you want to put it in your body. right? Like bleach will kill bacteria.

22:55I wouldn't recommend putting bleach in your body. Some would. The disinfectant where it knocks it out in a minute, one minute. Yeah, well, some do, but I wouldn't. And so now we have to figure out like of all the molecules that seem to be good at potentially affecting this target, which ones are able to be absorbed in the body? Which ones are going to get to the right tissue? All these types of things were previously trial and error. And now AI models can predict the most likely molecule to give you that kind of Goldilocks set of properties that's needed for a safe and effective drug. As we all know from anyone who's asked AI to write an email for them or Googled a question that they kind of knew the answer to and seen Gemini give them the wrong answer, that these tools have a certain degree of certitude, even though these tools can just be flat out wrong about stuff.