They warned about AI before it was cool. They're still worried

0:00AI used to be a thing of science fiction. I know I've made some very poor decisions recently.

0:09But I can give you my complete assurance that my work will be back to normal. And the genre is full of superhuman AI machines that become so smart, they turn against the humans that created them. Skynet begins to learn at a geometric rate. It becomes self-aware at 2.14 a.m. Eastern Time, August 29th. In a panic, they tried to pull the plug. Skynet fights back. Yes. That's an AI that could get out of control, but if you really think about it, it's much worse than that. Much worse than Terminator? Much, much worse. That's Keith Rotersheimer talking to NPR's Martin Costi back in 2011. He was a research fellow for what was then called the Singularity Institute for Artificial Intelligence.

0:53It's now the Machine Intelligence Research Institute, or MIRI. At the time, Rotersheimer was looking into the idea of a computer that was not only smart, but capable of improving itself. It's able to look at its own source code and say, ah, if I change this, I'm going to get smarter. And then by getting smarter, it sees new insights into how to get smarter. And then by having those insights into how to get smarter, it might have its source code and get smarter and get some insights. And that creates an extraordinarily intelligent thing. They called this the singularity because that intelligence could grow so fast, our human minds might not be able to keep up.

1:28In 2011, that still seemed like a long, long way off. But in 2025, artificial intelligence is seeping into everyday life with chat, GPT, and the like. Even proponents of AI, like developer Jonathan Liu, joke about the estimated probability of AI doom. What's my P-doom? I would say around 50%. And yet you're smiling about it? I'm smiling about it because there's nothing we can do about it. Consider this. While the AI boom continues and companies across the country are heavily investing in the technology, some researchers are begging humanity to pump the brakes.

2:10From NPR, I'm Juana Summers.

2:17Hey, a quick request before we rejoin today's episode. We have heard from listeners who say Consider This has become part of their daily routine, a way to make sense of things. If that is true for you too, take a couple minutes and leave us a review. It is a small thing, but it really does help people find this show. Thank you so much.

2:41It's Consider This from NPR. A superhuman artificial intelligence so smart, it can decide to get rid of slower-witted humans is a pretty terrifying concept. What was once strictly the stuff of science fiction is now closer than ever to being a reality. And if it does, some AI researchers have gloomy predictions about humanity's chances of survival. NPR's Martin Costi caught up with these so-called AI doomers. This is the main event of the evening. Welcome to a demo night in downtown San Francisco. Competitive events like this are a big part of the AI boom in this town right now. A chance for new developers to show off their new AI apps and maybe attract investors.

3:30Yeah, my name is Jonathan Liu and I'm the founder of Cupidly, which is an AI agent that swipes for you on Hinge. You describe your ideal mate to the Cupidly AI and it goes into the dating app for you to find a match. Or it did. Lou has now shut it down because app users were getting banned by the Hinge dating app. But Lou is typical of this crowd in his bullishness about AI and the prospect of AI eventually becoming as smart or even smarter than humans. I think once we do get superintelligence, hopefully we'll live in a utopia where nobody has to actually work ever again. But almost in the same breath, Lou also says that he sees a possibility that superhuman AI could end up killing off all of humanity.

4:15And he's not kidding. Just ask him for his P-doom. It's a term he'll recognize because it's sort of joking AI slang for estimated probability of AI doom. What's my P-doom? I would say around 50%. And yet you're smiling about it? I'm smiling about it because there's nothing we can do about it. This strange mix of optimism and fatalism has long been a part of the AI world. Even the CEOs of OpenAI and Anthropic, two of the most important AI companies, signed a public statement a couple of years ago that acknowledged the, quote, risk of extinction from AI. And the reason for this is a pretty straightforward, logical problem.

4:54If they were to build something that's smarter than us, how would they keep it on our side?

5:05That problem is called alignment, as in how to align AI with human values. And here in Berkeley, near the UC campus, there's now a cluster of people working on that problem and related AI questions. Nate Soares is president of MIRI, that's the Machine Intelligence Research Institute. That's the newer name for an AI alignment organization that NPR first visited back in 2011. I spent quite a number of years, maybe about 10 years, trying to figure out how to make AI go well. And for a bunch of reasons, that's been going poorly. Soares has now given up on trying to figure out that alignment riddle.

5:43He says the machine learning revolution of the last few years, which created ChatGPT and the like, is now moving things too fast towards superhuman AI. And he gets little comfort from the fact that this also means there are now many more researchers here who are focused on AI safety. Yeah, I mean, for one thing, I would not call it AI safety. I would say, you know, safety is for seatbelts, and if you're in a car sort of careening towards a cliff edge, you wouldn't say, hey, let's talk about car safety here. You would say, let's stop going over the cliff edge. That cliff, as Soris puts it, is a scenario in which AI gets more closely involved in helping to improve AI, accelerating a kind of feedback loop of self-improving artificial intelligence that ends up leaving humans behind as uncomprehending spectators, and then perhaps just obstacles to be swept aside.

6:33And that's why Soares and another Miri colleague have given up on alignment and are instead going the last-ditch route of publishing a book that begs humanity to slam on the brakes. The title of the book is If Anyone Builds It, Everyone Dies. Let that sink in and look around you. Does this all go away, really, in a few years? I mean, I can't tell you when, but it could be a couple years, could be a dozen years. But yeah, this around us is what's at stake. It's an extreme vision. Some critics say it's overblown that the current AI training methods can't even achieve human-level intelligence, let alone superintelligence.

7:15Others say the doomers are unwittingly hyping AI. One writer in The Atlantic ridiculed Sores and his co-author as Useful Idiots, whose doom saying makes AI look more powerful than it really is. And it's also just a lot to ask to get a booming new tech sector to restrain itself, maybe with government intervention. In D.C. right now, the conversation on AI is still very, very early. Mark Beal is president of government affairs for the AI Policy Network, a lobbying organization. There does seem to be at least an appetite to start measuring the risks and start to examine more carefully, you know, what threshold, what alarm bill might need to go off that would change that assumption about whether or not we ought to consider something as drastic as a pause.

8:02Government restrictions seem unlikely to Jim Miller. He's an economist at Smith College who's focused on the game theory aspect of AI development. He sees this as quickly turning into a race. If I am Elon Musk, I can say, you know what, I don't know if racing is superintelligent is going to kill everyone or not. But if it is going to kill everyone and I don't do it, someone else will. And if I end up killing everyone, I've maybe taken off a couple of weeks because OpenAI would have done it a week later. And then Trump and Vance can say, yeah, maybe this will kill everyone, but if we don't do it, China will.

8:32And for Miller, this isn't just an academic question. In his own life, he's decided to put off a risky surgery to correct a potentially fatal condition in his brain because he's an AI doomer and he's convinced that a superhuman AI is likely to end human civilization in the next few years. Or, if he's very lucky, that superhuman AI will spare us and then offer him a safer treatment.

9:02On the campus of UC Berkeley, the generation with the most at stake are setting up the information tables for their student clubs. At the table for the club devoted to AI safety, Adi Mehta says he has heard the Doomer argument, but he's focused on AI's more immediate risks. One thing that is more apparent for college students is that I can't remember the last time I did an assignment without using AI. It's automating a lot of our thinking away, which personally that's like a pretty big fear. Another club member, Natalia Trounce, says she's also just not that focused on doom. I think many things are possible, but it seems like it's not the most likely scenario at this stage.

9:44If I were to ask the average Berkeley student, is this, like, my life's going to be over in three years, just have fun now, or is it just... I feel like if it was, like, three years it's going to be over, like, it would have happened already. Walking back to the offices of MIRI, Nate Soares admits that with AI already such a normal part of life here, it's hard to convince people that we're about to go over that cliff. He says one hope is that maybe the rise of superhuman AI will be just gradual enough to be noticed and give people some time to react. Maybe it doesn't take a ton. Maybe the AI is doing a little better, getting a little smarter, getting a little bit more competent, getting a little bit more reliable.

10:24Maybe that'll make people a lot more spooked. I don't know. And maybe, just maybe, he and his fellow AI doomers are wrong about the danger. He says he would love to be wrong, but he doubts he is. Martin Costi, NPR News, Berkeley, California. For a more detailed look at AI and its risks, listen to our NPR Explains podcast, only available on the NPR app.

10:53This episode was produced by Mallory Yu with audio engineering by Ted Meebane. It was edited by Gigi Duban and Courtney Dorney. Our executive producer is Sammy Yenigan.

11:11It's Consider This from NPR. I'm Juana Summers.