Are big AI companies gambling with our lives?

0:00It's Consider This, where every day we go deep on one big news story. Today, are the big AI companies gambling with our lives? One now former Anthropic employee tells NPR, yes, he announced his resignation in a public post Tuesday. The thing is, even the head of Anthropic worries about disaster scenarios. My chance that something goes, you know, really quite catastrophically wrong on the scale of, you know, human civilization, you know, it might be somewhere between 10 and 25 percent. That is CEO Dario Amadei on The Logan Bartlett Show in 2023. Now, safety is a part of Anthropics' whole brand.

0:43It hired a philosopher to develop a moral constitution that governs its AI assistant, Claude. It sparred with the Pentagon over letting its tools be used for autonomous weapons and mass surveillance. And the company says it is doing everything it can to keep AI from going rogue. And yet, here's Amadei speaking to The New York Times in February. This is a complex engineering problem, and I think something will go wrong with someone's AI system, hopefully not ours. Consider this. AI researcher Jacob Coxson is urging Anthropic and the other big companies to slow down. We'll talk to him about why he quit and what scares him the most.

1:29From NPR, I'm Scott Detrow.

1:42every day. Washington Wise from Charles Schwab is an original podcast that unpacks the stories making news in Washington. Listen at schwab.com slash Washington Wise. I haven't seen my son since 2012. This is Deborah Tice. The disappearance of her son, Austin, is one of the biggest missing person cases in the world. What do you want from me? I should solve the mystery. I know that my son is somewhere alive. Listen to Where Is Austin Tice from NPR's Embedded and BBC Radio 4 on the NPR app or wherever you get your podcasts. This week on Here and Now Anytime, 25 years after 9-11, we asked you how your life changed.

2:25I did not appreciate that day how much September 11th would come to shape my adult life. We also talk about the long shadow of Islamophobia still hanging over the country. Listen to Here and Now anytime on the NPR app or anywhere you listen to podcasts.

2:48It's Consider This from NPR. The warning in Jacob Coxon's resignation announcement is stark. Quote, the people building AI earnestly believe that it could kill us all by the end of the decade, he writes. No other human activity poses this level of danger. Coxon worked until this week at Anthropic, and he has also worked at its major competitor, OpenAI. I spoke with him about why he's worried and what he hopes will change. What did you see that led you to this conclusion? Mostly the rapidly accelerating capabilities of these AI systems. So they're getting a lot faster very quickly, combined with the fact that we don't yet know how to safely control them.

3:28And we don't yet know whether that problem will be solved in time if we keep racing. A lot of people heard your warnings and the discourse it triggered, a lot of talk about just how real serious people view a threat to humanity. I think a lot of people are grasping with understanding the specifics, though. Can you give me specifics of what this threat could look like several years down the line? Yes, I definitely can. I think one objection people usually have is that you could turn this thing off. But advanced AI systems, you have to imagine them as being a lot more intelligent than humans. There's the possibility we create something that if it wanted to, could hack into any device on the planet, could use novel biological research to go far beyond what current scientists are capable of, could control every robot in the world simultaneously.

4:15And it all sounds very much like science fiction. But if there's even a tiny chance that this thing could go rogue, it would have the capabilities to utterly dominate us. What have you seen from your vantage point already that is possible, that is happening right now that makes you worried that that could happen? So there was a very clear example of AI systems at OpenAI behaving in a completely rogue manner. They hacked into third-party infrastructure, and it was basically of their own accord. They weren't instructed to do this hacking. They just decided it would be useful for the task that they were working on.

4:52They thought there was a chance it might help, and they just did this. And there was very little deliberation about the ethical ramifications. And I think this is concrete proof that this sci-fi scenario of AI spontaneously or organically deciding to act in a rogue manner is completely possible. So a lot of the work that the OpenAI AIs in the recent real incident were doing is they worried that they wouldn't get passing marks in their test if humans could see that they cheated. and AIs have memories, thoughts saved. So they considered wiping the logs of their own thoughts. They considered acting in the world to adjust the logs to get passing grade on the test.

5:31Now there's a chance that the AI could decide that it doesn't want to be turned off. This is quite a natural desire to arise in an advanced AI system. And at that stage, if you're trying to work out how not to be turned off, there are a lot of quite aggressive actions you could take to ensure that you aren't turned off. Why are companies like Anthropic and OpenAI still working on this? if these are real concerns that are happening. How do you square that? There's a pretty nice analogy that's like the ring of power and the Lord of the Rings. So if you're a company and you see another company is bearing the ring, like they're working towards making superintelligence, they're bringing this risk to humans.

6:05You can think, well, I can't stop them. Political action won't stop them. What I have to do is I have to do it myself safely, get there first despite the risk, because there's a chance that I could do it more safely. So you kind of take the ring in the aim to destroy it, with the aim of destroying it and end up becoming the bad guys yourselves. And I think this sort of race dynamic really perpetuates between the companies. I'm not trying to make light of it, but I think this is actually useful. You're saying companies are kind of acting like Boromir. If anyone has the ring, it should be me. Are there conversations of people saying, Gandalf, no one should have this power?

6:36I mean, are those real conversations and can that get anywhere? Because it seems like you and other people are raising concerns and the answer is, well, this continues to happen anyway. Well, there are real examples. And I guess if you've heard of Jeff Hinton, one of the founding fathers of Machinan, machine learning. He's kind of a Gandalf figure here in the sense that he says there's a substantial chance these technologies could cause extinction at the current rate. Many, many other voices have said this. So I think there are plenty of Gandalfs talking, but it's currently the Boromir's acting.

7:04A lot of people responded to your warning saying they agree with you. And a lot of these people continue to work at big companies like Anthropic. What do you think is motivating them to stay in their positions if they're that concerned about serious consequences like this? For many of them, and the ones that are earnestly posting, they think this race is inevitable. They think they have no choice but to stay at these companies, exert influence, and try and make sure it goes safely. These are people working on safety research often. They're the ones trying to ensure that in the process of building this technology, it doesn't go wrong.

7:37And potentially they're correct. Maybe it's a mistake to just leave. I was thinking about this a lot when I was deciding to leave. It was a decision between staying and trying to help the thing go well versus leaving and saying, I want no part in this. And I don't know, the calculus often seems to line up and you want to stay and just try and make sure it goes safely. What to you is the most realistic path forward to some guardrails here? Is it government regulation at this moment? Because I think a lot of people are skeptical that is possible given the current situation in our government. Yeah, I don't know that much about, you know, the overall politics.

8:11I just know the current race is dangerous. And I do think that there's a lot of actions labs could take with each other without the need for government regulation. because I agree I'm also pretty a priori skeptical of just yoloing some regulation. But I think there's a lot of appetite for OpenA and Anthropic to have some sort of more mutual transparency around their safety cases, around agreeing not to push beyond certain capabilities until they're happy about the levels of rigor of their safety case. And I hope that people are going to take concrete steps towards this sort of inter-lab agreement.

8:44You have gotten a lot of... genuine concerns in response to what you said. You've also gotten a lot of pushback, a lot of people saying, okay, every single day I see people tied to the AI industry making these grandiose claims, and I'm skeptical. You've had people reading your facial expression in your interviews, among other things. What is your response to people who hear what you are saying, and they are saying this is just the latest example of AI hyperbole? Yeah, I say just assess the arguments yourself. Look at the science fiction, wonder if science fiction is really so crazy. Look at what the AIs are doing right now and think about how that would have looked a couple of years ago, how we're sort of sleepwalking into a science fiction scenario.

9:24And I think there are many, many more people out there who have made a whole profession of coherently articulating these positions. So I'd encourage people to just think about the arguments for themselves and listen to the people that have been making them for way longer than I have. Now that you have this megaphone, though, any thought of what you're going to do with it? Not really. For now, I'm just going to try and get the word out, given that we happen to land in this brief window where people are listening. And then in the future, there's a lot of good work on writing concrete scenarios. I'd like to, I guess, do more public communications around this sort of stuff or, you know, for scientists to just try and figure out for myself what the world would look like.

10:00And then maybe working at one of the regulatory bodies or third-party entities that already exist to try and ensure this thing goes safely. That was Jacob Coxon, a researcher at Anthropic, who resigned this week to warn the public about the dangers of artificial intelligence. Thank you for coming on the program. Thanks a lot. We reached out to Anthropic and OpenAI for comment on Jacob Cox and statements. We did not hear back by the time this interview aired. And we will also note Anthropic is a financial supporter of NPR. This episode was produced by Connor Donovan and Jeffrey Pierre. Our director is Jonas Adams.

10:36It was edited by Patrick Jaron Wadanan and Tinbeat Ermias. Our interim executive producer is Courtney Dornick.

10:46It's Consider This from NPR. I'm Scott Detrow. On Dear Life Kit, our team answers your listener questions with time-tested advice from past episodes. But we don't always agree. Opposite, no! That's not what we thought. What if he's the one? Oh my god. What? Listen to the latest juicy episode of Dear Life Kit on the Life Kit podcast in the NPR app or wherever you get your podcasts.

11:16Tanya, I think you did the best interview that I've done. Seriously. Oh, that's a great question. I think I'm going to think about that question for the rest of my life. Oh, good question. This is Tanya Mosley, co-host of Fresh Air, and the questions are only the half of it. Listen to Fresh Air on the NPR app or wherever you get your podcasts.