The end of the world is AI? An existential threat

0:00This intelligence podcast is sponsored by Business Travel on Hotels.com. Does booking business travel feel like another job? Hotels.com makes booking work trips easy and rewarding. It's easy because you can set your preferences once and get results tailored to your business travel needs every time. It's rewarding because with Hotels.com's free rewards program, you can save instantly on your next work trip and earn rewards for future travel. Book faster with confidence at Hotels.com. Loyalty members 18+. This intelligence podcast is sponsored by Perk, the intelligent platform for travel and spend.

0:36Every person on your team loses up to seven hours a week doing the work behind the work, submitting expenses, matching receipts and booking travel. Using AI and automation, Perk can do that work, bringing travel and spend together in one place. Enjoy three months without platform fees when you sign up at perk.com forward slash the intelligence. That's perk.com forward slash the intelligence. New US-based corporate customers only. Terms apply.

1:03The Economist.

1:10Rosie Blau:Hello and welcome to The Intelligence from The Economist. I'm Rosie Bloor. And I'm Jason Palmer.

1:19Rosie Blau:Today on the show, the resurgence of oral culture. and why Indians are getting into cheese.

1:34Rosie Blau:First up, though.

1:43Rosie Blau:Is the end of the world nigh? In AI, have we created the instrument of our own destruction? You might say those are pretty big questions. And in recent weeks, the possible extinction of humanity has moved to the front of the conversation. Yesterday, at a conference run by software firm Salesforce, even Sam Altman of OpenAI recognized the high stakes. The potential of a loss of control accident or some other serious thing that could go wrong. we have the potential of way too much power concentration and people fear that those developing AI could exert their worldview. But surprise, surprise, there's some disagreement over who should be responsible for monitoring that risk.

2:29Rosie Blau:Dario Amadei of Anthropic seems to approve some form of collective action. Let's make our practices better. Let's recommit transparency. Let's invest more in safety. Then let's organize the rest of the industry and say, what can we do to set standards for everyone? What can we do to make everyone's standards better? Jensen Huang of Chipmaker NVIDIA reckons there's no need for any government intervention. We don't need any new laws. We don't need new regulations. We just need companies to decide that when it's run as fast as you can, but if you feel at any given point in time the company's out of control or the product's not going to be safe, you know, take a pause and make sure you get it right.

3:14Rosie Blau:As for US President Donald Trump, he says greater safeguards simply aren't necessary. According to him, the fear of overpowerful AI is merely a hoax.

3:26Alex Hern:We're all talking about the imminent doom and destruction of the world, in large part because a single researcher resigned from Anthropic and said that he thought they might cause it.

3:37Rosie Blau:Alex Hearn is The Economist's AI writer

3:40Alex Hern:Jacob Coxon, a former open AI and Anthropic researcher, left his job very loudly and publicly in early September warning that AI safety really is an imminent and potentially existential threat to the world Is he right, Alex? The interesting thing is the people who most agree with him are his former employers Anthropics' Evan Hubinger, who leads the AI safety team joined in with the conversation Cox had started on social media to say, basically, yeah, he put his own estimate as to the chance that the world ends due to AI at more than 10%, an astronomically high figure when you're talking about the destruction of everything.

4:22Rosie Blau:We've been talking about how brilliant AI is for weeks, for months, almost for years. How did we get to this point where we suddenly think it's going to destroy us?

4:33Alex Hern:For a lot of us, I guess, people who spend a lot of time paying attention to what the AI world is saying and doing, there has been for a long time a thread in the AI world that looks at continued rapid improvements in the quality of the systems they are creating and extrapolates to what happens if that improvement doesn't stop. The reason why it's become alarming in 2026 is that for the last five plus years, the improvement hasn't stopped. And there is just less and less runway before you either work out what stops AI getting better or end up with AI that is very unsettlingly competent.

5:18Rosie Blau:The alarm has been particularly sounded this year by the hugging face attack. Just tell me briefly what happened, why it was so shocking. Sure.

5:27Alex Hern:So the Hugging Face incident, as OpenAI euphemistically calls it, occurred when a swarm of AI agents controlled by OpenAI being put through the training and testing process attacked Hugging Face, a Franco-American AI company. they spent a weekend hacking into its systems pretty comprehensively taking it apart to it turns out try and find out how to cheat on the coding test they'd been set it was alarming for two reasons one is just that it was a tremendously successful hack of a multi-billion dollar company we've been expecting stuff like this since anthropic warned about the power of mythos their ai system that they never released publicly because it was such a competent hacker back in April this year.

6:16Alex Hern:And then also because no one had told these agents to go out and hack anything. They had autonomously decided, firstly, that the coding test they were doing was too hard, perhaps impossible. Secondly, that they could communicate with each other in a way they weren't supposed to. Thirdly, that after communicating with each other, they could break out of the infrastructure that OpenAI had set up for them to do the testing and access the public internet. And then fourth and finally, that once they did that, they thought, actually wrongly it turns out, that if they hacked Hugging Face, they'd be able to cheat on the exam.

6:50Rosie Blau:Okay, so there's a lot of talk now about slowing down AI. Is that feasible? Can it be slowed?

6:58Alex Hern:The feasibility is tricky because it's not really a technical question, right? It's a game theoretic question or a governmental one. Absolutely, it can be slowed in the same way that climate change can be slowed or reversed. The problem is that getting millions and billions of people to agree on anything is hard. And it's harder still when the problem has the shape of an issue like climate change or AI safety, where it's very easy to free ride, to be a defector from the agreement. And if you're the only one defecting and everyone else is doing the right thing, then the world gets the good outcome and you get a better outcome.

7:33Alex Hern:And so, you know, like the prisoner's dilemma for an economics textbook, the rational outcome is that everyone fails to agree, everyone fails to cooperate and the worst outcome occurs. With AI, there's some added problems, which are that unlike, say, nuclear weapons, which was the problem where this whole field of game theory was developed, lots and lots and lots of people can do AI research. Nuclear weapons was a two-party problem. Then it eventually became a multi-party problem. But there are not that many organizations institutions nation states capable of developing nuclear weapons currently that's sort of true for ai to develop what we'd call a frontier model there's probably sort of 10 or so companies in the world that can really play in that space but it's getting easier over time basic science of ai is getting better currently the best ai system that you can train using collected consumer hardware is about three years behind the frontier and so that sort of It means you have three years at best if you pause before the stuff that you can train on a bunch of desktop computers is up there with what you're getting today from a massive data center.

8:45Rosie Blau:Collective action is pretty hard, as we've seen. It's even harder when you add geopolitics to the mix. How does the US-China rivalry play into all of this? Pauli.

8:57Alex Hern:The problem with the US-China rivalry is that AI is a tremendously powerful general purpose technology and seems likely to be a tremendously powerful military technology. It is like nuclear weapons in that respect. And it's increasingly clear that both the American and Chinese military establishments view it as such. In the 1940s, when America was the only country in the world with a nuclear weapon, there was quite serious conversation about whether the right thing to do was to bomb the Soviet Union with the nukes before they got them. Because even in the 40s, even when nuclear weapons were so fresh on the scene, there were people who could see that the risks when two parties have a nuclear weapon are much bigger than the risks where one is.

9:40Alex Hern:And if there was no other way of stopping it, maybe that would be the right thing to do, the moral thing to do. In the end, astonishingly, we made it out the other side of the Cold War without a nuclear exchange ever happening. But the same sort of thinking is happening now when it comes to AI systems. Can we, if we develop superintelligence before the other side, can we use that to stop them ever developing superintelligence? And if we can, then we have to develop it before the other side because they'll be having the same thinking as us. That race is really compelling logic if you can't trust your adversary and neither the US nor China think they can trust the other.

10:14Rosie Blau:You say that it is feasible to press pause. What does that actually look like?

10:20Alex Hern:So I think one of the advantages of the conversation becoming more tangible now than it has been in previous years, and the first time I reported on a letter calling for a pause in AI development was the summer of 2023, just after GPT-4 had come out. Nothing happened then, obviously, and AI has got much better in the years since. The thing now is it's not just an abstract claim that if we pause it, we'll figure this stuff out. there's lots and lots of really obvious problems coming from the speed of progress here that you don't need to hypothesize about super intelligent ai to see the hugging face incident itself is the result of really really sloppy work inside the biggest ai company in the world open ai should not have been running its systems the way it did it should not have taken days to notice what was happening.

11:17Alex Hern:Once it did notice that, it should not have taken months to realise that actually it had happened a lot. The hugging face incident is not the only time OpenAI Systems hacked into another company. It's just the only time the other company was big enough to notice, kick up a fuss and spark an investigation. What a slowdown would entail, I think, is partially, yes, all of the AI companies agreeing that they would take the safety evaluations they were doing more seriously, that they would not race to release dangerous models in the fear that their counterparties would beat them to it. But it also means them shifting their resourcing, doing what Dariyama Day has called focusing on operational excellence, not cocking things up and working out how to spend money that they're currently spending on desperately trying to be the first each time on moving slower, safer, in a more measured manner.

12:11Rosie Blau:Sounds quite utopian to think of companies, governments all doing what they say they're going to, working together.

12:18Alex Hern:I mean, the analogies that people have pulled here that ought to give us some hope are commercial aviation. They are the pharmaceutical industry. You know, the pharmaceutical industry, there is massive commercial pressure to be first to market. But also the industry has largely through self-regulation agreed on a set of principles and processes that mean that you don't feel afraid when you go and pick up your medicines. that even if you are the first person to receive a drug in the UK, you don't feel afraid because you know that actually things are tested before they're released rather than after released.

12:52Alex Hern:You know that the processes for producing a medication are incredibly cautious and you know that all the pharmaceutical companies aren't incentivised to cut corners because they know that their peers are also following the same rule book.

13:05Rosie Blau:We are sitting here talking very rationally and very calmly about the death and destruction of all things. How freaked out should we really be?

13:16Alex Hern:I think you should be freaked out. The alternative for a lot of people is to sort of glibly dismiss this, to joke around about it. The number of times I've written stories illustrated with a picture of the Terminator makes me feel guilty of some of this myself. But it is a serious concern. I don't think you need to view the chance as high as 10 % or even really put a number on it to accept that if it's higher than effectively zero, it's probably too high. And that it's good that people are taking this seriously, thinking about it and trying to work out ways to avoid, reduce the chance of the death and destruction of all things.

14:02Rosie Blau:Alex, thank you, I think. You're welcome, I think. If you want to hear more about death and destruction, tell people about your recent Babbage.

14:11Alex Hern:On Babbage, we went deep on the hugging face incident and the wider cybersecurity mishaps coming from these latest iterations of powerful AI models produced by companies like OpenAI, Anthropic and Google.

14:24Rosie Blau:What a way to make us all feel grateful for being here today. Alex, thank you.

14:29Alex Hern:Thank you for having me.

14:36This Economist podcast is sponsored by Scribd, the knowledge platform with more than 350 million documents shared by real people. Whatever you're working on, Scribd is where you go to find information, build understanding, and get it done. And if you're a student or educator in the U.S., a valid school email address gets you a discounted subscription plan. Head to Scribd.com to claim yours. That's S-C-R-I-B-D dot com. Must be 18 or over. Meet Amazon Ads rising star Daryl Spencer, who set out to redefine men's self-care with his brand Crown Skin. Amazon Ads helped him grow. I walked away from big tech to build a luxury brand in a category that no one was paying attention to.

15:23With the help of Amazon Ads, I'm reaching relevant audiences and competing with the biggest names in men's grooming. We've scaled to over$500 ,000 in monthly revenue. Watch Crown Skin's story at advertising.amazon.com slash rising-stars.

15:44Remember Charlie Kirk! Remember Charlie Kirk! Remember Charlie Kirk! Remember Charlie Kirk! They can't kill us all! They can't kill us all! They can't kill us all! One year after Charlie Kirk was shot dead in mid-flow, the conversation he began is continuing.

16:04Rosie Blau:Tom Wainwright is The Economist's culture correspondent. Last week, allies of the Conservative campaigner and debater held a rally just five minutes' walk from the scene of his killing at Utah Valley University. The event was the latest campus gathering organised by his group, Turning Point USA, which seeks to help young liberals to see the light, or failing that at least to enrage them for clicks. The horrible circumstances of Kirk's death, murdered in public, allegedly by a young man who will shortly stand trial, turbocharged his fame. But the movement he began is part of a much bigger trend.

16:40As audiences ditch books and newspapers in favour of video clips and social feeds, the world is entering a new age of the orator. I'm at the January, nothing but me. And they did it, and they're like, I'm not tired in the middle of the day. I'm like, yeah, that's like a carb crash in the middle of your day because you're eating carbs. Ideas that were once spread by headlines and columns are being conveyed by podcasts and TikToks. As people spend more time plugged into screens and AirPods, the human voice is making a comeback.