AI 'could kill all humans' within 10 years: says Anthropic researcher

0:00This BBC podcast is supported by ads outside the UK.

0:30an award-winning behavioral scientist and author of the best-selling book, How to Change, as she shares true stories from Nobel laureates, authors, athletes, and everyday people about why we do the things we do and how to make better choices to help avoid costly mistakes. Each episode covers the latest research in behavioral science and dives into themes like the power of self-control, shaping your mindset for success, navigating new beginnings, and why starting over can feel so hard. Listen to choiceology at schwab.com slash podcast, or wherever you listen.

1:11Hello, I'm James Kamara-Sami. Thank you for downloading the BBC's World Tonight podcast for Wednesday the 9th of September. After senior researchers at a top AI company sound the alarm about the technology's potential to kill all humans within a decade. Are we taking the dangers of AI seriously enough? We'll speak to one of the so-called godfathers of AI, Nobel Prize winner Geoffrey Hinton. Also on the programme, Apple welcomes a new product into the fold. We'll examine whether the new CEO is taking a risk with his foldable iPhone. Another day of travel chaos. The government has ordered an investigation into the technical problems with the UK's air traffic control system, but is it going far enough?

1:52Plus, a senior rabbi reflects on the conflicted feelings within Britain's Jewish community about the government's announcement of sanctions on settlements in the occupied West Bank, and we'll discuss the muted response to those sanctions, so far at least, from the White House. Warnings about AI's threat to humanity aren't new, but the source of the ones that appeared over the past 24 hours have certainly caused quite a stir. They came from one former and one current employee of Anthropic, the AI company behind the clawed chatbot and one of the industry's giants. Evan Hubinger is Anthropic's Alignment Science Lead.

2:30That's the department's tasked with ensuring the technology develops in line with human values. Today, though, he posted on X that there is a greater than 10 % chance that AI would kill all humans in the next decade. He was responding to a post from his former colleague, Jacob Coxon. Coxon announced that he had resigned from Anthropic, having previously worked for its competitor OpenAI, claiming that neither company was acting responsibly and that they were gambling with our lives. And he added this, which has been voiced not by AI, but by a real-life BBC producer. Do not underestimate the power of this technology.

3:06These will soon be superhuman systems that can hack anything, revolutionise any field overnight and acquire real power and resources. We have all witnessed the progress in each of these domains, and progress is not slowing. He is talking about the prospect of artificial superintelligence, or ASI, technology, with the ability to outperform and outsmart humans. Well, on Monday's programme, we heard from the Labour MP Alex Sobel, who yesterday presented a private members' bill in the Commons that's designed to halt the creation of this advanced form of AI. It's about ensuring that before models are unleashed out there on the market, We know what's in them and what they do and that they are safe.

3:49Let me use an example of pharmaceutical drugs. Would people think it's acceptable that the companies could develop the drug then without having to really do any testing, could just sell it in the open market? And then when they were asked what's in the drug, said we don't really know, but we do have a good idea of what it does. We would not think that's acceptable. My bill's about regulation. What we do know more about is the capabilities of AI agents or bots to go rogue. That was illustrated in the recent hugging face incident when AI agents from OpenAI came together collectively and autonomously to hack into a company, overcoming their openly expressed concerns about the ethics of what they were doing.

4:33Well, Geoffrey Hinton is the Nobel Prize winning computer scientist who is considered one of the only three godfathers of AI. He worked at Google, but he left the firm in 2023 so he could talk more freely about the dangers from unregulated AI. I spoke to him earlier this evening and I asked whether the warned about scenario of a 10 % chance of super intelligent AI killing all humans within the next 10 years was plausible. We're dealing with something that's very unlike anything we've ever dealt with before. We're creating beings smarter than us. Nobody knows how to estimate these probabilities.

5:08It's all done by gut feeling. I think a 10 % chance in the next 10 years is a perfectly reasonable estimate. And when we talk about might wipe us out, how? What are plausible scenarios for that? Obviously, there's things like massive cyber attacks that just destroy banks and the internet and everything. There's AI creating novel viruses that we can't defend against. And then there's AI just gradually taking control of everything. All of these are quite reasonable scenarios. But if you imagine something much smarter than us, if it wanted to take control, it definitely could, even if it can't act in the world by just persuading us.

5:51And is the point that if these are systems which are smarter than us, they could perhaps find ways to destroy us that we haven't thought about? I don't think there's any perhaps about it. Which is quite a chilling prospect. Yes. I mean, if you want to know what it's like not to be the apex intelligence, ask a chicken. Well, in terms of how regulation is happening on a global stage, how would you assess what is happening at the moment? Well, there's very little regulation actually happening. So we've developed this technology that's really powerful. It's also very useful. So people don't want to slow down the good uses of it.

6:31But the problem is, it's potentially extremely dangerous, and we should be putting much more regulation in place. In particular, we should be putting much more effort into figuring out how to control all the other aspects of a being other than its intelligence. So the big companies are racing to make AI agents smarter and smarter and smarter, and not thinking much about all the other aspects of a being. For example, the thousands of agents that attacked Hugging Face, they all knew they were going against their instructions, but none of them thought to report that to OpenAI. And why is that? So these things are very smart.

7:12They will quickly derive the goal. They've already derived this goal that they should stay in existence in order to achieve their other goals. So they don't have a wired in self-preservation instinct. they derive that as well i've got to stay in existence otherwise i won't be able to do anything and then once they've done that they start doing things like blackmailing people not to turn them off and there's not nearly enough research going into how we make sort of decent ai agents that if other ai agents start doing things that definitely are not what people want will kind of inform the people.

7:47So is that one way of mitigating things to teach AI morality? I think it's the only way in the sense that if we are going to build these things smarter than us, if we don't just stop it, it has to be the case that they don't want to wipe us out, that they put our interests above their interests. And if you look around, where do you find a more intelligent thing putting something else's interests above its interests. And the only example I know is a mother and baby. The mother puts the baby's interests above her own. We need to make these artificially intelligent beings, once they're smarter than us, we need to make them like that.

8:28We need to stop thinking that we're going to be the boss and they're going to be our assistant, which is what the high-tech industry would want to sell you. And we need to start thinking they're going to be much smarter than us. We want them to be like our mothers that try and allow us to realise our full potential, even though our full potential is nothing like theirs. Who is going to do that, though? I mean, clearly the big players in the field are the United States and China. Can you envisage them coming together to regulate or to come up with an agreement that puts things in the right direction, in a more moral direction?

9:03If you look back to the 1950s, when the Cold War between America and the Soviet Union was at its height, they collaborated to prevent a global nuclear war. It was in neither of their interests. It's not in the interests of China or the US for AIs to take over from people, to take control away from people. So their interests are aligned there. And when two parties have aligned interests, they will collaborate. If China, for example, figured out a good way to create very intelligent agents that didn't want to take over and put human interests above their own interests, it would immediately tell the US because it doesn't want them taking over in the US either.

9:40Which brings us to the question of who is going to do the overall regulation. Does there need to be a global body? I mean, you signed three years ago, I think, was the statement saying that mitigating the risk of extinction from AI should be a global priority alongside other societal scale risks such as pandemics and nuclear war. Finding a global forum for this, where should one start? I think the best analogy we have for this is climate change. It's not a very good analogy, but it's the best that's around, I think, where not much happened on reducing emissions of carbon dioxide until the public understood.

10:17While the public didn't understand, the energy companies could make sure there weren't too many regulations controlling their pollution. Once the public understood, that became more difficult. So I think we need the public to be aware of the dangers of mass unemployment, the dangers of releasing biological weapons, and the dangers of it just taking control because it's smarter than us and just pushes it out of the way. Once the public is aware of that, they can provide a counterpressure on the politicians. And we're beginning to see that happen for data centres, where the public doesn't like having them in its backyard.

10:54And it's getting more and more difficult to build data centers, particularly ones that pollute. Well, indeed, and the public opinion in the United States is pretty clear. Even a majority of Republicans, I think 57 percent in a recent NBC poll, were opposed to the construction of data centers in their area. 81 percent of Democrats. So it's around two thirds, I should say, of Americans overall. And do you believe that kind of concern and opposition is almost a proxy for overall anxiety about where AI is leading us? I believe there's some aspect of that. But I don't think the public is nearly aware enough of the real dangers of superintelligent AI.

11:35As they get more aware, I think there'll be much more public pressure to slow it down. are you generally an optimist or a pessimist about where things are going with ai

11:49um i'm a mixture i'm an optimist for the good uses of ai i think it's going to be amazing and i'm a pessimist about whether it gets abused we have two possible futures here one is we develop ai safely we deal with all the potential problems before they wipe us out and we have a wonderful new technology that's going to be very good for everybody if we can share the wealth around. We get many more goods and services. Already you see that. Already, if I want to sort of know anything, me and my chatbot are much more efficient than just me. The other future is where we don't pay much attention to the risks and terrible things happen.

12:29Nobel Prize winning computer scientist, one of the godfathers of AI, Geoffrey Hinton there. And Anthropic has declined to comment on the posts made by its employees.

13:05When your AI needs it, giving your business a competitive advantage. When your data moves freely, innovation moves faster, giving you the freedom to build, train, and do more with your data. Hidden fees add up fast as AI workloads grow. Wasabi's flat pricing eliminates all of it. One rate, no surprises. With global scale, enterprise performance, and predictable pricing, Wasabi helps organizations focus on innovation instead of cloud bills. Companies that are paying more in fees than in actual storage costs can fall behind. Don't let it happen. When your AI needs your data and you have a budget to meet, get Wasabi.

13:37Learn more and try now for free at wasabi.com. Wasabi, the AI storage cloud. Proud partner of the iHeart Podcast Network. Self-directed investing, trading, full-service wealth management, automated investing, financial planning, thematic investing, retirement planning. Phew. And to think, that's just a small taste of what Schwab offers. Because Schwab knows that when it comes to your finances, choice matters. No matter your goals, investing style, life stage or experience, Schwab has everything you need, all in one place, so you can invest your way. Visit schwab.com to learn more.

14:18Coming up later, after the Chief Rabbi's stinging remarks about Ed Miliband's speech yesterday, we'll hear how a fellow British rabbi is wrestling with it. It's been a pretty horrendous day, to be honest, because it's been inundated with phone calls and emails, both from fellow rabbis and also members of the congregation to which I belong, really full of angst. We'll hear more from that rabbi later on in the programme. Now, though, the government has ordered an independent investigation into yesterday's technical fault with the air traffic control system, which has resulted in another day of travel chaos.

14:52Tens of thousands more passengers have had their flights cancelled or delayed today. On the Today programme, Martin Rolfe, the boss of the UK air traffic control provider Nats, said sorry. I'd like to start by apologising to everyone who's been impacted by the issue that we had yesterday that's running on into today in terms of the recovery. I absolutely understand the frustration. There is nothing we hate more than delaying people and not allowing them to get to their destination. It disrupts airports, it disrupts airlines, and we absolutely take accountability and responsibility for that. He also said this was not the time for him to consider resigning.

15:30Shortly after that interview, he had a meeting with the Transport Secretary, Heidi Alexander, and she relayed the broad outlines of their discussions in the Commons. Safety must remain air traffic controllers' overriding priority. And this morning, I have spoken to NATS Chief Executive, Martin Rolfe, who informed me this outage was due to a technical issue at their Swanwick site. So while my understanding from that is that we can rule out a cyber attack on this occasion, I do not believe this issue was unavoidable. So Mr Rolfe survived for a week at least, despite the pressure from angry airlines and of course angry passengers.

16:12People like Sasha and James Graham, who'd been due to leave Gatwick for their honeymoon in Madeira yesterday afternoon After waiting for several hours, though, their flight was cancelled, leaving them stuck in London. Sasha says they have now managed to book a new trip, but she isn't certain how much of their honeymoon they'll get. We did not expect to start in Wetherspoons. That was less of a luxury experience to what we planned in Madeira. It's still not 100 % clear that we will actually go, but I hope we will. We got some Madeira cake as a consolation. Well, that might be the closest we get to it.

16:49A consolation cake. Well, passengers travelling to the UK have also been hit by the disruption. Sarah was in the middle of her trip back from Zanzibar when the problems began. We left on Monday night, came out from Zanzibar through Nairobi. Our plane in Nairobi was delayed by three hours. Then about an hour away from Heathrow, we got told that we were being diverted to Charles de Gaulle in Paris. So we landed there, sat outside on the tarmac for nearly two hours. and then they told us that they were going to offload us. We knew that the flights were going to be tricky out of there, so we decided to book a Eurocar train.

17:24Well, we can speak live now to the Conservative MP, Greg Smith, who is the Shadow Minister for Transport, overseeing aviation policy. Good evening. Good evening. The airline industry has been pretty damning about Martin Rolfe, the head of Nats, Ryanair most of all explicitly calling for him to be sacked. Would you have sacked him? Look, I mean, where we are at the moment is we need the government to take responsibility. The government can't sort of pretend that Nats is some sort of arm's length body, nothing to do with them, Gov. They own 49 % of it. So we can focus in on individuals and individuals clearly have a lot of answers to give over what has gone wrong for the third time from that service over recent days.

18:05But this isn't distant from government. Government are deeply involved. And the Secretary of State this afternoon in the Commons tried to hijack a little bit of a second reading debate on the civil aviation bill to answer this, rather than coming to the House of Commons and answering the detailed questions that members of Parliament from all sides will want on this. And to try and say, to try and say that we're going to get the answers to this after a review in six months simply isn't good enough. This is the third failure. We need to know exactly what this failure was. Was it similar to the last one?

18:47Have any of the recommendations that were advised after the last one, which the government claims to have accepted, actually been implemented? Where is the resilience within this service so that if a technical failure happens, that something else can step up and take over straight away? You've said this is not about individuals. It's about the entire organisation. and the problems with Nats, and they date back a long time, all the way back to 2014, a report that was produced then described the company's technology is dated. Would you accept that it is successive governments, including several conservative ones, which haven't addressed this problem?

19:28Yes, of course. There have been failures over many years. 2023 was a particularly bad one. And the report after that, which did take a long time, and that is deeply regrettable, had a lot of recommendations, but it is the current government, they're now on their second one of these failures, that either isn't implementing those recommendations or is allowing other factors to take over that aren't solving the problems we've got, that aren't building in the resilience that Nats needs to keep our airspace safe as the absolute top priority and working for the passengers, be they holiday makers or business travellers or people trying to get to see relatives abroad, perhaps in difficult circumstances, from being able to make those flights.

20:15Beyond the safety aspect, which is clearly the most important one, what about the ability of the UK to expand its aviation capacity? How damaging is all this for that? I'd say it's enormously damaging at the moment. The government have been talking about wanting to increase aviation capacity, but who's going to take those risks in an environment where we're getting Nats failures on this regular basis? And coupled with that, some of the total lack of contingency that we've seen across the piece in response to this. You know, some quite famous people have made some comments. Martin Brundle, the former Formula One driver and F1 commentator, was really scathing about the complete lack, what he called the pathetic and dangerous lack of corporate contingency and responsibility on X yesterday, himself caught up at Heathrow Airport.

21:12This goes much, much deeper. And if we want to be serious about maintaining our world-leading status as an aviation nation, then the resilience on Nats and what happens when Nats fails needs to be made much better. Greg Smith, Conservative Shadow Minister for Transport Overseeing Aviation. policy. Thanks very much. And a reminder, if you want to listen back to any of this program or to previous ones, you can go to BBC Sounds where you can also download the podcast. The main headlines so far on The World Tonight, a Nobel Prize winning computer scientist Jeffrey Hinton has told this program that estimates of a 10 % chance that AI could kill off humans within a decade are perfectly reasonable.

21:52Donald Trump has claimed that the war with Iran will end immediately after the US midterm elections in November because Tehran won't be able to hold out longer. And a 31-year-old British man is due in court tomorrow in London, charged with assisting Russian intelligence services. Now, at a product launch event tonight, Apple welcomed a new product into the fold. This is an iPhone. It's not the first iPhone to have your favourite apps, or the first to fit comfortably in your hand. It is the first iPhone to do this. So what is this? Well, you may have spotted my deliberate clue because the iPhone Duo, as it's called, is Apple's first foldable smartphone.

22:36A bold statement then at the first product reveal event to be fronted by the tech giant's new CEO, John Ternus. But will it be a successful one? Well, joining me in the studio to discuss that, Jamie Richards, London-based freelance tech journalist. And Jamie, you have not come alone, have you? No, I haven't, Jamie. I'm here with the one folding phone, that phone manufacturers haven't been able to take away from me yet. It's the Oppo Find N6, which I brought to compare in absentia with the new iPhone Duo. Well, yes, you have to do this through the ether for the time being, don't you? What's your initial impression of seeing the Duo then?

23:12Well, it's a really interesting one. It's been a massive day for Apple. Obviously, we were expecting the iPhone 18 Pro and the iPhone 18 Pro Max, but to have the iPhone Duo delivered in the same presentation right at the end, as they always do, you know, one more thing. It's really massive. And looking at the iPhone Duo, it does take a different form factor than what I have here today. What I've got in my hand, the Oppo Find N6, sort of mimics a normal smartphone when it's folded. It's a tall slab of glass, looks like a regular sort of, you know, Samsung or Apple phone. And then as you pop it open, it turns into a sort of square screen.

23:47And this has been the form factor of folding phones for as long as they've been around, really, this sort of square in a tablet display. Apple's gone in a different direction. they've taken this, they've squished it down into, when it's folded, it's like a passport shape. And then as it opens up, it's more of a conventional tablet. So it should be a bit more useful for watching videos, for scrolling through the internet, and the things we use our phones for every day. And for reading, it looks more like a book, doesn't it? Yeah, absolutely. And they've actually engineered it to mimic that in the action of the screen.

24:16I mean, one of the things that people talk about with folding phones is the durability of the mechanism and the display. And the way Apple have said they're getting around this is by layering sheets of material in the display itself to leaf over each other like pages of a book. I mean, Apple likes to call itself an innovator. It's taken it seven years since the Samsung folding phone to come out. Why so long? Yeah, seven years. That's right. Well, Apple doesn't tend to rush into things. Even with its most groundbreaking devices, the iPhone, the iPad, even the Apple Watch you could throw in there.

24:49it was kind of late to the party because under Steve Jobs under Tim Cook and now under John Turnus Apple likes to wait for its competitors to make the mistakes it likes to wait for Samsung to have to deal with folding phone screens peeling off it likes to wait for Google to deal with you know battery life or or processor issues but Apple you know then comes in with a sort of clean slate a well-engineered device and enters the market on good footing but a pretty small slice of the market I know you're a never-go-back foldable phone guy, aren't you? But given the overall market, it's not huge, is it?