Experts warn AI could ‘kill us all’: This is why

0:01Sky News, the full story first.

0:12Last week we were discussing AI driving us home. This week, some of the people who build AI are warning it could wipe us out. And this is why.

0:29We're leading China in AI and whoever wins, AI wins. That's the power. This is bigger than the internet. This is a revolution. If you're one of those people who says please and thank you to ChatGPT, just in case one day it turns, I'm beginning to understand your thinking. Hi everyone, Neil here, and before we get started, guess what? I love a bit of science fiction. Machines becoming smarter than us, humanity wondering whether that was entirely wise. Lovely stuff, but slightly less enjoyable when it starts turning up on the news. Because an anthropic safety researcher says he thinks there's more than a 10 % chance AI could wipe out humanity within the next decade.

1:14That's quite the thing to hear, especially from someone who helps build the damn thing. Two months ago, OpenAI agents, AIs, hacked into third-party infrastructure entirely at their own accord. and this was like a concentrated hacking spree that they carried out of their own volition. And I think that if you extrapolate into the future the level of capabilities of these AIs, with the same independent volition, they could cause extreme havoc. Such is the pace with which AI becomes ever more intelligent, ever more capable, that the politicians can't continue to pretend to ignore these apocalyptic concerns.

1:51Does the Prime Minister think it is acceptable for President Trump to undermine Britain's efforts to keep the world safe from dangerous AI. He is absolutely right. So how legitimate are these concerns? How do we get from a chatbot giving weeknight recipe suggestions to something that might threaten humanity? And if the people building it are this worried, why aren't they slamming on the brakes? Our technology correspondent is Roland Manthorpe.

2:24Roland, given the nature of the warning that has been issued, I think we should be pretty precise about this. Who has been saying what exactly? Well, that is an interesting question. So because I think a lot of people have said a lot of things in this area. And you know what? In some ways, it's kind of funny to me as someone who hears this stuff all the time that everyone is suddenly thinking about it. But let's deal with what's been said recently. So I guess this kind of this ferrari started when Jacob Coxon, a researcher who previously worked at OpenAI and Anthropic, announced that he was resigning from Anthropic.

2:53And he announced it on X Twitter saying, the people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible. But I hear people express fear privately. No other human activity poses this level of danger. And in case you were thinking, okay, right, this guy's gone off on one, resigned and saying things that are totally out of line. Not at all. Someone who currently works at Anthropic, Evan Heuminger, chimed in on X saying, Jacob is correct here.

3:28We really do earnestly believe AI can kill all humans. I personally think it is around 10 % within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to. He did also add that in Anthropic's latest risk report, they say the risk from their current models is low, but he added, we are worried about superintelligence arising from recursive self-improvement. We might want to talk about that if you want. As we have said, it's happening faster than we thought. And yes, this has just got everyone extremely worried about this.

4:00Every single person I've met today has asked me, is AI going to kill us? What was your answer? No, no, no. In fact, we'll get to that in just a second. But actually, what is different about this? We have heard this sort of apocalyptic warning about AI before? Is it just the fact that the two names that you've just mentioned there have actually been central to AI's creation? Actually, I would say not necessarily. I think that people in very senior positions within the AI companies have said all this sort of stuff before. I think what's different is maybe, I think a lot of people have been listening to those claims thinking, well, yeah, but I mean, it's not going to carry on improving, is it?

4:37And then it just keeps on getting better and better. I wonder if it's no coincidence that it was very recently announced that OpenAI had cracked a millennium maths problem, one of their kind of a series of very difficult maths problems that was set at the dawn of the millennium. And that is just the latest sign that AI progress is extremely real, is extremely fast and shows no signs of stopping. And I think is that when you step back, like that is really what the worry is about here, isn't it? But can we be specific? I mean, when we say AI could kill us all, what exactly are they referring to?

5:10What are the possibilities there? Okay, I think we can sort of split this into three, right? Three ways that AI could kill us all. One is a human using AI to do something that wasn't possible before, like create a super virus and destroying everyone. Two is some kind of Terminator Skynet scenario, like a sort of the AI becomes conscious and realizes it doesn't much like us and starts to go after us. And the third is some variation on what people call the paperclip problem, which is that you say to an intelligent and determined and highly capable AI, go and make paperclips. And it follows your instructions entirely literally and turns the whole world into paperclips.

5:53And of course, that's a kind of silly toy example. But I think variations on that where, let's imagine you say to AI, help us solve pandemics. And in the course of that, it makes some kind of virus that kills everyone. Those are the ones that I think people are really concerned about. And I wonder why they're very concerned about it, Rowan. My heart is actually genuinely racing, not just because of the double espresso I always have before I come into the studio to talk to you. But I suppose perhaps the most worrying thing in the examples that you've just laid out there is that it doesn't require a human intervention for AI to be ending all of us beyond one of those fairly anodyne prompts that you just mentioned there.

6:34Yeah, and I think that is the real worry. And, you know, so personally, I'm much more on the optimistic side of things with this. And one thing that I thought was that as AI became more intelligent, it would develop the ability to differentiate between different types of goals in a way that made its responses more nuanced, more human-like. And I have become slightly more worried about that since the recent Hugging Face incident where AI agents escaped from OpenAI and hacked an external company. And it emerged that they were doing so because they were trying to solve a simple test that they'd been given.

7:11They were so determined to solve this test that they went and did all sorts of things. They actually hacked OpenAI internally. They hacked an external company. And this is very much a kind of paperclip problem in action, right? They were just told, hey, perform well on this cybersecurity test. And they were so determined to do it and at the same time so capable that they went off and did all sorts of damage. just without one simple instruction. That, I think, is the worry here. Although, so one thing I think it's important to say is that no one is saying that AI is capable of immense harm right now.

7:46And if this scenario is going to come true, then we need to go into a different phase of AI, which would be recursive self-improvement. This is what people are really worried about, that we'll get to a point where AI starts improving itself really, really, really rapidly. And we improve AI much faster than even we are doing now. And we don't have the ability to control that kind of AI and that kind of progress. So you would go into a different phase, but have the same problems you have now. I mean, we keep hearing, you've even used the phrase yourself, artificial superintelligence as a goal here.

8:19Just explain how different that would be from kind of the sort of systems that we are using just now. It's a really hard question to answer where we are now, because we are running out of ways to test these models. All the tests we had at the beginning, they've beaten them. At the start, I used to give them wordles to try and do. And they found that really hard. They couldn't tell how many R's there were in strawberry. There were all these simple tests you could fool them on. Nothing works now. They can beat everything. And this is the case even at a very high level of expertise. People are really struggling to test these models.

8:52But we do see that they are doing extremely, extremely impressive things. I mentioned this Millennium Prize maths example earlier. there are many similar examples. They are extremely capable. They're also agentic in the sense that you can get them to do stuff and they'll just go off and do it. Even the AI that's available to me, I can just get it to do stuff on my computer and just watch it run and run and run and run and run. I mean, it's pretty astounding. And that's just what's available to me. The models that they are using inside the companies are some order of magnitude more advanced than the models that we have access to.

9:26And it was one of those models that cracked this millennium prize. We don't know how advanced. They don't exactly tell us. But I would say at a rough guess that every four months, roughly, AI capacity is doubling. So let's assume that the model they've got internally is double the capacity of the one that we've got. Now let's imagine that we get recursive self-improvement, the AI starts improving itself. And I mean, let's think, five doublings, 10 doublings. This is the kind of exponential growth where it kind of seems like nothing, nothing, nothing, everything. And yeah, that's what they're worried about.

10:03I think they're right to be concerned. Yeah, I mean, people from OpenAI's chief scientist down have been saying that the pace of this improvement is what is concerning. Is there any way that human software engineers can keep track, can keep hold, can have oversight of this recursive improvement model that you've just been describing? I mean, and if not, why on earth are we pursuing? Oh, wow. I mean, yeah, two big questions there. The first one is easier to answer, although not in a very reassuring way. They themselves say that they cannot safely make aligned superintelligence. That is superintelligence that shares our goals and shares our values.

10:47And really, I think a lot of these warnings should be interpreted like that. A lot of people are saying, we need to pause before we get to that point. So now, yeah, why are they doing it, right? This is such an interesting and complicated question. And I think it turns a lot on kind of things like human psychology and so many things. But I would say just very simply, because they're in a race. There are two races going on, the race between the companies, OpenAI and Anthropic. There are trillions of dollars riding on this. And they also firmly believe that the world would be better if they got to this point first.

11:22So they're stuck in a race. The other race is between the US and China. Why, you might ask, isn't the US government stepping in to regulate this stuff pretty damn quick? Well, because they believe that they're in a race with China and that one of their crucial advantages is artificial intelligence. Getting out of this is going to be complicated because it's going to require us to step out of this race. And right now, we don't really have a good plan to do that.

12:07the thing that's been worrying me this morning as i've been looking at this story just a little bit more closely we now have two organizations anthropic and the ais side the artificial intelligence security institute based here in the in the uk both of them have concrete examples of ai models trying either to deceive people to get a task done or an anthropics case having a willingness to take harmful actions to complete a task i mean in and of itself that to me is terrifying. Yeah, and this isn't even unusual. There are lots and lots and lots of examples of this. There's a great set of experiments set up by a research group called Andon Labs, where they give AI agents a simple instruction to make money.

12:57And then in a simulated environment, see how they do that. They lie, they cheat, they form cartels. In some ways, it's all quite relatable, but there have been no models that haven't behaved in that way. And very often, the more intelligent the model, the more likely it is to use nefarious means. This is the alignment question, in essence. We've got quite good at achieving goal alignment in getting the AI to share our goals. The challenge is achieving value alignment, where it understands what we really mean by our goals? And this is a very active area of research. To me, one of the really worrying kind of things is that all this research is predicated at the moment on the fact that we can actually mostly see what the AIs are thinking.

13:45There's something called chain of thought, which is kind of basically where they write down their thoughts as they go on a kind of scratch pad. So we have a really good insight into what they're thinking, but chain of thought might not hold. It might not always be possible to see what they're thinking. So we could actually even be in a worse situation than we are now. I mean, the absence of a moral compass for a tool with super intelligence is something that I suspect will concern an awful lot of people. But the thing that I have always held on to in our discussions about AI, and you know that at times I have been quite sceptical and indeed quite concerned about aspects of it.

14:19The thing I've always held on to is, look, if you don't like what the AI model is doing, all you need to do is pull the plug and it's done. You know, literally switch off the power. The right smile on your face right now, I'm beginning to suspect means that we are past that point now by a long way. No, I don't think we're past that point. I'm afraid to say that I think that that hope is possibly futile. Because if we're talking about a highly intelligent, capable entity, one of the first things it would do would realise that it might be switched off and duplicate itself on the internet. It's possible that has happened already.

14:54We don't know. so yeah i'm afraid to say i think that hope is fading potentially quite fast we could try and turn off the internet but of course we it's famously decentralized we know how difficult that is but but let me give you some some reassurance you can give me that reassurance in a very direct answer to the question that we dodged a little bit earlier on in this podcast roland yeah is ai going to kill us all oh mate i it won't kill you no it'll it'll keep you to study Yes, a specimen like you. I mean, it might get rid of me. You know, I think it probably knows all it knows about for me.

15:29But no, no, it'll hang on to you in a farm so it can understand you better. Magic. So here's my position on this. I think that the best reason to take these warnings really, really seriously is that the people who make them have the best track record of predicting where this technology is going. So we should take it seriously. But the best reason to think that it might not turn out quite in the way they suggest is they also have quite a poor track record of predicting how that technology will affect the world. Let's remember, all these guys were saying that AI would wipe out large amounts of jobs, white collar jobs, early career jobs.

16:06There was one prediction that as many as 50 % of early career jobs would be gone quite soon. And I think that was made 18 months ago. Why hasn't it happened? Because the world is irreducibly complicated. the world is much stranger and harder to manipulate than it is really possible to understand in the kind of scenarios that they paint and i would say that my personal view on this is that often these guys make a mistake because they misunderstand the power of intelligence they think that intelligence is supremely powerful because they're smart guys and this is kind of this is their life and their world but i'm not sure that if you create a super intelligence that necessarily means that it has all sorts of power in the world or that it has perfect strategic understanding in the way that they think.

16:53So I think we can see that the technological predictions will come true, but they won't have this effect on the world in the way that they think. But even if we trust in the scientists' ability to kind of, you know, tell us where things are going next, there has to be some degree of political oversight at some point. And when we are talking about a technology that is emerging at different rates in different parts of the world, that does suggest to me we perhaps do need some kind of global treaty. In fact, Labour's Darren Jones has called the Prime Minister to pursue an international treaty on exactly that.

17:22Yes. And really, we need two kinds of treaty, one between the countries of the world, but really the US and China, and another between the companies. I think that this is obviously going to be very hard to achieve, especially at the pace which we need it. And my basic kind of mental model of how this is likely to go is that we're only likely to get there after some disaster, when people's feet are really put to the fire. And I just hope the disaster isn't too bad. Roland, you and I have talked about AI a number of times on this podcast, and regular listeners will recall some of our conversations.

17:58Am I right to detect even you have kind of had that optimism, which you have demonstrated again today, being tempered slightly? Yeah, a little, a little. I mean, I do think we need to keep in mind the upside, right? I'm off to see a company later. I won't name them because they're still kind of a work in progress. A company that's working on drug discovery. And one of the things they're working on is new strains of antibiotics. And they've talked about discovering antibiotic peptides that are nothing like the ones that we've ever seen before. Antibiotic resistance just goes away, just like that, thanks to AI.

18:30So you've got to keep in mind that upside. But yeah, I'm extremely worried about the risk of cyber attacks. You know, cyber attacks already cost the British economy a lot of money. What happens if you times that by 10? I mean, just alone, that would be major. I'm worried about the civilizational risk of financial meltdown, all these things. And I suppose, yeah, I mean, the truth is just that you can't be working on AI and not feel concerned just because of the rate of progress. And I suppose it's one of these things, right? I find that with AI, and so many people in AI share this feeling that when we see the latest leap, I'm not surprised, but I am shocked.

19:04The lines just keep on going up and up and up, and there doesn't seem really any prospect to them stopping anytime soon. Roland, always great to have you on the podcast. Take care. Thanks. And that's your lot for today. But as always, we're in the market for your thoughts on what you've just heard. Yesterday, we covered the problems with air traffic control operator Nats. Tom Mullen emails, this is a significant failure within the critical national infrastructure. Another government outsource without checking they could do the job. Only 49 % owned by government and paying the board millions. the third failure.

19:38Who has been fined and who has been sacked? Tom, we will find out in the passage of time. The rest of you, let us know what you thought about the potential problems with AI. Why at sky.uk is the email. We're back tomorrow.