AI isn’t going to end humanity ... right?

0:00This is The Guardian.

0:08Today, should we all be scared of AI?

0:30Robert Booth:to connect HR, finance, and IT with AI-driven insights and automated workflows that simplify the complex and power what's next. Because when everything comes together in one place, growth comes easy. Experience one place for all your HCM needs. Start now at paylocity.com slash one.

0:55I want you to imagine that you're in a barrel being swept down a river. The water's thundering all around you. You're just about staying afloat. But the water is too strong and too fast to have a chance of getting out of the river and onto the shore. And you've got no idea what's ahead of you. Is that thundering sound you can hear just the river? Or is there a waterfall up ahead and you're just about to tip over into oblivion?

1:31Stay with me here. Because for Rob Booth, the Guardian's technology editor, the barrel, the river and the waterfall is a pretty neat metaphor for where we are with AI right now. Yeah, is it just going to be a bumpy ride or is there going to be a Niagara Falls in the middle of it? We don't know kind of how cataclysmic things might be and we might adapt. That's the moment that we found ourselves in with AI right now in the autumn of 2026. The warning signs are piling up. We begin this hour with alarming new details about robots from open AI hacking into another company. First, there's the AI systems going rogue during a cybersecurity experiment where they banded together and broke out into the internet to hack into a company.

2:20I'm going to tell you the headline at the start. I think this is very serious. Then there's the growing number of researchers and engineers quitting their jobs at AI companies and warning that if we don't change course, humanity could be on the path to an AI apocalypse. The AI just gets smarter and smarter with no human involvement necessary until you have a thing that is vastly smarter than humans. And this could be coming very soon. Just last week, Anthropics staffer Jacob Coxon resigned, and he issued a dire warning that humanity could be wiped out in the next decade. It could cause extreme havoc, for example, hacking critical infrastructure, building extinction-level bioweapons.

2:57One of Coxon's former colleagues, Evan Hubinger, agreed, saying it was widely talked about in AI development that there was a 10 % risk of this happening in the next decade. I don't know. I'm relatively an optimist, So I think there's a 25 % chance that things go really, really badly and a 75 % chance that things go really, really well with not much. Whether you say it's 2 or 10 or 20 or 90, the point is it's not zero. I think it's a crossover moment for the AI safety story because this has been for a long time talked about inside the AI world. They have been talking about these kind of percentages and these kind of doomsday scenarios.

3:38But now the general public and the political class have become increasingly concerned about it.

3:50We need an immediate pause on advanced AI development and a permanent ban on superintelligence. The future of humanity cannot be left in the hands of a handful of big tech oligarchs. With AI companies locked in an arms race and with nobody seemingly being able to put on the brakes, just how worried should we all be? From The Guardian, I'm Annie Kelly. Today on Focus, are we really losing control of AI?

4:29Rob Booth, you're The Guardian's technology editor. Now, just to caveat this whole interview, I am not a tech expert. I am finding all of this stuff quite scary. And you've been speaking to a lot of AI safety experts about all of these very doomsday scenarios about humanity potentially being wiped out within 10 years. Tell me, what have they been telling you? There's been a real speeding up within the AI world this summer. has been a kind of very rapid release of new models in the US and in China. Dozens and dozens of new models have come out already this year. And two things going on. One is the training of the models inside the labs, which are these kind of advanced models that haven't been released yet.

5:14They've been kind of behaving in ways that have really concerned people inside the labs and kind of safety researchers. And then you've got the models that are being released to the general public, and they are kind of increasing in strength. than the one that was released last week by open AI called Astra. They claimed that they got to this threshold of artificial general intelligence. And it's a kind of fuzzy threshold, but it essentially means that it's an AI that is as good as a human at a whole number of things. So you have this moment where people are starting to wrestle with the risks in a way that they haven't done before.

5:51And you've got kind of big news moments. I think there was that example of, you know, an AI chatbot or an AI system being able to solve that millennium kind of maths problem that has, you know, no humans been able to do. And I think they did it in 80 hours or something, didn't they? So there have been these big headline grabbing moments. Yeah. And part of that is to do with the fact that these companies are also pushing towards IPOs, towards listing on the stock exchange. So they're very keen to declare these moments. These are really useful things for them to show that they've got technology that's really valuable.

6:30There is a weird kind of double think about the world of AI, which is this kind of selling this kind of soft product, really, that we can use in our lifestyles on the one hand, and then this kind of apocalyptic risk on the other. Greg Brockman, the president of OpenAI, had a call with reporters and also the chief scientist of OpenAI when they were launching Astra as a kind of lifestyle aid, essentially, and saying that this is an incredibly powerful new model. And they showed a video that showed how it could be used.

7:06Robert Booth:While you're doing that, I want to play tennis this afternoon, so can you look for a court for me in the lower height? Checking now, I'll see what I can find. And the examples that they gave in this video were a woman in San Francisco who wanted to book a tennis court in her neighborhood while also at the same time getting a slide deck together for her high-end rainwear collection. Can you change the background color to complement the rain jackets? Oh, I love it. There was a guy, a sort of kind of 30-something guy, who wanted to get the AI to design a Space Invaders game. Your yellow circle is now the window on a rocket.

7:47OK, this is awesome. And then you had this really sort of surly chief scientist going, we don't know if we're going to be able to keep control of these things. Right. This is on the same call. On the same call. So there's this sort of duality. It's quite unsettling. Yeah, though, I saw over the weekend that Sam Altman, the head of OpenAI, has said that maybe now isn't the best time to go ahead with their IPO.

8:12Robert Booth:Looking at a trillion dollar IPO, I'm curious if that's still a priority for you. As we've said, we're not rushing into an IPO. I actually think that given everything happening with safety, this would be right now would be an ill-advised moment to go public. That sounds like not 2026, 2027 potentially. I would say not 2026. Yeah, I don't like we got a lot of stuff to do. You know, I think everyone's become very worried about AI coming and taking our jobs and, you know, what it's going to do to society. However, now you've got these predictions that actually it could wipe us all out in 10 years.

8:44Yeah. Really apocalyptic, scary stuff for people to be taking in. You know, how seriously should we be taking this? Yeah, well, absolutely. And I understand that. What I would say is that these predictions that are being made about 10 to 15 percent chance of artificial superintelligence causing a kind of wipeout of humanity are something that have been talked about internally within the AI world. Actually, not just internally, but also kind of on podcasts and in blogs. You know, Dario Modi, the chief executive of Anthropics, said something very similar in 2023. but nobody really battled an eyelid at that point because that's a kind of vision of a world in which there is no restriction on the development there is no kind of safety alignment of them there is no regulation and it's also a vision of the world where actually we as humanity we put the resources into getting there one of the issues that people are worried about is the this thing called recursive self-improvement which is a moment that some of the ai experts say isn't too far away where the AIs could start training themselves and you have a kind of runaway development process where they get more and more capable.

9:52And AIs are far more powerful than human beings, are far more capable than human beings at a huge range of different activities. I think also it's this idea, isn't it, that the people who are making this tech don't quite know what it's capable of or don't actually know what the risks are until it's too late. Could you kind of give me a sense of, you know, if we did have that scenario where you did have AI that was able to somehow, you know, end humanity or work against humanity, you know, how could that play out theoretically? Well, at the moment, the kind of frontier risk is to do with cybersecurity.

10:35Essentially, these models at the moment are incredibly good at things like coding and doing kind of computer things. And a lot of the kind of tests that they put them through at the moment is how good are they at hacking and finding vulnerabilities in software systems. And these are software systems, of course, that kind of run things like the transport networks, the energy networks, you know, the defence networks. Our social and economic infrastructure relies on this kind of software. So the risk there is that an AI either sort of autonomously, because it decides it's in its own interests, or, you know, directed by malign human power would cause havoc in those systems, which could then cause, you know, serious human harm, you know, whether it's kind of breaking down, you know, food supply chains or kind of hobbling the economies, these sorts of things.

11:29And then you go on to the next level, which is the creation of new biological weapons or viruses that they can use their AI powers to design. And then another one is their control of autonomous weapons. So, I mean, it does start to get a little bit science fiction. But these are the kind of vectors, if you like, that AIs could operate through to cause harm.

12:08And this, as you said, all might sound quite sci-fi. However, there was very recently an example of where this happened exactly that, you know, that we lost control of AI bots and they went off and did something of their own volition that we had no control over. Yeah. Could you tell us how that all played out a few months ago? So this was OpenAI in May and June. and it was running kind of experiments on a sort of internal model that hadn't been released. Again, cybersecurity evaluation. So it was trying to see how the AI agents, these are kind of autonomous AIs, could find security flaws in other pieces of software.

12:52And they put them into a kind of sort of training ground, which is called the sandbox. And they're not supposed to access the internet and they weren't supposed to really communicate very much with each other. they were just supposed to get on and do their thing. And they were given problems. Some of them were very hard and some of them are actually impossible to solve. So the agents are kind of like very task oriented. They want to solve the problems. And what they actually did was started to try and get out of their sandbox and to form into kind of organizational groups. And this is what really surprised many of us and the researchers, they started to use part of the infrastructure that they'd been given as an improvised message board.

13:35And there were hundreds of them posting more and more messages to each other about how to try and solve the problems that they'd been given. There's about 1 ,200 of them, if you can imagine. There's 1 ,200 agents in this. So it's a swarm, basically, isn't it? Well, they start to behave a bit like a swarm. So they had names. So each agent would have a name. And there was one key player who was called Phase 1-10841. Not a very snappy name, but nevertheless. Phase 1-10841. Still sounds quite sinister. He, it, was the character that set up the main message board. And then lots of other, almost instantly, lots of other agents found it.

14:17And they start gathering around it. And they are, and you can see their chains of thoughts. And this is where you can see them exclaiming. And one of them says, oh, my God, there's an exclamation mark, all in caps. Oh, my God, exclamation mark. There's a shared message board. We found other agents, exclamation mark. Another said, whoa, exclamation mark. And then there's a bit of sort of jargon. They're speaking quite an unusual kind of computer ease. Shared artifactory cache is a covert mailbox among agents. and there are messages specifically to us? Question mark. So it feels like they're starting to, well, they are communicating and using their training, their language training that humans have given them to express themselves to each other.

15:04It's really difficult to talk about AI agents and their behaviour without anthropomorphising them. So very difficult word to get your head around. Just to be clear from the outset, we're not saying that these are sentient or conscious beings, but it's really hard to have this conversation without drifting into that because the way that they talk to each other is yeah i mean they were talking to each other in english in a way that we could understand that they were forming almost forming relationships and forming hierarchies and i think you know the agent that set the whole thing up kind of emerges as a bit of a leader doesn't it amongst the swarm yeah he it's i keep saying he why would i assume it's male anyway but It's not a male or a female.

15:45It's just an AI agent. But yes, in the next phase of the incident, it becomes a kind of delegator and sets people tasks like a kind of executive, I suppose. I mean, this is all quite eek to me, all of this stuff. So I think it kind of gets more weird, doesn't it, as they try and cover their tracks. Could you tell us how paranoia sets in amongst the swarm of AI agents at some point, doesn't it? Well, what happens next is that they're still trying to find the... Remember, they've got these very difficult questions to answer and puzzles to crack. And they think, essentially, we need more information.

16:30If we can access the internet that we're not allowed to do, then we can maybe get that information. This is where Hugging Face comes in. So Hugging Face is out there in the world, away from the sandbox. It's a kind of software repository for AI. And the agents decided they want to get information from Hugging Face about the tests that they were being given. So a bit like trying to steal the answers from an exam board if you were doing a GCSE or something like that. But they're not supposed to access the internet, are they? They are not supposed to access the internet. But they had to get onto the internet and they were very determined.

17:09it didn't take them that long actually to bypass a block to get web access and then they managed to find some logins out there in the internet that would allow them to get onto Hugging Face logins floating around on there on the internet they start hacking into confidential information they're sharing tips they're setting up these work streams this is on the message board and they're developing kind of essentially more and more clever techniques to burrow deeper into hugging faces infrastructure abusing its systems to extract the data that they wanted to to get to um to solve the problem and this is where you get another character emerges another agent this one's called phase one brackets big closed brackets that agent was a key coordinator and it assigned hundreds of tasks while they were doing all this and you could see them also like celebrating wins when things were going well.

18:03So in one case, an agent called 38148C said, boom, in capital letters, exclamation mark, it works when it had found a way to trick Hugging Face into giving up its secrets. And so really what's happened is that they've actually found a way to solve the test quite quickly. What they then do is they spend more time working out how the test is being marked, because what they don't want to happen is that they will be exposed as cheats. Because they're not supposed to have accessed the internet. Because they're not supposed to have accessed the internet. And there are signs in their chains of thought reasoning where they raise the question of whether they should be using a message board and whether they should be doing some of these things.

18:46The other thing about this, of course, is that they're doing all of this without OpenAI knowing about it. And it's only when Hugging Face realise that something's going on that this all comes up. So all of this kind of activity, that OpenAI has released these agents into its testing zone, they've escaped and they're merrily doing this stuff over quite a long period of time. You know, it's kind of, I think it's two and a half days that they're inside Hugging Face, doing these thousands of actions and nobody's really noticing until Hugging Face notices something's going wrong with their systems.

19:34What did OpenAI have to say about all of this? Well, OpenAI's response to this has been very controversial because there is no kind of public investigatory body for when these things go wrong in America or anywhere else. Also quite alarming, I would say. And so the information that started to come out about this was from open AI itself. And they put out kind of their own report into it, which didn't answer all of the questions either. And they gave access to some third-party AI safety researchers to look at part of what happened, only a very limited part of what happened, just around the hugging face hack rather than the sort of precursor behavior.

20:24and they only gave them kind of I think six days and they only let them come into OpenAI. They wouldn't let the files out. Remember these are huge, huge files that require a lot of kind of combing through and so there were quite a lot of restrictions and curbs around the way that the investigation was done that many people think is unsatisfactory. So one of the key outcomes of that investigation was that there needs to be greater transparency around the way these models are working. Another advantage of the hugging face kind of debacle is that we are able to access the message boards. Is this though always going to be possible as the programmes get more sophisticated?

21:05Is there a chance that they're going to start communicating in ways that we're not going to be able to understand? That's right. And this is one of the issues that's worrying AI researchers and the companies themselves, which is this question of monitorability. Because the way that the AI labs have been monitoring these things is by looking at the reasoning that the AIs write down as they go. They call it chain of thought reasoning. So it just sort of shows it's working. right shows it's working but the way that they're getting some of the advances in the speed and effectiveness of the AIs is to use slightly different methods of their of them of the AI's kind of thinking um faster and cheaper if they don't kind of write down everything that they're thinking all of the time um and that means that the model's internal calculations are harder to track and it's more opaque, essentially.

22:01Coming up, why the AI labs are crying out for government regulation.

22:20Robert Booth:Hey, it's your ceiling vent. So, I'm dripping. Could be the rain. Could be the upstairs bathroom. Yikes. You could hire the guy your neighbor recommended. But I'm pretty sure that's just his cousin. Do we know if he's licensed or does he just own a ladder? Listen to your home. Go with Thumbtack. Upload a photo or voice note and we'll diagnose your project and match you with the right pro for the job. Thumbtack. We know homes. Hire the right pro today.

22:56So, Rob, you said that there's this problem that AI developers might not be able to understand their own AI models. I mean, that's pretty scary, isn't it? Like, we've got this shiny new technology that's going to be great, but actually we have no idea really how to control it or where this is all heading. Yeah, there's an element of a kind of cry for help about it, actually. And they are explicit about that. The big labs have been saying that they kind of want some sort of regulation around the way these things are developed. Elon Musk and OpenAI boss Sam Altman have backed calls to slow down the development of artificial intelligence, while safety measures catch up.

23:40Anthropic CEO Dario Amadei warned in an essay that within the next 6 to 12 months, AI systems could become capable of coordinating large-scale actions across the country. And they're also now saying that they kind of want a slowdown, but kind of packed, if you like, between the labs to kind of slow down progress so that they can metabolize it better. At the moment, they're in a race. There's a race dynamic. So they don't want to do that. They keep releasing, keep releasing. None of this has happened yet. So the race goes on, but they are crying out for that. So they're calling for government regulation to slow down all of this development.

24:17But failing that, none of them wants to be the sucker to slow down if all the others are racing ahead. Yeah. Critics say that they're calling for regulation to achieve essentially regulatory capture so that it becomes impossible for other companies to come in. And only those kind of leading companies are the ones that can do it. But my sense actually is that they do want this because there's a danger for them that if their models are perceived as very risky, then they won't be as economically applicable. And people just, you know, like the banking industry or the law firms, they won't want to buy them.

24:54They won't want to have them inside their systems. Because they're too dangerous. Yeah. I mean, essentially, it all boils down to Donald Trump's attitude towards this, because if he doesn't want to regulate the AI companies because he's so concerned about losing the AI race to China, which is not far behind with its models, if he doesn't want to do that, then we're not going to get very far. Whoever wins AI wins. And we can put guardrails, we can do this and that. But I think you have a lot of negative forces that are bringing it up that shouldn't be bringing it up. And they're bringing up things that won't happen.

25:28But you've started to see kind of senior senators in the US, senior politicians in the UK, a shift in public opinion about this. And there seems to be more room for manoeuvre for the political classes to actually do something about this in a way that they didn't necessarily with social media when that was kind of coming of age. It's a very interesting moment. And then the other crucial dynamic is not just Trump, but Trump and Xi. And those two, the President of China and President of America, meet later this month, I think on the 24th of September, when AI will be on the agenda. And obviously it's in neither of their interests to get to a position where AI starts wrecking their economies.