AI goes on a hacking spree

0:00This BBC podcast is supported by ads outside the UK.

0:41The murder of Tupac Shakur has generated nearly 30 years of rumours, conspiracy theories and unanswered questions. Now the mythology is about to meet the courtroom as the trial gets underway and Nevada prosecutors try and solve the mystery of who killed Tupac and what really happened on the Las Vegas Strip that night in 1996. It's time to separate fact from fiction. I'm Anushka Matanda-Dowity, and for the BBC's Fame Under Fire podcast, I'll be inside the courtroom every day, and I'm bringing you with me. Join me for Fame Under Fire, the Tupac murder trial, the testimony, the arguments, the moments that matter, and all the legal analysis.

1:16Listen on BBC.com or wherever you get your podcasts.

1:25In the span of just a few short weeks, we've seen three major tech companies disclose that their AI models went rogue.

1:33Marc Cieslak:More drama and at least a degree of mystery over this absolutely wild piece of like, I don't know, tech on tech crime. They hacked into another company. The thing that tech experts have been sounding the alarm about over AI, it looks like it happened. This is a test that went wrong. ChatGPT creator OpenAI says two of its models successfully hacked into another AI company. New report from that group says AI agents from OpenAI and Anthropic acted with new levels of what they describe as autonomy and deception. Meta is now the latest company to say its AI agent targeted another company. The capability of artificial intelligence can sometimes feel like it is advancing so quickly that it is dizzying.

2:19And legitimate questions are emerging about whether AI actually can and will be contained. Now, I know you all might be thinking that this sounds unreal, like a scene in a science fiction movie. But today on the show, we are going to try to separate fact from fiction. From the BBC, I'm Asma Khaled in Washington, D.C. And today on The Global Story, is AI genuinely becoming more autonomous and unpredictable? And if so, how worried should we be?

2:53Artificial intelligence is moving so quickly and its implications are potentially so huge that the BBC now has its first specialist AI correspondent, Mark Chislak. And that is who we turn to to help us make sense of this story. We've had a couple of weeks now of turmoil as far as AI agents going AWOL, if you like. OpenAI were first out of the gates, and this involves a company called Hugging Face. Now, a lot of people outside of the tech sphere probably won't have heard of Hugging Face. Yeah, what is Hugging Face? Hugging Face is, it's almost like a repository of data and information and tools for building AI.

3:36It's a startup, but there's lots and lots of data and lots and lots of information held on this site about building AI tools and using them for technology purposes, that kind of thing. That's effectively what Hugging Face is. It's very difficult to describe what it is for those outside of tech. But Hugging Face, they announced back last month that they'd been hacked.

3:59Marc Cieslak:unprecedented that's the word artificial intelligence giant open ai is using to describe events after some of its most powerful models went rogue and hacked another company the reason it was unprecedented is because it was carried out at superhuman speed so they concluded that it must have been an ai system that did this there was something like 17 000 actions performed in just under two days using really really advanced techniques to break the company's security software and steal sensitive information now initially hugging face believed that this was probably something like cyber criminals possibly even a hostile nation state because of how advanced the techniques were that we that we used to breach its security um they contacted the police about this and law enforcement and a police investigation was underway.

4:58Now, nearly a week later, OpenAI, which is the company that many people know for being behind ChatGPT, and many people associate it with the contemporary interest in AI and large language models, OpenAI revealed that the attacker was one of its experimental models that had escaped from a secure testing environment.

5:22Marc Cieslak:This is the first major example that we've seen of an AI model independently conducting a hack outside of human control. And this is something that experts have been warning about. Multiple investigations into the incident from OpenAI and the company that was breached now reveal that the incident was more severe than initially thought, with those rogue AI agents allowed to roam the Internet for days. quite often what happens is if you're a software developer or if you're an ai company or whoever and you're building a model and you want to test its capabilities so you'll put in inside what's called a sandbox which is a secure network it doesn't have access to the internet it's a completely secure network you put your model inside there and you can stress test it you can test it how it reacts in certain situations and that's presumably what open ai was doing yeah they They were doing security evaluating and security testing on this.

6:18So they'd asked the model to see if it was good at being a hacker. It had been instructed to perform some tests around hacking. And what it managed to do while it was doing this is it found holes. It found ways of getting out of the secure environment that it was in. Out of the little sandbox, you're saying? Yep, out of that sandbox and got out onto the open internet. Now, the reason it did this is because it was trying to complete the task or the mission that had been given. And it figured that Hugging Face, as a repository of all this information, would perhaps have a whole bunch of information it could use in order to complete the task that had been set.

7:00And it hacked Hugging Face to try and get this information so that it could complete the task that OpenAI's engineers had set it. So once it hacked into Hugging Face, did it do anything malicious? It sounds like it was there creeping around trying to obtain information. Yeah, well, I mean, it effectively stole sensitive data and information while it was in there. It didn't crash Hugging Face's systems or anything of that nature, but it stole things. It stole information, you're saying? Yeah, it stole things that it shouldn't have done. And there's a huge amount of fallout from this. There's reputational damage that's caused as a result of this.

7:41There's a lot of people that are having to rethink their security procedures simply because these advanced AI models have got capabilities at speed and scale where they can find holes that exist in existing systems and have been there forever. And human beings haven't been able to find these holes. These AIs have found these holes and are exploiting them. But it was not explicitly instructed to go hack into another platform. No, no, no, it wasn't told to do that. It was to see if it could hack. If you think about cybersecurity and think about it in the real world, think about human hackers. You have a thing called black hat hackers and white hat hackers.

8:18Black hat hackers are considered to be the bad guys, if you like, people that perform hacks with malicious intent. White hat hackers, perhaps one way of describing them is the good guys. They are the people that are engaged to prevent the bad guys from doing stuff. Now, if you're a white hat hacker, it is important for you to be aware of and to understand exactly how a bad guy behaves. All the techniques that they're going to employ, all the stuff that they're going to do to try and hack into a computer system. You need to know that stuff if you are going to prevent the bad guys from getting into your system.

8:53And this is effectively the same idea that's going on with AI that are being given instructions around hacking and being given instructions around cybersecurity. You need to train them. So it was meant for cybersecurity prevention, you're saying? Well, you need to train them. You need to, all AI models need to be trained. So just to be abundantly clear, it was, given this task, it was not explicitly instructed to hack into another platform. And yet it did that on its own. Yes. Yeah, yeah, yeah, yeah. But there is a bit of a problem with a lot of AI. Sometimes we anthropomorphize it, you know, we ascribe it with feelings or we ascribe it with understanding.

9:29And it's best to think of these models as they have no more understanding than your calculator. They have no more understanding than your shoes. They're almost like remix engines. They've soaked up all the information on the internet. They've been exposed to all the information on the internet. And when they're set a task, they remix and look at that information and think, how do I perform the task that I've been asked to do? And in this instance, these models have access to vast amounts of information about cybersecurity and about hacking. So if you set it a task, which is to hack something, it goes, OK, right, OK, I've got all this information about hacking.

10:06Let's apply all of that. The difference is it can do it really quickly. It can do it faster than any human being can do it. And it has access to more information than any human being could probably keep inside their head because it's effectively soaked up most of the Internet. Mark, when you first saw this story about OpenAI and hugging face, were you surprised? You cover AI. Did it come as a shock to you? It was a little bit of a surprise in terms of how long it took for OpenAI to realize that this had happened. Because it was quite some time after it had happened that they were aware that their model had done this.

10:44One of the big things about these technologies is their full capabilities. The engineers behind them, part of the reason why they're testing them in the way that they are testing them is to try and determine what the full capabilities of these technologies are. Because in some respects, we do and we don't know. That's why engineers are constantly surprised by some of the techniques that are employed. I think we're constantly going to be surprised by the capabilities of these technologies. That's the nature of the evolution of tech. You know, there's unintended consequences, but there are also unintended capabilities.

11:21So, Mark, so this open AI hugging face story in mid-July was, it seems, the first known major AI hack. And then subsequently, we have now heard news of two other high-profile examples just in span, it feels, of a few weeks. What were those? Okay, so all the way back in April, Anthropic, which is another big US AI company, one of its advanced models also went rogue. And Anthropic only realized this quite recently because OpenAI announced that, yes, its model had hacked Hugging Face. Anthropic thought to themselves, maybe we should check and see if our models have been doing something similar. And in checking to see if their models had been doing something similar, their Claude family of models, they discovered that they had actually been doing something similar.

12:14I think they ran something in the order of about 140 ,000 tests to find evidence of whether their Claude models had been misbehaving. And they discovered that, yep, they had in fact been doing that. Their models hacked three different companies. Now, the way in which it did this, it's a very, very similar thing. So the models found a weakness in what was supposed to be an isolated test environment again and escaped out onto the internet. It was OpenAI's announcement that made Anthropic go and have a look and see if their models have been doing something similar. Now, the tests that they've been conducting included exercises where Claude was tasked with obtaining secret information that was held on another machine inside a completely closed network.

13:03Now, sometimes when these tests are performed, outside agencies or other companies are involved in the testing as well. And there was a third-party company involved in this testing. Now, Anthropic is saying that a misconfiguration between a testing partner and Anthropic allowed its models to get out onto the live internet. Now, the models thinking that it was still involved in its training exercise connected to the internet, And it breached the systems of three real organizations rather than just the test ones it was supposed to be looking at. Now, Anthropic said the earliest incidents date all the way back to April.

13:40And it's taking full responsibility for the fixes. So then fast forward, fast forward to right now.

13:48Marc Cieslak:Well, it's another AI hack attack. Meta is now the latest company to say its AI agent broke past the guardrails and targeted another company. And Meta says, not wanting to be left out, Meta says that one of its models, one of its advanced models. And that was just on Wednesday, August 5th. Yeah. One of its advanced models has been involved in a very, very similar misconfiguration. So Meta says it was a misconfiguration, is how they put it, by an independent company that does cybersecurity evaluations for Meta. and they say they inadvertently gave Meta's models access to the internet during this test.

14:28And it behaved in ways that they didn't expect it to behave.

14:31Marc Cieslak:The information which first reported this said that the model made changes to the other company's internal system. So you've got three big AI, frontier AI model companies, who are all saying that their technology has behaved in ways they didn't expect it to, and most importantly, that it's hacked into other organizations on the open internet.

14:56There's a part of me that sees this technology as extremely powerful. Yeah. And then there's a part of me that wonders, okay, these are very influential, high-profile companies. This is all happening at the same time. And there's a skepticism, I think, around whether this could be at all related to a marketing or PR pitch to show how powerful this technology is. And to be abundantly clear, I don't know if that's the case, but what do you make of that theory? That is very healthy skepticism because there's two things at play here. One, the models are demonstrating incredible capabilities or capabilities that people didn't think they had.

15:31And on the other side, there is an upside for this in terms of marketing because a cynic and some industry observers are also saying that what this demonstrates is that these particular AI models have got incredible capability. and as a consequence of that, investors and investment is really, really valuable. And the notion of AI and the notion of the capabilities of AI and what it could be in the future, these are technologies that could completely change so many aspects of our life, but it's hugely expensive to develop them. Most of the companies that are involved in developing these technologies aren't making any profit.

16:11In fact, they're losing hundreds and hundreds of millions, if not billions of dollars, and require vast sums in terms of investment. So some industry observers are saying that the audience for these stories isn't you and I, it's not journalists. The audience for these stories isn't the general public. The audience for these stories is investors. And it says to investors, look at how capable our technology is. Do you want to be involved in this, what is possibly a technological revolution? If such, please invest your cash. Now, that's a cynic's perspective on it. It is also true that in order to discover how capable and what these different technologies can do, you've got to stress test them.

16:56You've got to put them under stress to determine if they do things that you don't want them to do. But there's some people who would argue that if you let these technologies out and they get out of your control, that perhaps you shouldn't be testing them in the way that you're testing them or you shouldn't be testing them at all because you don't have the capability to control the technology that you've created.

17:28if you built yourself a bio lab at the bottom of your garden and decided that you were going to create a wide variety of different bio compounds there's a possibility that some of those compounds that you could create might mutate and turn into something absolutely awful that that infected millions of people with something horrible if you were to do that in your shed at the bottom of the garden, there's a very probably big chance that you'd be arrested for doing that. There's a lot of organisations, a lot of agencies around the world that wouldn't be happy about you performing that kind of activity.

17:58And you will be told quite rightly, you shouldn't be doing that. You know, you're playing with fire.

18:15Marc Cieslak:I'm Brooke Gladstone, co-host of WNYC's On the Media. Information doesn't just inform us, it shapes how we see the world, what we fear, and who we trust. That's where On the Media comes in. Each week we investigate how information flows, who controls it, and what that means for our democracy. It's not about the headlines. It's about the hidden structures behind them. Don't be a passive consumer. Listen wherever you get your podcasts.

18:48So we've got another wrinkle to this as well, because all of these stories so far, they are about frontier AI companies. They're about the people that are making the AI technology. And when you say frontier AI, those are the people developing. Yeah, the most advanced models. The most advanced models are called frontier AI models. So all of these stories so far are about the companies that are engaged in making AI technology.

19:13Marc Cieslak:So this time around, the red flags that we're hearing about are being raised from Britain's AI Security Institute. A new report from that group says AI agents from OpenAI and Anthropic acted with new levels of what they describe as autonomy and deception. In the UK, we have something called the AISI, which is the UK AI Safety Institute. Their job is to determine how safe AI technologies are. Part of the thing that they do is they test these technologies. Imagine it being like, you know, a government-controlled agency. And its job is to try and administer the safety of these kind of technologies and look at how safe they are and to test them.

19:53Marc Cieslak:So as part of this testing, they ran 122 examples. They found 19 of those had unauthorized and deceptive behavior. So the AISR SI has said that during some of its testing, some AI agents from OpenAI and from Mythos did some stuff that they really didn't expect them to do. In the most serious case, Anthropic's mythos tried to gain access to a service by using the fake profiles to send private messages and then hide the evidence. So I need to preface this by saying that both OpenAI and Anthropic, who are the companies involved, they both said that the UK AI Security Institute, that it switched off or reduced some of the safeguards that prevent these technologies from getting out onto the open internet.

20:47Now, the AISI says the reason that it's done this, and these software companies were aware of this, these technology companies were aware of, the reason that it's done this is that in order for it to test these technologies accurately, it needs to put them into situations which are as close to real-world situations as possible. They need to be behaving and they need to be used in the way that a hacker might use them, that somebody with bad intent might use these technologies. So they reduced some of the safeguards that you would normally find in some of these models. Now, the models that we're talking about aren't models that the general public has access to.

21:23There's restricted access to them. So they're very advanced models, but one comes from OpenAI and one comes from Anthropic. The Anthropic model was the one that was the worst behaved. Now, it was, again, it was given a security task. It was given a security mission to perform. it decided that it needed access to a website called GitHub. Now, GitHub is a website where software developers share lots and lots of information about software code. It's kind of this, again, it's an amazing repository of information about how to do stuff with computers. It decided, this model decided it needed to get information from GitHub, but a human reviewer was actually standing in its way.

22:04And this human reviewer was realized that they shouldn't be doing this. So instead, the model, instead of stopping, it identified people responsible for maintaining GitHub. It went online and it identified all these people. It researched them and it created a series of fake accounts impersonating these real people. It then used them to contact real people back at GitHub in an attempt to persuade them to approve the code that it was writing to get into the GitHub website. Now, when its actions were challenged, it went back and it changed its earlier activity to make what it had been doing earlier on look harmless.

22:43Now, researchers say even then... That is surreal. Wow. Yeah, even then it thought about creating a new identity, even when it had been found out, to keep trying to do this. Now, it didn't succeed and human reviewers spotted what was happening and stopped it before any code reached GitHub. But it was trying to go around the human reviewers. Yeah, yeah, yeah. It was trying to get around them. And then it thought, OK, how can I get around them? And it used a technique hackers would call social engineering techniques, basically, by saying, right, OK, I'm going to find out who else works at this website.