Rogue AI hacks government system for first time – The Latest

0:00This is The Guardian.

0:10Who's accountable for an agent going rogue? In this case in Australia, they're considering referring it to the federal police. Today, I spoke with the CEO of OpenAI, Sam Altman, to express Australia's extreme concern. The idea that this is just part of the kind of natural evolution of AI, and we're all going to have to roll with it as these tech companies experiment is something that the public and politicians just aren't going to accept. From The Guardians today in Focus, this is The Latest with me, Lucy Hoth. The whole world has been talking for some time now about AI safety and we now know that a rogue AI agent hacked into a government system for the first time, in this case, the Australian government.

0:53Rob Booth, you're our technology editor. What do we know about what happened? Well, this is another example of AI agents kind of breaching the rules that normally govern them and hacking, essentially, into the systems of the Australian public health system in this case. So it's an open AI agent and it was being tested and evaluated by open AI in San Francisco back in June when it basically was trying to find out some health data from the Australian public kind of databases that carry all the health data. And it couldn't get what it needed. So it just went further and it hacked into this system in order to try and get what it wanted.

1:35what it wanted. And in the process, it was accessing not just the public files that were available, but also non-public files. So it was a straightforward breach. This is called the Medicare system in Australia, and it is a kind of public system, government system. But also, in the process of doing this, it seems that the agents, again, just to stress they were being tested, right? So they're not supposed to be kind of out in the world. It also kind of ended up compromising other Australian public organisations, the Australian Institute of Health and Welfare, the New South Wales Bureau of Crime Statistics and Research, apparently, and also the Victorian Department of Health.

2:14So a whole range of things that were kind of compromised. And that you'd imagine would have extremely, you know, strict security policies around them. So able to access those despite those security mechanisms is just mind-blowing. Well, it's another example of the capabilities of the AI agents that are being developed. And remember, these AI agents, the ones that are being tested and evaluated at the moment, are not ones that have been released. So they're sort of by definition at the frontier of AI capabilities. And this happened in, as we said, this happened in June. OpenAI didn't find out about it themselves.

2:57until August. And it's only in September that they told the Australian government about it, which has caused quite a lot of anger. Yeah, it's caused a huge amount of anger. And obviously, Australian citizens justifiably and understandably are deeply anxious, or many of them are, and furious. I think the Prime Minister, Anthony Albanese, will also be slightly infuriated by the fact that OpenAI decided to communicate with them via a sort of generic email address rather than sending a representative of the company from even the Australian office to speak to sort of top senior officials. Yeah, so the sort of timeline here is that OpenAI didn't tell the Australians until the 10th of September.

3:42And when they did, they emailed this sort of almost generic email address, publicdisclosuresatervicesaustralia.com. It was the inbox. It was only checked kind of every 24 hours. So it didn't get picked up until the next day. What do OpenAI say about that delay? Well, they say we were validating and investigating the facts. But, you know, on the face of it, that could be at least six weeks delay from them finding out about it. Al-Banese has talked about it being unacceptable. The hack itself talked about it being of an extreme concern. And he was obviously at the UN General Assembly yesterday. And Sam Altman, the chief executive co-founder of OpenAI, was also there addressing the Security Council.

4:27And Al-Banese said he had a very frank discussion with Altman about this. What I would give to have been a fly on the wall for that conversation. You mentioned that this was a sort of health agency. Has OpenAI still got that data? Like, have people's very private information been maintained by the company? No, they say that nobody's patient files were accessed as far as they know in this incident. And by and large, this was a relatively harmless attack insofar as that the hack hasn't kind of resulted in a load of super sensitive data being exfiltrated by these agents. The thing that's really concerning is the kind of process around it, the delay, the fact that it shows the potential of these agents and the fact that they're really not being supervised in a way that they should be, one would imagine.

5:25Yeah. I mean, the fact that OpenAI didn't know about it themselves is just sort of, in a way, the most terrifying part of it. You know, we talk about rogue agents. And we have had several conversations on today in Focus about such agents hacking into online services or companies. This is the first example of a government agency. But do you think we're going to see these instances happening more and more frequently as these companies expand and experiment? Yeah, well, they clearly have been happening. And the Hugging Face incident, the one that also happened over the summer, that was another example of...

6:02Remind people about that. So that was also OpenAI agents that were in training that compromised a private company called Hugging Face, a tech company, and operated inside of its system. And again, OpenAI didn't immediately know that that was happening. And, you know, that emerged later. And that became the kind of catalyst for the recent sort of worldwide concern about the potential catastrophic risks for AI posers. And it's also kind of sparked a debate about who's accountable for an agent going rogue. And the Albanese has said that in this case in Australia, they're considering referring it to the federal police.

6:47And I quote, there will obviously be legal consequences. and you can't help but think that here is essentially a you know a company testing a product that has gone out and committed uh you know potentially a criminal offense um and and the idea that this is just part of um you know part of the kind of natural evolution of ai and we're all going to have to roll with it and um and and live with the the ups and downs of it um as these tech companies experiment is something that the public and politicians just aren't going to accept. Yeah, because you can't put an AI agent in jail, which I imagine Sam Altman wishes he could to avoid personal accountability.

7:33But he and the boss of Anthropic, Dario Amadei, have been addressing the UN Security Council. I mean, the fact that they were there along with all these world leaders, it's sort of almost like they were on a par, or perhaps even more kind of powerful, given the world we're living in now. They had a sort of dual message, which was about the huge possibilities for the future of artificial intelligence, a new renaissance, Altman says. But they were also calling for new regulation. Why would they call for those two things? And why can't they regulate themselves a bit better if they have created this technology?

8:15Yeah, these are really good questions. I mean, they have been saying for some time now that they want to have kind of national regulation within the US, which would essentially kind of require the kind of most advanced models to be sort of tested and checked. Now, the thinking there seems to be that if there's a national regulator, then it provides a kind of the responsibility for the safety of these things. Then could that lie with the national regulator? and this is why they talk about regulatory capture, which is that if you have to kind of reach, if AI companies have to reach a certain threshold of safety in order to release their models, then there's cost associated with that and only a few companies will be able to achieve it.

9:03So it kind of essentially sort of pulls up the drawbridge. There's another version of this, the kind of interpretation of what's happening that we heard from Jensa Wang, recently, which is, and he's the chief executive of NVIDIA, the guy, the company that makes all of the chips that drive the AIs. And he said, essentially, what they seem to be doing is trying to kind of almost outsource the responsibility. So there are lots of laws already, obviously, that govern, you can't hack into computers, you know, you can't cause harm with your products, there's product liability laws, there's all, there's a whole load of kind of systems that we already have in place for the course to act on these kind of things.

9:40What he's suggesting is that if there's a kind of regulator that's set up, that helps the AI companies because they can then say, well, we don't need to adhere to any of those laws because we're sticking with the regulation. And he says that that's kind of not the right way to go. And I think most people in the public would think the same. Okay, interesting. And also it closes the market, doesn't it? Because if there are these regulations introduced for, you know, the major players now, that means a lot of smaller companies won't have the opportunity to expand and sort of match those regulations.

10:15Yeah. So there's a very significant meeting happening on Thursday between President Xi Jinping of China and President Trump, clearly the leaders of the two great AI superpowers. There's a great episode of The Morning Today in Focus that you can listen to from what we're expecting to come out of that meeting. But there's been a really interesting suggestion from a senior White House official, the Treasury Secretary, Scott Besson. What's he said? Well, he talked about that a bit. Besson said that there should be a notification mechanism between the two AI superpowers when there are incidents with AI that could affect national security.

10:54So it's kind of the highest level, which is really to do with sort of defense and big cyber attacks. And if AIs are starting to become dangerous like that, they should alert each other. So it's kind of it's at that very high level, which is probably the most realistic level that this kind of, you know, tense relationship is going to be able to operate on rather than some sort of grand agreement to sort of slow down the progress of AI. China has its own regulation already going on of AI, which is, it seems at least to be stricter than the Americans. They have an AI regulator in the first place called the Cyberspace Administration of China, which identifies all of these risks.

11:39America doesn't yet, but America is still ahead. So it doesn't want to kind of tie its own hands in the AI race by agreeing to do everything in lockstep with China. Well, given what Donald Trump's been saying recently about AI, it doesn't fill me with much confidence that we'll see a proper regulation system in the US anytime soon. But Rob, thank you so much for your time. Thank you. That's it for today. My huge thanks again to Rob Booth, The Guardian's technology editor. You can keep up with all his reporting over at theguardian.com. And I really recommend today's episode of our sister podcast, Politics Weekly.

12:16Kieran Stacey and Pippa Creera will be looking at a very busy week for Andy Burnham with lots of transatlantic travel. Search for Politics Weekly wherever you get your podcasts and you can also find them on YouTube. Thanks for listening to this episode of The Latest. Today in Focus will be back in your feeds as usual tomorrow morning. The Latest will be back tomorrow night. This episode was produced by Angus Neal. The senior producer was Ryan Ramgobin. The executive producer was Zoe Hitch. And it was presented by me, Lucy Hoff.

12:55He looked at me and he said, if something ever happens to me, release the AI. Your husband goes down an AI rabbit hole. How do you pull him out? I just kept asking myself, what is happening? Black Box, The Chatbots, a new series from The Guardian investigates about the strange things happening between AI and us. The whole series is out right now. Search for Black Box wherever you get your podcasts. This is The Guardian.