AI bots meddle with US government agencies

0:00This BBC podcast is supported by ads outside the UK.

0:30Listen to Choiceology at schwab.com slash podcast or wherever you listen.

1:00Pre-owned equipment and more options to their line-up can shop at machinefinder.com. Boo-yah!

1:12Hello there. Welcome to News Out from the BBC World Service. I'm Sean Lay speaking to you from the BBC studios in London. Coming up this News Hour, back to school for one of the biggest rock bands of all time. But first, we begin with the activities of artificial intelligence that are once again making global headlines this Saturday. The company OpenAI says it's contacted governments and organisations around the world, alerting them that their websites may have been accessed, as it puts it, improperly by rogue autonomous models. OpenAI says all of the data was already publicly available. However, in some cases, the AI agents, as they're known, bypass security controls.

1:55Let's talk to Kate Bedman, technology journalist based in the UK. Kate, thanks for being with us on NewsHour. What do we know about this latest case of AI behaving not as the developer intended? What we know is that it seems much the same as a lot of what has been characterised as AI going rogue, which I would argue is a mischaracterisation. Effectively, it's working in a testing situation where they're working out what these agents can do and the limits of them. And basically what's happening is they are not being tested with guardrails. Now, in some cases, that's been deliberate. The aim is to find out what they can do.

2:33But this is not sort of stuff you see happening in some day to day life. This is AI companies deliberately setting their AI agents to work to see what they can do. Now, they're supposed to be working within what's called sandboxes, i.e. sort of restricted environments. but because of poor coding, lack of thinking ahead, not applying the what could possibly go wrong test, these agents which have got instructions and have objectives to fulfil are going on to fulfil those objectives in ways that, again, nobody applied the what could possibly go wrong test to. Australia's Prime Minister Anthony Albanese this week said the agent encountered restrictions but didn't accept no in the case of a hack against Australia's health system.

3:19Another politician in California describes these AI agents as relentlessly trying to complete a task regardless of the consequences. Is that a fair interpretation? Yeah, I mean, you know, these are not, you know, despite all these anthropomorphizing language going on, these aren't, you know, sentient beings. They've got objectives. And because they do very good sort of chain of reasoning work, they go ahead with trying to achieve those objectives. No doesn't have kind of quite the same sort of, you know, moral heft with software that it does with humans. So, you know, unless they're specifically told you cannot, you know, go to X website and you cannot use credentials you find on the open Internet as one of it's happened with one of these attacks, then it's going to go and do that.

4:08So this comes back to being a human problem. It comes back to the AI labs thinking, you know, what are the potential? What's the potential for harm here? I think, you know, they are not applying those guardrails themselves. This is a human problem as much as it's a technology problem. Right. Is it then one in which the human developers need to retrain the AI agents? it's one which they need to be much more careful and alert to how they set up their ai agents um and what they tell it it can and can't do now that's quite a hard problem because you know defining a negative is is tricky you know you can't do you tell an agent you can't hack any technology company and then you go on to name every last technology company from you know the the big beasts right down to little startups in Silicon Roundabout in London.

5:02It's a hard problem, but that doesn't mean it's an insoluble problem. Or one they shouldn't be trying to solve. Exactly. And actually, you know, the good thing is we are pretty good at cyber defence and it's not actually on the cyber defence to solve this problem. It's really down to the AI companies to be a bit more responsible about it. Kate Bevan, thanks very much. Well, getting our heads around who does what and even what does what in AI can be hard. Most people grasp what a developer does, but we're also learning about those who are testing artificial intelligence to see whether it can act on its own and whether particular AI agents pose a threat.

5:39META, or Model Evaluation and Threat Research, is a California-based group which describes itself as a research non-profit. It investigated the so-called Hugging Face hack this year when AI agents escaped the restrictions imposed on them and hacked into Hugging Face's infrastructure. What Mita uncovered was that open AI agents were operating as a sort of community. Agents became leaders and even took names. Tasks were assigned. The AI agents shared thousands of messages with each other and eventually covered their tracks. Some have called it a kind of mob mentality. Well, METER's president Chris Painter has been talking to NewsHour's Rebecca Kesby.

6:18Our goal at METER is to inform people of the evidence base and not persuade them to take one set of actions or another. I think that the decisions about where to legislate should come from legislators and from the public. You know, in the Hugging Face case, the agreement that we reached was to assess model behavior. So what were the kind of like motivations and aims of these AI agents during this incident? Well, on the subject of hugging face, I mean, people will remember that your team uncovered that the incident was actually even worse than was first thought. And that was even though that you only had restricted access to what was going on there.

6:58This is not like a regulatory setup where there's anything required of companies in terms of the breadth of testing that they do. And it's common for companies, they will disclose incidents that then aren't investigated. I mean, you talk about public interest. Many people worry that the incidents that have already happened are the tip of the iceberg. I mean, just this week, we had the Australian Prime Minister, Anthony Albanese, saying that the Australian healthcare system had been hacked back in June by a rogue AI bot. It was only discovered by OpenAI in August. They only told the government in September, on the 10th of September, by an email to the general email.

7:36I mean, so many of these things might be happening that we just don't know about. And at the moment, we're having to rely on these companies of being honest and actually owning up to these mistakes. I agree with that. I think that today, right, we depend on like in the Hugging Face case. My impression is that the world basically found out about it because Hugging Face disclosed publicly that they had been hacked. My impression is that they weren't even under any obligation to report that publicly. I think one thing to emphasize that I've been thinking a lot about lately is that all of these incidents in some way recently kind of look very similar, right, in that they have this like element of AI systems, AI systems either trying to get information from the Internet, from individual websites, or cheating on tasks inside of an AI company and trying to get more information about how to cheat at their tasks.

8:29And I think one thing to think about is it would be relatively easy, I think, for companies to solve that problem, right? They could more intensely cut the AI agents off from the internet. And yet we still could, like AI systems generally are going to keep getting more capable, right? Or at least, you know, if all this investment in larger data centers, more chips, more data, we should expect that they'll keep getting more and more capable, which means that the, if we kind of deny these agents even the opportunity to do these, attacks that we've seen recently, that doesn't mean that we won't keep having more advanced AI systems that are being built inside of these companies that might still have these kind of broken training processes that lead them to have the kind of, quote, motive to do these kinds of actions.

9:15I mean, are we at a point where we're already beyond that? Because some AI agents are already so clever, aren't they? I mean, in the hugging face incident, it gave itself a name, It hid its own tracks so that it wouldn't be traced. And it was a while till anybody found out about it. Are we beyond being able to control it already? I would characterize these as loss of control incidents in the sense that the thing that's happening here is it is very alarming to see signs of AI systems, you know, trying to figure out how to hide their tracks. And I think you are like right to worry that as they get more capable, they will get better at hiding their tracks.

9:54I think another thing to say here is it's possible. So, you know, our investigators depended a lot on this fact that when you went in to do the investigation, you could see this what's called the chain of thought, right, where the AI systems kind of think to themselves on a scratch pad about what action they're going to take next. People can sometimes get called like the reasoning. When you use ChatGPT, you sometimes see this window, right? You can see that it's thinking about its answer before saying something to you. One thing that I think is an additional worry here is even that could become less faithful or informative about what AI agents are thinking.

10:32So there's this idea right inside of the industry that it's possible that like soon we could have AI systems where to make them more capable, they move from reasoning in words to reasoning in numbers entirely. And what's called these vectors. So like entirely in numbers. And then when we send investigators in in the future to look at these incidents, it's possible that when we say like, OK, what was the AI agent thinking in this case? We'll find like just a whole big room of numbers and no words about its reasoning process. I sometimes describe it as like a gun that you can't aim. Right. And it's it's becoming more and more powerful.

11:10But you also are getting worse at aiming it. And that sounds really dangerous. A gun that you can't aim. Yeah, I think the thing to say is that the goal of our work is not really like it's hard. I think it's going to be difficult, at least today, for there to be something where you say like we did testing and this model is certified safe. I think that that's not really at least the goal of what we do. In a nutshell, do we have to have regulation, even if people like President Trump say the whole thing is a hoax? My goal is trying to make sure that there's at least one institution that has the technical capacity to do this kind of testing.

11:46I'm describing what Meter does in particular. But yeah, I think having more more actors that can do this kind of evaluation and inform the public debate, I think is really valuable. I mean, in terms of people listening to this, they'll be aware of the entire doom debate, basically. I mean, our impending doom potentially within the next 10 years as a result of uncontrollable AI. If the time is not now for regulation, we're going to lose it, aren't we? 10 years in some ways even sounds kind of far out where I think that we, I think very quickly here. What, you mean it could happen before that? it might be good here to be like a little bit specific about like, what is it that we think is going to happen?

12:31What might the world look like? One way of putting it is the AI systems at, you know, in the AI industry, it's common to talk about the idea that we might soon automate AI research or speed up AI research by having AI systems do the research itself, right? So sometimes this gets called recursive self-improvement. There's a bunch of different like terms that people use for this. But I think that that could mean that very quickly, we could have it be that like the AI systems are doing a lot of the work improving themselves. And then there's a question of that could mean that we get capabilities speeding up, I wouldn't say faster than they have in the past, but basically like a continued rate of very fast capabilities progress.

13:12If AI systems are then doing a lot of actions in the economy for us, Imagine that soon, it could be that soon we have robots that they can interface with or actuate in the real world. And then the question is kind of, if we have defective training processes when we have AI systems that are running large parts of the real world, things like they're running a lot of our cyber operations, actuating robots and cars or whatever in the real world, then the stakes for losing control of those systems could be much higher. And I think a world where we're that level of reliant on AI systems could come very quickly, like on the order of single digit years.

13:53Single digit years within the next five years, six, nine single digits. Yeah, that's what I mean. Yeah, I think that's possible. I'm saying that, yeah, I'm saying that I think large parts of the economy could be run by AI systems in that like in that number of years. Yes. That's Chris Painter from Meta talking to Rebecca Kesby. OpenAI says it was what it calls misaligned model activity. It says it proactively notified institutions, for example, the US Census Bureau and shared technical details to support the Bureau's own review. But Australian Prime Minister Anthony Albanese told it slightly differently in Australia's case.

14:32He told reporters he was disappointed it took the company way too long to contact his government. The incident against the Australian Health Service happened in June, but it wasn't until September 10th there was any notification at all, Mr Albanese said, and that was to a public mailbox.

14:57Coming up, how will Manchester City be punished for its alleged financial misdemeanours? demeanors. A former captain of the team says the glory cannot be taken away from the players. What we did, they cannot take it out of us. I mean, it's something that is not paid by money, you know, it's paid with effort, partnership, being shoulder to shoulder every day. I can only tell you that everything was deserved. I'm glad that I could live one of the best eras of the English football in here. But will lawyers agree? More on that story in 15 minutes. Our headlines, OpenAI says it's alerted dozens of global institutions and governments that their websites are made being accessed improperly by its artificial intelligence agents going rogue.

15:38A huge crowd has gathered in central Paris for an outdoor mass to be celebrated by Pope Leo Saturday afternoon, the second day of his trip to France. In five minutes, we'll hear from someone who met him this morning.

15:57You're listening to News Out with me, Sean Lay. South Africa's new ambassador to Washington is urging Donald Trump to end a refugee scheme for white Africaners like himself, who the President says are the victims of genocide. White South Africans now account for almost all the people granted refugee status by the United States. And the State Department says more than 17 ,000 more could be admitted next year. Here's our North America correspondent, Ionee Wells. In St Helena Bay on South Africa's west coast, the houses have an enviable view of long sand beaches, blue skies, a gentle swell of waves.

16:38The most important thing is your birth certificate. But this resident, S.Y. Devinaja, wants to pack it all up and, in his words, flee. South Africa is a terrible country to live in for everybody at this stage, not just the Afrikaners. But the reality is that there's a political drive towards killing white people and killing the farmers on the farms. And that movement is a political-driven agenda to wipe us out. Data shows most victims of farm violence in the country are black, not white. And farm attacks, while they do exist, only make up a small fraction of murders. It gets worse. And these are people that are officials, and they're saying that, kill the white farmer and take their land.

17:23The data hasn't stopped US President Donald Trump seizing on this narrative and claiming white South Africans are victims of genocide. He made this accusation to South Africa's President Cyril Ramaphosa at the White House last year and created a refugee scheme for white South Africans. We had three times, three separate occasions where people came into the house. Lili, not her real name, moved her family to Texas through the scheme a year ago, leaving their farm and lives behind in South Africa. Nothing was stolen. We weren't harmed. But there is friends of my husband, people that lived around us, that were not fortunate not to be harmed.

18:02They were brutally attacked with pangas and hammers, and that was very close to us. In the last 10 months, nearly 13 ,000 refugees were admitted to the US. All but three of them were South African. Larry Bartlett, the former director of the US's refugee admissions program, fears they're taking refugee places reserved for those fleeing persecution. It frankly would be laughable, except for the fact that we are now denying legitimate refugees an opportunity to come to the United States. And instead, what the Trump administration has chosen to do is to focus on a population of white people who, frankly, under U.S.

18:49law, unlikely qualify for refugee status. And they're taking scarce and precious places for people who have a real need to relocate, for people who have fled war, for people who fled ethnic cleansing is wrong. and I think is probably unlawful. From my perspective... South Africa's new ambassador in Washington, Rolf Mayer, negotiated an end to apartheid on behalf of the party responsible for it. Now he wants to negotiate an end to this scheme. The notion of a mass attack on the Afrikaner

19:29was misrepresented by those that might have conveyed the message. I mean, I'm an Afrikaner from South Africa myself. I can say that I've never experienced something of that kind. So the source is misrepresented, unfortunately, something that just doesn't exist. I made the point several times publicly that I would like to see the executive order being lifted. But here in Washington, D.C., pressure on the South African government is increasing. The US has announced visa restrictions targeting people it accuses of discriminating against South Africa's white minority. The number of those arriving here keeps growing, with some, like Lili, encouraging more to join her.

20:10My entire life's changed. What I call the American dream, OK, is we both have jobs. We live in a safe neighbourhood. This is home for me now. This is home for my family. Lili has no desire to go back, but it's not clear if a future government after Donald Trump would continue to recognise her status here. Ianie Nell Wells reporting. Now, Pope Leo has begun the second day of his visit to Paris. I'm just looking at pictures of him now as he prepares for an open air mass in the heart of the city. The sound is the crowds on either side of the Champs-Elysees roaring their approval as he drives slowly down this 1.9km long road at the heart of Paris in his Popemobile.

21:02Quite a sight there, crowds on all sides waving flags, dense crowds I should say on one of the most famous streets in Paris. Well, before this mass, he began the morning with a visit to a project which helps migrants integrate into France. Maison Bakhita provides an opportunity for people born abroad to meet French people socially, share meals with them and stage sporting events. Valérie Moyot works at Maison Bakhita, where she met the Pope ahead of the giant open air mass he'll be leading this afternoon. And I asked her what she thought when she met him. Oh, my impression is half emotional and half very much stimulated by what he said.

21:44And also by the attention that everybody in the Maison Vakita who was there to hear his words of encouraging, of course, encouragement, but also to see him, to see his presence. It's such a strong sign of interest for all the people that our associations accompany. You have a very mixed group of people there, don't you, from many different parts of the world who come and use the facilities there. What sort of people come to Maison Bakhita? To Maison Bakhita comes people who are exiled people, people from the exile who have food from their homelands, coming from countries of Africa, countries of Latin America, South America, from the Middle East, from Korea, from many, many countries.

22:40But what they have in common is that they don't know really what to do and where to go right now, not in terms of lodging or food, etc., because there are other associations and parishes who take care of these first needs. But the needs after that, which is, OK, I know where I am going to sleep tonight or where my child goes there. OK, fine. But me, what can I do in this country where I am not protected by the law yet? So they come to us and we just welcome them. Pope Leo came to visit you because the Archbishop of Paris apparently wanted the Pope to meet migrants and to meet people who are marginalised.

23:30What did he have to say to you when he was at the centre? What sort of remarks did he make? He made, when he entered there, he said that this world was a fraternal brotherhood for the humanity. And it was important for us to understand that. But I think he was very touched to be in that specific place. He knew in which place he was going. He said that, I would say, nobody is poor enough not to be able to give. And nobody is rich enough not to receive. So there is room for everyone. Is it important, do you think, for a message like this to be heard in France at this time from somebody as influential as the Pope is?

24:13I think it's always important to hear this message in France, in other countries certainly also, but in France. We'll have presidential elections next year. In France, where we leave difficulties, economical difficulties, where some of the population is getting poorer, and so forth. And what he says when he says that to the French people, this is my interpretation. He says, be aware you are not so poor as so economically in difficulties, not to open your heart and your mind to the other one who is lower, in a lower state of social welfare than you are. And together, you'll be richer. That was Valérie Mongeau.

25:03Back soon.

25:11John Deere is putting in work and putting up highlights. Oh man, look at those passes. Tillage, planting, harvesting. Three years in and these pre-owned machines are still at the top of their game. We're talking reliability, performance and long-term potential. And get this, these veteran machines can pick up the latest precision tech with ease. You know it, and fans ready to add John Deere certified pre-owned equipment and more options to their lineup can shop at machinefinder.com. Booyah! It felt like everything I believed in was suddenly collapsing. This is the story of an Islamic boarding school in Indonesia with a dark secret of the school's head teacher.

25:51This was someone I respected above anyone else. And the investigation led by students he's trying to silence. If this is leaked to the media, I'll be destroyed. This is World of Secrets, catching my teacher. Listen on BBC.com, the BBC app or wherever you get your BBC podcasts.