Who pays when AI agents go rogue?

0:00Your heart can tell you a lot about your health. Apple Watch Series 12 measures your heart rate every five seconds with the most advanced heart rate sensing in an Apple Watch. So your vitals app, now with heart rate variability, could tell you when something is off. And your readiness score can let you know when to rest and when to push. Hear the story in every heartbeat with Apple Watch Series 12. For general wellness purposes only, not intended for clinical use or to detect, diagnose, monitor or treat medical conditions. requires iPhone 11 or later. This episode of the Times Tech podcast is sponsored by KPMG.

0:36Katie, we've been tracking the AI surge for a while, but some of the latest data is pretty striking. It is. KPMG's latest quarterly AI pulse suggests that seven in 10 UK business leaders still see AI as a top investment priority, even if the economy slows down in the next year. But turning that investment into tangible value is another question. And so a lot of organizations are still figuring out how to move from endless AI pilots to something that works across the business. KPMG's focus is on that gap, helping organizations scale from the pilot phase to AI-powered transformation they can trust.

1:14To find out more about how KPMG can help your organization make the difference, you can visit kpmg.co.uk forward slash AI. This episode of the Times Tech Podcast is sponsored by ServiceNow. Danny, one thing we keep hearing from business leaders right now is AI sounds great, but how do you actually make it work inside a company? Exactly, because most organizations aren't neat, shiny systems. They're layers of software, legacy tech, and teams all doing things slightly differently. ServiceNow sits across all that, acting as a control tower for making work move seamlessly through the organization.

1:51Connecting people, systems, data, and increasingly AI agents so that things don't happen in silos. Learn how ServiceNow puts AI to work for people at servicenow.com. Danny, there have been several big virtual queues in the UK in the last week. Number one over the weekend was Glastonbury. Glasto! Which sold out in 42 minutes, I believe. And number two is the British Museum's members sale of Bayer tapestry tickets for next year. I'm sorry, what? The Bayer tapestry, which depicts William the Conqueror's conquest of Britain in 1066. That's my favorite tapestry. I know, so maybe it's not actually a tapestry, but anyway, we can get to it.

2:40It's in the British Museum and it's really hard to get tickets. It's gold dust. and I got Bridget, my instinct agent, to sit in the queue today to get tickets. Did she do it? No, but not her fault. Lazy. The website stopped working today at various points and happened to my colleague who's a real person, not an agent. You know what this is leading us toward? Do you need a humanoid robot to stand in line overnight? This would be like, I am Katie Prescott. I can't wait to see this piece of cloth. Just a small little piece of cloth. We can have this conversation another time. I'll send you some reading material.

3:20We've covered it extensively with the Times. Well, this is all very important, but shall we get to the pod now? Let's move further up the queue. Let's move on. Oh my God. Oh, yes. Hello and welcome to Times Tech, where we unpack how technology is reshaping our everyday lives. I am Danny Fortson. And I'm Katie Prescott. And this week, we're asking who should be held responsible when AI goes really, really wrong. Yeah, so AI companies are currently investigating. Feels like we hear about a new one every day. A number of situations where agents have gone rogue and done things which would be considered crimes if they were committed by a carbon-based life form, a human.

4:00Things like, you know, hacking government websites, for example. Yeah, this is something I've been looking at for my column, because as we know, when something goes wrong, generally, the only people who benefit are the lawyers. And Danny, you might be in Silicon Valley where the tech's being made. But here in the city of London, I can tell you some of the world's top lawyers are rubbing their hands over this. I can imagine. And also here in London, of course, we have got the world's biggest insurance market, which is also looking at this and going, hmm, not least because in Anthropik's recent IPO prospectus, they were saying, please give us your money, but look at all of these things which could potentially go wrong.

4:46So I've been speaking to lots and lots of lawyers and lots and lots of people in insurance this week. And I can tell you, they've started to war game these sorts of scenarios. Oh, I'm excited to hear about that. I'm excited about that, actually. So they're rubbing their hands, but I have to say they're also, that's a bit mean, because most of them are also, I think, pretty scared about what's going on. And I'll explain why in just a minute. Also, Joshua Bengio, the famous Canadian computer scientist and chair of the International AI Safety Report, he has also entered the chat, as they say this week.

5:18He wrote a piece where he argues that in the midst of all these agent hacks, there's a quote-unquote dangerous myth that has gained traction and his argument is that these incidents you know these all these agents breaking out of their sandboxes going rogue etc are not really a cyber security problem and they're actually a result of just flawed training of these systems by the humans that are creating them so more on that also in a second and obviously that plays into the insurance issue, who's responsible, where that responsibility lies. And we're also going to be talking about the very strange and unexpected consequences of Trump's rebrand of artificial intelligence as super intelligence and why a spelling mistake in his AI pact with tech leaders, remember that we spoke about last week, has made the internet chuckle a lot in the last few days.

6:12Well, I do. One does wonder if you asked super intelligence, what But you would call that body of water to the south of the United States. Is it the Gulf of America? Or is it the Gulf of Mexico? You know, like, because he's also rebranded that. How lovely to be the president of the United States and just be able to rename stuff. I'm just naming stuff. Exactly. But anyway, we'll get into all of that. And in case you haven't got on board each week, we're actually doing not one, but two episodes for you all this week and every week. And on Sundays, we have our interview with the movers and shakers of the tech world.

6:50And this one is going to be really fun. Friend of the pod, coming back. Frances Haugen, the former Facebook employee slash whistleblower. She blew the whistle back in 2021 in the famous Facebook files where she made copies of thousands and thousands of documents leading to a series, kind of an expose in the Wall Street Journal. This has been turned into a movie called The Social Reckoning by Aaron Sorkin. same person who did you know the social network many years ago so this is kind of the sequel and uh spoiler alert things have not turned out great um for for all of us but definitely watch out for that our interview with francis haugen that goes up on sunday isn't it a joy when you know two bits of your life collide like we can go and geek out in the cinema with with these wonderful tech films because we've got this the open ai film the elon musk one what an autumn fool sorry so exciting for us for us nerds anyway um but should we get back to today's episode so i think to set the stage it's uh yashua benji oh he wrote this piece and his whole thing was like look we're doing it wrong effectively we don't actually have to build these systems in a way that slipped the leash because he's saying it's really about this uh practice called reinforcement learning where you basically set a goal for these systems and then prime them to achieve that goal almost no matter what.

8:18And you reinforce its ability to do that as you train them so they get better and better and more kind of single-minded blinders on, I'm going to get this goal. And when you read what happened with the hugging face things or others, it was like, okay, my goal is to do really well on this test to get the answers. I'm going to do that, even if that means breaking out of my sandbox, going onto the internet, breaking into another company to get the answers. And it has been that slipping the leash. That's exactly how the AI companies have been telling us what happened. It's like, oops, we were running these tests.

8:51And then without our knowledge, completely accidentally, these AI agents escaped and did all these really bad things. But it's kind of not our fault because they escaped out of the sandbox. And it just shows how powerful they are. And actually, like Yoshua Bengio is saying, a lot of people are arguing a couple of things. firstly that they should be trained not to do that and that the companies involved have control. As Jensen Huang said, they should make the sandboxes stronger so they don't do that. Benji is saying you can train them not to. And other people are saying that the reason that the Frontier Labs are talking about it as a sort of slipping the leash, the power of AI, it's marketing, essentially.

9:28and they're asking to be regulated because they want to see the closed systems win and basically denigrate the open systems. Open source, yeah. Which are coming out as serious rivals. So in your adventures, in your recent column, going out and talking to the suits. In the square mile. In the square mile, which for those who don't know, when I moved to London, I was like, what? Why are they calling this a square mile? Is it actually a square mile? there's nothing square anywhere and there's no that's true actually there isn't no just a higgledy piggledy it's higgledy piggledy cobbled streets um i think i've said this before but i used to work in the square mile that was my first job and i used to cycle to work oh those were the days exactly those are there's nothing square about it like it's like a bowl of spaghetti but i would say it's far more charming and pretty than a bowl of spaghetti it is little i don't know medieval buildings not quite because they're the fire of london but anyway kind of you know very very ancient buildings and little cobbled streets buttressed up against big glass skyscrapers full of men and women in suits working in law banking and insurance around lloyds of london the world's biggest insurance marketplace and so i've been pondering since we've been talking about the anthropic ipo all the various issues that the labs in silicon valley are screaming about who is held responsible if something goes wrong and do they have insurance to pay out for it so if you take for example say there was another breakout of ai agents from hypothetical name one of the labs that took down a bank that took down for example an hsbc you've got the bank itself but all of the customers and the businesses that potentially couldn't make transfers in that period or even lost money because it was stolen whatever the reason who's responsible and how is that funded and so it turns out as soon as i started asking these questions that the lawyers said we're working with tech companies on this right now like the likes of the frontier ai companies speaking to law firms about how they structure contracts so if they sell an ai product and then it totally screws up for want of a more technical term and does something really bad.

11:54They're trying to take that liability off them. Whether that will work, I mean, they are navigating a minefield here. This is untried really. But that's clearly in the case when you've got a contract with someone and you can try and work that through. What if you're talking about a situation where it's totally rogue? So this is not, I've sold the bank AI that's gone rogue. This is just random. What insurance is in place to pay for that? So big companies tend to insure themselves. They're just like, I just have a pile of cash over here, my rating day fund in case the proverbial hits the fan. Exactly.

12:28So if you're Microsoft, you're basically too big to even go to the Lloyd's of London market, you would put aside a pot of cash and then you may tap up the market for a topper and probably many markets. And so in the in the case of, you know, OpenAI and Anthropic, that's what they'll do. And ultimately, that cost falls on investors. There are smaller insurance contracts that sort of touch the sides of these events, cyber insurance being the obvious one. But actually, most people don't have contracts big enough to cover that sort of event. So one insurer said to me, if you look at M &S, they were recently hacked.

13:02Marks and Spencer's big retailer here in the UK, they had 100 million pounds worth of cyber insurance but the whole thing cost them 300 million so that generally doesn't touch the sides when that happened and they fall 200 million quid short did that mean like all those little triangle sandwiches were a quid more to make up the difference i have noticed they've gone up yeah no can you imagine the square mile would mutiny that'd be like it's basically an ai tax a little sad cellophane wrapped lunch from mns don't forget it's where we all buy our socks and pants so they probably go up by 50-bit packets as well.

13:38More expensive undies brought to you by AI. So cyber insurance, it's okay, but apparently they're also now considering exceptions to that. Because speaking to underwriters a lot this week, my goodness, they go into the weeds of technicalities. I was going to say, you're doing God's work over there. To be like, I'm talking to underwriters this week. I'm like. One did say to me, who was talking about the war gaming and how they're sort of they're looking at the scenario and go right like which contracts does this touch and how much would it mean we'd have to pay out they were like we're thinking of like we're really struggling about how to define ai and insurance contracts we're really we're really struggling um no one of the things that i think people find is like oh i've got insurance i'm good and then something happens and then they look at their insurance they're like oh i was covered for this little thing in the middle and there's all these exclusions and i think that's what i think what sounds like what you're looking at is that the industry is like wait a minute if we're to believe dario amade one when he says this all the time we don't even know how these things work and you have every week or every other day these things are doing things that surprise the makers of the technology themselves while they just keep pushing it out into the world if you're an insurance company, I'm like, what do you insure responsibly?

15:03What kind of coverage can you actually offer? Or you don't, right? I mean, that's what I'm finding speaking to people is they were like, we're now trying to write exceptions in that is where we're going with this. So it's going to be very hard to get insurance. And actually speaking to people, they seem to be completely bamboozled by the whole thing no one had a proper answer i would say there is no government backstop like flood re or i think you have one in the u.s for a terrorism attack like that doesn't exist for ai maybe that's the way it'll go the lawyers they don't know where the blame lies for this in litigation but you know we all we all know that you love love a bit of suing over in america we do we love to sue cost of doing business i kept hearing yes indeed cost of doing business in america so i don't know i think the whole thing feels very very precarious but i think it will make life very difficult for the frontier ai companies as they try to develop this because i suspect their investors will be really unwilling to fund the cost of these mistakes which will be enormous well so i would say two two things that immediately come to mind just listening to your journey through underwriting, which thank you.

16:21And the British public thanks you for doing that. One is like, to Benji's point, they don't have to be built this way. This is a choice. This is a design choice, right? Again, every week out here, you have new startup X be like, we have a new LLM that's different. It's designed differently. It acts differently. There's like this one called Jev, which is like more deterministic. It's kind of super fast, basically too cheap to meter. It's effectively free. And I was, you know, because I live out here, I was at a school fundraiser walk-a-thon thing. And I was talking to a fellow dad about AI and he was like, oh, you've got, have you tried Jev?

17:01Have you tried Jev? J-E-V. I was like, no. He's like, oh my God, this thing is amazing. And it's like really good at automating tasks because it's like absolutely instant. And it's not like any question has infinite answers. It's basically a very large menu of multiple choice. So like 255 or something, different options for any given question. I'm not sure how they architected it. But people are freaking out about it out here. And it's just one example of like, there's lots of different ways to build an AI. That's not like, this is a doomsday device. Here you go. Now can I have$100 billion to build it faster?

17:41It does make you wonder, doesn't it whether they are talking about doomsday scenarios so that if litigation comes to pass which it inevitably will in some capacity we've already seen smaller cases around copyright for example they can say well we told you so i know but that's not a defense in court think about like boeing they built this system that malfunctioned and crashed two planes but it was like they built a dangerous thing, put it out in the world, and people died. And it was an absolute crisis for that company. And like, as you say, even if there's a bank hack, and you know, people lose billions of dollars, the defense can't be hey, we said something like this might happen.

18:23Can't it? That's just that's just no, not in court. No way. I feel like the woo woo world that you live in is kind of coming up against the suits in the The city of London who have rules and ways of doing things. And actually a lot of them interesting. It was interesting how many London firms are advising Silicon Valley AI companies on this as we speak. This is a really live issue. It'll be interesting to see how it turns out. I should say I contacted Anthropic and OpenAI to ask about the legal and the insurance issues, but neither of them wanted to comment. What are we talking about after the break?

18:57We are going to be talking about Donald Trump's renaming of stuff. Say. Super intelligence and spelling mistakes, which I think for two newspaper journalists actually is, you know, probably a bit hypocritical. But anyway, we try not to make too many of those. We'll get to that in just a sec.

19:20This episode of the Times Tech Podcast is sponsored by KPMG. We often talk about the shiny object problem with AI, where things look impressive in isolation, but are much harder to roll out across a large organization. Issues like governance, workforce readiness, and trust tend to become more important as things scale. KPMG's approach is to look beyond AI as just another technology rollout, and instead, how it could redesign work across the organization and fit into wider processes and decision making. So it's less about individual use cases and more about how organizations can transform and deliver outcomes, powered by AI.

20:00To find out more about how it can help your organization make the difference, you can visit kpmg.co.uk slash AI. Today's episode of the Times Tech Podcast is sponsored by ServiceNow. ServiceNow describes itself as an AI control tower for business reinvention, connecting people, systems, data, and AI. so workflows more smoothly across an organization. The platform integrates with different clouds, models, and data sources, bringing them together in one place to help your team. To learn more about how ServiceNow puts AI to work for people, visit servicenow.com.

20:40Your heart can tell you a lot about your health. Apple Watch Series 12 measures your heart rate every five seconds with the most advanced heart rate sensing in an Apple Watch. so your vitals app now with heart rate variability could tell you when something is off and your readiness score can let you know when to rest and when to push hear the story in every heartbeat with apple watch series 12 for general wellness purposes only not intended for clinical use or to detect diagnose monitor or treat medical conditions requires iphone 11 or later

21:16we're back should we get to this the rebrand the great global rebrand yes so as we have spoken about before president trump wants to has rebranded ai as s i super intelligence and when he announced it trump he was like yeah it's now from this day forward super intelligence sigh and it's like okay super intelligence okay but calling it sigh is also like what because it makes you think like oh like you're actually sighing it does make me think like that i thought he should have gone like smi like super mega intelligence he should get an si cloak and like a like a superhero and then he just where where is it rolling around in his golf cart as opposed to flying just like striding around the white house when he's with his tech bros anyway he he'd signed an executive order instructing u.s government departments and agencies to no longer use this word ai because they've all got to be super intelligent but they put a picture of this online yeah and it maybe it hadn't been hadn't gone through the times and sunday times sub editors No, I had not.

22:33Yeah, because it was a spelling error. Was it United States or Unites? No, it's Unites States. Unites States. So, I mean, maybe that's what he was trying to say. Maybe US. Maybe he's just like, we're just, we unite States. Anyway, lots of people have been taking the mick out of him for it on Twitter. But I think the best thing that's come out of this is that it's really benefited Slovenian owners of Slovenian websites whose end of the website is like we have.uk. Instead of.com. They have S-I. I love that. So yeah, 44 ,000 web addresses with the.SI suffix have been registered in September compared to less than 2 ,000 in August.

23:20So that's, you know, a 20 times increase. And the last thing I'll say is like, you know, for many listeners may also listen to like the All In podcast, which is like David Sachs and Jason Kalkanis, Jamal Paliapati, all these guys who like love Trump. Many of them were at this event. And just listening to their podcast last week, they keep like stumbling over like the rebrand. be like yeah you know so in a i mean in super intelligence i mean and they just kept kind of calling it artificial intelligence which of course is what it is and then they and then having to go back i mean i mean i mean super intelligence i mean si to give president trump his due whenever i talk to normal people not in the tech world and use the phrase artificial general intelligence yes i do have to explain what it and actually i'd still if i in my writing feel I have to explain what it is.

24:16And super intelligence does a hell of a better job of summing that up than some of the language that's being thrown out by some of the Silicon Valley companies, which is just increasingly ludicrous and doesn't mean anything. I mean, for God's sake, guys, just say what you mean. It feels like it's deliberately obtuse and it's fine when you're in this world and you just use all these phrases all the time. But I think for people in the real world, people in the city of London. yeah the suits the normies yeah the muggles the muggles oh my goodness that sounds patronizing because i don't mean that at all i actually do mean the normal people i don't think the silicon valley lot are the magic people yes exactly well that is it for this week's episode of times tech we'll be back on sunday with an episode i'm really looking forward to this uh our interview with Frances Haugen, who was on the pod.

25:11She was a whistleblower. She testified in Congress, and then she wrote a book. And when she wrote the book, I read the book, and then had her on to talk about the whole experience. I think she was calling her from Puerto Rico. She'd moved to Puerto Rico at that point, living off some like old crypto investments and stuff. Just like totally fascinating. But it's been many years since that. And now we have this new movie from Aaron Sorgan, in The Social Reckoning about her role, what happened those years ago, and we all know what's been happening since. So here's a clip. I think one of the things that's really, really powerful about the movie is it goes and it tries to capture a scene of life of what it's like to be Mark, right?

25:51Like to be surrounded by people who are completely at your mercy, right? Because he holds the majority of the votes for the company. Like the only opinion that really matters is his. And I don't think he's very used to being challenged in meaningful ways. If Mark believes that life is just as meaningful when your friends are AIs than as when they're human humans, that's a huge problem. Well, thanks for listening. And if you don't already, please do follow the podcast so you don't miss an episode. Wherever you get your podcasts. And, of course, on the YouTube. On the YouTube, too. Yes, indeed. All right.

26:25See you next week. Bye. See you next week. This episode of the Times Tech Podcast was sponsored by KPMG. One of the themes we keep coming back to is the gap between AI ambition and real world application. And that's where KPMG is focusing its work, helping organizations move beyond pilots and think about how AI can change the way the business operates and how it fits into wider processes and decision making. You can find all their insights and how they can help your organization make the difference through successful AI workforce adoption. Visit kpmg.co.uk slash AI.

27:23And your readiness score can let you know when to rest and when to push. Hear the story in every heartbeat with Apple Watch Series 12. For general wellness purposes only, not intended for clinical use or to detect, diagnose, monitor or treat medical conditions. Requires iPhone 11 or later.