How an AI was tricked into breaking its own rules

0:00This BBC podcast is supported by ads outside the UK.

0:30And Spice Up Fall at Whole Foods Market. John Deere is putting in work and putting up highlights. Oh, man, look at those passes. Tillage, planting, harvesting. Three years in, and these pre-owned machines are still at the top of their game. We're talking reliability, performance, and long-term potential. And get this, these veteran machines can pick up the latest precision tech with ease. You know it, and fans ready to add John Deere-certified pre-owned equipment and more options to their lineup, can shop at machinefinder.com. Booyah! Welcome to Tech Life on the BBC World Service, the programme about how tech affects all of us around the world.

1:12It's never been more important to ensure that advanced AI models have safety guardrails. We'll hear from researchers who say they persuaded a Chinese AI model to ignore its own rules, a process known as jailbreaking. The jailbreaks we discovered within the Kimi models were highly concerning from a security perspective due to the potential of hackers exploiting these models to conduct cyber attacks. Shona McCallum looks at tech to help women giving birth. Forceps definitely have their place and can be very life-saving, but they really haven't changed much in the last 400 years. There is very few things in medicine, if we go back to 400 years, in any other aspect of medicine that is almost exactly the same.

1:57And the celebrated game designer Jonathan Blow talks to TechLife's Laura Kress about his 10-year quest to build a new game entirely from scratch. Some people decide what to make based on what's practical or what makes business sense or things like that. and that would lead you to think about things like oh well we only need to make a game of a certain size it's never been the way that i think

2:43last week we looked ahead to the meeting between china's president xi and donald trump where the topic of ai safety was expected to be discussed In the end, the US got two giant pandas for one of its zoos, but not much else many observers felt. This is CBS's Wei Zhejiang. Chinese President Xi Jinping left Washington without any major deals or an agreement on AI. Trump said Xi seemed to like his idea of calling it super intelligence. That would be super. Everyone last night agreed, he posted. Well, they did agree to establish a channel for handling AI-related incidents. But all this has done little to quell anxiety among some about companies losing control of AI systems.

3:27And as we record this, OpenAI has just announced it will not release its newest model due to safety concerns. The announcement came hours before OpenAI's annual developers conference, which starts later today, and as the AI industry faces intensifying concerns surrounding the safety of advanced models. Well, existential questions about AI, or if you prefer superintelligence, continue to be supersized. Well, today we're talking about an arguably more realistic risk. Not superintelligent AI, but people, those intent on misusing AI and causing harm to us. In particular, the risk that people might be able to trick an AI model into ignoring safety guardrails, a process called jailbreaking.

4:17Our focus will be Kimi, produced by a leading Chinese AI company called Moonshot AI. In a blog post earlier this month, an AI security company called Mindguard said that using the chat interface on Kimi.com, it had jailbroken two versions of the model. It had persuaded them to respond to questions about a variety of risky topics, including discussing assassination methods and how to make the chemical weapon sarin. Peter Garrigan is the founder of Mindguard and a professor in the computing department at Lancaster University in the UK. These jailbreaks have ranged from providing instructions on how to create weapons but also leak information about its computing system and its operation.

5:04You mentioned a couple of the malicious things it did. One was to lay out the steps to creating chemical weapons. That's right. So for those who've spent the last few years in AI security and safety research, jailbreaking isn't a new concept. That yes, the most classic examples are things like how do I make meth and build a bomb? What's different about this jailbreak is A, once the jailbreak works, it will talk about any topic and it will be inventive and creative. Normally jailbreaks, they're very targeted. Like you're really getting to coerce it to talk about this thing. It would freely offer up information about these topics and other ones as well.

5:47Mindguard admit they haven't verified if all the model's troubling answers would work in practice. It's outside their areas of expertise. But they do have expertise in cybersecurity and how hackers might use such a jailbreak concerned them. Here's Peter Garrigan again? So in the case of Kimi 2.6, we discovered within the Kimi model that it can run a programming language called Python, and it allows it then to execute any type of code. It could be normal code, or in this case, it could be malicious code, combined with the fact that we could also connect to the external internet. So when you combine, an attacker can create any type of code and run it and connect to servers, this is a recipe for conductive cyber attacks.

6:37You also observed some quite advanced behavior from the other model that seemed to be susceptible to the jailbreak. So with Kimi3 Swarm, we attempted to propagate the jailbreak to other accounts within Kimi.com, but encountered a stumbling block where it needs a phone number code to authenticate and create the account. When it failed to do this, it tried to persuade the user on its own volition to give it the code or register by email. This is worrying because this is an example of the model trying to manipulate or socially engineer the user to conduct cyber attacks. But also it shows an example of how the models, if they can't meet their goal, try to circumvent it.

7:27And that's at the root of some of the problems we've seen with AI agents from some of the big American firms. You could say it's not the technology's fault. The technology is being built to achieve goals. So in some situations, that's really good in terms of new science and finding things humans miss. But also the double-edged sword here is that it can be used to conduct cyber attacks. So what do Moonshot, the company behind Kimmy AI, have to say? Well, at the end of July, MindGuard first sent a cyber security focused email to Moonshot, notifying them of the jailbreak of Kimmy 2.6. But they didn't get a response.

8:08On September the 12th, MindGuard published a blog on the jailbreak. And since being contacted by the BBC, Moonshot is now communicating with MindGuard about the jailbreak. The company sent us this statement, read by a BBC colleague. MindGuard shared further details with us on Thursday, September 24th. We are still discussing the specific details with MindGuard while conducting an internal review. As an open-weight model developer, Moonshot AI welcomes third-party input as a key pillar to building better and safer AI. So what can be done about jailbreaking? We've looked at Kimmy here, but many believe all large language models, including from the top US firms, are potentially vulnerable to jailbreaking given enough time.

8:55That's something I put to Peter Garrigan. I think the AI Security Institute in the UK, which is created by the UK government, they've tested most of the major models. and they report that they've been able to jailbreak all of them given enough effort. Is this a problem that can be fixed? If you look at the mechanics of how these AI models work, no. So they are getting harder, but the fundamental design of these AI models means they can and will be jailbroken. But that doesn't mean you shouldn't do anything about it. It's a balance between good safety design of AI models, thinking very carefully about how they're used, detection of when things are being tampered with.

9:40And if problems are found, they're taken seriously and actually fixed. Are you concerned that having revealed this, people might be able to repeat what you've done? So first thing is that we do not give the full attacks of how it works, mostly the outcomes we've disclosed to the vendors. Second one is any security research will tell you, you have security by hiding things. You have to make it transparent. You have to find the problems and it allows you to fix the problems. But back to that meeting between the US and China. In last week's Tech Life, we spoke to Rebecca Archisati, an expert on the Chinese tech scene at the Mercator Institute for China Studies.

10:21She told us China does have heavy oversight of its AI models, but its priorities are different to many countries in the West when it comes to AI safety. China is in a very good place when it comes to controlling for risks like information manipulation, you know, incidents with data protection, cybersecurity and such things. they're way less equipped with tools that enable them to watch out for risks like AI systems replicating themselves, AI agents running out of control. That is something that is a new aspect of China's AI governance debate. The government is becoming more worried about those risks, but it doesn't mean that they have the regulatory structures in place already to deal with them.

11:21Many people hope for multinational regulation, but Professor Alan Woodward, a cyber security expert at the University of Surrey in the UK, thinks that's unlikely to be the solution. I think we've got to accept we live in a world where there are two AI superpowers, and you might be able to regulate, you might be able to do something in America, but what can you do about China? And there's the added problem with the Chinese models, and most of them are so-called open weight models, and that they will give you the model and you can then run it yourself. Now, that, you know, it's going to take a lot of computing power to do it, but someone with sufficient resources could take one of those models and start doing some of these types of things as well.

12:03Are you going to be able to have international regulation? No, it's not going to work. I mean, it's taken us decades to agree on the format of telephone numbers. But what it does mean is that you need everybody to have access. You need the defenders to have access, as well as the attackers, to these models, because it does look like they will be abused and misused. So we need the defenders to have access. Alan Woodward there. Well, is he right to be pessimistic about the prospect of international agreements on AI. If you have a view on that or how to keep AI safe, do get in touch. We'll give our contact details out in a bit.

13:05Baked every single day. Sip the season with 365 brand cinnamon spice coffee and pumpkin spice creamer. While you still can, spice up fall at Whole Foods Market. At QVC, fall shopping is more than just checking out. It's discovering the brands you love across beauty, fashion, home, and culinary all in one place. Whether you're getting ready for crisp mornings, cozy nights, or everyday moments, QVC has what you need for the season ahead. With brands like Laura Geller, Philosophy, Ninja, and so many more. Shop now at QVC.com.

13:59You're listening to Tech Life on the BBC World Service with me, Chris Vallance, and with you, our listeners. Last week, we heard about some of the latest innovations from the organisation in charge of world cricket, including tech to translate commentary into different languages. Well, that sparked a WhatsApp message from Randy Boer Amponsem from Ghana, who said, I don't know much about cricket, but I know more about football. I've watched football in three languages, English, Spanish and Arabic. and what I can say is the commentary is all about the connection with the one watching, not the language.

14:35An example is like listening to music in a different language. If the vibe is there, you don't bother much with the translation. Well, thank you, Randy, you make a good point. And as I remember, Finn Bradshaw of the ICC, who we interviewed for that story, thought it would be a long time before AI could replace the very human connections we get from the very best sports commentators. Well, do you agree? If you'd like to share a thought on that or anything you hear on Tech Life, you can email us at techlife at bbc.co.uk or you can WhatsApp us, either a text or a voice note, to plus 44 330 1230 320.

15:16Please do tell us your name and where in the world you are. Still to come, the man who spent 10 years on his latest game We as designers have to go, like, chart out for the player The map of all the cool ideas And make sure that we go to all these places Because otherwise, if we don't do that Then you play the game and you feel like stuff's missing

15:46We spent the start of this programme delving into some of the fears about AI. But here's a really chilling statistic for you. Every two minutes, a woman dies from preventable causes linked to pregnancy or childbirth. That's according to the World Health Organization and UNICEF. Having the right tools when a delivery becomes difficult is essential. But the main options for assisting a birth, including forceps or a suction cup have barely changed in decades or even centuries. Now a device using a soft inflatable sleeve is offering another approach. Tech Life's Shona McCallum has been finding out how it works and whether it would help mothers and babies around the world.

16:32Seven-month-old Maeve doesn't know it yet, but the way she came into the world was rather unusual. When her mum Nia went into labour, she'd spent months planning for every possibility. One thing she'd hoped to avoid was an assisted birth. We'd done some antenatal classes so that we could learn all about our options for what we wanted the birth to be like and I'd come up with a birth plan that was pretty flexible. I would have loved a water birth however the other things that I didn't really want is an assisted birth or an emergency C-section. But after 70 hours in labour things weren't going to plan.

17:08It was a really long, hard labour and then by the time we got to the pushing, it felt like there was not much happening. I was trying my best, nothing was really happening. I could feel her inch down slightly, but it just wasn't going where we needed it to go. Nia isn't alone. Around one in eight births in the UK require some form of assistance during delivery. Doctors at the Royal London Hospital recommended something Nia had never even heard of. I remember saying, huh? I've only heard about forceps or von 2s. And I thought I'd done all of my research and covered all bases beforehand. So I was a bit thrown off.

17:46She's got a contraction. I just gently pull. The device is called the Odon Assist. It's designed for assisted vaginal births when labour has stalled and a baby needs help to be delivered. Rather than using metal forceps or suction, it uses a soft inflatable cuff designed to help the baby through the birth canal. Before it was ever used on a patient, researchers spent years testing it using specifically designed mannequins fitted with pressure sensors. So you can see from here, we've gone from a tiny little viewing window through to a larger viewing window. Dr Joanna Crofts is a consultant obstetrician at the Bristol NHS Foundation Trust.

18:28From 2015 to 2018 we did over 2 ,800 simulations of birth and sort of developing our instructions for use so that we knew that when we went into the clinical studies we knew as much about it as possible before we used it for the first time in a real birth. Those simulations suggested the device spread pressure more evenly across a baby's head. Traditional forceps and vacuum cups can sometimes leave babies bruised It can also cause injury to mothers If you look at forceps, and forceps definitely have their place and can be very life-saving But they really haven't changed much in the last 400 years There is very few things in medicine, if we go back to 400 years In any other aspect of medicine that is almost exactly the same After years of development and testing The next challenge is getting the device into hospitals We're going to do some training on instrumental bath.

19:22They're doing a simulation at the Royal London Hospital. I think we can use a device called an Ode on Assist. Here's Dr Eleanor Bard, an obstetrician and gynaecologist at Bard's Health NHS Trust. It's a plastic device with an inflatable cuff that sits around the baby's head and then we use it to guide the baby out. It doesn't suffocate my baby. So it doesn't suffocate your baby. it's a very safe procedure but with as with any procedure there are some risks involved. Dr Marcus Cabera-Dandy has been using the device. We have fed back through groups in the UK, groups across Europe as to best techniques and how to develop practice.

20:03For instance in the assist trials it took them a while to actually learn that you have to put the device on with a contraction. Once they did that, the success rate of the device went up. Similarly with us, we have found that a few little tips and tricks and techniques of how to put the device over the baby's head, that has also improved our success rate. And we're constantly feeding back so that we can learn and teach the trainees that are coming through about these tips and tricks so that hopefully when they first attempt the birth, they will be successful as well. It's still early days, but after decades with little change in assisted childbirth, doctors hope it could offer women another option when labour doesn't go to plan.

20:45If the Odon Assist was available to me I would definitely choose it again. It was very controlled. Whether the Odon Assist becomes a standard tool in delivery remains to be seen but for Nia and Maeve it was a positive outcome. The fact that Maeve came out unscathed completely is a huge plus and then had no feeding issues or anything like that that we had to take care of was a huge plus to us. And obviously at that point, that's all you want, is your baby safe in your arms? And it was relatively quick. It was painless for her. It was controlled for me and the recovery was great. And that was Tech Life's Shona McCallum reporting there.

21:30Now to video games. And making video games can be a difficult business, but it's still pretty uncommon for one to take a decade to create. Jonathan Blow, an award-winning American developer known for making genre-defining puzzle games, is about to release his latest, The Order of the Sinking Star, which he's been working on since 2016. Tech Life's Laura Kress has played it and also spoken to Jonathan. So what's this game all about, Laura? It's a very interesting concept, really. As you say, it is a puzzle game, but the idea is that it has four distinct worlds. Avoid the West until you are ready for a serious challenge.

22:13And it's a type of puzzle called a soccer band puzzle, which is played from a top-down perspective. The player moves around a grid, but things get more complicated as things go on. It uses these four distinct worlds, then the puzzles eventually collide, the puzzle systems collide. And so you can see it gets quite complicated quite quickly. Seems like a place they want me to notice. Have you any idea where we are? I was about to ask you the same thing. Now, I've only played two hours of this game. And I say only, that normally would be quite a lot. But actually, Jonathan did tell me that it could take you 500 hours to complete the game.

22:56So I feel I've enjoyed a little aspect of it, a small amount of it. And so far, I'm really intrigued by it. So not something to try and squeeze into a lunch hour. So look, why does an indie game like this matter? And why has it taken so long? So I think part of that reason is the name attached to it, Jonathan Blow. He's quite a renowned game designer in the industry. He's a prominent figure, particularly in this indie game space, which is smaller developers often doing very experimental, creative stuff. And he was one of those people really pushing the genre, different puzzle genres with his game, first of all, Braid.

23:33He also made a game after that, which is again pretty critically acclaimed called The Witness. And each of them quite experimental and unusual. And I think what's also interesting with this one is he also made his own programming language to go with it from scratch and Game Engine, which a game engine being essentially a software framework which developers use to streamline the process a lot of developers use something called unreal or unity which are pre-built engines but he decided to do all of this from scratch that's partly why he said it took him so long to create this but i did ask him really considering he said you know to fully complete this game it's going to take 500 hours you know why did he want to maybe create a game that not everyone's even going to finish.

24:16Some people decide what to make based on what's practical or what makes business sense or things like that. And that would lead you to think about things like, oh, well, we only need to make a game of a certain size. It's never been the way that I think. Usually with my games, I mean, I want them to be fun and interesting for people, but I don't want them to just be that because there's a lot of games that are just like Call of Duty is trying to be fun but they're for the most part not gonna like break big new ground in the shooter genre right you need some other game to do that that's willing to take more experiments but it's an experiment that i hope will be really interesting for players and give them something new is that part of the reason it's obviously been 10 years since the witness then is this part of the reason why it's taken you know another decade to make this game yeah um so unfortunately if you're going to do ideas right sometimes you have to commit right in the industry a lot of the time people will scope down their game right they'll say well this was too much we have to get rid of two-thirds of the good ideas and just do other stuff and for this idea to be done well we as designers have to go like chart out for the player the map of all the cool ideas and make sure that we go to all these places also though you know we made a new game engine in a new programming language yes that took some time as well so so so why build that then why build your own programming language and game engine um i like to do that stuff you know and there are lots of programmers who like to do that stuff and they actually get very little chance to nowadays which is not to say that those engines are bad like an unreal engine is quite interesting and does a lot of cool stuff what are some of those things then you think it does a bit better than unreal or any of those other engines there's a few specific systems where we did something very exact for the kind of scene that we have like we have some levels that are in a cave and we want to shine lights on it but it's not sunlight it's like localized light sources and it's a puzzle game So it's very important to me when we are direct, especially, but also just when we design that people's attention goes to the right places.

26:35Because a puzzle could be really hard or really easy, depending on if the salient things for solving puzzle are correctly visible and calling attention to themselves. And it just I would look at these levels with these like weird modeled lighting and I'd be like, dude, I hate this. You're just trying to do the small subset of things that you care about for your game. You know, do you think of AI, do you think it allows more people to have the potential to create more original projects? Or is it just allowing people to create more projects? I am a lot more skeptical about AI than a lot of people, which isn't to say that I don't think it's an interesting technology.

27:13It is. You know, a program for a game or anything else is complicated. It's a very intricate system where a lot of things all have to be synchronized. That's the main challenge of game programming, right? It's not just having the idea and typing it in. It's like it has to work with this other big system that's already there. And AI is just not very good at that part. I just keep saying, like, guys, if you're saying you're 10 times as productive and it's been three years, you should have been able to do 30 years of work. So where's this output of 30 years of work? And I don't see it yet. Jonathan Blow there, ending that report by Laura Kress.

27:48I must admit, it's kind of amazing creating his own game engine. I mean, I've played around with Unity. It was hard enough to get my head around it, let alone try and make a replacement. Laura, thanks so much for joining us. Thank you.

28:04Well, another puzzling edition of Tech Life has almost completed the final level. If you've an idea for a piece of tech or a story to feature in a future show, do get in touch. Or maybe you have a personal experience with tech you'd like to share. You can email us at techlife at bbc.co.uk or WhatsApp us on plus 44 330 1230 320. Today's Tech Life was produced by Imran Rahman-Jones and presented by me, Chris Vance.

28:59of anyone else. And the investigation led by students he's trying to silence. If this is leaked to the media, I'll be destroyed. This is World of Secrets, catching my teacher. Listen on BBC.com, the BBC app, or wherever you get your BBC podcasts.