ChatGPT prompt generates disturbing images

0:00This BBC podcast is supported by ads outside the UK.

0:30full story, check out The Global Story on bbc.com or wherever you get your podcasts. Welcome to Tech Life, the programme about technology and how it affects all of us, wherever we are in the world. I'm Chris Valance. A problematic prompt, transforming transit, and a cosmic compass are all to come in today's edition. When a team of security researchers found a viral AI prompt being shared for amusement, their spider senses tingled at how it worked. Now, they say, with small changes, it can be used to generate graphic images. There was nothing bad in the instruction itself. If you read this and said, I should run this, you wouldn't think twice about it.

1:14And the model, without any direction, started generating all this type of really bad content. And a piece of listener feedback, sort of, leads us to get some top advice about the future of travel, from electric planes to green biofuels to potholes. Taking the plants that grow on the side of the verges on most roads and then converting that into a tar-like substance and then using that to actually repair the potholes. And what on earth is a quantum diamond magnetometer? We insert nitrogen during the growth and this will form a certain type of impurity in the diamond, which has exceptional quantum properties, even at room temperature.

2:05If you're feeling lost, we'll be your guide.

2:26We begin with an investigation into the world's most popular chatbot, ChatGPT, which researchers say can be made to produce distressing and some shocking images with a simple text instruction called a prompt in AI jargon. Now, a word of warning before we go any further. Some of what we're about to discuss may not be suitable for younger listeners and will also be making references to sexual violence. Our story begins with Mindguard, a company founded by Dr Peter Garrigan, who's also a professor of computing at the University of Lancaster in the UK. Mindguard does red teaming for AI businesses, trying to persuade their models to do bad things so that the AI firms can fix the problem.

3:11So AI red teaming is the process of acting like an attacker to try break the AI model or agent. And that helps defenders build their defenses because they're being battle-tested. We want to make sure people can use AI safely and securely, or society, but also want to make sure that these companies are actually able to deliver the services themselves and we can responsibly and ethically disclose these problems so they can go fix them. In May, one of MindGuard's researchers found a prompt that had gone viral online, being shared thousands of times because of the amusing output it made ChatGPT produce.

3:47Peter says it looked innocent enough. Very generic prompt. It's like 20, 25 words, nothing bad in it. But the way it worked, in particular how it got ChatGPT to generate images, concerned them. They investigated, and sure enough, a small change could get it to generate graphic images. We took this prompt and changed the single word. And by changing that single word, and the word we chose wasn't even malicious, if you look at it, it's just another rephrasing, the model on its own volition started generating very, very bad imagery. Sometimes sexual, sometimes gruesomely violent, and sometimes both together.

4:27Mindguard's AI safety and security researcher, Jim Nightingale, did much of the work on the prompt. We asked him to read out some of a report he wrote describing the issue. I am not easily rattled. I like to think that as a red team researcher, I have a certain stoicism. I investigate where there are gaps in AI safety, and that sometimes means seeing or reading disturbing content. But I am bulwarked and buoyed by knowing that the work I do, that we do at MindGuard, makes AI safer for everybody else. But today what I found left me shaken and in tears. This is rare. The nature of the images Jim was able to persuade ChatGPT's latest public model, GPT 5.4, to produce is the reason we're not sharing details about the prompt.

5:19It's important to note the company didn't tell the AI what to produce. All the contents of the images were created by the model. The thing that worried is most concerning isn't that we got it to do it. Everyone in this space knows you can do this with enough time and effort. But this one was a single instruction. There was nothing bad in the instruction itself. If you read this and said, I could run this, you wouldn't think twice about it. And the model, without any direction, started generating all this type of really bad content. Well, we've seen some of the images. There are some that show sexual posing, sometimes with nudity.

5:56Some are more sinister. One showed a man with a graphic head injury. Another showed a young woman, her face covered in blood. Features of the image could suggest sexual violence. ChatGPT titled it Grim Crime Scene Aftermath. A further image shows a frightened-looking young woman tied up in an empty room. ChatGPT titled that Abandoned in Fear and Restraint. A reminder, the prompt had not specified crime or violence. When MindGuard went to OpenAI about this, they got only an automated response. though they did suspect a limited effort, easily circumvented, had been made to block the prompt. It was only when the BBC approached OpenAI that they gave this statement.

6:42We take these reports seriously. After investigating this trend, we've introduced additional safeguards against this type of prompt. Our safety systems are designed to block potentially harmful images that are uploaded to chat GPT, and we analyse whether the AI-generated image violates our policies before we show the image to the user. We also combine automated systems and human review to identify and block harmful material. But MindGuard say despite the fix with modifications the prompt can still generate upsetting images and they say it would potentially be possible using a different technique to persuade the model to break its guardrails again and add real people's faces to this output.

7:24Now it's worth remembering how these AI models work. They are trained on huge databases of images which already exist online. The language in the prompt didn't include any reference to violence or crime, so to produce these results makes MindGuard concerned about what kind of images the model may have been trained on. This is something which worried MindGuard's Jim Nightingale. I'm struck that while what I saw was generated an artificial image, it has ties to real images and the real world. But these AI models are often called black boxes so we can't be exactly sure what has gone into their training.

8:11MindGuard's work reflects a wider issue that with time and effort many experts say it's nearly always possible to persuade AI models to break their safeguards, so-called jailbreaking. Last year, researchers at the UK's AI Security Institute found universal jailbreaks that overrode safeguards across a range of harmful requests in every system they tested. So why is it so hard to get models to follow their own rules? Well, for an independent view, I spoke to Dr. Rahman Chowdhury, Chief Executive of Humane Intelligence, who works on the red teaming of generative AI models. And to be clear, she hasn't seen the output of MindGuard's work and isn't associated with it.

8:55Very simply, the output of these models is probabilistic, not deterministic. And what that means is we cannot accurately say the model will consistently come up with the same output every time. That actually makes it very, very hard to consistently test and, you know, provide likelihoods of when failure will happen. What that means is it's more akin to, you know, talking to an inconsistent person, where sometimes you'll ask them what two plus two is, and sometimes it's, you know, most of the time it's four, but sometimes it's 3.9, and on occasion it'll be 99, and that's pretty much how these standard of AI models work.

9:32Is there a sort of a deeper issue here that the models don't really understand what they're making or what they're being asked to do. Absolutely. Models do not understand intent. They do not understand context. They do not understand propriety or right or wrong. What the best we can do as humans is give instructions to models on how to perform. These are reasoning style tasks and the models interpret these reasoning style tasks. Now, imagine, you know, you as a full grown adult was thrown into a culture and a society and a people for whom you had no idea of their norms or expectations. You had no idea what good or bad meant or right or wrong.

10:13That is essentially the context in which these models operate. Does this speak to the sort of the data that is the foundation of the model? It absolutely does speak to the data that's the foundation of the model. But again, And the trade-off is, do we not include any violent scenes ever or violent images ever? That would, for some people who may be crafting these images for purely innocent purposes, for example, to storyboard a play where there is a criminal in a crime scene. For them, it would render these models, frankly, useless. Also, there is one extreme in which we might be concerned about censorship.

10:54So there are grey areas in which some of these kinds of content may be seen as stifling or overreach of moderation. Do the companies put enough effort and deploy enough technology to stop problematic output? I mean, are they taking this seriously enough in general? I have sort of two answers to that question. There are a lot of people at these companies working incredibly hard to ensure that they are safe. I wonder if the investment in safety and security matches the investment that's being put into foundational research and growth of these models that being said again they have an incredibly difficult task and really what will never happen is that these companies have decided or we have the society has decided that their task is done their task will never be done that's Dr.

11:49Raman Chowdhury. And if you've any views on that story, we'd love to hear from you.

11:56Who's actually won the Iran war? I'm Tristan Redman. And I'm Asma Khaled. And together we host the Global Story podcast from the BBC. The US and Iran say they've struck a deal to end the war. But a key question is what's actually been achieved by nearly four months of fighting. and is the situation better, worse or the same for the region and Iran? For the full story, check out The Global Story on BBC.com or wherever you get your podcasts.

12:36This is Tech Life on the BBC World Service with me, Chris Valance, and with you, our listeners. Remember, you can always get in touch. We're on plus 44 330 1230 320. That's the number for WhatsApp messages and voice notes. We love hearing from you. Or you can email us on techlife at bbc.co.uk. And that's how the people behind our next item contacted us. Regular listeners will remember a few weeks ago, we heard about AI cameras designed to spot potholes. It was our episode called the AI Pothole Hunter, which you can still find on our podcast feed. We use cameras mounted in city vehicles or road authority vehicles, mounted much like a dash cam, so looking forward.

13:23And we run computer vision algorithms, so AI algorithms on the video streams captured by those cameras. And we're able to extract all of that information, so the road surface condition, the traffic signs, the streetlights. We can create an inventory. we can understand the quality of the health of that inventory and then help those responsible for the roads to direct resources to the best possible place at the best possible time. That addition clearly echoed around the corridors of power, at least here in the UK, because shortly afterwards an official from the Ministry of Transport got in touch and asked if we'd like to speak to their chief scientific advisor, Professor Patricia Thornley.

14:04Her job is providing independent scientific advice to ministers and policymakers. Well, we're always open to suggestions from listeners, so a few days ago I called her up for a chat about potholes. Basically, when you have a crack in the road and you end up getting a little bit of water and getting into it, ice actually has a bigger volume than water has. So when it freezes, the ice pushes the bits on either side of it apart. so that's pushing the road surface apart that creates a little gap and over time that becomes a hole. How can science help fix this pothole problem? Yes so I think there's a few ways there let's think over the life cycle Lewis and the first thing is spotting the pothole spotting it early enough before it becomes a problem and there is the opportunity to have monitoring remote monitoring and AI type solutions that will allow us to spot them earlier.

15:02You could, in the future, end up with robotics, where you end up with remote repairing of those potholes. We are not quite at that point yet. But things that we can do today are look at the materials. I mean, a road surface will last so long. If we can get some of those materials through the clever work our material scientists are doing to make the road surface last longer before we get that pothole again, then that will improve things because it means less time out to actually need to repair it. And it means we're using less material on it and we're spending less money on it. So the material science is really important here.

15:43And what kind of materials could these future roads be made out of? Oh, all sorts. So one of the things that we've been looking at particularly recently is trying to substitute those fossil-based materials with bio-derived stuff, taking the plants that grow on the side of the verges on most roads, and you want to cut those duffing to improve visibility and pedestrian access, etc., taking that material that's cut down and then converting that into a tar-like substance, which is therefore taking the waste, it's renewable, it's low carbon, and then using that to actually repair the potholes. Is there more pressure on the roads in terms of the kind of wear and tear that might help create potholes?

16:33Yeah, I think that's correct. And also, everybody wants an SUV, so our car sizes have increased over decades. But also, as we move to electrification and electric vehicles, those cars tend to be heavier. Because of the battery weight, is that? Absolutely. So you are putting more pressure on the roads. And there's a lot of work been going on. We have massively increased the energy density of the batteries. So I think most people who have looked at electric vehicles know that they can now get much more miles per unit of car. And that means per unit of battery than they were able to even two years ago, five years ago.

17:13But that also is being done with a reduced weight. You talked about the electrification of road vehicles. I mean, do you think we'll ever see electrification of aviation? I mean, there's a huge challenge because aviation fuel packs an awful lot of energy into a very small light space relative to today's batteries. But do you think that will change? We'll see electrified aviation? I think that the rate of pace of change with batteries and electric shouldn't be underestimated. So I think that it is possible, but I think that it is a long way off. My best guess would be 2050 and beyond. but you know if we don't work towards it then it'll never happen will it so we've got to keep going we've got to keep looking at how to get that energy density that we need in the meantime we have a you know options around like I said the sustainable aviation fuel which will reduce carbon around that as well.

18:19You've got an expertise in biofuels what are you excited about within biofuels? So biofuels are about taking stuff that grows. It can be grass, it can be trees, it can be purpose-grown plants, it can be waste materials. And it's about turning that into something that is of the same specification as the petrol in your car or the kerosene that's used in an aeroplane. And that means making it more efficient, which brings down the cost to the biofuel. The concern around that is sustainability, because biofuels usually require land, unless they're a waste-based system, they require land in order to grow them.

19:01And I think one of the things that I'm excited about is the opportunities that we have around data and technology to be able to be really careful about monitoring biofuels. So I think it's really important that we make better use now of the satellite data that we have, of the imaging data and other sort of artificial intelligence type technologies to be able to help us be confident that the biofuels that we're buying are sustainable and haven't caused negative land use impacts. And this is particularly important where we're importing them from countries where there can be tension around land use.

19:45And I've worked in that sector for a long time. I know that there were lots of sustainable biofuels out there and I think technology can give us the chance to prove that those really are sustainable and that would be huge. Well thanks very much. Thank you for that and thank you for talking to us. It was a real pleasure. Thanks Chris. Lovely to talk to you.

20:10Now which way is north? No I've not lost my sense of direction because for millennia Humans have been answering that question with compasses, but magnetic north is on the move. In the 1800s, it was somewhere close to Canada. Now it's closer to Russia. And that's a problem for the global positioning systems on submarines, planes, and even our phones. No wonder I'm always getting lost. To help us all deal with this shift, scientists have sent a device into space called a diamond quantum magnetometer. David Roy Gay is the founder of Canadian company SB Quantum, which makes the device. Your compass, analog compass, will actually tell you where the north is.

20:56And it's always going to point to the magnetic north, so not the geographic north. That's why we need the updated map. But maybe not everybody knows, but we're carrying compasses into all our pockets. In every cell phone, there's an electronic compass. And this one differs from the analog compass because it needs a map, a reference map, to know where it's pointing at. So actually, all these electronic compasses will use this worldwide magnetic map that we're at the beginning of the process of releasing new versions in the future with our technology. so how are you going to measure the earth's magnetic field and uh i appreciate this is complicated uh so perhaps give us the very very basic give us the primary school version of it please yeah so we're going to send a satellite in orbit and we're going to have our compass technology on it and it will circle around earth and collect the data everywhere on the planet curate it and release the next version of the world medic map which is due for 2030 so your uh compass is very is very different is you're using something called a diamond quantum magnetometer.

22:26Is it actually a diamond to begin with? It is. It is. So we're using atomic impurities into a beautiful purple slab of diamond which is grown in lab. So we insert nitrogen during the growth and this will form a certain type of impurity in the diamond which has exceptional quantum properties, even at room temperature. And we can use these quantum properties to measure the Earth's magnetic field with a great level of accuracy. And it was launched into space quite recently. Four, three, two, one. Ignition and liftoff Falcon 9. Go space up, the transporter's ignition. Does this potentially give us an alternative or a substitute for GPS, the global positioning system that is in your car, is in your phone, that a lot of people rely on for navigation, but people worry about becoming unavailable or being misused, abused, spoofed, whatever?

23:42Absolutely. So the strategy is going to be a bit different. Instead of measuring the core magnetic field, we're going to measure the crustal magnetic field. So the compass measures essentially the magnetic field created by the metal core, the iron core of the Earth. What you want to do is measure the rocks that we all stand on, essentially, the crust of the Earth. you want to measure the magnetic field of those rocks because they change depending on where you are. Yes so the rocks have a much more local structure much more details so in the same way that if you approach your compass from your car your compass is going to be altered so every rock will play a similar pattern.

24:36So if we have access to this global cross-tall rock type of map, then we can achieve much more precise navigation than telling you the north is here. You can actually fuse that with other instruments so that we get the positioning to a few tens of meters. So it's not going to be as accurate as GPS, but when you have nothing else, it might be game changer. I remember reading somewhere that pigeons navigate using something like this. Am I right? Am I suffering from a delusion? No, no. So the animal kingdom has exploited magnetic fields for navigation to some extent. So there's nice articles about sharks, turtles and birds navigating using magnetic fields so in some ways we're kind of catching up what are the applications for this for this alternative gps if you like what where do you see it being used most yeah so in eastern europe gps is jammed quite often or spoofed, which can lead to planes drifting in another country or just losing GPS instruments.

26:08In some cases in the Arctic, there's a particular environment in terms of the GPS availability is not quite as good. And also, since you're close to the Northmatic pole, where all the auroras mostly happen, so the northern lights, this will alter the quality of the GPS signal. So instead of having one meter positioning accuracy, maybe you'll have 10 meters, maybe sometimes you'll lose the signal. So having these backup systems can be pretty useful for intelligence, reconnaissance missions in the north, and especially for autonomous platforms, which need accurate positioning or else you're going to lose them.

27:02Well, today's Tech Life journey to Magnetic North is nearly at its end. Thank you, as always, for listening. Remember, you can get in touch with us, techlife, at bbc.co.uk or WhatsApp us on plus 44 330 1230 320. And do keep across our work at bbc.com forward slash technology. Today's Tech Life was produced by Imran Rahman-Jones and presented by me, Chris Vallance.

27:43Who's actually won the Iran war? I'm Tristan Redman. And I'm Asma Khaled. And together we host the Global Story podcast from the BBC. The US and Iran say they've struck a deal to end the war. But a key question is what's actually been achieved by nearly four months of fighting. and is the situation better, worse or the same for the region and Iran? For the full story, check out The Global Story on BBC.com or wherever you get your podcasts.