Deepfaking disability

0:00Rowan Ings:This BBC podcast is supported by ads outside the UK. I'm Sam Peranti, and for the past seven years I've been granted access to a team of US special agents. Their job? To track down those who create and trade in sickening images on the dark web. When I hug my kids, that's your fuel, that's your why. From the BBC World Service, World of Secrets, the darkest web tells their story. Search for World of Secrets wherever you get your BBC podcasts. This is not the future we were promised. Like, how about that for a tagline for the show? From the BBC, this is The Interface, the show that explores how tech is rewiring your week and your world.

0:49Rowan Ings:This isn't about quarterly earnings or about tech reviews. It's about what technology is actually doing to your work and your politics, your everyday life. and all the bizarre ways people are using the internet. Listen on BBC.com or wherever you get your podcasts.

1:10Rowan Ings:Just a heads up. This episode contains references to pornographic material and AI-generated content involving minors. Wow, you look incredible.

1:19Nathalie Jimenez:I mean, how come you're back to your old look?

1:22Rowan Ings:What can I say? I missed wearing black. This Instagram video appears to show a young woman with Down syndrome wearing a grey top, a black skirt and lip syncing to the audio you just heard in what appears to be a bedroom.

1:34Nathalie Jimenez:But in reality, it's a deepfake that's been made for profit as part of a disturbing trend.

1:40Rowan Ings:The Down syndrome profile was stealing this stuff from TikTok. These who ever watches this, this person is 16, they don't have Down syndrome. Like, this has been altered with AI. I'm Rowan Ings. And I'm Natalie Jimenez. This is the documentary from BBC Trending. In this episode, we're going to hear how people who have been deepfaked have been affected by it, speak to content creators with Down syndrome about how it impacts them, and try to trace the people profiting from deepfaking disability.

2:11Nathalie Jimenez:Deepfakes are AI-generated or manipulated videos or images that look almost real, and they are all over the internet. Deepfakes have been used for years to create non-consensual intimate images of women.

2:24Rowan Ings:Over the last three months, we've been monitoring a trend that deepfakes down syndrome for profit.

2:30Nathalie Jimenez:We found deepfake Instagram accounts with thousands of followers that led to OnlyFans accounts with hundreds of likes. The deepfakes are being used to grab people's attention on Instagram to get them to subscribe to accounts on OnlyFans, the most widely used adult subscription platform where creators sell intimate images and videos of themselves.

2:49Rowan Ings:Just one day, it just kind of popped up out of nowhere. just looked off. It just looked off to me. It's nothing I had ever seen before on Instagram.

2:57Nathalie Jimenez:That's Justin, a dad of two based in California, who says not long ago, he barely paid attention to social media. But now he calls himself a part-time deepfake hunter. And it all started when, scrolling through Instagram, he stumbled on a deepfaked video.

3:13Rowan Ings:You look lonely. I can't fix that. I was looking at the comments and I see all these people praising this person, They're like, oh my God, you're so beautiful. You're so, you know, hot and all this. And what are these people thinking? Like, did they not know this person is even real? The profiles that were doing this just to get people to kind of subscribe to like the OnlyFans accounts.

3:33Nathalie Jimenez:Since then, Justin has set up two accounts on Instagram dedicated to exposing hundreds of deepfaked profiles. He posts the AI-generated videos, which aren't themselves explicit, alongside images of the real influencers whose faces were altered. And he's learned to spot the telltale signs.

3:51Rowan Ings:There'd be like little facial glitches, like just kind of like shimmering, glitching like on the cheeks and the forehead. I also noticed, too, that when a lot of the times when the faces were turning, the AI can't handle when heads turn because then the filter will kind of fade away.

4:06Nathalie Jimenez:But nothing really shocked him until one day he found something he says he never expected. And it's been causing an uproar online.

4:14Rowan Ings:The darkest I've seen as far as this whole AI face thing is going.

4:17Nathalie Jimenez:What Justin saw was a video of a young woman wearing a bikini with a face swap that made it look like she had Down syndrome, a trend that in April, he says, started flooding his feed, including some really disturbing reactions.

4:30Rowan Ings:Hey, I mean, if she's down, I'm down.

4:34Nathalie Jimenez:But that's not the only thing that shocked him.

4:36Rowan Ings:I noticed them popping up. She herself, the girl, Alice, she commented under one of my comments saying, hey, that's me. This person's using my videos. The Down syndrome profile was stealing this girl's stuff from TikTok.

4:47Nathalie Jimenez:The person Justin saw commenting on the profile saying this was her body was a 16-year-old called Alice from London. And the conversation in the comments, Justin says, was disturbing.

4:59Rowan Ings:I think Alice mentioned in her comment that she said that she was underage. Majority of the comments were all very inappropriate. We looked at this deepfakes account and several similar accounts. Across all of them, the comments contained highly sexualized comments.

5:12Nathalie Jimenez:We wanted to speak to Alice to find out how she felt about what had happened, so we contacted her, and she agreed to have a chat.

5:19Rowan Ings:Hi, hello, nice to meet you. Hi, Anna, thank you so much for having us. Thank you so much for coming. Hi, Alice, how are you? So nice to meet you. That's Natalie and me meeting Alice. Her mum was there too, greeting us when we met them for the first time in person.

5:36Nathalie Jimenez:When we first reached out, we had no idea the story was actually closer to home. She happened to live just a short drive from our London office.

5:44Rowan Ings:Alice has not spoken to the media about this before. It all started in April when a friend from school sent her the deepfake profile after seeing it on Instagram. When I first saw it, what was going through my head was just like shock and humiliation, I guess, because the first thing I noticed, obviously, was the OnlyFans link in the bio and then obviously the AI that was being used to edit my face. It was just really strange to see.

6:11Nathalie Jimenez:Alice told us she reported the account to Instagram, but received an automated response saying it did not violate their guidelines. The bio didn't directly link to OnlyFans, but instead pointed users to a link hub, which contained a link to an OnlyFans profile.

6:26Rowan Ings:Social media platforms and pornography websites have policies to deal with deepfakes. You're not allowed to post explicit deepfakes on Instagram or Facebook. But the deepfaked videos which used Alice's body weren't sexually explicit, though some showed her in a bikini or in a crop top.

6:44Nathalie Jimenez:OnlyFans has a rule banning non-consensual, unlabeled deepfakes. But as far as we could tell, there wasn't any AI-generated content on the account, which this deepfaked Instagram profile pointed to. We'll dig into that more later. What has stayed with you the most from all of this, looking back?

7:02Rowan Ings:It was quite shocking to see that they knew my age. I mean, I DM'd them multiple times saying, like, I'm 16, like, can you get this taken down? And I got no response. And people did also comment on the account. Please, whoever watches this, this person is 16, they don't have Down syndrome. Like, this has been altered with AI. And so whoever was posting it was aware of my age and chose to do nothing. So I think that was quite scary to see as well, that complete disregard for, like, my age, my concern, my feelings. It was a total invasion of my privacy. Did you go to the police at any point? Actually, on the day it happened, I called the non-emergency number just to kind of inquire, like, are there any legal ways to pursue this or just a way to contact Instagram and get it taken down?

7:47Rowan Ings:And I was pretty much just told, like, we sympathise with you, the account looks bad, but there's nothing you can do. There's no laws currently surrounding AI, although they are trying to introduce one. It was just surprising to hear that there was nothing in place that I could do about it. In a moment, we'll hear from experts who say this is not just invasive. It is part of a broader system of exploitation. You're listening to the documentary from the BBC World Service.

8:14Nathalie Jimenez:You're listening to the documentary from BBC Trending. We've been investigating a surge in AI-altered videos known as deepfakes, fueling a new wave of exploitation in the online adult content industry. One trend in particular has gone viral. Deep fakes on social media portraying people with Down syndrome and generating profit on adult content platforms.

8:38Rowan Ings:After meeting with Alice, we took our findings to an expert, senior researcher in AI ethics, Eleanor Drage, at the University of Cambridge. We showed her the deep faked images of Alice. Could you sort of describe what you're seeing and just what your initial reactions to it are? There's a girl wearing normal things for a teenage girl, more low cut. She presents a little bit with Down syndrome. It says that she's 18 and we know at least the person whose body they're using is not 18. We can see that she's got tens of thousands of views and they've got pre-offensive things written on them. The other major issue with this of course is that it's taking someone's data without their consent.

9:24Rowan Ings:It's using it to capitalise on disability as a way of making money. That's the kind of core of the problem. This isn't what good sexual expression means for anyone because it's not real and it's not safe and it's actively harmful to people like Alice that have had their whole profiles, their lives taken and messed around with. What's key is that Alice's video only appeared on Instagram, but when we clicked through to the linked OnlyFans account, we found a different woman using the same name, Miel, but her face was always hidden or cropped out. Each AI profile we found on Instagram followed this same pattern.

10:05Rowan Ings:We've deduced that this is a way around OnlyFans' no unlabeled deepfakes policy, which bans non-consensual AI-generated content. And since we can't know if the woman on OnlyFans consented to appearing on OnlyFans, Eleanor says that this points to something bigger. So if there's these multiple different women that are part of a single profile, You have a web of exploitation. It's not about an individual anymore. It becomes a much bigger story. What I was really shocked about in Alice's interview, she also called the police and they said there was nothing they could do. And Instagram did nothing to take it down.

10:40Rowan Ings:Could you explain why there's so little recourse for victims in this kind of situation? Unfortunately, until we have really excellent data protection in the UK that we are implementing really well as well, then this is continually going to happen. And the question of where the buck stops is super important. And while responsibility is distributed, everybody has to do something. That doesn't mean that everyone should do nothing.

11:11Rowan Ings:We wanted to find out who was behind these kinds of deepfakes and if any of them could be held to account.

11:17Nathalie Jimenez:We started digging into the online world of AI OnlyFans managers, or AI OFMs, who make money using AI-generated influencers on adult content platforms and focused on French OFMs. Digging through Instagrams, YouTube, and finally Telegram channels one of them stood out, not only because of the sheer popularity of it, with over 7 ,000 members, but because of the man behind it, Dorian, a 27-year-old from France living in Dubai.

11:49Rowan Ings:You guys, I hope you're doing well and I hope you enjoy the video because I will give you a lot of value. But right now, we will do a little bit of lifestyle, bro. Look at this. That's the very first time I drive a Lamborghini. It's going to be crazy. Let's go. Welcome guys into this new video. We are in Croatia. Today I'm in Santorini. We made 20k with that model.