Why deepfake wizards went underground

0:00This BBC podcast is supported by ads outside the UK. Decisions made in Washington can affect your portfolio every day. But what policy changes should investors be watching? Washington Wise is an original podcast from Charles Schwab that unpacks the stories making news in Washington right now and how they may affect your finances and portfolio. Listen at schwab.com slash Washington Wise. That's schwab.com slash Washington Wise. Running a business shouldn't feel like surviving a software group project. One app for accounting, another for inventory, another for sales. And somehow, none of them talk to each other.

0:44That's where Odoo comes in. An all-in-one business management software that brings every part of your business together. From sales and accounting to inventory and marketing, all in one powerful platform. No messy integrations, no bouncing between tabs, And best of all, no spreadsheets. Stop managing software and start managing your business with one unified system. Try for free today at odoo.com. That's O-D-O-O dot com. I know what you're thinking. You're thinking we're probably going to do a whole segment on Sydney Sweeney being naked, Sydney Sweeney's boobs, the politics of Sydney Sweeney's boobs.

1:23But actually, we're not. No, instead, we're going to be talking about a whole range of other naked people, naked women, but not Sydney Sweeney and not in a funny way. Sports. Sports. Sports. Sports. Sports. Sports. Sports. And actually, Matt, the theme of this episode is the impact that social media and technology is having on women and the representation of women and women's rights and everything else. We're going to talk about the kind of content which is probably having overall a more corrosive impact on the perception of women, but that it's the kind of content that you might never have even come across and you might not have heard about yet.

2:04Right. We're going to be talking about the decentralization of deep fake pornographic images being created of women and also the rise in Western influencers traveling to Afghanistan to make content.

2:25earlier this year on top comment we spoke about how grok and other chatbots were being misused to deep fake nudify often women in public life also kids though minors and there was a load of controversy about it and a lot of pressure put on the social media companies particularly x because grok um its chatbot was one of the places where this was happening but the thing is and we find this all the time. Once something moves off the main social media sites, I think there can be this misconception that the issue is resolved or that it's gone away. And actually this deep faking of women in public life in particular has almost become so normal and more fragmented.

3:04And that's what we're going to chat about today. And that word you just used, nudify, which I hate that that has now become a part of our lexicon. It kind of indicates that it isn't just grok, as you said, but there is a kind of flourishing deepfake commissioning community online that's way deeper than anyone realises. And we're going to get into that and also discuss the legislation because it's not working. Yeah. And there's a load of data that's come out making this exact point that the issue of deepfaking of women politicians in public life isn't something that has gone away or changed. In fact, you're 33 times more likely if you're a woman parliament not a man, to be depicted in or associated with deep fake pornography.

3:52And of those EU member nations that were studied as part of this research, one in 14 women versus one in 500 men were found to be depicted in or associated with deep fake pornography in this way. And this is only becoming more of a concern, I think, because AI is so accessible and it's so much more convincing, right? Like an AI fake of someone even a year ago, two years ago, wouldn't necessarily convince anyone around them that it was actually a naked image of them or actually a video of them. Whereas now, it's all just so much more realistic. Yeah, exactly. And you yourself have been the subject of deepfakes, haven't you?

4:33Yeah. And it's something that's been flagged to me before. But actually, when I was getting ready for this episode, I decided to have a look again at what was happening online right now so when you search for this kind of thing often it's framed as like an only fans leak so it'll be suggested that someone's got content from your exclusive only fans obviously i i don't have an only fans um but if you search my name and that kind of thing a whole load of websites come up and these websites have kind of variations of like the word fap the word nude like they're all those sorts of names what you would expect and what i think is a change that I've noticed in recent months is how many of these websites exist so these are not mainstream social media platforms these are just websites you click on them they present in a similar way to a porn website but obviously like when I'm looking at this content of myself this is not consensual right like I've not shared these images with this website they've found them from my public social media profiles so there's this picture of me I'm wearing like a dress it's a picture that was taken at Christmas and it says marina spring only fan leaks photos and it's got like a hashtag number and then it has in yellow with a fire emoji undress ai and if you click on that it takes you to another website which you actually have to give id in the uk and prove that you're of age to use it e.g it's treated like a like a porn site but essentially it's a site that people can use to edit your photos so that you look like you're naked i i really cannot think of a reason like a valid reason anyone would ever need to use that because it is not the same as like consensual pornographic content or consensual OnlyFans content or what have you.

6:08Like this is all images of me that I have not consented to them being used in this way, let alone being nudified, as we would call it. Yeah. And I haven't seen this image, but you were saying that like, it's shocking how realistic these things are. And I think that's the real difference as well, because I'd seen deep faked imagery of me before. And often it looked like my head had just been edited onto a different body, right? whereas the the strange thing i don't know if strange is the right word the kind of violating thing about the recent imagery that i've seen is that it is an actual picture of me you obviously know like i know the original photo so i know that i wasn't naked um but you can even i for a split second kind of believe it which shows you how convincing a lot of this stuff has become because if someone saw this and i was showing it to um a colleague who i'm close with in the office and her reaction was like, oh, wow, that's really convincing.

7:04Obviously, she's not actually seen me naked, but they've kind of done it proportionally to look like me, which is really, I think it's that bit that feels kind of icky. I mean, this is also what makes the images quite weird, right? And I think this is a telltale sign for spotting this stuff. It's like you are in a situation where you would not be naked. Like, you know, it's like me. On the beach or whatever. It's like me. This is obviously not funny, by the way. and it is violating and not very nice but it's also quite absurd like if you could see this picture it's me standing in the kitchen with a glass of I think it's sangria I'm in like my friend's kitchen I think we were watching the Eurovision this was several years ago right and I was obviously wearing clothes yeah but in the picture I'm just standing there with a glass of wine not wearing clothes I think I think if I if a picture if a deep big of me naked holding sangria in someone's kitchen watching Eurovision Leap.

7:58I think people would assume that I was naked and that's not it. You're talking about this in a very blasé way and you're laughing about it, which I think is healthy and good. But like, this is one of those moments where you kind of have to take a step back, take a breath and realize what it is we're talking about here, that it's now so normal for you as a journalist to have people like look through your social media, take a photo of you and make it naked, like, why are men doing that? Like, why are people doing that to you? Well, exactly. And, you know, you could search for basically any woman in public life's name and the same sorts of terms, and you would find images on these websites most of the time.

8:37And that's not just here in the UK. That's all over Europe. It's in loads of different places. The reason I can laugh about it is because this image is not being distributed super widely, right? Like, it's not something that people in my actual life are seeing or that people who listen to Top Comment or follow my investigations or podcasts or whatever are consuming. Instead, it's being shared with clearly a niche audience for whom there is appetite for this sort of stuff, a niche anonymous audience. But the crucial issue here, I think, is the way that it undermines me as an investigative reporter.

9:08It plays into this narrative that my appearance is sort of more important than the journalism and the work I do and that it's fair game to be derogatory and offensive and sexualize women in public life. And actually, Matt, one of the women who was included in this research, so content about her was analyzed and featured. She's a politician in the Netherlands. She's called Suzanne Kroeger. And she makes this point, which is she's less concerned about the impact on her as an individual because it is kind of, I mean, it's gross and offensive and a bit absurd. What she's more worried about is the wider impact on women in public life and particularly women in politics and how this undermines their credibility and also actually could put other people off getting into politics.

9:53I was alerted a while ago about the fact that there was deep fake porn content using my images, which is, of course, a criminal act. It's illegal and deeply, deeply disturbing. And to me, this is not about our individual cases, however disturbing they are, but it's about democracy. because attacking female parliamentarians is really undermining democracy. In a time that we see that women's rights are under pressure, we need those female voices in the public debate, in politics. And we need girls and young women to really aspire a career in politics, to serve the public good, to be part of the public debate.

10:34We see that female politicians across the political spectrum are being targeted. I have colleagues from different political parties that are a victim of this. So it's not about politics. It's about silencing women in the public debate. And that's what's making it extremely disturbing. I mean, we've sleepwalked into a world where it's a normalized feature of being a woman in public life that this will happen to you, which I just think is utterly insane. And, you know, when faced with such an almost psychopathic act that someone would create this image, it proves what I think to be the case, which is that this is yet another symptom of a kind of very modern phenomenon of extreme radicalized misogyny of men who actually hate women, who want to see them destroyed.

11:23That is a particular feature of the online world we live in today. Where are these images coming from? So obviously we mentioned Grok earlier and that Grok became a tool that people earlier this year could use to more easily edit publicly available images of women, usually high profile women or women who have some kind of public facing role. But actually we also saw it happening to the average woman who just happens to have a picture on a public Instagram or a picture that had been shared. And actually, to some extent, it's much worse if you're a woman who's not in the public eye because you don't necessarily have any mechanism by which you can deal with it or talk about it.

11:56And we should say that. But before we saw, and I think this was the case over several years, really, how it was mainly a combination of websites and then also forums where people could interact with one another anonymously and generate this content, like 4chan, for example. Right. So the 4chan website, which was it was kind of a hub for all this content until recently, was an almost commissioning system where you could ask this community to create a pornographic deepfake of someone. And it was kind of ran under this hierarchy where the people creating the deepfakes were called wizards. And this board on 4chan was shut down in May after all the negative press coverage around it.

12:39and I think that that kind of dynamic and community point is really crucial here because if there are other people who want this same content as you and I think you could probably say this of most dark content on the internet it it perhaps legitimizes it in their eyes it's like oh there are other people who enjoy this sort of fetish if we want to describe it like that which obviously is derogatory and abusive and not nice um but but it's like oh well there's a community of people here who all think this is okay and also crucially and that this was definitely the case on 4chan, we don't have to say who we are, right?

13:09It's anonymous. Exactly. And I think, and this is one of the reasons why Laura Bates or her book, Men Who Hate Women, which explores these kind of radical online communities is so interesting because she kind of makes the case that these extremist groups have more in common with kind of domestic terror organizations than they do with kind of online communities because they truly hate women and their misogyny is so radical that they actually actively want to ruin their lives. You mentioned this earlier, but the dehumanization is a kind of a prerequisite for this, right? Because they're not seeing this as doing it to a person.

13:44They see them fully as like an object. And they also often have this kind of grievance justification for what they're doing, right? So this kind of incel or pickup artist or red pill idea that women owe men attention and sex and that women are punishing them by not giving it to them creates this kind of ready-made script for treating these deep fakes as kind of payback against women as a whole. What you're identifying there is that there's actually probably two main genres of people that distribute, create and consume this content. There are the people who have a sort of sexual fascination with that content and perhaps wouldn't even necessarily see it as misogyny or hating women because they see it almost as the same as as pornography or that it's kind of fair game if you're in the public eye and this is just normal now which obviously it's not and then there's the the group you're describing which is people who actually are doing this because they want to ruin women's reputations and lives and they want that content to be distributed and they want people to be convinced that an embarrassing and humiliating and debasing image of that woman is circulating i mean when you look at the manosphere matt do you find that this is a tool that is readily used or actually is it almost like too simple for the manosphere no it absolutely it's it's huge amongst the manosphere because it's a punishment for women right it's it's punishing them and it's the but it goes even deeper than that men will share genuine um leaked videos of women to punish them and there you you begin to realize when you see enough of these posts of men doing these things that I do think that it's not just about the kind of sexual excitement of it.

15:27It's not just about the kind of bro humor of it either. It really comes from a place of hating women. There's something particularly new and modern about this online extreme misogyny that I think we haven't yet figured out as a society. And does revenge porn for you fit into this wider landscape, right? Because I guess revenge porn is often people who are known to one another rather than it being this slightly dispassionate, I'm going to do this to someone I don't know. I think when it's revenge porn, like sharing amongst a small group of people, maybe that's different. Maybe that's a very personal, but revenge porn often is shared amongst online communities of men who encourage each other to share that stuff and then revel in the punishment of that woman.

16:11That's where I think this turns into an extreme misogynist group. And one of the issues here is that in a positive way, because there's been a lot of pressure on the major social media companies, we're seeing this stuff go, I mean, go underground maybe feels a little bit strong, but it's fragmenting, it's popping up on these websites that are anonymous. And in some ways, that fragmentation is a real headache for legislators and for legislation, because ultimately, it becomes harder to police or regulate these sites which almost pop up in a sort of whack-a-mole fashion and like obviously when i reach out as a reporter to these to these people they're not going to reply to me unless they have some sort of interest in promoting their site there is this data act which came into force earlier this year in the uk and it made intentionally requesting or creating sexually explicit deepfakes of an adult without their consent a criminal offense um and that is the case even if that image is not published but it is created um but the problem is first of all that requires then figuring out like who the individual is who's made this content if i just search the stuff on me now there are multiple different websites so the amount of resource and time it will take to identify all of those different people and then hold them accountable for doing this which is technically illegal is a lot of time and effort and and not something that it feels like there is resource to do and then you've got the other issue which is not tackling the tech itself so obviously there's been pressure on the social media companies but like what about for example the fact that this undress ai sort of app exists like should that be allowed like is that ever used for anything normal probably not and if you look at countries like italy for example deep fakes have been criminalized but a victim has to complain first which by the way like the photos are so humiliating.

17:57I don't even want to show like you or the producer or basically anyone in the office other than one person who I felt like I could show the image to, right? You can see why it's not necessarily that appealing for women to come forward. And the enforcement will only target the individuals who created the content. So not even the kind of lurkers who might consume or engage with that content. Like they're kind of fine to do that, right? Yeah. And when this happens to a woman, the content, it leaves the original creator and it becomes, it kind of, it spreads and it spreads. And this happened recently.

18:27It's not a deep fake. These were real videos that were leaked of a former college student named Maddie Kowalski, who had kind of pornographic videos that were filmed without her consent leaked by what's called the Burnerverse, which is kind of a loosely connected online community of sports fans and frat bros. And they shared these videos and these images in an incredibly degrading way. That woman, it then becomes her life for years trying to get this stuff removed, fighting the tech companies, fighting the men behind it, tracking them down. It becomes expensive and it becomes like a full-time commitment.

19:05And she doesn't have a normal life from that point onward. That's what I think is so frustrating for people about this story is it will be much easier to stop this stuff at the source. And crucially, as you say there, the approach is always to react afterwards. It's to take stuff down. If you look at other examples of legislation, say in the US, they have this Take It Down Act, but also in France and Australia, they have similar legislation. It's always responding to this stuff. And so like you pointed out there, whether this is, we're talking about deep fake content that is not real, or whether we're actually talking about sexually explicit content that is real, ultimately the approach is to say, right, let's get rid of it once it's already been shared, often gone viral, or at least had a significant number of views.

19:47and none of this is dealing with the source and there will always be people who will distribute this sort of thing but because it's websites and because accountability is hard and because it's tricky to figure out who these people are everyone kind of gives up and it becomes normal and exactly like you say it's kind of funny and weird and absurd because it's about me so I can sort of say that but it's also gross and dehumanizing and not normal and yet it has become completely normal and it did not shock me in the slightest when I found it.

20:17John Deere is putting in work and putting up highlights. Oh man, look at those passes. Tillage, planting, harvesting. Three years in and these pre-owned machines are still at the top of their game. We're talking reliability, performance, and long-term potential. And get this, these veteran machines can pick up the latest precision tech with ease. You know it and fans ready to add John Deere certified pre-owned equipment and more options to their lineup can shop at machinefinder.com. Booyah! Self-directed investing, trading, full-service wealth management, automated investing, financial planning, thematic investing, retirement planning.

20:54Phew. And to think, that's just a small taste of what Schwab offers. Because Schwab knows that when it comes to your finances, choice matters. No matter your goals, investing style, life stage or experience, Schwab has everything you need, all in one place, so you can invest your way. Visit schwab.com to learn more.

21:42with local people there who refer to themselves as the Talibros. They run kind of tourism companies that invite influencers there. And they're kind of inadvertently, or sometimes advertently, is that a word? Making propaganda that paints Afghanistan out to be a lot less bad than it is. Come to Afghanistan. Enjoy the beautiful mountains, the beautiful tanks, and come stream. Holy shit, bro. Uber Black? Salaam Alaikum. Alaikum, you ready? I'm ready bro. Don't believe the media guys, don't believe the media, look at the Afghanis. Women and children out by themselves, without his not allowed. Everyone says they must be accompanied by a man, maybe this is not true.

22:25This was also our first Taliban interaction and they invited us for some green tea and wanted to hear about our experience so far. They were all super friendly and just wanted to make sure we were feeling safe here. We're staying at the Taliban bro's house. We're starting before we eat. Onions, you just slam onions? Yeah, I saw the law. Just slam the whole thing right before eating? Yeah, yeah, yeah. Your wife doesn't work, right? No, no. She's a queen. She doesn't even go outside to buy the groceries. I do everything. Yeah? Afghanistan is blue, it's green, it's lush, it's mountainous, and it's got ducky boats.

22:55So the people in Afghanistan are essentially, would it be fair to say, using these influences? A bit like, and I'm sure the influencers would very much debate this, but almost like useful idiots who will present their country, perhaps as they see it, but without asking any of the complex or difficult questions about, for example, women's rights? Is a lot of the content kind of saying, oh, no, no, no, no, it's not as bad as you think? Yeah, kind of. I mean, look, the distinction between the influencers from the West who are making this content and those who are hosting them, the Talibros, they claim to be kind of tourism companies.

23:29We reached out to one of these Taliban bros. And we asked him several questions about why he kind of makes this content and invites influencers over. And he just didn't answer any of them and said, do you want to talk about the Taliban or do you want to talk about tourism? Right. So he claims that his focus is tourism. Taliban officials told the New York Times last year that 14 ,500 foreigners had visited Afghanistan since 2021. Most of them are obviously men. This is a country where, of course, whenever an adult woman leaves her home, she's obliged to conceal her voice, her face and her body.

24:03And formal education past the age of 12 is restricted. Books written by women are removed from universities. None of this, as you can imagine, is covered fairly and accurately by the Taliban content. So these influencers are kind of becoming useful idiots for the Taliban to paint their regime in a more flattering light than a legitimate journalist or even like a travel content creator from back in the day and Anthony Bourdain would do. And the Taliban have always denied that they, you know, pay these creators to post the content. The content's not sponsored. But ultimately, in the same way that influencer trips, brand trips work, you take someone somewhere, you present a certain view of it, you facilitate an influencer, and then their messaging often matches up with what you would ultimately quite like them to say.

24:50But you don't have to tell them to say anything. They can just turn up and post that content. And I think that is particularly appealing to an influencer when they feel like they can expose, quote unquote, the truth about what's really going on. The establishment, the state, the cabal, whatever you want to call it, are telling you that, you know, what's happening in Afghanistan is really harmful for women. But look, look, look, I'm going to show you the real truth, even if they might be being taken for a ride by the people welcoming them in the first place. Exactly. Yeah. I mean, this is about challenging the mainstream media.

25:20A lot of the content creators, like, for example, there's one of them is titled a video, Fuck the Media, I Went to Afghanistan, and titled another video, Afghanistan is Not What You Think. So the genre of content that this is, which is edgelord travels to dangerous place, has always been popular, right? Again, you could argue that that kind of travel content is what Anthony Bourdain did. It's what Vice did, right? It was like, let's go to this country that you've never heard of that's dangerous, and we're going to lift the lid on what's going on there. People love that kind of stuff. But these people are completely sanitizing the image of Afghanistan.

25:55And in this kind of, you know, fight against the mainstream media, this is what you've got instead. Instead of vice, you've got the talibros. So I hope you're happy, people who hate the media. But I mean, if you just look at some of the... I thought you were going to go into your sad little life, Jane. You won, Jane. Oh, my God. Enjoy the money. I hope it makes you very happy. Dear Lord, what a sad little life, Jane. You ruined my night completely so you could have the money, but I hope now you spend it on getting some lessons in grace and decorum because you have all the grace of a reversing dump truck without any tyres on.

26:36We've never done that meme on top comment before. Come dine with me vintage. I also think, by the way, so the Taliban's become increasingly apt at courting Western influencers and sanitizing their image through that. But the right and the kind of conspiracy theory, manosphere section of our internet in the West has also been courting the Taliban. And I do think that a lot of this started with Andrew Tate. So the Taliban, surprise, unsurprised surprise, like Andrew Tate. Yes. I mean, Andrew Tate has claimed that the Taliban have offered him political asylum, right? You know, he said, oh, the Taliban, they bring law on order, you know, on the Taliban side, they've participated in a Twitter space, like a live discussion to discuss the detainment of Andrew Tate along with his brother.

27:24And they said, you know, in this Twitter space that they're concerned for Andrew Tate and they say Westerners need Andrew Tate because we are oppressed by feminists. So you have this bizarre Taliban Manosphere axis of anti-feminism going on that I think has informed a lot of these content creators who go there. I mean, for Manosphere influencers, like Jalalabad is the best way to get a huge amount of attention online. And people like, for example, Lord Miles, Miles Rootledge, the influencer, he claims that he was kidnapped by the Taliban when he made contact there, but that they treated him really nicely.

27:58And he called in some servant. He brought tea and biscuits. And that's kind of the start of my journey of my eight months in Taliban jail, but also befriending the Taliban, basically. and I'm still the Afghanistan guy to this day. They're getting millions and millions of views on this kind of stuff. And if we're thinking about the mutually beneficial relationship here, so the influencers know that they can get quite a lot of views and likes and engagement for this sort of content, the Taliban also know, or the Taliban and the people facilitating this, know that these influencers know how to make content that is good and will get likes and views and therefore will allow them to project a certain image or message to the wider world, right?

28:40Oh, they definitely understand social media. I mean, just one video that comes to mind. Basically, there's two people with bags over their heads, reminiscent of the execution videos of Daniel Pearl and James Foley by Islamic State. And the people behind them say, we have one message for America. And in your mind, you're thinking, oh, this person is going to get beheaded. And then they lift up the bag from their heads and say, welcome to Afghanistan. We have one message from America. Welcome to Afghanistan.