The Facebook grifters making money off AI Holocaust victims

0:00This BBC podcast is supported by ads outside the UK.

0:30us. And then we'll be learning about Francis Galton and the racist, discredited pseudoscience of eugenics with Desiree Birch. We'll meet many medieval saints in their bone boxes with Rachel Parrott. So if that sounds like your sort of thing, listen to You're Dead to Me wherever you get your podcasts. Thank you. Bye.

1:00It was an emotional response. It wasn't until I started reading some of the comments that I did question it. It wasn't a, this could be false, this is false, this is fake news, this is AI driven. It wasn't like that because it was beautifully written. In early June, J &B from the south of England commented on a Facebook photo and post about a young girl called Miriam Gold, who died in Auschwitz, the most notorious Nazi concentration camp. The picture is black and white and shows a dark-haired girl in a colored jumper and apron, knitting while looking into the camera. What Jane didn't know then was this Miriam the Post described never existed.

1:44There were several Miriam Golds who were killed in the Holocaust, but from what we could find, the one described in the Post wasn't one of them. During the Holocaust, Nazi Germany systematically murdered six million Jews. Millions of others were killed and persecuted on the grounds of ethnicity and race. The photo and story we were looking at, however, were created using AI, drawing on the well-documented histories of Holocaust victims. The post seemed very real for Jane, though. In fact, she was so moved by it, she, like 1 ,500 other people, commented. Part of her comment reads, I have met in the past some of the Holocaust survivors.

2:26They were truly amazing. and my uncle was part of the convoy that went into Auschwitz to free them. He told me things later in his life of what he saw, man's inhumanity and cruelty, to its fellow mankind. My family were all part of the fight against all of this, so I do understand. There were so many wrongs in the world. What Jane had come across is commonly known as AI slop, a type of post, typically of an AI-generated photo and text that is flooding social media platforms. Some of these posts, like the infamous image of Jesus made of shrimp, are silly, absurd, and easy to spot. But with others, it's becoming harder to tell.

3:09Jane feels bad for not recognizing it was AI-generated and says she was rightfully called out for engaging with that content. It just never crossed her mind this could be fake as it felt very real to her. Many of the AI posts about Holocaust victims that I found mimicked the posts uploaded by the Auschwitz Memorial on their Facebook page. From the end of June onwards, the memorial felt compelled to publish a series of posts warning about these images, saying they are a dangerous distortion and disrespect victims and harasses their memory. This really is instrumentalization, manipulation, distortion that can lead to many disturbing things.

3:49This is Paweł Sawicki, the spokesperson for the Auschwitz Memorial in Poland. It can damage the trust that people have in what they see, because we already started getting comments on our Facebook posts that, oh, this is an AI-generated photograph. This is post by the Auschwitz Memorial about real Holocaust victims receiving these comments. Pavel also warns to be careful if you come across photographs from inside Auschwitz. We don't have visual representation of almost all the stories that happened in the camps. The perpetrators didn't take photographs. Taking pictures for the prisoners was extremely risky.

4:26And in the story of Auschwitz, we only have four photographs that were taken by the prisoners in the camp. Jews who were thrown into the crematoria risked their lives to take a few pictures so that people could see what happened there. And here we have somebody making up the stories for some kind of a strange emotional game that is happening in social media. This is not a game. This is a real world, real suffering and real people. Pavel told me that he and the memorial have been dealing with AI slop for a while now. He says when he reached out to Meta, Facebook's parent company, he was told that there were no violations of the guidelines with these posts.

5:05The only thing they could look into is that there are some administrators of these profiles that do not seem to be humans. What both Pavel and Jane wanted to know is, who is behind these posts and why are they making them? I'm Christina Folk, and in this episode of the documentary from BBC Trending, we're tracking down a network that is behind a large number of these fake AI Holocaust posts and other similar ones. In the process, we'll uncover their ways of working and communicating. But first, you may be wondering, why would anyone do this?

5:41The answer is quite simple. Money. This is a growing economy. We found a global network of grifters and spammers that's emerged in the past year and learned how they're exploiting Meta's content creator program. By their own admission, some of these creators steal people's identities or outright fabricate stories for engagement. We found profiles impersonating government agencies, businesses, and other unsuspecting online users in order to target people in the UK, US, and across Europe. The amount of profiles I came across was so vast, I brought on a colleague for support. He has previously investigated how unscrupulous content creators have used AI to turn a profit.

6:27Last year, when AI Slop was beginning to take over Facebook, a lot of people just saw it as funny, but some of the experts were quite nervous about how many people were falling for it. This is Kevin Nguyen from BBC Verify. He's done a lot of reporting about social media grifters and says what I'm looking at is called content farms. So creators flood social media with as much material as possible to get as many eyeballs as possible. There used to be a lot of stolen and repurposed images. Sometimes it was cringy memes. Yeah, most of us have that relative that shares the like and comment unless you hate freedom kind of posts.

7:03AI Slop is just an evolution of these schemes. But how does that make money and who pays for that? The same entity that runs the platform, Meta itself. It explained how in a video announcement late last year. In case you missed it, Facebook has introduced Facebook content monetization, a new program that makes it easier for creators to earn money on Facebook on more content. Meta, like most social media companies, pay users to feed its platform with content. In December, the tech giant said it would be streamlining its paid programs under one umbrella called the Content Monetization Program. The exciting part, it said, is that you get paid based on performance.

7:42The better your content performs, the more money you earn. The marketing for the program showcased influencers such as musicians, dancers, educators. The person who made the announcement in that video for Meta is a doctor and mental health advocate. But it seems content farmers saw it as another opportunity. What I wanted to know is why the Holocaust was such a reoccurring theme. So Kevin and I started mapping out every AI Holocaust photo we could locate. This led us down a Facebook rabbit hole.

8:15There were dozens of AI Holocaust images, like a picture of a baby being left on the train tracks outside of a camp, or a girl inside the Auschwitz camp playing the violin. Some posts had tens of thousands of likes and shares. And they were being distributed across hundreds of patrons and groups. Some of these accounts are posting more than 50 times a day and had names like historical figures, history haven and timeless tales. They weren't just uploading Holocaust themed images either. It was anything vaguely history related from the Second World War or even the recent floods in Texas. What we could observe is that it was often themes and pictures that would evoke an emotional reaction like the one Jane had.

8:59When Kevin looked into some of the pages who posted these, he found something unusual. A lot of them were clearly coordinating with each other. This is when we started to look into a particular network that appeared to be responsible for a lot of these posts. They were connected to each other and as it appeared, worked quite closely together, closer than what we had seen before. They also organized themselves in Facebook groups dedicated to understanding the content monetization program. Here, they answered each other's questions and offered tips on how to get around Facebook's rules. These groups also quickly turned into a marketplace for buying and selling Facebook pages, something that Meta strictly prohibits.

9:38What we also found is that a lot of them were quite misleading on the pages they ran. When you go back far enough into these pages' histories, you can see what they used to be. This page called 90s Talk, which is sharing a lot of these fake Holocaust images, used to pose as the official Tennessee Fire Department's office in the US. This other page posed as a commercial fishing business. This one said it was a viral animal video page. I ascertained this was one of the first steps in crafting content to target specific types of audiences and then make the page eligible for the content monetization program.

10:12Meta also typically pays more views from places like the US and the UK compared to Asia. Although a lot of these pages claim to be based in New York City, we found people operating them were from Nigeria, Vietnam, Thailand and India, and a lot of the Holocaust AI slop was being driven by accounts that wrote in Urdu, the national language of Pakistan. Many of these Urdu accounts appeared genuine and were quite open about what they were doing. They posted pictures of their earnings on their own public Facebook profiles. Some creators claimed to be making tens of thousands of dollars through Meta's paid creator program.

10:49One account in particular uploaded a screenshot showing he had racked up more than a billion views across his images and videos in just a few months. Almost all of his posts were AI generated. Others were sharing how they did it, which led us to this.

11:09What you're hearing is a Pakistani-based creator and kind of social media broker named Fasal. To our surprise, in this video, he's explaining how he's buying and selling Facebook pages. Posted at the bottom of the video was a WhatsApp number. So we decided to get in touch with Farsal directly. That's coming up next. This is the documentary from the BBC World Service.

11:39Hello, Greg Jenner here. I'm the host of You're Dead to Me, the BBC comedy show that takes history seriously. Every episode, I pair up a top historian with a fantastic comedian, and we have a lovely, funny, fascinating chat about a different subject from world history. And in this new series, we're beginning with an epic voyage through the story of Homer, the Iliad, and the Odyssey. And that's with Kyle Smith-Bino joining us. And then we'll be learning about Francis Galton and the racist, discredited pseudoscience of eugenics with Desiree Birch. We'll meet many medieval saints in their bone boxes with Rachel Parrott.

12:12So if that sounds like your sort of thing, listen to You're Dead to Me wherever you get your podcasts. Thank you. Bye.

12:23This is the documentary. I'm Christina Feulk, and for BBC Trending, we are investigating how AI-generated pictures of Holocaust victims are generating views and revenue for content creators while misleading audiences around the world. We not only wanted to know why they were doing it, but if they realized the harm they could be causing. So we needed an in into the network, someone who could share the inner workings of the scheme. As we focused on the network based out of Pakistan, we started to work with our BBC Erdo colleague, Uma Dras Nagyana, who helped us track down some of the creators offline.

13:02We tried to reach out to those that make these Holocaust AI images. asking them if they are aware of the harm they could be causing. They did not respond to us.

13:18Then Uma managed to get hold of Farsal Rahman, who we had identified being part of the network that is getting paid through the Meta Content Monetization Program. Farsal admits to having created AI Slop in the past, but says he has not been creating these AI Holocaust images. He says he has a computer science degree, and this scheme has become his full-time job and only source of income. The key is traffic. It needs to be active and related to the content you are posting. If you are working on history content, then you should have traffic related to history, meaning a page with good, relevant followers.

13:58The more views you get, the higher your earnings will be. Fasal says with good quality content, And if the page has between 100 ,000 and 300 ,000 followers, the earnings are between around 530 to 1 ,060 US dollars per month. And often, these creators own multiple pages at the same time. In Pakistan, where the average monthly income is around 294 US dollars, this has become an alternative way to make money. However, Meta's content monetization network, which is invite-only, does not allow pages in Pakistan to join, says Fasal. He and the other creators in the network get around this by using software that makes it appear as if they are based in a different country, as well as email and PayPal accounts based out of the United States.

14:47Fasal told our BBC early reporter Uma how he sets up Facebook pages that impersonate others and gets content that engages with international audiences.

15:00Those pictures we get from different websites like Pinterest. If needed, we also generate them using AI. So in this way, people's privacy could be breached, meaning there could be copyright violations. Doesn't that make it illegal?

15:21People upload content voluntarily. Obviously, the purpose is for someone to see it. So, if we take that content and post it on our own page, there is usually no problem with that. But when you create or lift images like this from somewhere else, don't you feel it's unfair or wrong to use them?

15:47Basically, most of the work we do involves using general content, things that are widely available. We focus on viral content such as wrestling, cars or history. The content performs well. The earnings are decent because it is international, which people from everywhere watch. Now we knew how they are making the money. And Fasal just mentioned they pick history as a theme because it goes viral. often not researching the context of these events they are posting about. And Uma put the question to him about why the Holocaust seemed to be a focus of creators. And I tell you something else, why is it that so many pictures focus on Holocaust victims?

16:35I did not get it. What do you mean by Holocaust? You don't know about the Holocaust? No. My colleague Uma explained the Holocaust to Fasal, who responded by saying they don't post any graphic images. Fasal also says they choose content that resonates with the followers on their pages and points out why a closely knitted network is essential for his line of work.

17:03More people can handle the tasks, so you earn more and work becomes simpler. The work involves counting content, editing copyrighted material, uploading and monitoring the page for any issues or problems. If a team is handling it, these tasks are much easier to manage. So we work together as a team. Although it is strictly business for Fasal and the other creators in the network he's part of, the harm is real for the Auschwitz Memorial. When we looked through the comments under these AI slope posts, we saw several that outright denied the Holocaust. As we heard earlier, Paweł Sawicki was told in a phone call with Meta Poland that the AI Holocaust posts did not violate their guidelines.

17:47But many of the accounts we looked at posted similar Holocaust content. These accounts had clearly engaged in misrepresentation and the trading of Facebook pages. These are both in violation with Meta's own policies. We laid out what we found, along with some of the accounts and groups, to Meta. About a day after, we noticed that some of the accounts and pages we sent them had been taken down. And we also noticed they had taken down the posts the Auschwitz Memorial flagged in the first place. But only after, we pointed them towards it again. And then, they responded with this statement. We removed the pages and groups shared with us and disabled the accounts behind them for violating our policies on spam and inauthentic behavior.

18:33Meta said the content of the posts themselves did not break any of its rules, only the accounts. Meta rewards engagement in their content monetization program and allows AI-generated content. Whilst they're not intentionally encouraging users to post false stories of the Holocaust, the creators post these sort of AI slops to get higher rewards. We also mentioned in our initial email to Meta that the accounts were just one part of a much larger network. But other accounts and groups, even ones that were much more prolific, were still up and running. During our investigation, we kept checking Facebook, and my timeline is now full of AI Holocaust slop by new pages we haven't even come across before.

19:18So, what is all this AI slop doing to our online ecosystems?

19:26Sunny Leo is from the Stanford Social Media Lab and says moving away from human moderators to algorithms means losing the nuances and missing things. She says it changes the makeup of social media platforms and moves even further away from their founding ideal of a town square where people meet and exchange ideas. It's more like Times Square in New York. It's really loud, it's crowded, and all those kind of signs really just for people's attention. But if you think about the quality, maybe not there. She also says AI has a lot of positive potential, but it's being used to produce harm. It's not just stealing a page or somebody's image, but stealing their story, their traumas and their identities, but also fundamentally distortion of human history.

20:14Holocaust denial and distortion is not new. It's always there. But now deepfake and AI is another tool they use to do that. It's important to keep in mind that AI is a tool though, so not inherently bad or evil. Can it be used for good then when it comes to the Holocaust? I put this question to Dr. Robert Williams of the International Holocaust Remembrance Alliance. He also is the director of the USC Shoah Foundation that has previously used AI to keep stories of survivors alive and make them accessible in different languages. I don't think any individual or organization knows the exact approach.

20:52The concern I have is there are some organizations, very well intended, that are taking Holocaust content and using AI to manipulate that content. So it appears that survivors are speaking words through pictures of them when they were children. It contributes, I fear, to a sense that Holocaust history is somehow fabricated. We don't quite know the solution to making people use and understand the content better, but I think any form of extreme manipulation is something that we should shy away from. Dr. Robert Williams speaks a lot with survivors and families of Holocaust survivors, and he's discussed these fake AI-generated Holocaust images with them.

21:39What I have seen is the impact that this has on Holocaust survivors and their descendants. They are incredibly concerned by what they're seeing. They don't quite understand what they're seeing. And I've heard from several survivors themselves a certain sense of sadness that this has been allowed to happen. They feel like their efforts haven't been enough. And that's a very sad thing to consider because the last of the survivors will soon leave us. In my conversations with Jane, who you heard from at the beginning of this program, she had commented on an AI-generated image. She had a lot of questions.

22:22One was particularly important to her. How can we spot AI-generated content? I put this question to Sunny Liu from the Stanford Social Media Lab. And although she thinks no one can be expected to play detectives every time they open social media, she gave a few tips. Fundamentally, it's just ask ourselves questions a lot. Why I feel this way, why I see this content? I should pause and then check, especially for the content. They're really important to us. Millions of content we see today, but for the ones really important, has real consequence to us. I think for us, it's really just pause, think and check.

23:00That's it for this episode of BBC Trending with me, Christina Völk. Many thanks to Kevin Nguyen and Uma Drasnagina. This program was mixed by Andy Mills. Our editor was Flora Carmichael. Thanks for listening.