The AI Exodus Begins

0:03Hello, and welcome to the 404 Media Podcast, where we bring you parallel access to hidden worlds, both online and IRL. 404 Media is a journalist-founded company and needs your support. To subscribe, go to 404media.co. As well as bonus content every single week, subscribers also get access to additional episodes where we respond to their best comments. Gain access to that content at 404media.co. I'm your host, Joseph, and with me are the 404 Media co-founders sam cole hey emmanuel mayberg hello and jason kebler hey so one small bit of housekeeping we are having a party slash gathering slash event in los angeles july 30th wednesday at 6 p.m i'll put a link in the show notes but sam anything else you want to shout out now while i'm just doing it or should people go to the event page if they want to see us hang out and and talk about 404 go to the event page and buy a ticket or email one of us for the code if you're a supporter um if you're a subscriber in the supporter tier which means you're paying supporter of 404 media you can get in free um and we have the password so we'll have a post up on the site soon with the actual password for subscribers to get in but um until then you can just hit us up but yeah we're excited yeah sounds good again that's july 30th wednesday at 6 p.m check the event page for all the latest information but it'll be great to see some of our subscribers jason were you going yeah it's uh it's gonna be at a place called rip space which is like a hacker diy space um we'll probably do some sort of like live podcast live q a something like that we're sort of nailing down what it is we're going to do probably by the time you hear this there'll be more information on both the event page and our website uh but yeah we'd love to see you and there'll be a party after yeah sounds good all right we have as ever a bunch of stories to go through here the first section is going to be a series of the um pieces written by a manual all about ai uh porn civetai hugging face all of these different platforms are hosting it and there's there's been a ton of developments over the past few weeks and months in this world so the first piece is uh the headline of the first piece is a16z backed ai site civetai is mostly porn despite claiming otherwise so emmanuel civetai is this popular ai platform we've spoken about it a bunch you've done a bunch of investigations into it we've especially covered how it's used to you know uh host models and generate pornography what is civetai's stance on that do do they see themselves as this big provider of pornography or not well they definitely didn't used to and we can kind of talk through where they landed these days.

3:15But historically, Civitai, which is a platform where people share AI image generation models, they've always allowed adult content. They've always allowed AI models that generate the likeness of real people. They just didn't allow people to post models or images that combine both of these things, right? So no non-consensual content, that policy was always there. A lot of my reporting has been about how they failed to properly enforce that initially on their own platform, but then especially off platform. Also, while I was talking to them for all of this reporting, whenever I managed to get their PR person on the phone or talk to anyone there, they said that my reporting is unfair because yes, there is adult content on the website and they support free speech and they think people should be allowed to have porn models and have models of real people but that's really a minority of the content that is on the site and i have said in my stories and i have told them directly that that just like as a user experience as someone who goes to their main page if you look at the top models, the top images, the latest models, the latest images, it's just a never-ending stream of adult content.

4:43But I didn't have the data to back that up. So I've always kind of presented what a user sees versus what they claimed. They made this case especially strongly after I published this piece that showed that Civitai's cloud computing provider called OctoML had a lot of internal discussions about the kind of content they were generating for Civitai, and they were worried that they were generating child pornography for Civitai. And that really freaked out the company and they then went out a kind of like a mini pr tour with their ceo justin mayor and one of the things he did i think the day of me publishing the story right so like in order to coordinate with the deadline that i gave them he gave this exclusive interview to venture beat and he said uh let me pull up while you're pulling that up this is back in 2023 right this is an investigation you wrote pretty early on or 2024 maybe but quite early on in our history right yeah so this is end of this is December 2023 and VentureBeat writes that contrary contrary to those figures showing 60 percent of content on Civitai as not safe for work a figure derived from 50 ,000 images today users on Civitai generate 3 million images daily and the company says quote less than 20 % of the posted content is what we would consider PG-13 and above.

6:22So this is Justin Mayer, the CEO of Civitai, explaining to VentureBeat that the kind of audit that OctoML, this cloud computing provider, did on the content they were generating, they saw, they thought it was 60 % of the content was explicit. They say that it's less than 20%. Cut to recently, I wrote a story about Civitai changing a lot of its policies because of pressure from payment processors. So in May, they stopped, or they banned rather, all models that generate the likeness of real people and also specific types of porn that the payment processors thought was too extreme. I wrote about this.

7:10I published a story. A researcher from the University of Zurich reached out and said that she has been scraping Civitai and she has like really comprehensive data on what exists on the platform. And we can dig into some of the numbers if you want. But basically it shows that their claims that it was only less than 20 % of the content are absurd and the content is primarily used for adult content. Yeah, this is really, really interesting because you have your reporting. over years at this point where it's just like there's a lot of porn on here you know and i think you did a story a long time ago as well which basically showed that one of the the driving forces of capital and innovation in the ai industry is to develop porn basically and of course we all know that um from sam being the original reporter to cover deepfakes as well like it's always here all the time so you have the reporting you have um not to undermine it or anything but your experience of the website is more anecdotal right as you say you you log onto the website i don't have the data yeah yeah yeah exactly so i see a lot of there's a lot of poor models on here you finally have this data and we'll talk about what those numbers are in a second but you said this researcher was scraping civet eye um i don't know whether they got this granular with you but sort of how did they do that did they look for all images that were marked as not safe for work or did they grab videos and analyze them did they break down how they did the scraping exactly or did they not get the detail uh they didn't technically explain how they like gathered all the data i'm not sure what the specifics there are but what they the data is composed of more than 40 million images that they took from the site each of those images has a bunch of metadata that Civitai itself generates.

9:12This isn't the researchers annotating the data or analyzing it or anything. It's data that Civitai itself tags as using its own rating system of like R, PG-13, R, sorry, X, triple X, right? That's kind of its own method for tagging images. And then there's other tags that could include like names of celebrities or the type of sexual act that's an image uh and and things like this they also did something similar to the models themselves and they uh did that for 230 000 models i think are in the paper i've been in touch with the researchers and i've looked at some data that is more comprehensive than that i think they're up to like 400 000 is uh as what they have but it's not published yet and like in in a journal or in a draft yeah So it's interesting that it's based on the classifiers, where it's Sifiti itself saying, this is not safe for work, which, well, that meme, that gif of they just submit it, it's almost like that in data.

10:19So the research of Scrapesat has all these images, has all these classifiers very handily attached to them. Well, what's the research showing? What numbers are we talking about here when it comes to the prevalence of pornographic content on Civitai? So in December of 2023 is when I published my story about OctoML. That's when Justin Mayer gives this quote to VentureBeat. He says less than 20%. The researchers say that by October of 2023, 56 % of all the images were tagged not safe for work or higher, right? So that is already like more than double than what Mayer claimed. The actual number is likely higher because as both the researchers noticed and as I have noticed and reported, not everything is properly tagged.

11:22Some people self-identify their content as not set for work. Others do not and maybe try to get it under the radar. If you have some non-consensual content or a non-consensual model up on the site, maybe the people who are doing that are not tagging it properly because they're trying to evade moderation. So the actual numbers are higher. Now, I'm looking at a graph of kind of the distribution of adult content and safe for work content from the first quarter of 23 to the fourth quarter of 24. and Mare was incorrect or misleading in his statement back when he made it and back when I was talking to him for my reporting.

12:09But from that time until now, the number of overall images and models on the site has absolutely exploded, as has the proportion of adult content. It's like at this point, it looks like 80, 90 % of everything that is on the site is adult content. I mean, that's staggering, right? Yeah. I mean, it just, it shows that the reason the hearing from this person and reading the paper was very validating is because it shows that, like, my user experience of Civitai and what I thought the site is, despite what it was presenting itself as to the public and investors, it's just porn. It is just a porn site.

13:03I feel very comfortable saying that it is a porn platform. And actually, when I reached out to Justin Mayer for this story, he said he's standing by his venture beat comment, which I think are completely wrong. but he does now admit that somewhere around the beginning of 2024,

13:27the reason people use the site has changed. And I think that is probably because it's like, you can't deny it anymore. Just like by looking at the numbers, by looking at the site, it's just undeniable that it is primarily used for adult content, or at least was until these policy changes. Yeah. I mean, that's really, really interesting. and then just the last question on this article, then we have a couple more to touch on related to this. But you brought up these policy changes and we'll go into more detail about the exact why of this with the payment processes in a bit. But basically Civitai banned models related to real people in May, which is obviously a huge seismic change for any sort of AI platform.

14:09What happened when Civitai banned AI models based on real people, according to this researcher who's constantly scraping civetire? Like, did it fall off a cliff? Like, what happened? I don't think, first of all, I should say this is another very important number and another very interesting finding of the paper because they just recently changed the policy. So the researchers have all these models in a spreadsheet with the metadata and also the links to where the models live. So once Civitai announced that it was removing real people content from the site, it was very easy for them to check how many models were removed overnight.

14:56And the actual number is more than 50 ,000, which is another kind of interesting, you kind of, when I report about Civitai, it's like, you know, there's Taylor Swift and there's Natalie Portman and like, you know, all these big celebrities that have multiple models, and then you see there's a bunch of YouTubers and lesser-known Twitch streamers, but you're like, what is the actual number? And it's much higher than I thought. It's like, I can't even think. Obviously, there are multiples. But tens of thousands of people have bespoke AI models to create their likeness. It's just really kind of a shocking image, a shocking figure.

15:36But to your question, I don't think they still, they have like post policy change usage number and post policy change, like content distribution. They can just see what was removed that was posted before the end of 2024, I suppose. Yeah. So then this just leads on to two other stories. and the title of this podcast is something like the AI Exodus Starts. And that's where this comes in, in that Hugging Face, another AI model hosting platform, you have a headline here, Hugging Face is hosting 5 ,000 non-consensual AI models of real people. So are these some of the models that were on Syvati and then they got removed and now they've moved over here?

16:30Like, What's the deal here? Yeah, so basically credit card companies come to Civitai. They say, hey, you have to remove all this content, all these real people models, or we're not going to work with you. They are not working with them anyway. Still, even after they remove that content, maybe Sam wants to talk about this, but we saw something similar with Pornhub. It's kind of really hard to undo a payment processor cutting you off after they make that decision. and that's where Civitai is at the moment. But when they initially announced this, they were like, hey, we're making these models invisible.

17:10The people who made them can have access to them for an unspecified short time. And then we're just nuking them off the site entirely. And the moment that they announced this, the Civitai community immediately mobilized and started this Civitai, or actually the Civitai archiving project already existed in the form of a Discord channel because there was other policy changes that kind of made people feel like something was coming. And they were starting an effort to archive anything because they were afraid that Civitai would remove it. Once this news came out, it really kicked into high gear and they started grabbing everything that they can in order to make it available elsewhere on the internet.

17:54and because of this research that I talked about in the previous story I had a spreadsheet with all the links of all these real people models where they used to live on Civitai and the archiving project created a website where you take that link you enter it into the website and it directs you to a mirror where that model was re-uploaded to Hugging Face. So Hugging Face is like an AI tool and resource sharing platform. It's very popular. It's got like a multi-billion dollar valuation, a partnership with Amazon. They've been very outspoken about wanting to be like an ethical AI company and what that means to them.

18:42I talk about that a little bit in the story. they don't have any specific policy against models that generate the likeness of real people, but they do have a lot of kind of vague language about AI being, needing consent in order to be ethical. And I told hugging face that I have this data that I know of at least 5 ,000 models that are hosted on their site and not just random models, models that I know were used to create non-consensual porn models that were removed from a different as we just discussed porn site because they were used for this purpose and i just haven't heard back from them after multiple attempts uh you haven't heard back from the ethical ai platform company about the unethical stuff happening on their platform that's right yeah and i've talked to them before and they've replied to my emails and i don't know why they're stonewalling me about this one but they are i think i can say we're recording unusually we're recording this podcast on a monday um by the time this podcast is out my story will be out and we'll see if they have anything to say then but i i tried really hard to get any comment from them and i think if they asked then i would give them all the links and then they can decide what to do about them but they were just not interested in doing that um and then we'll just touch on your last story very briefly but yes i just wanted to ask sam for a moment.

20:11This last one is about payment processes and sort of the power that they have in the porn industry, right? And for those who don't know, well, sorry, I'll ask this to you, Sam. For those who don't know, what do we mean by a payment processor exactly? And why are they so important to the adult industry? So as Emmanuel just alluded to, Pornhub has had this problem and a lot of porn sites have to abide by the rules of payment processors. So, I mean, when you're talking about like the different like payment processors and gateways and banks, it's like kind of, it ladders up through the institutions that we're talking about.

20:53But so like, for example, Stripe has really strict rules against porn and sexual content. um visa and mastercard stopped processing payments for uh pornhub because uh there was allegations of abuse material on pornhub um you know it's like chase has have rules about not safe for work content adult content it's because they're considered high risk quote-unquote high-risk material um so in the same in a similar vein as gambling or guns um they place sexual content in a similar kind of risk category as as a bank and then that trickles down to payment gateways and the processors and things like that so um you know it's like it's if it's working ideally what you get is like these banks are not having to be accountable for uh abusive content like what Emmanuel's talking about.

21:54What happens a lot of the time is the banks end up acting as moral arbiters for porn. So you have a lot of consensual adult content creators caught up in the rules against abusive content. These payment processors don't want to take the time to sort out the difference in a lot of cases and just ban it all. And we've been talking about this for years happening to like consensual and safe and um legal adult content and it's like these other platforms like subatai and um some of the other platforms that are now having trouble with payment processors they're fucking around and finding out at this point it's like this is you thought you could kind of just like get away with doing whatever and you were above it because or cool bros or whatever the logic is.

22:50I don't even know. It was making a lot of money, so surely it was too big for them to give a shit. But I think now they're finding out that the same rules apply to them. And I don't know. We can't really put these things in the same bucket as legal, safe adult content. But they do a lot of the time have to abide by similar rules yeah and it's coming for them now and i and i guess just to wrap it up a manual what is what is the impact from these payment processes um deciding they don't want to work with civetai or another ai platform like what happens after that sort of action has taken place and what's going to happen now very quickly i would just say i think last week i talked a lot about how the reason I was covering Civitai is that to me, it seemed like a critical piece of internet infrastructure in the entire practice of producing non-consensual porn.

24:01And like, what happens if you remove that? Where do people go? And something I saw people say after the Civitai announcement around real people is that they're just going to go to tensor.art which is a site i covered like i think the first week we launched but it's a site that's identical to civetai basically it's the same services same ui you go there you download models you can generate images yada yada a lot of people move their models there for a while tensor art had like a tool that even helped people import their civetai models automatically. And on Friday, TensorArt announced that they are getting the same pressure from payment processors and they're no longer, they're saying temporarily, I don't know how they resolve this, but they're saying temporarily, no more real people content.

24:54They've disabled this Civitai importing tool and they're trying to come up with some sort of solution. I'm very skeptical that they They can find a solution and still have the payment processors work with them. Because as I've said, sometimes even if you fix every problem, they don't want to come back. So I'm just going to continue to track like a huge, like there's a huge vacuum now and how non-consensual content is made online because of Civitai policy changes. And I'm just tracking where all this energy and all these people are going. And that's probably going to be some more reporting in the future.

25:30Yeah, totally makes sense. All right. We will leave that there. When we come back after the break, we're going to talk about one of Sam's stories about preserving history, very much in the vein of the Trump administration continuing to wipe things off the internet. But this is now sort of IRL archiving. We'll talk about it in a minute. We'll be right back after this.

26:03Time, skills, and money. When it comes to growing your business, we all know that engaging with our customers is crucial. But the question is always how. Most of us don't have the time, skills, or money to put towards keeping up with marketing every day. We're too busy running our companies to begin with. Well, let me introduce you to Constant Contact. Your mini marketing team awaits. Constant Contact's award-winning marketing platform is here to make marketing way easier and way more effective for small businesses like yours. The best part? You don't need to know anything about marketing. Your struggle with expensive, slow, and unmeasurable marketing is over.

26:43With their all-in-one platform, you can create and manage attention-grabbing campaigns in just a few clicks. Email, text, social media, events, landing pages, you name it, it's all in one place. You also get automated sending, real-time reporting, and tools that actually help drive sales. So you're not just marketing your business, you're growing it. And if you ever get stuck, friendly phone support is included with every plan. That's why over half a million small businesses trust Constant Contact to stay connected, top of mind, and ahead of the competition. Constant Contact, helping the small stand tall.

27:18Every podcaster, content creator, and business knows that in an increasingly fractured information ecosystem, it's getting harder to reach your audience. Constant Contact can help. Get a free 30-day trial when you go to ConstantContact.com. Try Constant Contact free for 30 days at ConstantContact.com. ConstantContact.com. You know what doesn't belong in your epic summer plans? Getting burned by your old wireless bill. While you're planning beach trips, barbecues, and three-day weekends, your wireless bill should be the last thing holding you back. That's why I made the switch to Mint Mobile. With Mint Mobile, you can get the coverage and speed you're used to, but for way less money.

27:59And for a limited time, Mint Mobile is offering three months of unlimited premium wireless service for 15 bucks a month. So while your friends are sweating over data overages and surprise charges, you'll be chilling, literally and financially. Say bye-bye to your overpriced wireless plans, jaw-dropping monthly bills and unexpected overages. Mint Mobile is here to rescue you. All plans come with high-speed data and unlimited talk and text delivered on the nation's largest 5G network. Use your own phone with any Mint Mobile plan and bring your phone number along with all your existing contacts. Ditch overpriced wireless and get three months of unlimited service from Mint Mobile for$15 a month.

28:40I switched to Mint Mobile from one of the big incumbents and the only thing I noticed was a much smaller bill. This year, skip breaking a sweat and breaking the bank. Get this new customer offer and your 3-month unlimited wireless plan for just$15 a month at mintmobile.com slash 404media. That's mintmobile.com slash 404media. Upfront payment of$45 required, equivalent to$15 a month. Limited time new customer offer for the first 3 months only. Speeds may slow above 35GB on the unlimited plan. Taxes and fees extra. See Mint Mobile for details. This limited edition inbound September 3rd through 5th brings attendees to San Francisco for a one-time only West Coast event with insights they won't find anywhere else.

29:29Just revealed, the inbound 2025 agenda is now live. From the Agent.ai workshop, From Idea to Agent, and Dwarkesh on AI's Future, Research-Backed Bold Predictions with Dwarkesh Patel, explore 200-plus sessions all created for your growth. Get fresh perspectives on innovation from a dynamic lineup, including Sean Evans, the host of Hot Ones, Creative Force Amy Poehler, tech reviewer Marcus Brownlee, and AI pioneer Dario Almede. They'll bring their unique approaches and expertise to Inbound 2025. Cut through the noise with focused, actionable takeaways on the latest marketing, sales, and AI trends that give businesses a competitive edge in today's rapidly changing landscape.

30:12Network with decision makers in San Francisco's AI-powered ecosystem. where innovative technologies are creating entirely new approaches to business. Experience firsthand how San Francisco's technology ecosystem is revolutionizing content creation, distribution, and monetization through AI and innovative tech solutions. Secure your spot at inbound.com slash register. And if you want the VIP experience at Inbound, don't wait. VIP tickets are almost sold out. Get exclusive perks that help you make the most of every moment, including a welcome party, exclusive networking opportunities, and early registration access to limited capacity sessions.

30:51Registration for limited capacity sessions begins in August. Don't miss this opportunity to secure your ticket in advance, start favoriting your must-sees, and be ready when reservations open. Session reservations for VIP ticket holders begin August 5th, and GA opens August 12th.

31:16All right, and we are back. Sam, you wrote this one. The headline is, Save Our Signs wants to save the real history of national parks before Trump erases it. First off, can you just give us a little bit of context of what is the Trump administration doing to national parks and the history around them? What's the issue exactly? Yeah. So like many things these days that are just like bad news, dark news, started with an executive order in March. Trump issued this order called Restoring Truth and Sanity to American History. Already off to a really promising start with that title. Um, the order mostly like targets the Smithsonian museum institution and that network.

32:12Uh, it makes some really bizarre claims about, uh, the museums being part of this like revisionist movement, um, that they're trying to paint historical milestones in the negative light for acknowledging racism in American history, things like that. um so the the order is mostly about that but because it also mentions the department of the interior that catches national parks and monuments in that net also so um i can just read part of what the the order says because it's written like so many of these a very kind of slippery, vague way. Yeah, I'm sure it was Lord de Hellenbeck, but also very wordy.

33:06I mean, who knows? You know, it's nothing was in all caps. So I'm sure it got at least one proofread. But yeah, it says that the Department of the Interior must take action, whatever that means as appropriate and consistent with law to ensure public monuments, memorials, statues, markers, or similar properties within the Department of Interior jurisdiction do not contain descriptions, depictions, or other content that inappropriately disparage Americans' past or living, including persons living in colonial times. Interesting carve out. And instead focus on the greatness of the achievements and progress of the American people or with respect to natural features, the beauty, abundance, and grandeur of the American landscape.

33:54So, I mean, if we can use our contextual thinking skills, take action. I think we can assume means remove or edit monuments, memorials, and plaques basically at national parks or anywhere that's under the Department of Interior's jurisdiction that say anything about anything other than how pretty the landscape and the monument is. And not disparage people, which would rule out quite a bit of American history if you can't really tell the truth about what was going on. Yeah, so purely hypothetical situation, you can have a sign that says, wow, trees look nice, mountains look good, but you can't say anything like, well, the indigenous people of this land are XYZ or something like that.

34:44Is that basically the thrust of it? Okay. Yeah, I mean, it's and again, it's like this, the order doesn't really use any very specific examples for national parks. But given the the attack on like, so called DEI, the way this administration has been moving up until this point, we can assume that it means, you know, things like talking about indigenous people being slaughtered in this country, things like racism and civil rights. um, and all of that, that actually built a lot of the, um, helped build a lot of the park system. Um, black indigenous people played a huge part in building the parks.

35:26Um, and you know, we're many of them were there first, um, things like that. It's just like, there's a lot of history going on in the parks that we just, um, are apparently no longer allowed to acknowledge, um, environmental justice, um, women in leadership, things like that, that the administration has targeted in the past already online, we can assume probably counts in this case too. Yeah. I mean, I haven't really been covering any of the sort of removal of history by the administration, but I don't know. That just drives me crazy, the idea that you were going to delete history. Jason? I'm curious if this stuff would be foiable.

36:12Theoretically, it would be. I think the crowdsourced aspect of it is very cool and like makes it a group project. But I, but theoretically, like the national parks offices should have copies of this somehow. Like they had to go get them printed, et cetera. And theoretically, all of this could be obtained by FOIA, which is, yeah, I don't know if they brought that up at all. I think that this is maybe easier than FOIA. like FOIA is a roll of the dice at this point but um you know a FOIA about this and maybe we'll file some would theoretically capture like um attempts to take it down or like alternate versions of the um like what's being changed etc like if it were um responded to appropriately yeah totally and um I just looked it up and got the email address or the portal for the national park service so we can do that or do it through muck rock or or whatever um but what these people are doing is sort of this other archiving project so with all of those problems in that context enter save our signs this campaign's then launched um but what's the goal of this uh campaign sam and who is involved exactly and how are they going about it what are they doing yeah so um like seeing like the writing on the wall so to speak not to use a corny pun for this particular story that was pretty good um the writing on the placards um data preservationists from the safeguarding and research and culture project and the data rescue project uh launched this web portal called zero signs um they're asking people if you're at a park if you're at a national park to just take a picture of pictures of um placards and signs and monuments and that um they see as they come across them to preserve them to archive them essentially because they could be under threat we don't again we don't really know what's going to go on like the the take action line is very vague but um you know it's um it's something where they're the government is asking for um visitors to fill out this qr code survey to report signs that they think are negative so it's kind of like a counter protest to that saying you know if you're gonna if um trump administration is gonna ask people to essentially snitch on signs that address um you know anything other than beauty and grandeur of the american landscape and you know flag these placards that could fall under whatever the administration's definition of negative is then they're going to turn around and say you know what we're going to archive this and make sure it's preserved in case it goes anywhere in case it um is removed and the negative quote-unquote negative content must be taken again using the words of the the order taken action against by september 17th is what the order says so they're launching this to um basically say we have to hurry up and preserve this in case something happens to it um in the future but i mean the trump administration has totally gutted the park service at this point.