We're Not Ready for Chinese AI Video Generators

0:03Hello, and welcome to the 404 Media Podcast, where we bring you unparalleled access to hidden worlds both online and IRL. 404 Media is a journalist-founded company and needs your support. To subscribe, go to 404media.co. As well as bonus content every single week, subscribers also get access to additional episodes where we respond to their best comments. Gain access to that content at 404media.co. I'm your host, Joseph, and with me are two of the other 404 Media co-founders, the first being Sam Cole. Hello. And the other being Emmanuel Mayberg. Like, hello. I think Jason's on a plane right now.

0:42I hope Jason's on a plane right now. I left Jason behind in Austin. We had a busy night. Last night was our big takeover and after party. Flipboard kindly hosted us for a shindig downtown for a self-by. So yeah, it was a long night. My voice is a little hoarse, so I apologize. guys. It's got a little fry, more than usual. And I'm a little hungover, but we're here. Yeah, I just got in this morning. So it was an amazing time. It was really good to see everybody. I feel like I get just as hungover by speaking loudly at a party than I do from the alcohol. Yes, it's a whole... And yeah, I'm an inside cat.

1:28I do not go to networking events much unless we're throwing them. So it was amazing to see everybody and meet everybody and met a lot of podcast listeners, which was very cool. Yeah, it was a good time. You heard that some people only knew about the event through this podcast, right? Yeah, yeah. A few people came up and said, oh, yeah, I heard about this happening on the pod. I was like, that's amazing. We plug it on the podcast because we know a lot of people listen. But it was cool to see how many people actually are tuning in. so yeah yeah and it brings up one thing i'll just say super briefly which is that clearly some people only listen to the podcast rather than reading the website and at first when we sort of realized that i found that quite strange and then i realized wait i do that myself i listen to the verge podcast every single week twice a week if they've uploaded without fail but then i generally don't read the website just because i prefer to digest it that way so if you are one of those people who only listens to the pod.

2:33Thank you very much. We really, really do appreciate it. And the pod is growing. Um, with that said, let's get to our stories because we have a bunch of complicated and interesting stuff to get through. The first couple of stories, which we're kind of putting together, they're written by Emmanuel and the headline of this first one, Chinese AI video generators unleash a flood of new non-consensual porn. Um, Emmanuel, this is something you've been working on for a long time. What is the top line of your investigation? Is it about the guardrails of these Chinese developed AI models? What's the top line?

3:10The top line of the investigation is that there are a bunch of AI video generators that are available via apps that you can get via your web browser or the app stores. And I don't think many people know about them because they come from smaller, lesser-known AI companies. And to explain for a bit why I have been working on this for so long, I think we need to travel back in time to, I think it was February of last year, when OpenAI revealed Sora, which is their AI video generator. And I think that's the first time that people saw kind of high-quality AI-generated video. And that really blew people away.

4:01It blew me away. They kind of came out with those video samples out of nowhere. And that tool wasn't available at the time. And it wasn't available for a long time. It is available now if you pay. They were more just showing it off, right? Yeah, they were just showing like, look at how powerful AI videos can be. And they definitely delivered that message. What I did that day is immediately go to the chat rooms where I monitor the communities that create and share non-consensual AI-generated pornography. and I wanted to see how they reacted. And obviously they were kind of salivating over having access to those kinds of tools, but it wasn't really a concern, A, because OpenAI didn't give them access and they rolled it out very slowly and additionally, very safely.

4:54OpenAI is notorious for having pretty strong, some would say overbearing, unnecessary guardrails around all their AI tools. and that is true for Sora as well. But what happened in between the time that Sora was announced and today is that a bunch of other competitors in the A market rushed to launch competitors. And at first, these competitors seemed not nearly as good as what we saw from Sora. But as everyone who's familiar with this beat by now knows, AI kind of develops at a very fast pace. And now there are a bunch of AI tools that produce pretty damn convincing video. And for reasons that we can speculate on here in a minute, they just have really, really, really bad AI prompt guardrails, right?

5:59So people might remember one big story we did. God, it's 2023 now, I think. No, it was also 2024. But that was around people using Microsoft Designer and kind of writing prompts that tricked it to generate non-consensual images of Taylor Swift. And that loophole we've seen replicated in many AI tools since then. But overall, the big AI companies realized that people were using this loophole and abusing it and have gotten a lot better at having guardrails against that sort of abuse. These newer AI companies, not so much. And over the months, I've seen people find these AI tools, find the loopholes, and just generating, at this point, mountains of non-consensual videos of celebrities.

6:56so yeah it sounded like it almost started with the taylor swift stuff that we reported on obviously and you got and i think sam as well on that piece as well got more information and then everybody saw it because it went viral on on twitter or whatever now you're saying these other uh predominantly chinese developed ai models um are basically being used for the same thing but to a much much larger degree it's not just some taylor swift stuff it's all of these different celebrities. When it comes to the video generators themselves, what are they doing exactly? Do you give it a video and it auto-completes it?

7:38Do you give it a photo and it animates that? Do you give it a text prompt? What does the user put in to then make all of this stuff? It's both. You can generate something out of what appears to be nothing but it's not really nothing because it's pulling from huge data sets that the AI model was trained on. But you can basically write a text prompt and generate a video that way. Or, and I think this is important, you can do image to video, which you give it a still image, and then in the text prompt, you write how you want the AI tool to sort of animate that image. And the latter is harder to moderate because the text prompt, you can fairly easily filter out terms that you don't want people to use.

8:38And those can be names of celebrities, nicknames of celebrities, and a bunch of sexual terms, right? That's like a fairly easy way to filter out a bunch of bad content. You could do that with images, but that is much more complicated because then you need to train other AI models to recognize a person in the image or recognize nudity in an image. And that just takes a lot more effort to filter out those kind of visual prompts. It just appears that the AI tools that I've found, most of which are developed by Chinese companies, are not doing that very hard work of visually detecting images that are used to animate pornography.

9:26Yeah, because it could be a red carpet photo of a celebrity or something like that. It's like, what, are you going to ban all red carpet photos? Well, then they'll just find another photo or something like that, right? so what are people making exactly is it like sort of lewd images where maybe they get a celebrity flashing their breast or something like that or is it um you know more full-on pornography obviously there's degrees here i mean it's all non-consensual it all sucks but like what are people making broadly so i think if i went into these chat rooms and uh put all these videos into a spreadsheet and counted what is the most popular type of video i would say it is probably videos of female celebrities taking their tops off the reason for that i think is that there's a very popular tool called Pixverse, which is just...

10:29I think, to be fair, is used for non-harmful reasons by a lot of users. But it's just an easy, accessible tool. You can get it on the web. You can get it via the Apple App Store. So it's very easy to access. And this community has figured out how to abuse it in this specific way. They found the specific written prompt that you can use to create that kind of video. And it's really easy. So that is the most common one. But what I saw is that people move from tool to tool depending on what kind of video they want to generate and what kind of vulnerabilities they're finding in each tool. So Pixverse is good for that.

11:15Then there's this other tool I talk about later, which is actually from an American company called Pika. And that one can produce straight up videos of oral sex fairly easy. And it looks a bit janky. It definitely looks weird. But it is also, I think, would be fairly horrific to find someone doing this to your likeness. And again, it just takes an image or or something like that, depending on the tool, but it's very low effort if they have the prompt workaround. What specific Chinese tools are we talking about? And are they like little upstarts? Are they like open source projects? Are they well-funded operations?

12:05Like who are these or what are these tools? And sort of where do they come from? So it's this new crop of AI tools that are doing this exact thing I think I talked about in the beginning, which is OpenAI presents Sora, and everybody sees the potential in that. And rather than be careful and release it very slowly and safely, their tactical move is to just get to the market first, get something that is honestly probably not as powerful as Sora, and definitely not as safe. but it's really easy to access and there is a demand for this kind of tool and just getting there first and giving people access, that's just a better business strategy or the only competitive business strategy that they have.

13:05They are well-funded. They all have millions of dollars in venture capital. for example this app that i talked about pixverse it has some notable people like the person at the head of that company used to be the head of machine vision at byte dance which is the company that owns tiktok so they're new companies it's like a new generation of ai companies we're seeing similar type and scale of company in the u.s uh but in this case they happen to be almost exclusively chinese yeah and there's a line in the piece where you say 404 media is not sharing the exact prompts that produce these videos um can you explain why i mean i think it's obvious but i think it's useful for people to hear why you don't include these exact prompts but then also can you just mention so you won't mention the prompts but you'll mention the companies like is that because well they're massive multi-million dollar companies like why wouldn't we name them like what's the thinking there yeah i mean as sam knows very well uh when covering this kind of thing you're always trying to walk this very complicated line where you want to report about an important issue you want to name and shame basically these companies and hopefully apply pressure not just on them to build better protections, but also build pressure on Apple and Google who are making these apps accessible via the App Store.

14:45But at the same time, we obviously don't want to teach people how to create very harmful content. So we're kind of sharing the responsible parties here, but we're not sharing the communities where people will teach you how to do this or the specific prompts that will generate those harmful videos yeah sam what do you think of that balance between highlighting something because there's a public interest is important while not amplifying the bad stuff like how do you figure that out in your head yeah i mean it's just so hard to to write about something and especially to illustrate what's going on without saying what's going on, like saying it plainly.

15:31I think when we kind of beat around the bush, so to speak, and try to use euphemisms or descriptors that aren't exactly, not the prompts, but naming what these companies are. For example, last week we talked about Instagram gore. It's like saying exactly what people are seeing without being cute about it, I think is really important as journalists. Um, so it's, it's a hard calculus. I mean, it's, um, definitely something that I struggled with a lot early on, kind of figuring out when to, um, be very blunt about these things and name the companies and when to not. Um, and I think a lot of it comes down to it's, I think just, I think a lot of people's reaction when they see like, oh, you're talking about like a telegram group or an AI model or a tool or whatever it is, it's like, they're like, oh, I haven't heard of that.

16:26So you're amplifying it. It's like, no, actually tens of thousands of people have heard of this. You just haven't. Lots of people are using this and this company is making a lot of money on those people. And it's doing, in a lot of cases, real harm to a lot of people. Just because you haven't heard about it doesn't mean it's not already a huge thing. Maybe it'll become more of a mainstream thing and more pressure will be put on it to, like Emmanuel said be taken off the app store and things like that but that's also out of our hands it's not really a part of our job to to do that so yeah it's a tricky thing for sure it's something we think about i think every time one of these stories comes up yeah i mean i think the stakes of this example i'll say are lower but almost the quintessential one i always remember is when gawker first covered the silk road website you know people were debating like oh you shouldn't do that because now people will know to go do it and say i don't know man the revolution in marrying bitcoin and the tor anonymity network to allow the borderless online exchange of narcotics is probably something that's worth putting in an article and it applies here when it comes to like the unbridled use of this ai technology and and on that so just to get back to the the models we're talking about a little bit, Emmanuel.

17:44So it's mostly Chinese in this article. There are some US ones or one you did mention. Is it more that the new generation of ones without guardrails just happen to be Chinese and that's sort of why Chinese isn't the headline? Or is that sort of a commonality when it comes with Chinese companies that they just don't have these guardrails? How exactly does the China element play into it? Because, of course, you're always careful, and we all are here, but recently we had the DeepSeek stuff, and people lost their fucking minds over the Chinese element. That's a little bit different, because it's like, well, you're giving data to a Chinese company, blah, blah, blah, but is it more, they just happen to be Chinese here, or what's that?

18:32So I think there's two things that are happening. There are... American companies that are doing this. There is like American competition, but I think there is less. And we've written about some of them. I wrote a story about one app called Dream Machine. Sam wrote a huge scoop about... I forget the name. The AI video generator that we got the training date on. Runway. Runway. Yes, thank you. Runway. We're forgetting all our stories. That's a long time ago. A lot has happened. Yeah. So they exist. But I think one thing that is definitely happening is that... It's just that the Chinese companies saw an opening.

19:19We're in this great competition in the AI industry between the US or the West and China. And it was just a place for them that they can get ahead a little bit. And they did. And I think that's one thing that is happening. The other thing that I think is happening, and I haven't been able to prove this, and I didn't put this in the article, but I feel comfortable saying it here. And I invite people who are listening who might be interested in safety or red teaming and might be able to teach me about this, honestly. But I do suspect or I wonder if there's a language barrier problem here where the American companies are just better at building what we call the semantic filtering, right?

20:02Like the word-based filtering of prompts where the Chinese companies, since they're initially built for Chinese markets and are prompted in Chinese, maybe have fairly good filtering in Chinese, but the English language filtering is not as good. I was wondering if that's one issue here, but I don't know that for a fact. That's super interesting. Yeah. I think there's one more thing you wanted to mention on this story, Emmanuel, before we move to Alibaba. There was the apps and the models, right? Right. So just to transition here to the next story, So far, we've been talking about apps. These are user-friendly, consumer-grade, anyone-can-use-them, are advertised-to-the-average-user type of tools.

20:56The other thing that is happening at the same time is a rerun of what we've seen with this website called Civitai and these more open models as opposed to apps. So there's two Chinese companies, Tencent and Alibaba, which are two of the biggest tech companies in the world. And they have released essentially the video version of Stable Diffusion, which Stable Diffusion also does video. But Stable Diffusion is basically an open weights model. It's an AI image generation model that you can tinker with to customize it and make it better at producing specific type of images. And Tencent released this tool called Hanyun.

21:50I hope that's how you pronounce it. And Alibaba produced this other model called WAN. And they're just the exact same thing. They released all the documentation. There was a GitHub where you can go and download the code and tinker with it. And as soon as this happened, very rapidly, the exact same thing we saw with AI images happened. The models were adopted by the Civitai community. They were modified to create videos of highly specific sexual acts and fetishes, and then also videos of very specific small-time YouTubers and Twitch streamers, Instagram influencers. and while Civitai at this point is pretty good at preventing you from posting non-consensual content to its website it also makes it incredibly easy to like I'm going to take this AI video model that has been designed to create videos of blowjobs and I'm going to take this other AI video model that's been designed to recreate the likeness of this Twitch streamer that I like and you kind of put them together and make non-consensual videos, which are also of much higher quality than what these apps that I talked about do.

23:15But at this point, they are more difficult to produce. You need to navigate Civitai, know how to run these models, either do it locally on a fairly powerful GPU or rent that GPU time in the cloud and setting up the workflow for that. And it's not impossible. I could figure out how to do it. Anyone can figure out how to do it. But it is several degrees more difficult than just downloading an app and clicking generate. Yeah. And I think just the last thing on that before we take a break is when Alibaba released this open video model and then obviously then got used for porn, as you reported, what was the actual intention by with releasing this like what why did they want to release it what were they hoping it was going to be used for that's a complicated uh question to answer because it gets into this greater debate of like why is mark zuckerberg releasing llama as an open model right so the theory is that it becomes uh widely adopted across the world and then question mark, question mark, question mark, monetize it somehow.

24:32It's how that works out. That's kind of like above my pay grade, but you're just like, the plan is to make it open. So as many people as possible adopt it. So the technology is developed by a community along with the company and has a lot of investment from that community. And then you sell them something. I don't know how that last stage works out. But it's just the open model of AI. Step for profit. You don't need the in-between steps. That's just how it works. All right, we'll leave that there. Really, really amazing stuff. When we come back, we're going to be talking more about AI. This entire episode is about AI.

25:14I mean, we're going back to almost our 2023 roots. But it's going to be about how police are using AI. when it comes to analyzing seized evidence. We'll be right back after this.

25:42I don't know about you, but I like keeping my money where I can see it. Unfortunately, traditional big wireless carriers also seem to like my money too. After years of overpaying for wireless, I finally got fed up from crazy high wireless bills, bogus fees, and quote-unquote free perks that actually cost more in the long run and switched to Mint Mobile. Switch to Mint Mobile and you'll save in the short and the long run. It's easy to switch. You can keep your device, your phone number, and your money. Say bye-bye to your overpriced wireless plan's jaw-dropping monthly bills and unexpected overages.

26:19Mint Mobile is here to rescue you with premium wireless plans starting at$15 a month. All plans come with high-speed data and unlimited talk and text delivered on the nation's largest 5G network. Use your own phone with any Mint Mobile plan and bring your phone number along with all your existing contacts. Ditch overpriced wireless and get three months of premium wireless service from Mint Mobile for$15 a month. If you like your money, Mint Mobile is for you. Shop plans at mintmobile.com slash 404media. That's mintmobile.com slash 404media. Upfront payment of$45 for three-month five-gigabyte plan required, equivalent to$15 a month.

26:57New customer offer for first three months only, then full-price plan options available. Taxes and fees extra. See Mint Mobile for details. We're journalists, not business people. At least, we weren't until we started 404media. Starting the business meant figuring out how to open up an online merch shop, which is something that I was honestly dreading. Then I found Shopify. Shopify is the all-in-one online and IRL store solution that you can get set up in minutes. Nobody does selling better than Shopify, home of the number one checkout on the planet. Shopify is not so secret secret. It's ShopPay, which boosts conversions up to 50%, meaning way less carts going abandoned and way more sales going.

27:41If you're into growing your business, your commerce platform better be ready to sell wherever your customers are scrolling or strolling. On the web, in your store, in their feed, and everywhere in between. Upgrade your business and get the same checkout we use with Shopify. Sign up for your$1 per month trial period at shopify.com slash media, all lowercase. Go to shopify.com slash media to upgrade your selling today. Shopify.com slash media. This episode is sponsored by BetterHelp. Therapy can feel like a big financial investment, with a single session costing anywhere between$100 to$250. For most people, that can be out of budget.

28:25But with BetterHelp, you can save on average 50 % per session. And more than anything, it's an investment into your mental health, and your mental health is worth it. With therapy, you can learn positive coping mechanisms and learn how to set boundaries with yourself and others. Therapy isn't just for those who have experienced major trauma. It's also for anyone who wants help with navigating their everyday life experiences just a little bit better. With over 30 ,000 therapists, BetterHelp is the largest online therapy platform. The platform has served over 5 million people around the world. And with BetterHelp, you can switch therapists at any time.

29:00And if you're busy like we all are, you can join a session with just a click of a button. Your well-being is worth it. Visit betterhelp.com slash 404media today to get 10 % off your first month. That's betterhelp.com slash 404media.

29:30Okay, so this next story is from Joe. The headline is, Celebrite is using AI to summarize chat logs and audio from seized mobile phones. I think, let's start with, what is Celebrite? A notorious company in our little world, but for people who don't know, what is Celebrite? Yeah, so Celebrite is an Israeli company. I mean, it has US subsidiaries and stuff, I'm sure, but predominantly an Israeli company. and it's basically ubiquitous in the world of law enforcement so when a police officer seizes a mobile phone um maybe that's at the border with cbp maybe that's ice when they arrest somebody maybe it's a cop a traffic stop what they'll often use is a piece of technology from this company called celebrate and you know it comes in lots of different forms there but the generally the tool is called, I pronounce it UFED.

30:28I'm not sure if that's entirely correct, but I like that it has FED in there, UFED. And you plug the phone in. If it needs to, it will crack or bypass the password or the passcode requirement. And then it will download all of the data on the phone. You know, in some cases, able to get deleted chats or chats that you thought were deleted, it but they weren't forensically deleted necessarily um and then it takes all of that information and the police officer can you know safely store it for later you know if there's i don't know a murder investigation and you have the phone of the victim you want a forensically sound image of that phone or if someone's crossing a border you just want to download everything on their phone and rummage through it without a warrant because that's what authorities um are able to do there's a funny sidebar about how celebrate started which is that when you would go into the apple store and you were like an android user but you wanted to change to apple ios apple would have celebrate devices in there because it could help transfer your data i don't think that's the case anymore because i don't think that's necessary but it's just funny especially around the san bernardino attacks of 2016 and there was all of the fbi versus apple stuff there's a lot of coverage about celebrate because people were speculating that celebrate was the um company that unlocked the phone for the fbi but um i think it was kim zetter at the intercept actually did a really good profile of celebrate and how yeah you go to an apple store there's actually a celebrate in the back probably but yeah it's a very very common tool and company across law enforcement.

32:13Its main competitor being Greaky, which has probably taken actually a sizable chunk of its customer base, I imagine. Every once in a while, someone on Reddit posts a picture of the Celebrite device at the Apple store, and it's like the Leo DiCaprio meme pointing, like, oh, there it is! So, Celebrite is doing what every other company in the world is basically doing right now and they slapped AI on it, which is what your story is about. What does that mean for them? What does it mean in this context for them to use AI? Yeah. So they've slapped AI into their products called Guardian. Again, that's not the one that's actually getting the data from the phones.

32:55But after the cop has extracted the data, they upload it to this system, like an evidence sharing system. This is a bad analogy, but it's almost like Google Docs for cops where you can collaborate across the cloud live with each other on a piece of evidence um it's like that but for stuff that's been taken from mobile phones so you'll have all of the chat logs in there the voice memos the photos all of that sort of thing and what guardian is now capable of doing um with this slapped on ai it can summarize all of that material so rather than a police officer have to read through every single text message or listen to every single voice memo, Guardian can use AI to potentially summarize it.

33:42Now, I haven't seen sort of a Guardian-produced AI report, and I would absolutely love to see one if anyone gets hold of one, but that's the way they frame it, in that it can really speed up investigations. It can save cops time. It can do all of the promises of generative AI, essentially. But this isn't a school kid trying to generate an essay. This is a cop generating a summary of evidence seized from a mobile phone. And that is very, very different, in my opinion. The stakes here are a lot, lot higher when it comes to sort of this use of AI. Do we know what cops think about this new feature?

34:30Yeah, so there's a couple of testimonials included with Celebrite's announcement. And I should say, this actually came from a press release from Celebrite in February. I'm not really in the habit of reporting on press releases, but it seemed like nobody picked this up. I was searching around for something, probably to do a Celebrate because of all of these leech documents we get from Grey Key and Celebrate, that sort of thing. And then I came across this February press release saying, yeah, we're putting AI into this product. So included in there are testimonials from police officers. Now, this is very, very common.