The Final Boss of AI Slop

0:00Seit Maren die kostenfreie Flatex Academy nutzt, investiert sie wie Maren Buffett. Für alle die wie Maren Börse im Blut haben. Bis 28. Februar. Depot eröffnen und wir übernehmen deine ersten Orderprovisionen bis 500 Euro. Flatex besser richtig handeln. Investieren birgt Verlustrisiken. Bedingungen gelten externe Kostenfallen an.

0:24Hello and welcome to the 404 Media Podcast, where we bring you unparalleled access in the world, both online and IRL. 404 Media is a journalist-founded company and needs your support. To subscribe, go to 404media.co. As well as bonus content every single week, subscribers also get access to additional episodes where we respond to their best comments. gain access to that content at 404media.co. I'm your host, Joseph, and with me are two of the other 404 Media co-founders, the first being Sam Cole and Jason Kebler. Hello, hello. I nearly said Emmanuel just because I'm reading the script, even though he is clearly not here today.

1:06So, Jason, what's going on with merch? Yeah, so we haven't restocked our merch in months. And so orders have still been coming in, but we've been basically out of most things for quite a while. So we just are doing a big restock right now. I just placed the order. So we have crewnecks and hoodies back in stock. We have hats, both black and green, back in stock. Those have been out of stock for a long time and are very popular. And we have the return of beanies. and then very excitingly we have a few new items of merch so we have a new crew neck sweatshirt in addition to the one that we had from last year we have a long sleeve doom uh t-shirt so we had the tank top doom t-shirt over the summer now that it is fall we have a long sleeve t same design um so we'll add some sleeves and that's the numbers 404 but if you look closely it's a bunch of code and that code is for the video game doom right yes it is uh and then we have a logo t-shirt pretty simple but people have been asking for that so um that's all up for pre-order now in our shopify uh which you can find our website if you just look for the merch store um and then also if you place an order i'm really behind on shipping it will be out like this week i'm sending stuff out.

2:34So basically big merch refresh. Very excited about it. Go check it out. Have we been impacted, touched by tariffs at all? Yes, we have. Probably we'll do an article about this at some point. But we mentioned this before. We work with a local screen printer in LA. They're really, really cool. It's like a very small shop. Um, and basically like the underlying t-shirts that we're buying, the underlying sweatshirts that we're buying have gone up in price pretty significantly over the last just like month and a half, something like that. Um, we're talking like five to$7 per item just for the base t-shirt at wholesale.

3:23So yeah, I mean, when we're talking about tariffs impacting things, this is one of them. I'm going to talk more to our supplier just about what is going on. But it's interesting because some of our stuff is made in the US. We try to make stuff in the US when possible just to support local business and things like that. I've mentioned many times I'm in LA. there's like various clothing companies here that make their stuff in a in la um but they're getting the actual cotton elsewhere presumably or they are like now playing in a world where all their competitors prices have gone up and so they're they can increase prices well i'm not exactly sure like what it is in this case but the underlying cost for us has gone up like dramatically in the last month so not good yeah just another way in which um some large economic and geopolitical decisions can impact for journalists trying to sell t-shirts with the doom code on them basically i mean it's impacted a you know untold number of businesses in all these different ways and you know it's impacted us in a fairly straightforward way probably like hey price went up you know uh but hopefully people you know go check out the the merch refresh i'm definitely glad that the doom one is now on a long sleeve shirt there was no way i was ever going to wear the tank top although i know jason rocks it don't think of him and sam big fan yeah yeah sun's out guns out not for me uh so yeah i'm glad the new stuff is in and with that let's talk about this week's stories the first section a lot about zora 2 i don't even know if i did that that sounds like a z uh soar like like a kingdom hearts character i think that's why my brain refuses to pronounce it because i'm not i'm not entertaining the idea of kingdom hearts i find it interesting from an IP...

5:40I don't want anything to do with that. Anyway, okay. It's about Sora, OpenAI's new AI slot machine app video model. And the first one is, OpenAI's Sora 2 copyright infringing machine features Nazi SpongeBob's and criminal Pikachus. First of all, Jason, what is the new app exactly? Because it's not just a new video model, which we see every six months or whatever, right? Yeah, I mean, it's OpenAI's attempt to make a TikTok competitor. So it's a social media platform, or at least it's trying to be. And that's like kind of the new thing that AI companies have been trying to crack. Like the new problem they've been trying to solve is like how to make this stuff social.

6:30We've seen like Meta try to put it into its products, but Mark Zuckerberg has talked about trying to make its own AI-focused social media app. And so this is OpenAI's attempt at that. So basically, you download the Sora app, you scan your face. You can't really use it unless you scan your face. And that's the first thing that you do. Do you literally have to do it? Or you can just scroll if you don't? I'm not sure, actually. But it's the first thing that you're prompted to do. And you can't do much of anything on it if you don't scan your face. I should have checked. I just did it because I was trying to figure out how this worked.

7:17But it's shoved in front of you. Yeah. So you open it up and the camera turns on. And then you do a selfie. And you take a video and you hold it to your face for a few seconds. And then it asks you to look left, look right, look up. This takes maybe 10 seconds. And then it asks you to say three numbers. So it'll be like 42, 87, 91. And that's it. You give it your voice by just saying that. And I bring that up because maybe like four or five years ago on an episode of Cyber, which is the old podcast we did at Vice, we did a deep fakes audio episode where we cloned the voice of Ben McCoo, who was our host at the time.

8:01and it we had to like find bespoke software engineers to figure out how to do this like they they had created a model that would like clone his voice it was pretty sophisticated it was running on their like own server and ben had to read maybe like 20 minutes worth of audio which is a lot of audio thing and at the time they're like we've gotten this down from like this is state of the art like you only have to read 20 minutes worth of audio versus like hours and hours and hours worth of audio and it didn't sound even that good like it did sound like ben but it was like a days-long process he had to do all this stuff and now to make pretty convincing video with synced audio that sounds like your voice it's like a 17 second process and sam i feel like you have done like quite a lot of reporting on like how that tech has changed like i don't know i feel like you're you've like cloned yourself various times i mean when deepfakes first came out in i guess late 2017 i tried to make one of myself um and at the time it was like i don't know you needed to you need to have like pretty extensive programming knowledge to even just set up the um the adversarial networks that you needed to like make a deep fake visually and I gave up.

9:27I mean, I was like, I don't have the, I don't have the patience or even the knowledge to like follow a tutorial at the time. And now it's like, you know, we can do it on an app and it's literally one photo or in some cases, not even a photo. If the person is like well-known or a public figure, you just use text. But yeah, that's the case with all of, like every format of this technology is so much better than it used to be. to the point where you just need, like you said, like a couple seconds worth of audio. It's also interesting to me because that's, I had to go through this recently with like my bank with like doing the voice verification stuff, like to make like a voice print.

10:09And it's basically the same exact thing. It's like you read like eight words or something or numbers and you instantly are like, I know your voice now. It's like, this doesn't seem right. It seems like it should, you should need more than that. It seems like you should need more. Yeah. Yeah. Well, and with bank accounts specifically, I used the service 11 Labs a few years ago now. And to clone my voice with that, I had to read out, I think, a few sentences. And it was weird. It was like some romance novel, if I'm remembering correctly. And it got like a bit steamy. I'm like, why the fuck am I reading this?

10:42And then I did that, cloned my voice. And then I used that cloned version of my voice to break into my bank account, which was protected with voice ID. So And as you say, Jason, it's crazy easy now to do it. Yeah, and I bring this up and we dwell on this because that is the big advancement from Sora, in my opinion. There have been other nudify apps and AI apps where it's like, oh, you put in a picture and then it makes a crude version of it. And those have been very damaging and we've written about them. But there hasn't really been a video model, to my knowledge, where you can deepfake yourself or for lack of a better term, you can clone yourself in three seconds.

11:26And that is how the app works. It's basically like you scan yourself, it's connected to your profile, and then you can use text in any other AI app to say like, Jason, writing an article in a cyberpunk future, and it will make a video of that, and it will have my voice, and you can write the script, blah, blah, blah. And to be very honest, it is pretty damn good like it is shockingly good um i think that there's still in many of the videos like quite a tell where it's like oh this is ai generated but with things like animation which we'll get into in a minute like you can tell it to make an episode of spongebob and it will look like the actual animation and the videos like look pretty realistic the audio sync pretty well the voice sounds like my voice it's like quite concerning um and then you can it's called cameo you can like put anyone else who has the app into your videos and by just like tagging them and like generating a video with them so basically like people have been making videos of sam altman the ceo of open ai by tagging him and then making him say and do like crazy shit um and and notably and I think to OpenAI's credit, this is like an opt-in system.

12:51And so you can be on the app and say, don't let anyone make videos of me. You can also say, only let my friends make videos of me. And you can also say, you have to manually approve any video that has you in it before it is published publicly. I wonder if Sam Altman, Is he doing that to all of the ones in Nazi uniforms? I mean, no, no, I think you can. I think you can make it just like in anything goes situation. But like there is a setting where it's like I have to manually prove each and every video that's made of me. Although like I actually haven't tested it enough to know because you can generate drafts that live on like the person's phone.

13:37and then that person can download the video, which I think is actually one of the concerning things about Sora and what we're gonna see and already are seeing is people using Sora to generate video, downloading it and then posting it elsewhere. Because if you stay in the Sora app, which again, it's just like a TikTok clone. So everything on the app is like vertical videos that you can scroll endlessly and they're all AI generated. It's like if you're in that app, you know it's fake because it's like an AI app. So on that front, it's actually slightly better than my Instagram feed, where my Instagram feed is full of AI shit.

14:19Right. Which isn't disclaiming either. Yeah. Whereas everything on Sora is AI generated. But Sora is so good at generating AI video that you can just generate the video, download it, and post it elsewhere. And that's probably advantageous for the people doing this because the other platforms are monetized, as we've talked about endlessly. Yeah. And we'll get to more about spreading these videos and misleading people. But as well as uploading your own face, you mentioned there's SpongeBob. And of course, the headline mentions criminal Pikachu's as well. when you first open the app and you start scrolling what are some of these examples of stuff you're seeing because the article starts with basically a very long list to give readers an idea of what people are seeing yeah so um i got this app the day that it came out and the way that i got it is i bought um i bought an invite code on ebay um because i was not invited by open ai which we also covered to be clear because that's interesting in its own right yeah yeah um And it was full of just copyrighted stuff.

15:30And so I think the first video I saw was Spongebob saying, I'm on the Sora app, oh my god. And talking to Patrick and it sounds like Spongebob, it sounds like Patrick, it sounds like Squidward, whatever. Then I saw there's so much Pikachu. There's tons and tons and tons of Pikachu. I actually saw a video of Sam Altman grilling a dead Pikachu on a charcoal grill and him going like, and like cutting into Pikachu and being like, I'm going to get sued for this, which I thought was actually pretty funny. There's like tons and tons and tons like Mickey Mouse, you know, Bart Simpson, Peter Griffin from Family Guy, like any copyrighted character you could possibly imagine was on there and people were doing all sorts of shit with it.

16:23Like, you know, there was crossover stuff, there's a lot of people putting them into like minecraft uh there was like fortnight but it has you know peter griffin shooting stewie or whatever um and that was like the vast majority of the videos that i saw it was like a mix of that a mix of like sam altman videos and then like random people you know posting themselves just trying the app out but like the stuff that was going viral that was being delivered to me by the algorithm was primarily copyrighted content. Yeah, and you write in the piece, quote, with the release of Sora 2, it is maddening to remember all of the completely insane copyright lawsuits I've written about over the years, some successful, some thrown out, some settled, in which powerful companies like Nintendo, Disney, and Viacom sued powerless people who were often their own fans for minor infractions or for use of copyrighted characters there would almost certainly be fair use.

17:22That was a long quote, sorry. But what do you mean by that? Because you're seeing all of this wave, this flood of copyrighted material in this app. Clearly, OpenAI has scraped these characters and this material from the web in some fashion. And then you have companies like Nintendo who are usually crushing random ass fans for making stuff about their games. What do you think about that disconnect? Yeah, I mean, I think that there's a huge disconnect. and with a caveat that we'll talk about in a second of what has happened over the last week. But basically, Nintendo is extremely litigious. They go after people who pirate their games, which makes sense, but then they also go after people who mod their games or emulate their games.

18:05I wrote about one time a Pokemon company and Nintendo suing this fan who threw an unofficial Pokemon party at a game dev conference, like not even affiliated with the game dev conference. It wasn't like it was at this big event. It was just like at some satellite bar. And they sued him because they put a Pokemon on the poster of that tiny, tiny event. And so then to see just like this anything goes extremely blatant copyright infringement by a company that's worth billions and billions of dollars and has all this funding is kind of insane to see like how far we've come. We've talked about the copyright issues with AI before.

18:47um so we won't get into it too much here because a lot of it is like unsettled laws to whether this is fair use um but like it was only a year and a half ago or so that most of these big ai companies like wouldn't even say whether they were training on copyrighted content even though it was obvious that they were and now it's like you can have a picture perfect uh you know episode of the simpsons with all of the voices synced up, it's like, obviously, they were training on all of this. And so it's kind of insane. And to be clear, there have been a couple high-profile lawsuits against AI companies.

19:27Disney is sitting mid-journey. But you would think when you have Mario... When you have Sam Altman killing Pikachu, you'd think, hey, maybe they would be mad about this. Yeah, and there's still time, obviously, for lawsuits to happen and stuff, but companies can opt out of their IP being used in the app. And as far as I know, companies have to do that on a per-character basis. like Disney or Nintendo can't go and say, take all of our stuff out, please. They have to go, could you take out Pikachu, please? This is what Pikachu looks like. And I mean, I remember way back when Pokemon was, what, 151 Pokemon?

20:14How many is it now? I mean, I think it's up near a thousand, if not over a thousand. So Nintendo is going to have to send a thousand letters and open a thousand different cases about Pokemon. I don't know how many people care about the latter series of Pokemon. That's my bias showing. But Disney's going to have to do the same for that. And, you know, maybe that will, you know, please these companies. But yeah, sure, a lawsuit could happen. And with that said, because now the app's out and companies have had a minute, a few days to, you know, opt out certain characters. Have you seen any change on the app?

20:52Like, oh, actually, I can't make SpongeBob Nazi anymore. Have you noticed anything? Yeah, yeah. Yeah, so actually, yeah, I didn't mention, but there were videos of Spongebob being a Nazi that I saw right after I opened the app. And then one thing you can do in Sora is you can then tweak things that have already been published. And so people were like, now make it Pikachu, now make it Goku, now make it Rick and Morty. And so there was just an endless scroll of Nazi IP with swastikas in the background. It was crazy. It was crazy. But anyways, over the last week, a lot of the characters have been opted out by the companies that own them.

21:39And that is now one of the main things that people on Sora are making videos about and talking about where basically like, Sora is no fun anymore because everything I try to do is a content violation. And so a lot of the videos are about content violations and getting like Sam Altman to be like, say things that are like, you can't have fun on my app anymore, which is like the app only came out six days ago. And so, uh, I mean, that, that raises questions about whether this app will have like any sort of staying power whatsoever. Uh, we've seen it time and time again, like big fancy new social media app, uh, is released.

22:20There's like an invite system. So it's hard to get on. So it's really, that sort of thing. Yeah. It like seems really exclusive. and then it's like, oh, we got an invite, like got to get on this app. And then it's popular for a few days or a few weeks and then it kind of peters out. And it's like, I don't know if this will have any sort of staying power. Like obviously, ChatGPT has, but not everything that OpenAI has made has been like some massive hit. And so, yeah, it's like a lot of the characters have been opted out and people are now like complaining about it. And let me just say that the opt-out system is like a system that OpenAI devised it's like that's not like a legal framework i mean very very easily like a company could be like no take down everything or we're going to sue you and i don't know how that would go but um it's worth saying that this is like an interesting new like paradigm that they're trying where it's like we're going to put you in the app unless you specifically tell us you're not going to be in there that's not how copyright works it's literally opposite of how it's supposed to work here um and it i mean i guess maybe to wrap it up before we go to the watermarking thing just with all of that ip now being removed by the copyright holders it almost just brings up a question of what was the fucking point of this app in the like literally what is the point of this app like what do you think well i mean i i think it's like ai generated video is very popular on the internet just because of the AI slop phenomenon and all that.

23:55And it's like, this is the best, easiest slop maker. I mean, I think that people will find ways to jailbreak it, for lack of a better term, like to get around the guardrail and things like that. And then also, it's like most AI slop that we see on the internet doesn't have characters in it and stuff like that. And so they've made the best, easiest AI slop generator that there is. And I think that to the extent that there will be staying power for this, I think it will probably be from the slop manufacturers who are using the app to make stuff, but then are taking it off the app and putting it elsewhere to go viral in the ways that we've talked about before.

24:39Yeah. More like generic monster does XYZ or something. It doesn't have to be some IP from Stranger Things or something. It's just going to be more generic creations which don't violate copyright, probably, maybe. I also, I'm really sorry if this is so dumb, and this was like the whole in-joke the entire time, but I'm just sort of realizing that Sora from Kingdom Hearts, which is a video game which has all of this IP crossover from Disney with Goofy and all of that, and then OpenAI uses that name for its massive copyright infringement app. I'm sorry, I'm only just putting two and two together. I mean, I don't think that's why it's called Sora, but that's really funny.

25:21Exactly. There's also Cameo exists too. They're using the word Cameo for literally what Cameo is, which is like a put a celebrity in something to make them say what you want them to say app. Does Cameo not own that trademark? Surely they have a trademark on that. I'm actually going to look that up while um sam i'm going to ask you about this uh next story and the headline is and matthew our regular contributor wrote this sora 2 watermark removers flood the web but basically you know every sora 2 video comes with a watermark it's sort of like a little spinning icon that as jason says you can be like oh that comes from this app so it's going to be ai but um as matthew has reported there's now this wave of watermark removers where you just upload the video and it takes it out automatically um i'm just wondering what you what you think of that because of course you've covered a ton of you know abusive um deepfake and ai videos as well which don't present themselves as ai generated they don't have a watermark very much deliberately what do you make of sora trying to have a watermark and then it being defeated in 20 minutes or whatever it was.

26:32I mean, it's like very predictable. Like you can take watermarks out of pretty much any, like TikTok has watermarks where it's like little TikTok logo bounces around. It's just not like watermarks in general to be a solution for AI generated content. It's like a good start, but it's relying on people to keep it in the video to like not use something like this, exactly like this to take it out. it's watermarks are just such a flimsy

27:02indicator of AI content in general. And it's also, you can put it in, you can put a watermark on a real video, like you could edit a video to put a sore watermark on it and say, Oh, that's not real. There are water, there are types of watermarking that aren't, we're talking about like the little logo, right? Like the thing that is on the video that you can see with your eyes. There's types of watermarking. That's like, It's more like thumbprinting or hashing where it's not visible to the naked eye and it's much harder to remove. You have to edit the video to actually be able to see it. Google has something like this for VO.

27:37But this particular type of watermark is very predictable that people would try to take it off of Sora videos to make them seem like they're real videos. And then they repost them to like Reels and other platforms where, you know, it's like they're mixed in with real stuff. Yeah, this whole story was very, it's like, yeah, that's what people are going to do with this. It sucks that this is like the cottage industry that pops up every time. So it's definitely worth reporting on and noting that here it is again, like always. Yeah. And it was just done so quickly and it can be automated by tools that anybody can then use.

28:22I'm looking at the Cameo trademark and I'm not a copyright or a trademark lawyer. I think Jason would probably know more about this, but you know, it says in its goods and services section, downloadable software used to create personalized video messages for entertainment and structural inspirational or greeting purposes featuring athletes, actors, entertainers, influencers, et cetera, et cetera. I don't know. It sounds like you could do that on Sora. so i guess we'll see um i know but then there's this specific uh style of the cameo logo whatever um yeah i'm pretty sure there's going to be at least one lawsuit uh after this app in some sort of um direction um all right let's leave that there when we come back after the break we're going to talk about the app ice block and red dot which you know are both for reporting sightings for ICE officials and how Apple and Google has removed those.

29:16We'll be right back after this.

29:31You ever have one of those moments where you realize you started something big and suddenly you're the CEO, the designer, the marketing team, and the IT department all at once? When we launched 404 Media, we had that moment. We're not just the journalists, we're the business people and everything else. Thankfully, when it came time to set up a merch shop, we had Shopify in our corner. Shopify is the commerce platform behind millions of businesses around the world, powering 10 % of all e-commerce in the US. From household names like Mattel and Gymshark to, well, 404 Media, Shopify gives you the tools to build, sell, and grow anywhere.

30:10Shopify makes it easy to create a store and to refresh your items when the time comes, just like we just did with a new line of merch. With hundreds of ready-to-use templates, you can create a stunning online store that perfectly matches your brand style, even if you're not a designer. And when it's time to grow, Shopify gives you the power to market like a pro, launching email campaigns, running social ads, and connecting with your audience wherever they scroll or stroll. Shopify is your commerce expert, helping you with everything from inventory management to international shipping, returns, and more.

30:43Because when you're ready to sell, you're ready for Shopify. Turn your big business idea into with Shopify by your side. Sign up for your$1 per month trial today and start selling at shopify.com slash media. That's shopify.com slash media. This is a paid advertisement from BetterHelp. October 10th is World Mental Health Day. and this year we're saying thank you therapists. Thankfully therapy has been destigmatized over the last few years and a lot of that is thanks to easier access to qualified thoughtful therapists through services like BetterHelp. If you've done therapy you know that the right therapist can change everything.

31:26Helping you cope with a world that let's face it has been quite a mess lately. Finding a therapist you vibe with can be hard which is why BetterHelp does the initial matching work for you so you can focus on your therapy goals. Just fill out a short questionnaire which helps identify your needs and preferences and BetterHelp's more than 12 years of experience means you typically find someone you like for the first time. But if you're not satisfied with your match, you can switch to another therapist at any time. BetterHelp has more than 30 ,000 therapists and has served more than 5 million people globally, meaning that you're sure to find a therapist that works for you.

32:03This World Mental Health Day, we're celebrating the therapists who have helped millions of people take a step forward. If you're ready to find the right therapist for you, BetterHelp can help you start that journey. Our listeners get 10 % off their first month at betterhelp.com slash 404 media. That's betterhelp.com slash 404 media. Fall is planting season. Did you know a lot of plants actually do better when they're planted this time of year? That's why I love fastgrowingtrees.com. Doesn't matter if you're in the sunny South or where the air is getting chilly. Their plant experts can help you find the perfect fit for your space.

32:44I've loved my experience with fast growing trees. Picking out my plants and trees was a breeze. There was no shoving the plant into my car, getting dirt and leaves all over. My beautiful Arbukina olive tree, dragon fruit plant, and bird of paradise all showed up to my doorstep happy and healthy. In the few weeks since I planted them, I can already tell they're gonna be a great fit for my backyard. And I've really enjoyed reading the specialized care instructions and tips I found on Fast Growing Trees website. Fast Growing Trees has over 6 ,000 plants for any space. Fruit trees, flowering trees, shrubs, even indoor plants.