How Deepfakes Destroyed a High School

0:00It was so painful to get those emails. Calling the event rumors and speculation when a crime occurred was almost worse than the crime itself.

0:30As well as bonus content every single week, subscribers also get access to additional episodes where we respond to their best comments. Gain access to that content at 404media.co. I'm your host, Joseph. And with me are the other 404 Media co-founders. The first being Sam Cole. Hello. Emmanuel Mayberg. Hello. And Jason Kebler. Hello. Hello. So let's mention, yeah, if you are in Los Angeles, we are doing an event at the LA Public Library, like the central one, on Sunday, May 31st. I guess the best way to learn about it would be, we'll include a link in the show notes. But Sam and I, as well as our friends, Dexter Thomas and Gita Jackson, will be talking about AI and AI Slop and how it is changing the internet and all that sort of thing.

1:24So it's Sunday afternoon. Please come out if you can. Link is in the show notes. Yes, get the details there. All right, with that, we'll get to this week's stories. The first one is one that Sam wrote that she's been working on for a long time and it really paid off. It's a really, really deeply reported piece here. the headline is how deep fakes tore a high school apart um i'm actually going to add a question to this sam before we get into sort of the story itself and the specifics and like what happened here just broadly like what do you think about deep fakes and schools at the moment and sort of the value of this sort of story because i feel like we we hear that deep fakes are all over schools boys are using it against girls non-consensually all of that what was the idea here to be to like focus in on a specific like example and case or like what was the idea here yeah i mean i so yeah like you said schools dealing with deep fakes is not a new topic four four has written about this quite a bit we've been covering this as it comes up for the past several years emmanuel jason wrote a really good story based on police reports in 2024.

2:43So that's two years ago now about how a school failed to report deepfake harassment among their students and how the police just kind of also bungled it. And everyone was kind of like, who dropped the ball and where? And since then, it's been something that it's just in the news so much lately, I think that people start to kind of stop paying attention to it. It's like, oh, another school is dealing with deepfakes. Another school is dealing with like synthetic nudes and these nudify apps that we hear about all the time. So I think there's value in just taking a beat and breaking down what is actually happening when this happens.

3:21It's not even that this particular school was an extraordinary case or that it was like really uniquely fumbling the situation because these schools are failing students and the police are failing students in the same way over and over and over. It's a pattern. So I think there is value in just breaking down where exactly what points in this kind of system fail, where they actually are lacking in good response and, you know, breaking down actually like, you know, at what points were the images talked about? At what point were they reaching the victims? Was the school notified? Was the police notified?

4:06I think it's really helpful. And I think that's the only way that you can actually prevent something like this happening again is learning from the ways that other schools have responded and how parents respond, how police respond. So that was the point of the story is just kind of getting in basically from the moment where it started. So I think I reached out to one of the parents from like a Facebook thread a couple of months ago and was like, you know, hey, I saw that this is something that you guys are dealing with in real time. It's not resolved. It's something that is actually still being like litigated amongst you debated about what the response is.

4:42So I reached out directly to one of the parents was like, hey, do you want to talk about what's actually happening and then kind of follow through as this goes on? So that's where the story took us. And I would say it's still not totally resolved. It's still very much in progress, but it's the sort of thing that takes a long time to follow. Yeah, exactly. And that is part of the story, the amount of time that is sort of elongated here. So let's start with the creation of the images themselves. What exactly happened? How many people impacted? Like right in the beginning, what is literally like the starting moment of this, you know, of these deepfakes taking over the school?

5:21Yeah, so it was the first week of December. and among the freshmen at Radnor High School, which is in Pennsylvania, it's one of the top schools in the state, it's on the main line, which is like a very affluent area, just to kind of paint a picture of the student body and what we're dealing with here. It's a well-resourced school. It's, I would say, upper middle class. It's a good school. But in early December, these kids started talking amongst themselves on Snapchat, which has disappearing messages and elsewhere, talking about how one of the boys, one of the boys in the high school had made images of his classmates.

6:02And they were discussing the fact that his parents caught him and he had to delete the app. And they were talking about how he had spent$250 on this app and on making these images and how he was caught and his friends were like, you know, you're dead. Like your, your parents are going to kill you if they find out what they, what you were making with that thing. Um, and so from there, the next morning, uh, the girls who were in the images had to go to school and face other kids talking about them. Everyone had been talking all night about how these images were going around and like they were, they were saying, you know, we've seen this, we've seen that.

6:46It was really unclear about what was actually being spread and what was actually just kids talking. But the girls were dealing with, you know, high school stuff. It's like the kids were, the boys were high-fiving each other. They were like, you know, harassing them the next morning after a night of just like being bullied over these images. And they were really upset. They were texting their parents. I talked to one of the parents extensively about this and she was saying that her daughter was just like, this isn't okay. The kids are defending their friends. They're lying about what's happening.

7:22So that was kind of what they walked into the next day. And I don't know. I mean, I remember high school. That's a nightmare. Yeah. High school is a nightmare without AI-enabled apps that can instantly nudify you or whatever. And now you have all of that and this. Yes. And just to clarify, what was the app capable of exactly? Is it one of these undressing apps? What is it? Yeah. So I'm told, and again, I don't have definitive proof of this, but I'm told by the parents and by people who are familiar that the app was called Movely, which we know what Movely can do because the Tech Transparency Project did a big report about nudify apps recently that I wrote about last month where Moveli was one of them.

8:13And Moveli is one of those apps that on the face of it looks like, put your face in Spider-Man or in a cat picture. Or even put yourself on the beach, stuff like that. But then once you pay up for the subscription, you can put someone's face into whatever you want. And it's like sexy undressing videos. And this is how they're marketing the app to paid users. So as I understand it, the images were highly sexualized images of minors, of girls, young girls. And they were all between, I think, 14 and 15. So freshman girls. and it looked like nudes of them going around and being shown off by the boy who made them.

9:09So that's what we're dealing with. Movely is still online. It still has the website, but I haven't been able to log in. I can't buy a subscription anymore. I think after the Tech Transparency Project support, Apple took Movely off the App Store. but there are tons of other apps like this out there tons of other apps specifically targeting young boys and men for this sort of content or be like make her boobs bigger, make her butt bigger, make her do this, make her take off her shirt, stuff like that so that was Moolie's entire entire gig yeah, I mean every day essentially every day we learn about a new one And I think I got a text yesterday from a 404 media reader being like, Hey, here's another app I got recommended on Reddit or something.

10:01And it's exactly the same thing. And they're recommended on social media. It's like on TikTok and on Instagram, Reddit. We've now done many stories about this. Like Emmanuel has done a bunch of stories about the fact that these are being advertised often to kids, often on Instagram, often on TikTok. And the Apple App Store has removed a lot of these apps. There's been academic reports about this happening over and over again, systematic attempts to document this problem. And yet, every time I go on TikTok, I'm still getting ads for similar apps. And they're often really blatant, actually. The advertising is very blatant, what they're for, where it's like, make anyone nude or like replace anyone's face with anyone else's.

10:56But then yeah, again, like when you look at the app store description, it seems like innocuous or like an app that's just for fun. And it's they have not gotten a handle on this problem from what I can tell. Yeah, it seems like it. So those images are made. They're spread around the school. As you said, the girls are obviously um i imagine pretty traumatized uh by that and it's really really horrible what happens next does the information sort of get to more parents like sort of what's the next stage of this because there's parents there's the school there's law enforcement but which bit comes next exactly so the parents tried to get on top of this from the start um they emailed the school they called the school uh they were like you know please don't bring my kids into a one-on-one meeting with anybody until, you know, we have a chance to talk to you.

11:49You know, they're, they're having a hard time already. It's like, it's, you know, it's, they're an hour into the school day and they're already facing so much just trying to get, get to their class. So the parents are, are just immediately in action. They're like, we have to, we have to get a hand on this. We have to figure out what's going on. What the school does is, um, well, first they, they take too long. Basically, they take several hours to get back to the parents. They say, you know, we heard from a safe to stay hotline, which is the it's like a hotline where you can report harassment if you're a teenager or anybody for like cyberbullying or like a shooting threat or something like that.

12:32But they someone had reported to the hotline that this was happening. So the school was already kind of aware of it, but they were they were just kind of scrambling also behind the scenes. The school sent a statement, which is in the story, but they didn't answer any specific questions about the response and things like that. But they sent an email. A lot of this is happening over email, honestly. They sent an email a couple of days later where they addressed rumors. They called it rumors, which I think was upsetting and inaccurate. Because the images are real. Well, they're not real, but they're real.

13:11Yes. Right. Yeah. Yeah, it's definitely happening. But the principal emailed and said, you know, there are rumors about AI generating videos that are going around. We understand how upsetting and serious the situation is. We're treating it with highest level urgency. um you know they're they're trying to kind of get a handle on on the response uh they say that they contacted the police as soon as it was brought to their attention um and that the police were actively involved um the police also didn't reply to any way or they didn't they didn't answer any questions specifically um they didn't uh they directed me to the da's office which took took over the case.

13:52But they said that there were supportive measures in place for the girls. So there's five girls in total dealing with this at the school. But those supportive measures were you can leave class early. You can eat lunch separately. You can isolate yourself basically from your bullies if you want. but not but the school didn't in that particular at that particular time like take the bully out or the boy who generated the images out as far as i can tell he wasn't suspended or anything um he didn't show up to class the next day i think his parents were like you're not going to school the next day um but the response was definitely not not great because it was isolating the girls They were being punished for something that was happening to them.

14:45They were told that they could request accommodation, basically. They were like, you can come to the school counselor if you want and talk about this. Which is not, again, it's part of the pattern that we see over and over at schools. It's like they give the targets of this stuff the option to, you know, isolate and be away from, you know, their bully, but they don't actually respond to the actual trauma that's happening. So that's where that stood for a long time. And then the parents started going to school board meetings, which is where a lot of the story was reported. It was just watching school board meetings and watching how the board is grappling with this.

15:28The parents are coming to school board meetings and saying, hey, we need to change the policies. We need to talk about the response, all this stuff. And those are public, right? Yeah, they're on YouTube. One of the biggest points in the school board meetings that parents kept coming back to was the way that the school and the police jointly were responding to these things. It was so painful to get those emails calling the event rumors and speculation when a crime occurred was almost worse than the crime itself. I mean, if you're going to put out a statement like that, make sure the facts are right.

16:06I mean, at a bare minimum. I mean, the victim center thing. Great. But to say these images move and dance, I mean, for crying out loud, this is pornographic material. This is not Mickey Mouse. This is not Disneyland. We don't have snuffleupagus moving around on a stage. This is pornography. And this email was so damaging and hurtful. It was the school administration and the police co-signed this email. And the email said that the police had concluded their investigation. So it had been a month. And said that there was an incident off school grounds outside of school hours. Which again, kind of doing a little cover your ass.

16:47The student used an app that animates images, making them appear to move and dance. Which downplays what actually is happening on this app pretty severely. You know, it sounds like it's some kind of goofy like elf yourself app. It's not. It's realistic sexualized images. And then they say in this email that after the investigation, they found that there was no evidence shared with law enforcement depicting anything inappropriate or any other related crime. which was just galling to the parents and to the the girls because they were saying there was no crime they were saying there was an investigation but they have all these screenshots they have you know knowledge of these things happening over other apps over discord places like this they know that the app was used but the police just say they didn't find anything which is shocking.

17:45So I guess just on that, before we get sort of the bigger takeaway question, it's like, what is the legality around this? Obviously, it will depend on the images. So it can be hard to say, like, obviously, possessing child abuse imagery is illegal in the United States. And that does extend to AI generated imagery. We've seen like, you know, more and more cases of that recently i guess there's like a cyber bullying element here like and a harassment element like legally where might this case fall into and i appreciate stuff might be ongoing but like where do you see it potentially yeah for sure i mean there is like a legal precedent for this is the the frustrating thing and that was was really frustrating about this email because they were like there is like something happened and we need to figure out what it is and you know like where are the where are the images now like did he delete him he said he deleted him off his phone um where are they you know you guys like the police relied on teenage boys to forfeit evidence is kind of the the takeaway from this which is crazy you know anyone should know that that's that's insane um and then just before we get into like the actual the law here um just to close the on that email a week later after that initial email where they said nothing happened the police released a statement and said that a juvenile offender was charged with the crime of harassment.

19:12So something did happen. And they kind of flip the story at that point. So, I mean, it's, I think it's hard because first of all, we're talking about minors. We're talking about images that the parents did not see. And that apparently the police didn't try to actually find as is what's what the parents believe at this point um in pennsylvania pennsylvania actually has pretty good laws against deep fakes um they they criminalized deep fakes malicious deep fakes in 2024 um they're they're among the states in the u.s they're a leader in this sort of thing um and then in 2025 a man was actually an adult man was charged with 30 counts of felony possession of child sexual abuse material that were AI.

20:04And AI generated child sexual abuse material has been a federal crime for years. So it's, there's a lot of confusion about like, you know, because it's fake, is it really a crime? It is, it's a felony. Child sexual abuse material is very serious. uh so i the i think a lot of the the confusion and the hesitancy in these cases is because they're children they're children making images of children um but you know we have precedent of kids actually being charged with serious crimes you know 13 year olds being charged i think it was 2023 where the first case of a child being charged with child sexual abuse material happened And they were arrested.

20:48They were charged with third-degree felonies, which is itself very wild. It's like the kids are being failed in every direction here. A 13-year-old being charged with a third-degree felony, that's not solving, obviously, it was three years ago, that didn't solve the problem here. Right. And they may not understand what they're doing. I don't think they do, yeah. Right. Or they may not understand that it's obviously a horrible thing to do to another human being because we're talking about literal children who don't get it. Yeah. And the adults don't even seem to understand. You know, it's like no one is no one is actually taking this as seriously as it is.

21:29And that's something that I talked to quite a few experts about this for this story. And, you know, just bringing home the fact that this this is a serious crime. It's being enabled by Apple and Google through the app stores. uh it's being pushed by social media just handed to them um and then when it when they actually take the bait they use the they use these apps uh there's just immediate handoff to police which there should be you know police should get involved at that point um but there's no there's so little preventative education about this at this point it's like when you get caught but you will get charged with a felony and your life is ruined.

22:14And these girls' lives are ruined because they are so traumatized at this very tender age where they're now confronting sexual abuse material of themselves being spread in their school. They can't study. They're in therapy now. They're leaving class early, stuff like this. So it's just in every direction, these kids are being let down by the adults in the room. And I don't think you can wait until there's a serious crime committed by a 15-year-old or 14-year-old or 13-year-old to actually take some action on this. I think schools need to realize how serious this is and start talking to kids about it, you know.

22:56And not just like, here's an AI class, here's like an AI education class, which is what a lot of the schools end up doing. um actually talking about consent and sexuality and having some literacy around what you're actually doing with these apps and what you're what you're up against is is a serious thing so i don't know it's just i again that's why this sort of story i think is important to to really dig into and examine because the the failures start before it actually happens It's long before that. It starts when you hand a kid a phone and say, good luck. I think it's tricky because none of us would want to excuse the behavior.

23:44But I also agree that, I mean, imagine you're a kid. it is hard to understand that it's a serious felony when it's being pushed at you from the biggest social media platforms in the world. And then all you have to do in order to partake in it is download it from the Apple App Store, which is readily available on your phone. And it's just so frictionless. And it's like your way to arrive at a felony is so frictionless, which is not the same. I mean, we've reported on drugs being offered on social media and guns and things like this. But that is way more in the underbelly compared to something that's like a paid advertisement.

24:29And then I guess the other thing I wanted to say is that we report on all the many ways that you can produce non-consensual content. and there's a spectrum. There's people who roll their own models and make their own models and they're using their own GPUs and all that stuff is extremely hard and probably impossible to stop because you can just make your own DIY method of making non-consensual content. And then there's people who are subverting existing image models and using all these hacks and that's more available. But whenever we write about one of these abuse cases in high schools or junior highs, it is never one of those solutions.

25:22It's always the lowest hanging fruit, which is there's an app advertised on TikTok, you download it from the app store, and you use it for money. And people often ask me, and I'm sure Sam, you get the same question as well. It's like, what do we do about this? And it's like, that's a big question. And ultimately, the answer is, you can't stop it completely. You probably are never going to stop this completely. But what you can do is take care of the lowest hanging fruit, which is these apps on the app store and the advertisements on social media. And we don't know what that world looks like because we haven't tried that really.

25:57These companies haven't really tried that. And maybe kids do... get more sophisticated and start using their own GPUs to make this stuff. But the least we could do, we haven't done, which is just taking care of these apps. Yeah, that makes total sense. All right, let's leave that there. Everybody listening, definitely go read the article for all the much deeper specifics as well and sort of the play-by-play of what exactly happened. After the break, we're going to talk about a company called Bus Patrol. We'll be right back after this.

26:44This episode is sponsored by Incogni. When people think about identity theft, they usually imagine some giant data breach making headlines. But the reality is, a lot of the information that scammers use isn't stolen in a hack at all. It's already out there. Your phone number, address, email, age, family members, old accounts, things you would never even think about. Well, there's entire websites built around collecting and publishing that information. And the easier it is to find, the easier it is for scammers, impersonators, and identity thieves to target you. That's why I've been using Incogni.

27:20I've been using Incogni for years because we're the target of a lot of these attempts to steal our data, to use it to dox us, to harass us, etc. And Incogni has been amazing at helping us remove that personal information from the internet by contacting data brokers and people search sites on our behalf and demanding that they delete my data. And they don't do it just once. Incogni continuously monitors and sends removal requests because your information has a habit of reappearing online over time. One feature that I really like is custom removals. Let's say you find a specific website with your personal info on it.

27:58Instead of trying to figure out who to contact and how to get it taken down, you just give that to Incogni. And then their privacy team handles the removable process for you. It's a huge differentiator because privacy isn't just about reducing spam or making sure that you don't get hacked. It's about shrinking the amount of personal information online that's available to people who shouldn't have it. The less information that's floating around about you, the better it is for you and the harder it is for bad actors to build a profile on you. Take your personal data back with Incogni. Go to incogni.com slash 404media and use code 404media to get 60 % off an annual plan.

28:36Again, that's incogni.com slash 404media and use code 404media for 60 % off an annual plan today. As I've gotten older, one thing I've realized is that there's so many stories our parents never tell unless someone actually asks them. They don't usually volunteer that stuff unless you bug them. Stuff like beyond the usual family stories. What was their first job? What were they afraid of when they were younger? What decision changed the course of their life? And the weird thing is, you don't always think to ask those questions until years later, or maybe you don't ever remember at all. That's why I love StoryWorth as a Father's Day gift.

29:12Most Father's Day gifts end up in a drawer. StoryWorth doesn't. It gives your dad a year-long experience and gives your family a book full of stories he'd probably never think to tell on his own. Each week, StoryWorth sends him a question about his life. He can then answer however he wants, by email, through the web, with a voice recording, or even through a guided phone call. And you can help choose the questions too. Pick from their library, write your own, or let StoryWorth create personalized questions based on his life. The coolest part is that you get each story as he tells it. Then after a year, StoryWorth compiles everything, his words, his photos, his memories, into a beautiful hardcover book.

29:51Families have already created more than a million books with StoryWorth and they recently introduced an unlimited plan so you can give StoryWorth memoirs to friends and family throughout the year. I got a StoryWorth last year for my family and it's been really amazing to learn all these things about their high school and college days that I never knew. So get something for the people that you love to hear in their own words about their memories before they fade. This year, give dad a gift that captures who he really is before the stories get harder to remember. Father's Day is Sunday, June 21st.