How to Fight Back Against AI Bot Scrapers

0:00If you like 404 Media, you probably followed our reporting about how big tech has made consuming the news a mess. Social media and search algorithms have made finding and understanding the news exhausting. It's tough scrolling through timelines full of content that have been reverse engineered to perform well in an algorithm or that you're only seeing because of targeted advertising. That's why I like Ground News, a news platform that doesn't use manipulative algorithms and which lets me quickly see how news outlets around the world are covering a story. Ground News helps me get a snapshot of the entire media ecosystem, left, right, and center, which I found really useful as someone who's scanning the internet all day.

0:40I found Ground News to be particularly helpful when I was trying to parse through Donald Trump's massive legislative agenda. On the left, Ground News surfaced an American Prospect article called, 10 Bizarre Things Hidden in Trump's Big Beautiful Bill. In the center, it also surfaced a great analysis from Higher Ed Drive called, One Big Mistake, Higher Ed Sounds Warning Over GOP Budget Law. And then I also was able to find a bunch of news, takes, and analysis from right-wing outlets without having to dive into the muck on X. Ground News let me do all that research really seamlessly without having to endlessly scroll through social media's echo chamber reinforcing algorithms.

1:21Go to groundnews.com slash media today to get 40 % off the Ground News Vantage plan and get access to all of their news analysis features. That's groundnews.com slash media for 40 % off the Ground News Vantage plan for a limited time only. Groundnews.com slash media.

1:45Hello, and welcome to the 404 Media Podcast, where we bring you unparalleled access to hidden worlds, both online and IRL. 404 Media is a journalist-founded company and needs your support to subscribe, go to 404media.co. As well as bonus content every single week, subscribers also get access to additional episodes where we respond to their best comments. Gain access to that content at 404media.co. I'm your host, Joseph, and with me are 404media co-founders, Sam Cole. Hello. And Emmanuel Mayberg. Hello. Jason's not here. He's still on vacation. Yes, listeners will know we all took last week off, which is only possible due to our paying subscribers.

2:28We have never done that since we launched in August 2023. At this point, it was rejuvenating and ready to get back into a ton of stories all ready to go through now. And then we're going to have a bunch more for next week as well. So let's get straight into it. This first one is one that Emmanuel published, I think today or yesterday, the open source software saving the internet from AI bot scrapers. So we spoke about the problem a little bit on an earlier episode of the podcast. Can you just lay out that problem, first of all, before we talk about this solution? So what is happening with AI scrapers and who is it impacting exactly?

3:13I think one of the funny things about taking a vacation after not doing it for so long as it warps your perception of time. And I'm like, when did we have that podcast? Was that two weeks ago or the early 90s? I don't remember. But yeah, it was like, I think it was two weeks ago. I wrote this story about the first good study that I've seen about how AI training data scrapers are really messing with libraries, museums, and any other form of resource or archive that's open to the public. So you have all these AI companies. They need all the training data they can get in order to train their AI models.

4:00So they send these bots to crawl the entire web and hoover up all that data. And there are so many companies doing this now, and they're doing it so rapidly that if you're just a university or even a big university with an online library, all these bots suddenly hit your website. They're doing it millions of times a day and they crash the websites eventually, making them unavailable for the human users that they were initially made for. So that's kind of largely the problem. Yeah, they're being bombarded basically in these, maybe not intentional, but essentially DDoS attacks where there's all of this traffic coming and it's knocking these systems offline so nobody can really enjoy or read them or get anything from them.

4:53So that's the problem. You need some sort of solution, some sort of mitigation. I mean, hopefully the mitigation doesn't take them offline, which some people have unfortunately it has to do. So enter this tool that you've just written about called Anubis. What is it and how does it work? How is it trying to address this problem? So I wrote that story that I just talked about and a bunch of people reached out and said, hey, you should really check out Anubis. There are several solutions that people are trying in order to prevent these AI bots. One that we've talked about and that historically has worked is robots dot txt which is a file you can put in front of your website that tells bots not to crawl it just like no automated tools should crawl this website that used to work that was pre the generative ai boom and this data mining gold rush suddenly people just don't give a fuck and they're like scraping the sites anyway so yeah go ahead is is that the reason it doesn't work because again you're totally right and plenty of people brought this up that it used to work pretty effectively.

6:04You have that file on your website and it tells people please don't scrape this. Is it just that people don't care now or there's way more people doing scraping or a combination of both? Why doesn't robots work anymore? Because it's not a rule, you know what I mean? It's not like if you ignore robots.txt, the internet police is going to come to your house and arrest you. Not yet. Not yet. Once they give us badges, maybe we will do it. But no, it's just a norm that used to be respected and I don't know, we should probably talk about this at some point, but the internet used to... There's protocols and there's norms.

6:40And the internet used to run on a lot of norms. And these are just being ignored entirely now because it's going to be very profitable for you to build a successful AI tool. So if you need to jump over the robots.txt fence in order to get that training data, you're going to do it. Who cares? So that no longer works. You have some other big companies that are trying things. Cloudflare. I think they announced a new method that they're trying today where they're blocking AI bots by default. I'm not sure how exactly that works. They've also done this mischievous thing where... Jason wrote about this.

7:17Some other people are doing this. Once you... Once an AI bot comes to your site, you send it down a rabbit hole of never-ending links to garbage websites and you trap it in this infinite loop. So that's one solution. Anubis is the reason people I think reached out to me and are very happy with it and it's getting widely adopted is it's lightweight, it's open source and it's fairly effective in a less expensive way that we can get into later. Yeah. And I think the Cloudflare one as well, I mean, hate to hand it to Cloudflare, a company that enables a lot of bad stuff and is kind of all over the place when it comes to its ideology and its decisions around content moderation or whatever.

8:12Oh, okay. They actually did something pretty interesting. I'll give them that. But there's also a way that publishers may be able to get funds potentially through that Cloudflare mechanism as well. It will direct people to, well, do you want to pay this publisher to scrape this material? Yeah. So you asked me how it works, which I lightly dodge because it's complicated and I don't want to get it wrong. But basically, these bots, the scrapers that are trolling the entire internet, they don't look or behave like a normal internet user. When that traffic comes to a website, it doesn't look like a human user who is using a browser in order to access the website.

9:03Because there's no browser involved. It's just like a very lightweight, automated tool that is accessing the site and taking the data. What Anubis does is basically put a check before you can access the site where it is looking for a type of cryptography that JavaScript does and has been doing. And all web browsers have been doing since around 2022. Maybe you could talk about this a little bit. a lot of people really hate JavaScript, and there are browsers that don't use it. And there are people who disable JavaScript across the board. So if you're one of those people, you're not going to be able to access sites that use Anubis.

9:50But the vast majority of people are using Chrome or some other type of browser that is constantly going through this cryptographic check to make sure that the user is there and that they're using a browser. And basically all Anubis does is look for that once it verifies that it's a browser, that it's probably a human, and it lets you through. The reason that is clever is because there's a cat and mouse game that's happening between admins that are trying to protect their websites from scrapers and the AI companies. So the reason that the creator of Anubis, Shie Yasuo, was willing to tell me how this check works is because she knows that the AI companies can't really implement the solution.

10:45it would be too computationally expensive for all these ai's papers to pretend to run chrome before they access a website and that's why it's kind of like a simple lightweight clever solution yeah it is really smart um and i don't want to oversimplify but maybe it's just a way for people to understand but like you know it's a more sophisticated capture in a way where a capture is you're doing proof that you're not a robot, you're solving a little puzzle. Of course, there are entire companies of people in various countries who will get commissioned. I think commissioned is putting it very charitably.

11:27They'll be paid very, very low wages to click through captures and then provide the solution to that. And of course, plenty of AIs can actually bypass captures now. This is like a much smarter way of going about it, it seems. And you mentioned JavaScript, and I'll just say briefly, it seems like a really good trade-off to me that, you know, you have this JavaScript running in your browser, you're using an ordinary web browser, it lets you through. You're right in that. Some people don't like to use it. For example, when you download the Tor browser, there is a very easy-to-flick on-and-off setting to disable or enable JavaScript.

12:04and the reason for that is you know you can leak some personal data about your device to websites running certain pieces of code and you know it's i guess it's a little bit less now but you go even five six seven years ago malvertising was just a massive thing where javascript was constantly used to deliver malware to people's devices simply for loading a web page so it doesn't have the best wrap but i don't know you're an ordinary user with a fully up-to-date um uh web browser and you're accessing a website seems like a pretty good trade-off to filter out all the ai bots really so you mentioned you spoke to the creator um what did they tell you about why they decided to make this like what makes somebody wake up and then just go i'm gonna make anubis The scrapers came for her.

12:57That's why she did it. Like all the other libraries and archives that were in this previous story that I wrote about how AI scrapers are messing up their infrastructure, she has a Git server where she keeps some of her work. She keeps it open so other people can access it. And one day she was trying to access it. It didn't work. And she looked at what was going on and some kind of Amazon AI scraper was hitting it hundreds and hundreds of times. and forcing it to reset and then eventually go offline. And that's when she set out to make Anubis. That's amazing. So she makes it, it sounds like for personal use initially, did she upload it publicly at the same time?

13:45I'm just trying to get from how she makes it to, apparently a ton of people are now using this. What happens in between those two points? Yeah, I think it's just word of mouth. She did make it available. It was on GitHub. And I think people started to use it. And it seems like what really made it take off is that... I know that you're previously a Linux user, so maybe you're familiar with this. But Gnome, which is a popular open source implementation of Linux, started to use it to protect its libraries and all of its publicly available data. They picked it up and it went from there. And then a bunch of people in the open source community, honestly, started to use it because it is open source, I think was part of the attraction.

14:27And then, yeah, it's been downloaded. When I wrote the story, it was 200 ,000 times. I imagine it's a lot higher now because the story got some pickup and people were really excited about it. UNESCO's website and infrastructure uses it. Some universities use it. You sent me a story after I published where Duke had a case study about them dealing with scrapers and they also came to anubis for the solution i think that's what they ended up doing yeah i'm trying to find it right now um barge's member off the top of my head in that it wasn't even just that they were using it they were like almost reviewing it and being like this is how effective this was um for you know protecting uh our property and protecting us from scrapers and it seems like it worked um pretty well yeah and i guess like just the stress Like 200 ,000 people downloading something.

15:20Okay, not the biggest number in the world, but they're not random, normal users. They're probably web administrators or something, you know? And 200 ,000 web admins or sysadmins is a lot of people, I would say. Yeah, right. It's not 200 ,000 individual users and their little blogs. It's 200 ,000 users, many of which are the admins of incredibly valuable online resources that are accessed by millions and millions of people. Yeah. So I guess just the last thing is, what do you think happens now? Is it just that more and more people use Anubis or is it also, I mean, obviously both can be true, maybe more and more solutions start cropping up like Cloudflare as well.

16:16And there's just sort of this ecosystem of all of these, hey, screw you, stop scraping us solutions. Yeah, I think it's hard to imagine that Cloudflare is not going to provide some sort of workable solution to its customer base and other internet infrastructure companies are probably going to do the same. I think the question is how popular this open source alternative gets. And I don't know the answer to that, but I know that it has a lot of momentum. I know that a lot of people are using it. A lot of notable organizations are using it. The developer says that she's not yet at a place where it can be her full-time job, but she wants it to be.

17:05And I don't know. I can definitely imagine that happening. I can imagine her getting enough donations to make this a full-time job, hire a contributor, and have it be an open-source protocol that is very popular and is used more and more as this problem persists. Yeah, the problem isn't going away. And it's probably not even getting smaller or plateauing. It's going to get worse, if anything. And the other thing I forgot to mention is that not everyone can or wants to use Cloudflare or another solution from a big company. A lot of people on principle, or for practical reasons, want an open source free solution yeah that's fair and i guess the last thing i'll say is um of course we try to block ai scrapers as well this is a uh a big reason why we require email address um signups on the site you know an article will go out there and we'll pull it behind the free wall where you have to provide the email and you know and then stuff is obviously payable as well.

18:18But this, it seems that it has stopped some scraping. And the way that we can see that is a fewer AI generator ripoffs of our articles, essentially, which is not the best metric in the world, but it's a pretty good one. And I'm sure we'll look deeper into it as well of how much are being scraped and whether it's going down or plateauing or whatever. But that's what we do. All right, we'll leave that there. And when we come back, we're going to talk about another story of mine, which is about ICE and its new facial recognition app. We'll be right back after this.

19:09today's episode is sponsored by better health workplace stress is now one of the largest causes negatively affecting our mental health with 61 of the global workforce experiencing higher than normal levels of stress unfortunately for most of us work stress is going to stay in our lives but you can learn how to more healthily deal with it. Lunchtime resets, under-desk treadmills, and vacations can help, but they aren't long-term solutions to stress. Therapy can help you navigate whatever challenges the workday or any day might bring. With over 30 ,000 therapists, BetterHelp is the world's largest online therapy platform, making it really accessible and flexible.

19:47You'll definitely find a therapist that works for you and fits into your busy schedule. If you need to switch therapists, cancel, or reschedule an appointment, or get in touch with your therapist, you can do it with the click of a button. As the largest online therapy provider in the world, BetterHelp can provide access to mental health professionals with a diverse variety of expertise. Unwind from work with BetterHelp. Our listeners get 10 % off their first month at betterhelp.com slash 404media. That's betterhelp, H-E-L-P.com slash 404media. I do a lot of online shopping in bed or from the couch.

20:23Half the time I want to buy something, I have this sinking feeling. My wallet is in the other room. This has actually become less of a problem lately because I've noticed more stores have the purple ShopPay button. When I see that button, I know checkout's going to be a breeze because it's powered by Shopify. In fact, Shopify makes everything easy from buying stuff as a customer to becoming a small business owner and running your own store. Shopify is the commerce platform behind 10 % of all e-commerce in the US. From household names like Mattel and Gymshark, as well as another brand you might know, 404 Media.

20:59Shopify gives you that leg up from day one with hundreds of beautiful ready-to-go templates to express your brand's style. And forget about the code. Shopify helps you tackle everything you'll need in one place. From restocking inventory to managing payments to handling shipping, analytics, and more. And don't forget that purple ShopPay button used by millions of businesses around the world. It's why Shopify has the best converting checkout on the planet. If you want to see less cards being abandoned, it's time for you to head over to Shopify. Sign up for your$1 per month trial and start selling today at shopify.com slash media.

21:39Go to shopify.com slash media. Shopify.com slash media. This September 3rd through 5th, check out Inbound, which brings attendees to San Francisco for a one-time only West Coast event with insights you won't find anywhere else. Inbound 2025 is the only place that you're going to find Sean Evans, the host of Hot Ones, Creative Force Amy Poehler, tech reviewer Marcus Brownlee, and AI pioneer Dario Amadei. They'll each bring their unique approach and expertise to Inbound 2025. And this just in, the Inbound 2025 agenda is finally live from the agent.ai workshop, from idea to agent, and Dwarkesh on AI's future, research-backed bold predictions with Dwarkesh Patel.

22:28Explore more than 200 sessions all created for your growth. At Inbound 2025, you'll get to network with decision makers in San Francisco's AI-powered ecosystem, where innovative technologies are creating entirely new approaches to business. And you'll get to cut through the noise with focused, actionable takeaways on the latest marketing, sales, and AI trends to give your business a competitive edge in today's rapidly changing landscape. At Inbound 2025, you'll experience firsthand how San Francisco's technology ecosystem is revolutionizing content creation, distribution, and monetization with AI and innovative tech solutions.

23:06So secure your spot right now at inbound.com slash register. That's inbound.com slash register.

23:24All right, we are back with one of Joe's stories. Headline is, ICE is using a new facial recognition app to identify people, leaked emails show. So, Joe, you've been doing a ton of really good reporting about ICE and about facial recognition separately and sometimes combined. This, to me, is something that was not even on my radar yet as far as what police were doing in the field. And then realizing that they were sticking phones in people's faces and then also using that as a facial recognition app was just like a one-two. Like, oh my god, this is insane and horrible. um so yeah why don't you just explain maybe for a second before we dive into the app itself um for people who might not be familiar what exactly are police doing or ice agents or officers or whoever doing in the field that people were like oh shit what are they doing what are they doing with their phones before we get into the app even yeah yeah that's fair so um as many listeners will know ice is conducting operations all around the United States, but especially in Los Angeles.

24:42And of course, there were protests in response to that, and Trump deployed the National Guard, and then later the Marines. Those protests have kind of petered out for the moment, but there are all of these sort of individual flashpoints where ICE will raid a Home Depot or a 7-Eleven or somewhere else and then the community will get around it um ice will be wearing masks they'll be heavily armed they may not identify what agency they're from uh their badge number or anything like that and there's a few videos coming out around that time of ice officers repeatedly taking their mobile phones and sort of not shining almost shoving them into people's faces like very deliberately and clearly filming or taking a photo of people.

25:35And there was one very stark example where a protester was following ICE in his vehicle. From the video that was published, it didn't look like he was interfering with the operation, it didn't look like he was getting in the way of federal law enforcement or anything like that, but he was following them. These apparent ICE officers stop, they get out of their vehicle, they crowd around his car, and this is like it seems on a busy road, they then start asking him, what are you doing, why are you following us? And then two, maybe three of them get out their phones and keep pointing it at his face.

26:15Like it looks like they're taking photos of his face. And that's what I thought it was at the time, and it might still be, and we'll get into the nuances of what exactly we know and what we don't know um but people saw that and they were asking on social media and elsewhere why is ice why do they keep taking photos um of us which i know a fair question to to ask but then i got these leaked emails from inside ice revealing the agency has a new facial recognition app so maybe they're using that we don't know if in all of these videos um they're doing that but it absolutely adds new context to that and the fact is that ice does have this app now as you know i'm sure we'll get into yeah so what is this app like you said might or might not be what is in the videos we don't know but we know that it's being used or at least ice has it um what's it called and what does it promise to do yeah so it's called mobile fortify and it promises to allow ice agents to identify people in the field by simply pointing their mobile phone um at them which obviously is the point of facial recognition that's obviously the entire purpose of the technology but it's one thing to have facial recognition, I don't know, on the camera outside a military base to make sure who's coming in, or, you know, facial recognition at a retail store, which is still very controversial, but it's different to have it immediately and accessible in the hands of federal law enforcement who are grabbing people without due process and putting them into this black box system and giving them a capability right into the palm of their hands.

28:14They can identify people on the street. And we've, of course, ICE has facial recognition from other companies as well. We don't know who made this one, whether it's in-house or made by somebody else. And other law enforcement do as well. But it's the immediacy and it's definitely the context of Trump's mass deportations that makes this interesting. And of course, these emails that I got, ICE was announcing the existence and the availability of this app to all of the personnel in Enforcement and Removal Operations, ERO. And that's a part of ICE that deals with deportations. This isn't HSI, which does child abuse investigations.

28:58This is ERO, the deportation part of ICE. Okay, gotcha. So you found this through FOIA documents? Or how did you figure out the details of this app? I was leaked emails from inside ICE. And these emails were sent to all personnel inside ERO. So that's a very widespread of people, which I think is interesting because that's a very widespread of people that may or may or may not be using this tool. It's at least been advertised to them inside the agency. And of course, with this new budget that US lawmakers have just passed and then Trump's signed into law, ICE just got another$6 billion for surveillance capabilities and I think$150 overall,$45 for more facilities to store.

29:59people is deporting, all of that sort of thing. So it's clear they're ramping up their surveillance tech. And I think this is just one part of it. Yeah. Okay. Gotcha. Yeah. Sending it to everyone in ERO is very, very crazy. So we know that Clearview, as you mentioned before, uses facial recognition tech. But how is this similar? How is it different? Is it different? um what are we working with here in comparison since we already have that as kind of like a an example yeah so the way that facial recognition tech works really broadly and obviously there'll be some differences between different companies and that sort of thing but broadly is that it has this massive database of images of people's faces or maybe it has a massive database of of the hashes of people's faces, like the cryptographic representation of it.

30:57But a lot of the time, it's just going to be literal photos. That's the case with Clearview AI, which is a company that Kashmir Hill and New York Times revealed several years ago at this point. And they had, as she described in that coverage, and as the person who provided the public records to her they had sort of crossed the rubicon of facial recognition where to build that database of faces clearview scraped social media it scraped the web so people's venmo accounts uh social media accounts uh web pages all this sort of thing and created this massive database of billions upon billions um of faces um that's what makes clearview apparently pretty powerful and apparently very popular with law enforcement at a local, state, and federal level.

31:48So that's how it usually works, and that's how you usually get this sort of data. With this new ICE app, Mobile Fortify, it's not using images scraped from social media. Well, it appears to be using, and again, there are still questions we don't have all of the answers to, but it's the emails I've got said, this ICE app is using the system from customs and border protection that takes a photo of somebody every time they enter or leave the united states you imagine you go to the border and that could be uh you know at la airport or whatever or it could be crossing uh via car from mexico or wherever you're probably almost certainly going to have your photo taken especially if you're you know on a travel visa or a work visa or something like that, your photo will be taken.