Blocking AI crawlers to save the internet, with Cloudflare’s Matthew Prince

0:00Scaling a business in the US can feel like a maze. Each state has its own payroll, benefits, and compliance rules, pulling your focus away from growth, which is why founders use Deal. Deal is the professional employer organization, or PEO, that gives you a dedicated HR expert, plus Fortune 500-level benefits for your team. The National Association of PEOs says businesses can grow twice as fast if they use one. So, if you're scaling, make it simple with Deal. Go to deel.com. slash MOS and get up to three months free. If you're in private capital, you understand that your next great deal starts with who you know.

0:41That's why over 3 ,000 firms trust Affinity, the CRM platform purpose-built for private capital. Affinity's relationship intelligence reveals your team's strongest paths to founders, investors, and decision makers using insights It's hidden in emails, calendars, and meetings. With powerful AI tools like Affinity NoteTaker and Deal Assist, you'll spend less time managing data and more time closing the right deals. Learn more at affinity.co slash marketplace. We go to war every day with Russian hackers, Iranian hackers, North Korean hackers, Chinese these hackers who are trying to get in and thwart our systems and get around it in various ways.

1:28The AI companies are a piece of cake in terms of blocking the real hackers that are out there. That's Matthew Prince, CEO of Cloudflare, the$60 billion tech firm that facilitates more than 20 % of all online traffic. Cloudflare just released a buzzy new tool that blocks AI crawlers from scraping content from websites. I wanted to talk to Matthew because Cloudflare's tool could have dramatic impact on AI firms, on news and content sites, and on the future of the internet. Plus, he has an insider's view on the state of cyber warfare right now in the age of AI. So let's get to it. I'm Bob Safian, and this is Rapid Response.

2:21I'm Bob Safian. I'm here with Matthew Prince, CEO of Cloudflare. Matthew, thanks for being here. Thanks for joining us. Really appreciate having me on. So I have to say, what Cloudflare does, an internet infrastructure provider, it isn't always seen as the sexiest tech. But you do touch more than 20 % of all online activity, right? You stand between website visitors and a website or an app itself managing traffic and content so the experience is smoother and faster and more secure. But recently, you released a new tool that's got a lot of folks buzzing, a blocker for AI crawlers, the bots that scrape content from websites without their consent.

3:07You've called this new tool the biggest thing you or the company has ever accomplished. You actually gathered your team atop the World Trade Center and pushed a big button to launch it. Yeah. So, you know, I feel incredibly fortunate to have built what, you know, today is a$60 billion company on the back of the Internet. And we became aware about 18 months ago of a new threat. And we'll, I'm sure, talk about that. But to the internet, to the web, to content creators, which was being posed by these AI companies. I don't think intentionally. I don't think they're, I think most of them are actually trying to do the right thing and out for preserving the ecosystem.

3:50But they created this real, real challenge. And so when we realized that there was something we could do about it, we spent about a year talking to everyone in the content creation space, everyone sort of in the AI space. We actually signed a lease on a new office space in New York, which is the top occupied floor in the World Trade Center. And we hosted a party, and at midnight we pushed a big red button. It was me along with really the leaders in the publishing space who recognized the challenge that was there. Every publisher from the Associated Press to Ziff Davis and everything in between, we're going to change the rules of the road.

4:33and say that if you're not paying for content as an AI company, then you don't get that content. And the threat that you saw that a lot of folks like me in journalism and media see, that it was that sort of the old bargain that you could crawl my site and then you'd send me traffic back, basically, that the AI companies, that's not really what their model is, right? The original deal that Google made with content creators was give us your content and then an exchange will send you traffic and you can create value with that traffic. Either you can sell something, the content itself or a subscription to the content or, you know, an e-commerce business that's on the side.

5:19You can sell ads against it. You can get value from just the ego of knowing that someone is consuming what you created. No one gets paid for editing Wikipedia. And we have really good data going back at least 10 years on sort of how that bargain has changed. If you just look at Google, you saw it get about three times harder to actually generate traffic from the same sort of content that you created because Google changed the user interface. They added an answer box at the top of the page. So if you type in, you know, when did Cloudflare launch, instead of you having to click on one of the links to find the answer, it answered right on the page.

6:00And again, it's better for users. We shouldn't dispute that this is better for the Google user at getting the answer right there than having to, you know, like sort through documents. But it meant the answer box made it three times harder to get traffic. They then, in the last six months, added AI overviews. overviews, and the AI overviews took the answer box and they essentially put it on steroids. And so today, it's almost 10 times harder to get that actual traffic from Google for the same amount of content you created. And this isn't just Cloudflare data. Pew Research just came out with a study that lines up exactly the same and says that the minute you show an AI overview, it's less likely that people click on links.

6:44And again, that is better for the Google user, but it is worse for the content creator because it means that you can't sell a subscription, you can't sell ads, and you can't even get the ego boost of knowing that people are reading your stuff. And that's the good news. Google is trending towards something that looks more like OpenAI. Today, OpenAI is 750 times harder to get traffic from than the Google of old. Anthropic is 30 ,000 times harder to get traffic from than the Google of old. And so if publishing, if content creation is struggling today at it being 10 times harder, I worry that it won't survive at 750 or 30 ,000 times harder where instead of reading original content, we're reading derivatives of that content.

7:31And if people don't have the incentive to create content, they're not going to create content. I mean, reporters need to eat. Journalists need to eat. Academics need to eat. Researchers need to eat. So there needs to be some business model or at least some value creation model behind the future of the web. And it's not going to be around traffic because an AI-driven web doesn't drive traffic. Well, and the irony, right, is that the AI itself needs the content that folks are creating to be able to make those answers. And now, you know, who knows where they're going to get their answers from? That's the key.

8:06And 80 % of the major AI companies use Cloudflare. in their infrastructure. So we know them pretty well. And we have had over the course of the last several months conversations about this. And what they have all said, with a few exceptions, but almost all, is we agree content creators need to get paid for content, but it has to be a level playing field. What nobody wants to do is that they pay for content where all of their competitors get it for free, right? Or the flip side where somebody gets it for free and everybody else has to pay. Because what you ideally want, you want a lot of people creating content, but you also want a ton of different buyers.

8:51So you want to make sure that, you know, new startups can participate, you know, established players that have been in some other space aren't at a disadvantage to someone like Google, who's been able to, you know, crawl the internet with impunity for an enormous amount of time. You want to make sure that someone who's hyper-regulated like Google can still compete with new startups. So creating that level playing field is incredibly important and ensuring that there's a healthy ecosystem. If we can figure out how to do that, if we can figure out how to say, listen, everyone has to pay. The rules are the same and fair across the board for everyone.

9:27I'm encouraged that the AI companies have actually said, yes, and we're on board because ultimately we know that the fuel that fuels our engines, the thing that is critical is that original content that's out there. I mean, JustAnthropic will scrape a site 60 ,000 times for every one visitor that's there. And what's key is that's real cost, right? Someone has to pay for that traffic. And so just from a pure fairness perspective, they should be compensating creators that they're pulling that content from. And this blocker tool that you came out with, is the tech itself hard fundamentally? I mean, I'm sure it's not easy, But like, how much of it is about sort of the tech versus like the strategic decision that you guys made to like, all right, we're going to challenge this, which seemingly no one else was trying to do.

10:17It's hard to say what's hard anymore. I mean, we process, I mean, something like 20 trillion internet requests a day, which is just an inconceivable number. So just the mere fact that we do that is hard. And if you think about who we go to war with, like what our primary business is, we started as a cybersecurity company. And so, like, we go to war every day with Russian hackers, Iranian hackers, North Korean hackers, Chinese hackers who are trying to get in and thwart our systems and get around it in various ways. And so it was kind of funny. When we first started talking to publishers about this, like, it was almost this sort of nihilistic kind of, oh, my gosh, what are we possibly going to do?

10:59There's no way we can stop it. These guys are so smart. They're a bunch of nerds. They're in Palo Alto. So we can't ever possibly block them. And I remember thinking, like, we block the North Koreans every day. And they're not subject to the Computer Fraud and Abuse Act. And they don't have a C-corporation that you can sue, right? The AI companies are a piece of cake in terms of blocking versus the real hackers that are out there. That said, I think the hard thing isn't the blocking. The hard thing is making sure that you're doing it in a way that creates a level playing field. Google is a huge challenge.

11:38And here's why. Google has one bot, and that bot does five separate things. It does brand safety checks. So if you're Procter & Gamble, you don't want your ads being run against pornographic content or something like that. And so the bot checks a page to make sure it's not porn and then only releases a Procter & Gamble ad if it's not. So it does that. It does traditional search indexing. It does data mining for the answer box. It does data aggregation and collection for AI overviews. And it does training for Gemini. So it does, one bot does all of these things? Five things. It's all in one. Yeah.

12:19And they give you three different options today. They can say you block everything, but that sucks because now you're not in Google search results, right? Now you're not able to put any ads on your pages because Google powers so much the ad ecosystem. You can allow everything, but that sucks because now your content is being taken from you and you're not getting any value exchange for those three things, for AnswerBox, AI Overviews, or Gemini. Or you can block Gemini, the fifth thing. They've actually given you the ability to block that one thing and not block the other four. What we think is we need to create really a taxonomy and an industry standard on what responsible crawling is.

12:59And that's super hard. But we're working with standard organizations like the IETF to set down what those standards are. And we're very hopeful because, you know, if Google is sort of the hard case, on the other side, who's doing this really responsibly is OpenAI. OpenAI says, okay, we have one bot for training. We have a different bot for search. We distinguish between those things. We publish clearly who our bot is. They've been willing to They go into actually cryptographically signing that their bot is their bot so that we can tell that someone's not forging it or something else. They follow the sort of the equivalent of like the street signs that are out there and they call it a reasonable rate.

13:40And again, I think that that's a very good example of a company that is trying to do the right thing. And before you release this first round of this tool, because clearly you're continuing to develop it, did you give the AI companies a heads up? Or some of them, I mean, some of them are clients, but I can imagine that there are some AI companies that are like, how do we get around this new tool? You know, I think there are some bad actors out there. And I think it'll surprise some people who the bad actors are. Some of them won't be surprises. But there are some folks who basically, if they can't get through one way, they try a bunch of other things that start to resemble what the North Koreans do.

14:23Now, we, again, are really good at this. So we're tracking them. We're monitoring them. And very soon, we will publish and we will name and shame who is actually a bad actor in this space. And we will take from what has been basically posting a speed limit sign that says, you know, don't drive more than 55 miles an hour. We'll take that speed limit sign that we have that we put up on July 1st, and we'll make it into something that is actually much more strict, where we're saying, listen, we're taking away your car. We're not allowed to drive on the road anymore. And the same time that we do that, I think we're going to highlight some of the companies that are doing it right.

14:56Because you want to actually say these are companies that are good actors that are out there. And so as we start to separate the good actors from the bad actors, and by far, most of the AI companies are actually good actors. There are a handful that are doing some pretty shady stuff. And again, we'll shut it down and hopefully they'll reform. I understand you're exploring sort of a pay-per-crawl model with some of the content publishers, which to me sounds a little bit like a toll on the highway there, right? That you have to pay a toll if you want to come through? Well, again, just simple first principles.

15:29If you are generating a huge amount of cost by crawling somebody, but you're not giving them any benefit, then step one is block them. But if that's still – if that's valuable to you, then once you've created scarcity, then there can be a market, right? There's already a lot of content that AI companies are paying for. OpenAI, again, is one of the more responsible players in the industry from everything that we've made, has done deals with a lot of content providers saying we'll pay for the content. But Sam can't be a sucker. He can't be like I'm paying for it but all my competitors get it for free.

16:05And so I think that comes back to why the reception from the AI companies was actually a lot more positive than people might assume. Because what everyone wants, everyone agrees the ecosystem has to be healthy. Again, almost everyone agrees, a couple of bad actors. But almost everyone agrees the ecosystem has to be healthy. There has to be some compensation for taking content. And it's not going to be traffic anymore. It's going to be something else. Now the question is, okay, how do you pay? And I think a lot of times big AI companies and big publishers are just going to negotiate deals themselves.

16:34So if you're Condé Nast, you go out and do an open AI deal or a Google deal or something else, and you can negotiate it yourself. We don't have any – again, we can provide the blocking technology, but we don't have any role in that. I think for the smaller AI companies or for the long tail of publishers, that Cloudflare can hopefully sit in between and help negotiate what is the best deal. And we don't know exactly what that will look like yet. It could be a micropayment every time a page is accessed. It could be something that's closer to the Spotify model where there's a pool of funds and that gets distributed out to all of the different content providers.

17:09I don't know. That will develop. But step one in any market has to be scarcity. If you don't have scarcity, you don't have a market. And the bad actors that you refer to, I know you're not prepared yet to name them. But are these like companies that are non-U.S. companies, you know, China and other places? or these are? No, we have some very bad actors that are venture backed, you know, very successful companies. And we have reached out to them directly and said, listen, we see you doing this. You got to stop or we're going to shut down your access to the roads. And we know a lot of their investors and we've reached out to their investors and we've given them every possible chance.

17:53And so now we're at the final stage. Next step is name and shame them. And again, that's not something that I'm excited about doing, but I do think that we've got to get, we have to establish this level playing field. Let's just acknowledge that Matthew is really poking the bear here in challenging how AI companies operate, going after the hottest players in tech who are driving global markets and the global economy. But change is necessary, at least from my perspective in media as a content creator. So what will the internet business models of the future look like? We'll talk about that after the break.

18:34Stay with us.

18:39Real leaders don't back down when the stakes are high. They innovate, they push forward, and then they take the stage at the Masters of Scale Summit. Join us in San Francisco, October 7th to 9th, to hear from the CEO of the New York Times, scientists using cutting-edge technology to find cures, the leader of crypto powerhouse Coinbase, a retired four-star general, and many, many more. Apply now at mastersofscale.com slash apply25. That's mastersofscale.com slash apply25. If you're ready to take your startup from idea to impact, then AWS is your launchpad. From data storage to machine learning to secure app hosting, AWS gives you the tech trusted by the world's fastest scaling startups.

19:28And here's the best part. Join AWS Activate today and you could score up to$100 ,000 in AWS credits tailored to your stage and network. Go to aws.amazon.com slash activate and start building. Expanding your business in the U.S. can feel like a maze. Every state has its own payroll, benefits, and compliance rules, which can pull your focus away from growth. That's why founders use Deal. Deal is the professional employer organization, or PEO, that gives you a dedicated HR expert plus Fortune 500 level benefits for your team. The National Association of PEOs says businesses can grow twice as fast if they use one.

20:12So if you're scaling, make it simple with Deal. Go to deel.com slash mos and get up to three months free.

20:45where internet business models and incentives are moving to, and whether it's likely to be as robust as what we've had in the past. Let's jump back in. I'm encouraged by analogous markets that have developed. Before Steve Jobs launches 99 cents per song with iTunes, the music industry was about an$8 billion industry in terms of what their annual revenue was. if you just look at Spotify, last year, Spotify distributed$10 billion to musicians. And so even though everyone thought, oh, digital is going to kill the music industry, it actually ended up being pie expanding. And so I'm actually optimistic.

21:29All of us are going to have subscriptions to a certain number of AI agents that are out there. That revenue that goes to the AI companies, some portion of it has to go to GPUs, some portion of it has to go to AI researchers. Some portion of it has to go to your other overhead. But some portion of it should also go to content companies and content creators. And my prediction is that the algorithms and the GPUs will turn out to be commodities largely. And that how AI companies will differentiate themselves is access to unique content that they have and they have alone. So imagine Taylor Swift is about to release a new album, and she does an interview with some journalists, and they are willing to give that interview to one AI company exclusively for a week, how much is that worth?

22:18Probably quite a bit, right? A lot of people are going to sign up if you haven't. It's like, if Netflix is willing to spend tons of money to produce movies, like there's going to be value in actually creating unique content that your AI has that another AI doesn't have. And so I'm actually optimistic that we might be at a golden age, at the precipice of a golden age of content creation. If we do this right and we get the incentives right, it might be that instead of us all sort of worshiping sort of the deity that Google taught us to worship, which is traffic, which has always been a really bad proxy for value.

22:54If instead we find a way to compensate creators based on when they actually create something which is worthwhile and advances this human knowledge, that we can actually do some real good in the world at the same time that we help the content creators get paid more. I hope you're right, Matthew. I sure do, because it does sound like the world I want to live in. Think about everything that's wrong with the world today. And again, I don't care whether you're a Democrat or Republican and conservative or liberal. No matter what, we've gotten more polarized. We've gotten more anxious. this, everything feels urgent and panicky.

23:34And if you ask, like, why is that the case? This is going to be too strong a statement, and I risk pissing my friends at Google off, but sort of it's all Google's fault, right? I think Google has been a net positive for the universe. I think they're good, but they did teach us all to worship this deity of traffic. The question is, how do you get traffic? And so there was a whole series of companies that came along, the Huffington Post and BuzzFeeds and others of the world, where they would create a piece of content and they would literally beta test different headlines for which ones created the largest cortisol response, the largest stress response in readers, because that was the thing that was most likely to get a click.

24:16I don't know that that's good for the world. And again, Google just started it. Facebook took it to the next level. TikTok is now taking it to a whole nother level. But that world in which we are just trying to stimulate a cortisol response, and that's what success in media is, I think is really, really, really dangerous. If, on the other hand, you think about all of the AI companies that are out there, it's not a perfect representation, but it's a pretty good representation of a map of what human knowledge looks like. And I picture it like a big block of Swiss cheese, right? So big block of Swiss cheese, lots of cheese, but lots of holes in that cheese.

24:55If we could actually help AI companies say, okay, and this is actually a technically solvable problem. When a new piece of content comes out, if we say, hey, does this sort of just add more cheese to where we already have cheese, or does this help fill a hole? We can say that you get compensated more by filling those holes. I think that's actually the way that you advance human knowledge. Again, I think Spotify has done some just amazing things. One of the things they do is they actually publish a list of queries that people have typed into Spotify of music they want to hear that they don't give very satisfactory answers to.

25:31So let's say that you want to hear, I don't know, a disco about how much fun it is to dance with your cat. There's no song out there like that. They publish that list. And there are music creators out there that look at that list and they write songs based on what people are searching for that is sort of this unmet need in the market. And they make tens of millions of dollars a year writing songs based on unsatisfactorily searched Spotify queries. If we can get to that place where, again, instead of us thinking, how do we write content, which is yet another story about what crazy things are happening in politics or how, whatever it is that makes people just get up in arms and click on things.

26:20If instead we can say, what is it where there's an unmet human need to advance human knowledge? And we can reward content creators for that. I think that could, again, be the golden age of content creation. You mentioned earlier that your heritage at Cloudflare is from cybersecurity. I want to ask you about the topic of cyber warfare. I mean, wars today, they emerge in cyberspace before they're on the battlefield. We've seen that in Ukraine. Your detection tools have been utilized in a lot of these situations, right? Like, you're a partner in these cyber battles? Yeah, I mean, I think we're a defensive partner.