Can The Web Survive Generative AI? — With Matthew Prince

25:26And so I want to run this other idea by you, which is that if we had the same amount of content that we have today, that's a great idea. But what we're seeing now is this explosion of content creation that's made through generative AI. Like it's kind of funny, like every time you see like these suggestions that we're talking about, YouTube's making these suggestions because clearly there's traffic to be had. I'm sure there are already YouTubers today that are feeding that into ChatCPT, spitting out a script, running that through VO3 and Google, and then posting the videos and cashing in on traffic.

25:58So there's just going to be, and we're in the middle, I believe, of this explosion of content. Actually, you probably have better data on that than my suppositions. it almost feels like a DDoS of the web where like, you know, if there, if the ability to create content is constrained by a human's ability to create content, then you have something to bring to these AI companies. But if human plus bot content starts to become the norm, there's going to be so much, then even if you're creating high quality stuff, it's not going to matter very much to these generative AI companies. What do you think about that?

26:30I think it's still, So first of all, I think that there's the pure AI generated content. There's lots of research that shows that training AI on AI data is sort of like that old Michael Keaton film, Multiplicity, where basically every copy of something gets worse and worse and worse. And again, that feels like that's going to still be the case for quite some time. May in the future robots be able to go out and do interesting reporting from the field. May they be able to do interesting research for sure. But today, I think that that interesting research, that interesting original content, that interesting insight that comes from the work that right now only journalists and researchers and others can do is still the most important thing for filling in those gaps in the Swiss cheese of AIs.

27:17What is just, again, high volume, low value content? My hunch is that that's, if we score it correctly, going to be exactly what it is, which is low value content. And so it should be rewarded very minimally. I like to ski. So I live part of the year in Park City, Utah. You're in the right place. I care enormously about the snow forecast. There is a forecaster in Utah named Evan Thayer. He writes these incredibly precise weather forecasts where he will literally tell you it's going to snow this much on this run and this much on this run. And again, I actually pay for his content because that's super value for me.

27:58I am going to be more willing in the future to pay for an AI that has actually licensed Evan's content back from him than I would to pay for an AI that doesn't have that content. Because again, that content is going to be super useful and unique and valuable to me. And so I think actually what it will do is we as we have more AI systems that are out there is it will cause you to look for more original creative content. And that's going to be the thing that the AIs are going to be the most willing to pay for. And that, again, I think is actually a beautiful thing where we're instead of creating incentives to create more and more salacious headlines and chase traffic, we're creating incentives to create knowledge that fills in those places in sort of the Swiss cheese where there might be holes.

28:42Taken in aggregate, all of the AIs are probably a pretty good representation of what human knowledge looks like. And so if we can score them and say, okay, here are the gaps in human knowledge and here are the places we need to fill in, that actually gives a really rich place for creators to look to create content which advances human knowledge. So you know DeepMind is working on weather forecasting right now. this example that you gave of Evan Thayer, the forecaster in Utah. Are we that far away from just telling an AI, hey, like you're tapping into the deep mind model on weather forecasts. I want to ski this route today.

29:20What's happening? I think we're probably pretty far away from that. But again, I think Evan is going to be always better using the tools of AI plus his local knowledge to make this better. AI just becomes a tool that creative people use in order to tell stories better, get better information, do more research. And again, I am skeptical that in the short term, at least, that we're going to have real value that is created by training on purely generated content. Okay. Shape the future of enterprise AI with agency. A-G-N-T-C-Y. Now an open source Linux Foundation project, agency is leading the way in establishing trusted identity and access management for the internet of agents, the collaboration layer that ensures AI agents can securely discover, connect, and work across any framework.

30:16With agency, your organization gains open, standardized tools and seamless integration, including robust identity management to be able to identify, authenticate, and interact across any platform. Empowering you to deploy multi-agent systems with confidence, Join industry leaders like Cisco, Dell Technologies, Google Cloud, Oracle, Red Hat, and 75 plus supporting companies to set the standard for secure, scalable AI infrastructure. Is your enterprise ready for the future of Vigentic AI? Visit agency.org to explore use cases now. That's A-G-N-T-C-Y dot O-R-G. So we've talked about your solution.

30:58Let's dive into the technological side of it a little bit. We are a tech podcast, so we should do that. So Cloudflare, security company, helps websites stay up on the web despite all the threats. Yep. And let's just, at the very beginning, kind of talk about the threats that you see to websites, who's trying to take them down. Yeah. What's happening on that? Yeah, so protecting websites is part of our business, so is protecting employees as they go out across the internet. So Cloudflare is fundamentally kind of a network that is built with all the performance, reliability, security, availability, and privacy guarantees that, frankly, the internet should have been built with had we all known what it was going to become.

31:40But obviously, back in the 60s, 70s, and 80s, when we were laying down all these protocols, we didn't think about those things. And so Cloudflare is basically reverse engineering the Internet in order to give it those performance availability, security, reliability, and privacy guarantees on top of what is there. And so today, one of the main uses for Cloudflare would be to you're putting a website or a web application or anything online. You want to make sure that it's safe from different sorts of threats. And so what are the threats that we see? I mean, every day we go to war with the Chinese government, the Russian government, the North Koreans.

32:21I mean, everyone is trying to hack into our customers because who are our customers? Some of the largest banks in the world, some of the largest governments in the world. And they are all constantly under threat and constantly under attack from these organizations. The media companies actually were a pretty small part of our business. We had some media companies that used us, but it wasn't a big piece of it. And what happened starting really 18 months ago is that those companies said, hey, I know we hired you in order to stop the Chinese hackers, but we have this new threat that's there. And frankly, my initial reaction was, publishers, they're always whining about the next new technology, like what's going on?

32:58And over and over they said, just pull the data, pull the data, pull the data. And it was only when we actually saw the data and we saw how AI companies were taking content without giving anything of value in return, that they were actually adding enormous amounts of load and in some cases taking whole websites down because of the amount of traffic that they were sending to it. Right, they basically DDoS the websites. DDoS the websites, you know. It's crazy. Not intentionally, but that was the point at which we said, listen, maybe there is something that we can do here. And, you know, at first, I think a lot of the publishers were saying, oh, this is so hard.

33:35There's no way we can stop it. You know, there are these nerds and they live in Palo Alto and they're so smart. What are we ever going to possibly do about it? And I just kept saying, guys, we go to war with the Chinese hackers. We can stop some nerds with the C Corporation. And I think it took a while for that message to really get through. But now that it has, it's been really rewarding to see that the vast majority of the world's publishers, major publishers have said, we need to change the model. We need to be compensated for our content. And Cloudflare has the right idea in terms of the technical solution to do that.

34:11By the way, folks,$60 billion company listed publicly. So it's one of the bigger cybersecurity companies on the New York Stock Exchange. But I want to ask you, okay, so we're going to get into this technological solution. But what you said is interesting because what if these AI bots, do you ever think there's a world where these AI bots ingest not just the publishers, but the banking websites as well? Like, are you like a natural enemy to having everything go through that? because if everything goes through ChatGPT, then these other sites that you secure might not need your services. I think, I mean, there's going to be some gatekeeper for how agents and other things access various services online.

34:51And I think that the challenges in each of those cases are different. In the case of a bank, you might want to say, I want to have guardrails that are in place. I want to make sure that this is actually a customer that's accessing an account. I want to make sure that they can only conduct transactions that have been authorized by an actual human being or something like that. Cloudflare actually provides those guardrails and makes it so that a bank can say, I want to expose my infrastructure to AI, but do it in a way which is safe and secure. I think publishers have a different challenge. And so in our case, a way of thinking this is we have a whole bunch of developer documents which are on our website.

35:29We want those to be in AI. We want coding platforms when someone says, oh, I want to use Cloudflare to build X, Y, or Z for it to be able to spit that out. What we've done is we've actually tried to identify with real narrow precision, what are those pages that are on the web that have some indication that they are going to be monetized? And generally that is look at, is it behind a paywall or does it have some sort of an ad unit on it, like a banner ad or some sort of ad that's there? If we detect that, then we're we're blocking it by default, but we're not doing this. Again, there's value for AI.

36:07And we want to make sure that AI is actually getting the data that people want to have in it. So the About Us page on the New York Times probably should go into the AI system. But in brand new article, you know, with breaking news, probably should be restricted. And again, unless the AI company is actually paying for that content. I guess the way I want to ask it is if everything goes into ChatGPT, what's left for you to protect thinking outside of the media world? Well, again, I think that 80 % of the AI companies are customers of ours. And so we protect them as well. Okay, sounds good. Just wanted to ask that.

36:41I was curious about it. But let's talk. Okay, now, so you're going to build a technological solution that will block crawling. Yes. And so robots.txt, which is this code that you put in like the header of your site, if you don't want to be crawled, that wasn't working? Yeah, I mean, I think robots.txt has two problems. The first is some people just ignore it. And so if you ignore it, then you can still crawl all you want. And there's even some big legitimate companies that completely ignore robots.txt. And we're really good at basically being able to say, okay, here's what robots.txt says. How are you actually following what sort of the rules of the road are?

37:23And if the answer is yes, then robots.txt is a great solution. But in the cases where somebody is ignoring it, then we need to actually put in place additional technical barriers to restrict their access. And so that's exactly what we're doing. The second problem with robots.txt is it's not granular enough. So take the Googlebot, for example. Google's crawler does five different things, at least. One is it checks if you have an ad on a page, make sure that if you're putting an ad for Procter & Gamble product up, it's not against a pornographic site or something like that. So it does brand safety checks.

38:01The second thing that it does is crawls to index for traditional search, the 10 blue links that are out there. The third is that it crawls to create answers that are in the answer box. The fourth is that it crawls to create answers that are in the AI overview, the newer thing that they've rolled out. And the fifth is that it crawls in order to ingest content in order to put it into Gemini. It's a lot of crawling, a lot of crawling all through one crawler. And for lots of different reasons, they don't want to split that out into various crawlers. But right now, they basically make you have a choice.

38:30They say you can either block Google entirely, in which case you can't run ads, you don't appear in search, but you don't appear in the AI overviews or Gemini or other things. Or they've recently added a tiny flag, which basically just says, I'm not going to use this data just for the Gemini piece. But you still appear in AI overviews. You still appear in Answerbox. We think there needs to be more granularity where there is a difference between taking content and transforming it. And a license should say, you can't do that without my permission versus just taking that content in order to do brand safety checks, taking that content in order to do traditional search.

39:05And so what we've proposed and we're working with the IETF as well as regulators is extensions to robots.txt to give it that granularity. And that actually then allows us to further test to watch, you know, if does this robot behave in an appropriate way? And if the answer is yes, then maybe it gets more permissions to do things online. If the answer is no, then we will put more restrictions and blockades in place to stop what are, again, badly behaving robots. So what you're going to do now is, in addition to that, put a wall up, technological wall. That's right. No crawling. Sorry. Enough of you haven't respected robots, TXT, no entrance.

39:43That's right. And so the original, like we're all familiar with like 404 errors when something is not found. Success on the internet is a 200 response that comes back to you. There's actually an original, one of the protocols set out a 402 response. And that response says payment required. And so we're actually tapping into that exact original specification to say when a robot tries to access a page where there's an intent to monetize it, so it's either behind a subscription or it's got ads on it, that there is an ability for us to say 402 payments required. And then there's a negotiation. And in some cases, and at first that's going to be largely large publishers with large AI companies doing deals like what Reddit has done or what the New York Times has done or what others have done where they have licensed the content.

40:31And then certain robots get access to that. But in other cases, and I think over time, that will be a dynamic process where maybe a smaller AI company or a smaller publisher will say, hey, here's what I would charge for this content. Cloudflare will surface like how valuable that content would be for that particular AI. And then the AI companies can decide, is that worth it or not? And it might be a very small transaction, maybe a fraction of a penny or maybe a few cents, or in some cases, content that is really valuable might be worth hundreds or thousands or millions of dollars. You could imagine Taylor Swift is about to release a brand new song and the lyrics get published.

41:08How valuable is that for an app for teen girls who are lonely and want to talk about things? It's probably pretty valuable and especially valuable if you could have exclusive access to it for some window of time. And so that's the sort of thing where I think a marketplace over time can develop where original valuable content will get compensated and there will be a clearing price in the market once we have that scarcity that's created by that wall. Okay, so it's not just a blocker. It's also this marketplace where you're going to have publishers that will sell their content. So that's a way where you could have useful, effective chatbots and potentially a flourishing web.

41:44Exactly. And that's and that I think is what we're trying to play for. Again, my utopian vision of the future is robots should pay a lot for content and humans should get it for free. Right. And so to kick this off on June 3rd, as the day turned July 1st, you had a party on the top of the World Trade Center, where a bunch of publishers pressed a red button to get this thing going. And that includes some very big names, Condé Nast, Time, The Associated Press, The Atlantic, Adweek, and Fortune are all going to be part of this. And a lot more. Frankly, there hasn't been a publisher that we've talked to who hasn't said that this is a change that needs to happen.

42:24You're on the right path for it. And so across the board, not only the kind of 20 % plus of the web that sits behind Cloudflare already, but I think another 20 to 30 % that are these major publishers that are out there are all on board in doing that. And what I think has been encouraging is at the same time, we've been having conversations with the largest AI companies. And all of them agree that content creators need to be compensated for their content. They all agree on that. The devil's in the details. And some of them are pushing back in various ways. But I've been really encouraged that as we have talked to the largest leading AI companies, the largest technology companies in the world, they're actually leaning into this.

43:03They all recognize that content creators need to be compensated. And I think over the months to come, that's when the hard work will go down around how do we actually create this marketplace in a way which is fair for all of the different providers in the ecosystem, treats everybody in a way that has a level playing field, still allows new entrants, doesn't just reward the largest companies with the biggest budgets that are out there, make sure that, you know, legacy providers like Google are treated the same as, you know, newer providers that are there. That's all going to be really tough. But I am incredibly encouraged by the conversations I'm having, not just with the publishers who are all on board, but actually with the AI companies who recognize that something needs to change.

43:47That's interesting that they're recognizing this because the sense that you get is that you hear these announcements of deals, like OpenAI paying X million to the Wall Street Journal to be able to include their articles or Dow Jones. And the sense you get is that they're just kind of payoffs to not get sued. Like Sam Altman very clearly is not happy with the New York Times pursuing OpenAI and especially the actions that the Times are taking in their lawsuit, like forcing OpenAI to preserve their chat logs, which I think is wrong. But it is interesting. So what do you think about – are we going to see an evolution from these one-off deals to this marketplace-style world?

44:26Well, I think that – I mean we've seen this story many times before. I mean, Napster was along, it was a wild west. There were a bunch of lawsuits from the publishing, the music industry targeting Napster and the like. And then along comes iTunes, which starts out as 99 cents a song, but eventually evolves into what is much more, much closer to a Spotify model of a subscription and a pool of funds that then get distributed out to all the creators. So I think we've seen this story before. And I think that one of the things that's really important is that OpenAI and others are willing to pay for content.

45:04They do the deals that are there. And I don't think it's right to just say, we'll do a deal to avoid lawsuits. Again, I think that when you talk to leading AI companies, they understand that people are doing the work to create content. They need to get compensated for that content. And if it's not going to be through subscriptions or ads or ego, it's got to be through something else. And so exactly how that happens, we'll figure out. But what I know won't work is if OpenAI is paying for your content, but you're giving it away for free to everyone else. It's not going to work. OpenAI eventually is like, listen, we want to support you.

45:38We want to help you out. But we can't be the suckers. We can't be the only ones paying where you're giving stuff away for free. And so scarcity is needed in order to actually have value in any kind of market. And so I think that the people who have actually leaned into this the most heavily are the ones that have the existing deals with some but not all of the AI companies because they realize that for those deals to be valuable, for them to renew for more, there has to actually be scarcity where they're getting something of value. You can't charge open AI, but give it away for free to Anthropic.

46:12Something needs to actually restrict it and say, everyone needs to pay. Everyone needs to be on a level playing field and figure out what that looks like going forward. Could there be some collateral damage with the solution like you're implementing? For instance, I'm looking at the names of these publications, Condé Nast, Time, the AP, the Atlantic. I imagine they get a lot of traffic from search as it is today. So if you put this blocker up, does that impact their SEO, for instance? Yeah, so we've been very, very careful to say that the traditional search today is not blocked. And even AI-driven search today isn't blocked.

46:47But you're going to see us give publishers the tools to differentiate between search indexing and derivative content. So the way I would think about this is the Google experience today. It may be that a publisher says, I still want to appear in the 10 blue links, but I don't want to be in the AI overview or the answer box. And the granularity of being able to say, OK, Google, I understand you use one bot, but we need that to be treated similarly. And again, I am hopeful. And in my conversations with Google, I am increasingly hopeful that they understand the importance of this and giving that granularity.

47:21But if for some reason they don't, I am also 100 % certain that regulators are paying a ton of attention to this and that around the world, you will see them force Google to split their crawler out into announcing exactly what it is doing. Again, I think that that's kind of the, hopefully we get to an agreement with Google way before that has to happen. But that's inevitably, I think Google is going to have to say, you know, if you don't want us to use your content for derivatives, you have a way of controlling that while still appearing in search. Okay. A couple of big picture questions before we leave.

47:58How much bigger is the web getting? And is the web sort of accelerating and the size increases that we see? Yeah. I mean, it's actually been, by all the measures that we can see, it's actually kind of plateaued and is actually flattened out in terms of content. You see fewer domains getting registered. You see fewer new websites going online. I think a lot of that has moved to individual platforms. So more of that on a YouTube, more of that on a Facebook, more of that on a TikTok that is there. And I think part of that is because those tools have provided content creators easy monetization tools to allow them to not have to think about some of those problems.

48:39I think that in an ideal future, you would want content creators to be able to be free from those platforms to earn more themselves and but still have abilities to monetize that content in interesting ways and so um again i think there are lots of people who are working on on that problem i actually think google has been you know one of the organizations that has again created what was the business model of the last 30 years of the web but the business model of the next 30 years of the web is going to be different and we've got to think about it in a different way it's not going to be banner ads it's not really probably going to be subscriptions it's going to be something different.

49:15And so this is our attempt at one solution, but I doubt it will be the only one that emerges. Now I'm curious what I'm going to do because it's the one person content operation. So anyway, you should certainly be charging AI to license your voice. Can I sign up to your product? For sure. Absolutely. Okay. I'm going to email you after this. And then when it comes to cybersecurity, obviously you talked about how you're dealing with all these governments that would like to hack into sites across the web. Have they been able to use generative AI tools or automated coding to get more to become more effective at what they do?

49:48Yeah, I mean, I think that that anytime a new technology comes out, bad guys are going to use it as well as as good guys. And so we have seen and we will continue to see some horror stories around, you know, the family that was tricked by some gang in order to wiring their life savings because something someone that sounded like their their daughter called and said, I've been arrested in Mexico. You know, I need I need to pay to get out or other things. I think we are seeing a real rise in, especially out of North Korea, North Koreans posing as if they were applicants to various jobs. And then that is, you know, allowing them access, which they can then use to do any number of nefarious things.

50:33All of that, again, assisted by AI. So I think that's been sort of on the bad guy side. The good news, though, is that the good guys, folks like Cloudflare, we have been using AI as well in order to not only detect these things, but get smarter at detecting attacks earlier in the process. That's working for you. At the end of the day, who wins in the AI risk is whoever has access to the most data. And I just think that the good guys are always going to have access to a lot more data than the bad guys. And so far, I feel like we have made the web more secure with AI over the course of the last two and a half years and stayed way ahead of the attackers.

51:09Although, again, there are going to be horrible stories. There are going to be problems that are there. I think that it is going to be harder and harder to trust that something that you're seeing online is actually real and will have to turn to other ways that are more secure about verifying things like identity and authentication. okay last question for you we have 60 seconds you mentioned you're a believer in this technology what does the next couple years in ai look like to you are we going to hit agi anytime soon like what are the what's the timeline you're thinking i mean i i don't i don't i'm not i i am i am so i believe today that 99 out of cents out of every dollar spent on ai is just being lit on fire but that one cent that's out there is going to generate real returns it's very hard to figure out what's kind of just a total waste of time versus what's not um you know we we see a lot of data about how much you know ai systems are really um are are being used not not so much for businesses today a lot of the business applications have been very tough to uh take on but but a lot of times just for for like loneliness and social interactions and things like that so i would imagine that a lot more of those things are going to develop and those will be sort of the first uses.

52:25I think the business applications are actually going to take longer. And in places where it's easier to verify the output as being legitimate, it's going to be easier. So coding, like we see that our engineers are significantly more productive by using AI tools than they were before. That's not causing us to hire any less engineers. It's just meaning that every engineer we hire is that much more productive. We have a huge backlog of things to do and AI is helping us do that. On the other hand, I'm still quite skeptical that the AI customer support agent, that is a much harder problem, or the AI lawyer, that is a much harder problem because it's just harder to tell whether something actually worked or didn't.

53:06There's no debugger in those spaces in order to figure out if what the AI is creating was actually true. And so I think you're going to see just huge leapfrogs in what are things like coding, but I think it's going to take longer for us to do things that are a little bit more difficult to verify. Very interesting. You're deeply optimistic about the technology, but still think 99 cents are wasted. It's going to be very interesting to shake out. Matthew Prince, great to see you. Thank you for coming on the show. Thanks for having me on. All right, everybody. Thank you for watching and listening.

53:34We'll see you next time on Big Technology Podcast.

53:48You