


Cloudflare’s stance on “Content Independence Day” and how publishers/brands can survive the shift from human-driven web traffic to AI/agent traffic. It covers pay-per-use for AI scraping, AEO (answer engine optimization) readiness tools, disallow AI training controls (separating search vs training), and the broader “crawl-to-referral” breakdown.
AI agents now generate over 57% of web traffic; the crawl-to-referral ratio is tens of thousands of times worse than before (e.g., OpenAI ~1,700 scrapes per referral; Anthropic ~73,000:1). Publishers need new business models so AI can’t “summarize what doesn’t exist.” Cloudflare argues AI should pay for content and that enforcement must be network-level, not an honor system.
pay-per-use marketplace with declared crawlers; converting content to Markdown for better machine readability; Google’s mixed crawler issue and Cloudflare’s “one button” controls to restrict training while allowing search.
Lara Cohen, VP Strategic Partnerships for Media, Creators, and AI at Cloudflare; former editorial director at US Weekly, and partnership leadership roles at Twitter and Linktree.
Written by AI. May contain mistakes.
Discussion on the impact of AI on original journalism and publisher content.
“that goes into good journalism, and also what it takes to keep that work going.”
Introduction of Lara Cohen and her diverse career in media and technology.
“What is one thing about you that not a lot of other people already know?”
Lara discusses her work at Cloudflare supporting publishers amidst AI changes.
“But I, you know, I spent yeah, I started my career as in media as a journalist myself and then made, you know, the switch to Twitter.”
Exploration of Cloudflare's initiatives to protect publishers' content and adapt to AI.
“referral traffic for their business model and what was going to happen with that all being sort of taken and synopsized by AI.”
Discussion on the implications of AI-generated web traffic surpassing human traffic.
“I want to talk just for a moment, though, about what you do day to day, because it is such an interesting title.”
Discusses how the shift from human to bot traffic impacts publishers and their monetization strategies.
“So before we get into some of the tools that Cloudflare has built to help and quantify, and they're really interesting.”
Explains Cloudflare's pay-per-crawl model and its implications for AI companies and publishers.
“So one of the things that Cloudflare has been working on, which is really cool, is a pay-per-crawl model released last summer.”
Discussion on Cloudflare's AEO readiness tool and its relevance for marketers.
“And let's talk about what Cloudflare is building on the marketing and brand side.”
How marketers can leverage proprietary data for better visibility and optimization.
“No, I mean, it's funny because, you know, we were really trying to build tools sort of like for whatever your goals are as a publisher, as a brand, anyone who sits under CloudFlare.”
Cloudflare's new feature allowing sites to opt out of their content being used for AI training.
“Again, knowing exactly what's happening in your content in terms of how it's being discovered by the answer engines, it's your own proprietary data and that's what it should be rooted in.”
How publishers can maintain visibility while protecting their content from AI misuse.
“like pretty significant, the disallow AI training setting.”
Examining the evolving relationship between brands, AI agents, and consumers.
“So if AI agents, this is something we talk about so much internally as an edit team, just how shopping is changing.”
Discussing the challenges publishers face regarding Google and their reliance on traffic.
“So we're winding down and we only touched on Google here and there.”
Explore how publishers are coping with reliance on Google and AI crawlers.
“threats to publisher traffic because publishers can't really say no to Google.”

Get the gist, explore an idea or ask about something said.
On AdExchanger, what are the key points of this episode?
Automatic transcript. May contain errors.
0:12Lara Cohen: Welcome to Ad Exchanger Talks, the podcast devoted to examining the issues and trends in advertising and marketing technology that matter most to you. Today's episode is sponsored by Basis, the leading intelligent operating system for autonomous advertising. Its enterprise AI solution transforms campaign briefs into strategies and media plans that integrate directly into omnichannel activation. You can learn more at basis.com.
0:50I'm Alison Schiff, editor-in-chief of Ad Exchanger, and I care a lot about the work that goes into good journalism, and also what it takes to keep that work going. But it gets harder to do when AI tools use original reporting and publisher content to answer questions without sending readers to the original source. What choices do publishers have? Can they protect their work and get value when AI uses it? My guest this week is Lara Cohen, the VP of Strategic Partnerships for Media, Creators, and AI at Cloudflare. And she has a lot to say about that. But first, I'd like to take a moment to tell you about our upcoming Top Women in Media and Ad Tech event.
1:29Join us on the evening of November 3rd in New York City as we celebrate the leaders, innovators, and builders moving advertising and media technology forward. The Awards Gala will take place at the Lighthouse at Chelsea Piers, honoring an extraordinary group of women across AI, commerce media, privacy, programmatic, measurement, and CTV. Look forward to seeing you there. Hey, Laura, welcome to the podcast. Hi. Good to see you, Allison. Good to see you as well. What is one thing about you that not a lot of other people already know? Something that I could not find out just by Googling you because there is one very fun fact about you that I'm going to bring up, but that is Google-able.
2:09Let's see. I'm a deep thrift head. I'm like a big vintage Levi's connoisseur and I've started a side hustle in the last couple of years of like basically because I had bought too much vintage of like doing some reselling of vintage appearing at like vintage pop-ups and stuff which is a very fun creative outlet actually um so it's very cool I was just at housing works yesterday over the weekend and I bought an Armani blazer for$33. Yeah. No, I mean, like, once you get into it, you kind of get addicted and you are less into buying things fast fashion because you realize everything was such better quality made.
3:00Yeah, this one was, I think it was from the 80s. I don't know. It does kind of make me feel like Don Johnson when I wear it. I want to roll up those sleeves, but I like that style. Yeah. Listen, listen, it's all back. Fashion is cyclical. So, you know, Don Johnson vibe will be, or maybe it's big this year. Who knows? Quality maintains itself. Yes. And that's going to be a fun segue that I'll put a pin into because we're going to be talking about what quality means on the internet and how we make sure that we have quality content going forward in the future, which I think is very important. It's Armani's AEO rating.
3:41Exactly. But before we get into that, I want to do a quick career rewind on you. So you spent 11 years at Winter Media and you were eventually the editorial director of US Weekly. So traditional media, tick. Then you spent time at Twitter when it was still Twitter and you orchestrated one of the most. I know. Yeah. And when you were there, you were instrumental in getting that Ellen DeGeneres Oscar selfie happening, which is pretty cool. So social media tick. And then you left, came back to Twitter for a second stint, left again after the whole Musk acquisition. There's probably a story there.
4:26And that's a Drink story, Allison. I don't know if that's podcast friendly, but yeah. I will take you out. Yes. A couple of years as VP at Linktree, a VP at Linktree working with millions of creators. And then a couple of years ago, you come to Cloudflare as VP of strategic partnerships for media creators and AI at an internet infrastructure company, which I think is really interesting. So not on my bingo card, for sure, to to be at a deep security infrastructure tech team. But I, you know, I spent yeah, I started my career as in media as a journalist myself and then made, you know, the switch to Twitter.
5:10And I, you know, the through line of my career is that I've been really interested in helping, you know, creators of all size and scale, publishers, journalists, media, best use technology to, you know, connect with their audiences. And, you know, that's what I did at Twitter, building up the partnerships team there. And then, you know, at Linktree. And then to be perfectly honest, like I wasn't so familiar with Cloudflare until they released their content independence day announcement in 2025. At the time, I had become really sort of fascinated about what was going to happen with publishers as we shifted from, you know, a search engine economy to an answer engine economy, knowing that so much traditional media was reliant on referral traffic for their business model and what was going to happen with that all being sort of taken and synopsized by AI.
6:11And so when Cloudflare made that announcement, I was like, wow, this is really interesting because they have the ability to be sort of carrot and stick here to really kind of create some friction that I thought was necessary to publishers, you know, protecting their content, understanding what was happening to their content and, you know, sort of recreating, reinventing a new business model. And so I sort of made a mental note like, oh, I got to track someone down who knows someone who knows someone at Cloudflare. And then coincidentally, literally the same day, a former boss of mine from Twitter connected me with my boss now, Stephanie Cohen, who's the chief strategy officer, and was like, you know, you should talk to Alara about this space.
6:52And yeah. And so now I am more technical than I ever thought, you know, was possible starting my career at Us Weekly. But, you know, I really think I'm really happy to sort of be at this intersection now because I think it's really important. It was clearly destiny. Yes, clearly. Clearly. I had CDN in my future and I just didn't know it. So I'm sure all of our listeners know what Cloudflare is, but it might be a little abstract to them in the way it was to you before. So a network security company that routes roughly 20 % of all global internet traffic, which is a pretty mind-blowing stat. So in a very literal sense, you're part of the plumbing of the internet.
7:43But what I think is interesting and important is that Cloudflare has decided to take a stand in favor of publishers, which I applaud. Yes, yes. No, I mean, I think that like Cloudflare is the way I describe it to folks who, like me, were not sort of so familiar with what the company does is sort of like the good guy bodyguard of the Internet in a lot of ways. You know, historically, we've protected websites from, you know, DDoS attacks, spam, you know, all these like malicious actors on the internet. And a couple of years ago, we sort of made an observation. There are a lot of publishers who sit under us and that like referral traffic from Google and like other, you know, human traffic was declining on the internet.
8:26And that was having a really massive effect on publishers' business model. And so, you know, I think at Cloudflare, like, we really believe in the future of technology. You know, we want AI to thrive and to achieve all these promises that, you know, it's all the potential that the founders of these companies talk about. But we strongly believe that in order to do that, like, in order for AI to thrive, original human content also has to thrive. Because, you know, it's not really artificial intelligence. It's really collective intelligence. You know, it's trained on all the work that, you know, scientists, journalists, creators do.
9:07And so in order for AI to continuously get better, like all that content needs to get better. And that means that like there has to be a new business model that's invented. Right. You can't summarize what doesn't exist. Exactly. Or you can, but it's not the output is not good, you know. And so like AI just trained on AI is not going to be good for the society is sort of like what we fundamentally believe. The snake will eat its own tail eventually. Yes, yes. I had a flash, by the way, when you said the good bodyguard of the Internet. I'm like, are you the Kevin Costner of the Internet? Oh, no, I don't want to sing.
9:46My heart will fall on. I will make you. This will devolve. This podcast will devolve quickly. Yeah. I want to talk just for a moment, though, about what you do day to day, because it is such an interesting title. Like what kind of strategic partnerships is Cloudflare striking with creators and with media companies and with AI? Yeah. So I think that me and my team sort of work on both sides of the house, I think, and are really attempting to sort of bring the sides closer together. I think part of my day is spent working really closely with publishers. And so I think there's really been a pretty dramatic change over the year plus that I've been there.
10:27I think when I started, there was such a lack of awareness with publishers of like, they knew that their referral traffic was dropping, but they didn't quite know why. They hadn't sort of made the connection between, you know, AI summary generation and like the lack of like human visitors coming back if they could just get an answer generated to them. So I think, you know, a lot of the focus has been on sort of working on tools for them so they understand what's happening to their content and then they can make decisions about whether they want to allow AI traffic, bot traffic to, you know, see their content, access their content, or if they want to protect it in the hope of monetizing it.
11:11You know, and publishers are making different decisions based on like their own goals. I think that a lot of that work too has been, you know, historically like Cloudflare's customer has been really like a deeply technical, you know, like CISO adjacent person. But all this data is super relevant now for different teams than ever before. You know, like when I started at Cloudflare, truthfully, like being on the less lesser technical side, I would see like our bot management dashboard. And I knew that there was like all this relevant information in there for publishers. But it was not presented in a way.
11:48It made me want to black out. You know, I was like, what is this? But, you know, for publishers, it's really helpful to understand, you know, just how much OpenAI is trying to access your content, just how much Anthropik is trying to access your content, you know, what your human to bot referral traffic is, like what your most popular URLs are. And so presenting that content in a way that's palatable to business teams, marketing teams, et cetera, working on tools so they can sort of understand the value of their content, you know, what's working and what's not. So it's been, you know, on that side of the house, that's sort of been the publisher side.
12:24And then on the AI company side, you know, I think it's been about, you know, we drew this line in the sand with Content Independence Day. But, you know, beyond that, like, what can we offer as Cloudflare, you know, sitting across, you know, 25 plus percent of the internet besides the content that will help them be more efficient and smarter in terms of like accessing their content in a way that like serves them. So how can we give them sort of a roadmap of what's under us and sort of point them at the most valuable content that's like high quality, you know, when something is new and fresh and refreshed, et cetera.
13:02So it's more of a push and not a pull. Because it's our belief that if we can sort of give them this roadmap of like where the valuable content lives and make it more efficient, that savings and cost and compute can be generated back to the publisher. We do live in particularly wild times, though. I'm just going to toss this number out there from data that Cloudflare published this year. So AI agents now generate more than 57 % of web traffic, which I'm not good at math. It's like that's more than half. Yeah, more than half. Yep. Yeah. I think it's in the 60s now. Yeah, yeah. So that's crazy because I think the 57 % stat was from just a handful of months ago.
13:45So it's just happening very fast. And 2026 is the first year in history that machines outpaced people on the internet. But the web was built around human traffic. That's the entire premise of digital advertising, of media, of e-commerce. So before we get into some of the tools that Cloudflare has built to help and quantify, and they're really interesting. And some of them are brand new approaches to how publishers can make money. But what happens to all of that, the web built around human traffic, when humans are no longer the majority of traffic? I mean, does it require a full-on rethink from scratch?
14:25Or can you apply, I don't know, Band-Aids? Yeah, I think it can be sort of a crawl, walk, run. I think that like... Crawl. Very funny. Ha ha ha. Didn't even mean that. Look at that. Yeah, because I think, well, first of all, I think that like, you know, the conundrum that a lot of publishers are in, and frankly, in a lot of ways, publishers are just sort of the canary in the coal mine because this is now sort of something that, you know, all brands are tackling and everything else, you know, is that you can't sort of, you're not going to ditch, you know, what has worked for you with like traditional human traffic right away because you still want that human traffic.
15:07Like, you know, your Google referral traffic is still important to you. So you're not going to, you know, scrap everything and just optimize, you know, your content for being machine readable only. I think what we've been trying to enable is like some really lightweight things for, you know, publishers, retailers, et cetera, to be able to do to optimize their content for machines while also, you know, and really making that more light touch. So like, you know, being able to put your content into Markdown, for example. So it's like more machine readable and accessible. It's like just sort of like a really easy way to sort of make your content more palatable to machines without, you know, degrading, degrading, you know, your human experience.
15:57But even so, like the implicit deal on the web that you let bots crawl, they send you human readers. And that's kind of how it how it goes. That bargain doesn't exist anymore. Totally. I mean, like, I think that, like, you know, the premise that, you know, every every publisher and retailer sort of gave their content to Google for free in exchange for, you know, the the human traffic back to look at ads, you know, buy subscriptions, et cetera, that promise was just sort of broken. And I think as a result, you know, it's not like the publishing industry was in particularly strong footing before this moment.
16:41You know, there had already been such like a hit, you know, from social media and from everything else in terms of like, you know, the business model for publishers online. And so I think that it's really been – publishers have had to act really quickly to sort of adopt to a new – figure out a strategy here in terms of like what their – how they were going to show up and what that meant to them in terms of like how they were going to make money in the new internet economy. And it's critical because Cloudflare has done some estimations. The crawl to referral ratio is like just tens of thousands of times worse than it used to be.
17:27So OpenAI's scraping to referral ratio is something like 1 ,700 scrapes for every one referral back. Anthropic, it's like 73 ,000 to one or something crazy. um so yeah for ad supported publishers and media companies that that is not great but one thing you guys have been working on it was illuminated to me like when i saw i saw a tweet uh pretty recently about how the artist formerly known as twitter x now was like referring i forgot what the actual statistic was but like you know exponentially more traffic back to publishers than any of the LLMs, which I know when I was at Twitter, and that was at a time when we were, I would say, more publisher and media friendly.
18:12And we were really making a concerted effort. You know, we were like a media group chat, really, like a journalist group chat all the time. And we still were not driving meaningful referral traffic back to publishers to help them support their businesses. And that was like a problem that we were constantly reckoning with. And so the fact that now, you know, in the post-Elon X world, like the fact that Twitter, like, you know, X now is driving some referral back to, um, so much more referral traffic back to publishers and the LMs like really sort of crystallized for me just how deep the problem is, um, for, for publishers and ad traffic right now.
18:55Oliver, the dog is barking for publishers. Yes, yes. He's fired up. Rightfully so, Oliver. He cares about their plight. Yes, yes. So one of the things that Cloudflare has been working on, which is really cool, is a pay-per-crawl model released last summer. The idea that AI companies should actually pay, crazy, for the content they're scraping. You were personally very involved in that whole initiative. A year in, is it working? Are AI companies actually paying? Yeah, we've shifted the model into what we're calling now pay-per-use. Paper crawl was a catchy name, first of all, but also I think like, you know, sort of like an idea that like now that we've gotten deeper into it has sort of shifted because I think a crawl is just like one metric to sort of like decide what the point of action is.
19:47And I think what we found is that like letting the demand side, the buy side sort of determine like what the use is, they're more inclined to come to the table to pay for the content. So we do have buyers in the marketplace now. We have, you know, a range of buyers who are in from, you know, there are a bunch of the AI search companies that are all now sort of competing to be like the Google of the agentic era, which is great for publishers to capitalize on because they're competing to build an index really fast and to compete with Google. And they know to do that, they want the highest quality information.
20:28And so they've been a really good buyer. There are some enterprise companies that are interested. If you are a large investment bank and you're building your own model, it's really important for you to be trained on content that you know is truthful and accurate and can be attributed back. And so that's an interesting model. And there are a couple of other use cases of, we have a couple of announcements coming up of really interesting folks who are entering the marketplace on the buy side. Who sets the price though? That's something I've been wondering. How does a publisher figure out how much their content is worth to a crawler even for a use?
21:07I mean, I just know from the time I spend working on my own stories. I think it's really valuable, but it's my own thing. And so I think publishers have probably the best sense of what it's worth, but also maybe an inflated sense sometimes, if I'm being honest. Yeah, totally. I mean, I think that like, you know, a lot of this is just like traditional marketplace dynamics of like how a market gets set, you know, and that's usually set by the person who's paying for it. And then, you know, the way our mechanism with pay-per-use works right now is that like, you know, anyone under Cloudflare will be able to get a ping basically saying that like if your content is protected, like, hey, like so-and-so wants to pay for your content at this, you know, point of use at this rate.
22:01do you want to accept, decline, or negotiate? And so I think that we're really trying to get out of the dynamics of letting people determine their price. I think that in terms of understand, I think what I'm proud of from the Cloudflare side on the analytics side, getting back to sort of like the publisher tools that we've created, I think having a dashboard that shows you exactly how much OpenAI is trying to access your content, exactly how much their competitors are trying to access your content. You know, certainly for the larger folks who have been doing direct deals with the LLMs themselves, like that data has been really helpful in sort of, you know, taking away some of the data is asymmetry at the negotiating table, because I think that, you know, for a long time, like, publishers were just like flying blind in this space with like how much they should charge.
22:56You know, on the other hand, I do think that, you know, sometimes publishers may have an unrealistic expectation for what their content is worth, that they're looking to, you know, sort of make a one-to-one replacement of, you know, what the loss of their ad revenue. And as I like to say, like, I'm sorry, I can't, as much as I want to, I can't turn back the hands of time and like make it 1995, you know, it's a different time. So I think what we're looking for in the marketplace is really like, you know, good actors on both sides who are truly trying to experiment and get this right here. You know, we have really high parameters for who we let into the marketplace from the demand side.
23:41You know, they have to declare their crawler. They have to like be showing up honestly, you know, not trying to like hide their crawler. So I think that like we're – I think we're at the stage now where there's like experimentation happening and I think we will get to a metric that feels good for everybody. Knowledge is definitely power in a negotiation. Totally. So we're going to take a quick break and when we're back, we're going to talk about lots of other tools that you guys are putting out there to help publishers. So stick with us.
24:25I'm Sarah Sluice, Editorial Director of AdExchanger. And with me today is Michael Olson, EVP of Client Development at Basis, which is supporting our podcast this month. We're going to talk about something that doesn't get enough attention, paying invoices. What is the cost of slow financial reconciliation and who is the most impacted? Hi, Sarah. Thanks for having me. The cost is time, cash flow, and trust. When financial reconciliation takes too long, payments get held up, and finance teams, whether they're in agencies or ad brands, are just chasing down numbers. People are spending a ridiculous amount of time figuring out why two systems do not match.
25:10Agencies probably feel it the most because they're stuck in the middle carrying the float. They have the client asking questions on one side and media partners on the other. The longer it takes, the more time everyone spends fixing the past instead of focusing on performance and health and growth of the business. So you outlined a very frustrating problem. I would not want to be that person chasing down invoices. Is this primarily a technology problem, a workflow problem, or a people problem? It's all of the above, but I'd lean towards mostly a workflow problem. the industry definitely doesn't have a shortage of technology or data.
25:51We have a shortage of systems that actually work together. And when they don't, people end up filling in all the gaps manually. They end up downloading spreadsheets, comparing reports, sending emails, chasing down discrepancies. It's a mess. Technology is obviously part of that answer, but another piece of software by itself just simply doesn't fix it. It's imperative for technology to actualize omni-channel data monthly and push to reconciliation systems. As an industry, we have to connect the tech stacks interoperably, end-to-end, and then allow the technology to do the heavy lifting to align the data between the contracts, media plans, media transactions, and the payments.
26:37Okay, so it sounds like technology and workflow can make this just a bit easier. What does automated reconciliation look like? A lot of automation talk right now, and if I'm thinking about the future agentic state, ideally, reconciliation just happens. AI is constantly looking at delivery, spend, invoices, payments, and then flagging issues as they happen instead of somebody finding them three weeks later. And in a lot of cases, it should be the AI should be able to fix those issues without a person ever touching them with the correct harnesses in place. People should only have to get involved when something unusual happens, which is the human oversight that gets away from this huge month end exercise that costs a lot of time and resources and much closer to knowing where you stand financially at any given time.
27:31This is the ideal state Basis is going towards for our connection with Prisma by MediaOcean, a platform used by numerous agencies for financial reconciliation. Okay, so it sounds like by connecting those two different platforms, we can and adding agentic AI, we can make it a lot easier and people won't have to chase down invoices all the time like they used to. So thank you so much for sharing with us what that would look like. And thanks to Basis for supporting our podcast. Thanks for having us, Sarah.
28:08All right, we're back. And let's talk about what Cloudflare is building on the marketing and brand side. So in August, you launched an AEO readiness tool. AEO, as everyone knows, I think, answer engine optimization, which is all the hype right now. AEO, GEO, all the EOs. All the EOs. Yes, yes, yes. Having a moment. And it also means that there's a lot of snake oil flowing. We get pitched constantly with people who say they know exactly how to show up. And I just don't think that's true because no one knows the answer. No one knows the answer. Yeah. But vendors are making some very confident promises about being able to get a brand recommended by AI without a whole lot of data behind it.
28:56I mean, SEO has been around for decades and marketers are still arguing about how to do it. And now there's AEO and there are no rules like left. So what's a marketer, what's a marketer to do? And how does this tool work? Yeah, yeah. No, I mean, it's funny because, you know, we were really trying to build tools sort of like for whatever your goals are as a publisher, as a brand, anyone who sits under CloudFlare. And so for some people that may mean, you know, understanding what's going on and then protecting your content so you can monetize. But some people, especially like, you know, if you look at retailers or, you know, anyone, lots of other different, even some publishers are looking to like optimize their content.
29:38Like they want to make sure that they're, you know, showing up prominently in ChatGPT and that their brand is showing up, you know, in the parameters that they want their brand to show up. They want to have some control over how they're perceived on the answer engines. And so we realized that the same data that we were feeding back for the folks that we're looking to protect it, also really valuable for people looking to optimize it. And yeah, I think that there's, as in every sort of moment and shift, there are companies that sprout up that make a lot of promises that are sort of leveraging people's insecurity in the moment.
30:14And so we really wanted to root our tools in your own proprietary data. And so again, Again, knowing exactly what's happening in your content in terms of how it's being discovered by the answer engines, it's your own proprietary data and that's what it should be rooted in. In fact, the AEO tools that are in the market right now, a lot of them, the good ones, honestly, will ask to import your CDN traffic so they can help these determinations. We feel like as your CDN provider, we should be able to give you that data too. So everything from our AEO tool is, you know, rooted in data and in your own data.
30:59And then, you know, on top of that, we layer on things like, you know, queries to understand, you know, what your citation rate is, how you're showing up, what your share of voice is compared to other things like you on the market. And so layering on to that, you know, that data on top of what you know can help marketers sort of make decisions like where to double down, where they need help in terms of like, you know, changing, you know, it's a really good experimentation tool to see, you know, what's working and what's not. That information could feed, like you're saying, very nicely into your marketing strategy because you're tracking prominence.
31:40So how early in an answer you show up and how much of it you own, share a voice against competitors so you can see who's winning the prompts your brand is losing. Exactly. Exactly. Help you make actually informed decisions that are based on data. If a brand checks their share of voice and they find out that they're losing the prompts that matter most to them, what could they actually do about it? It's one thing to know and it's one thing to be able to take some steps. Totally. I mean, our goal there is to then sort of give you some tools to like actually increase your visibility with AEO. So again, like as I said, like we have a tool that makes it really easy to like move your content into markdown.
32:19So it's like more machine readable, you know, you can sort of test different prompts to see, you know, how am I showing up versus other competitors? And you can sort of know like what's working for other people and like what you should, what you should sort of invest in strategy wise yourself, knowing like which specific URLs may be generating the most content, well, generating the most attention from the LLMs will then help you sort of optimize and repeat that behavior if that's what your goals are. Going back to publishers negotiating and also protecting themselves is another Cloudflare announcement I want to talk about for a minute, which feels like it could be like pretty significant, the disallow AI training setting.
33:06So basically any website can opt out of having their content used for AI training, but they can still remain fully indexable for search. And until now, it's pretty much been a forced all or nothing choice. How does it work exactly? And how does Cloudflare enforce the distinction between you can crawl me for search and you can't use this or you can't use that to train your model? Yeah. So again, going back to that first original content independence day announcement, you know, when we sort of drew this line in the sand, it was like AI, you know, AI offer on and what we've discovered, you know, in the year since then.
33:46And this is like such a constantly evolving space. So we have to move in the time, you know, in at the pace that everyone else is moving. But, you know, ideally we would want, you know, for publishers and again, for everyone, retailers, et cetera, you know, you may want your content to be discoverable in an agentic era. You know, as you mentioned before, more than half of internet traffic is non-human. You know, you want to show up in this new world, but that doesn't mean that you want to like give the milk away for free without getting people to buy the cow, you know? And so most of the large language models have different crawlers on the internet that declare their, like declare their intentions.
34:31You know, and so like OpenAI is actually like a really good actor in the space where they have like, you know, different bots with like different intentions of like what they're trying to do, like a training, a training bot, you know, a bot for search, et cetera. And so we've given more granular controls back to the publishers to be able to, you know, identify what they want from different crawlers and like, you know, make the distinction between search versus training. And it's also not an honor system, right? Because when I heard about it, I thought of robots.txt, like that comparison came to mind, which is the old standard.
35:11I mean, it still exists for telling crawlers what's off limits, but that is largely an honor system. Yeah. And AI companies have a mixed record. Yeah, pool rules. It's like when you go to the pool, they say like, don't bring food, don't run, whatever. Don't pee in my pool. Yeah, exactly. So I think doing it at the network level, this is going back to Cloudflare sort of being the good guy, bodyguarding the internet. It's like you actually have enforcement backing up. It's not a suggestion. Like there's someone there to actually protect your content and enforce the rules that you're dictating. I want to change directions slightly again.
35:45So if AI agents, this is something we talk about so much internally as an edit team, just how shopping is changing. So if AI agents are increasingly the first point of contact between a consumer and a brand, it's how they're doing research, they're chatting with a bot to figure out what to buy. if the agent is doing the discovery and the comparison shopping and forming the shortlist, and hopefully you're on that shortlist as a brand, does that mean brands are going to be marketing to machines in the future or that they have to think about that as an important first step before they can even get to the person?
36:26Because I'm thinking almost of an agent, weirdly, as the bodyguard of the person that gives the agent instructions about what they want to buy. Yeah, I mean, it's interesting. I think retailers are really in an interesting moment right now because I think, you know, Amazon, for example, has been super clear that they don't want to disintermediate their relationship directly with the human. Like they are being extremely protective of their content to the outside world. You know, and even now as we've shifted to, you know, personal agents, you know, with Muse and with Dot, et cetera, like they're saying like, no, like we don't want to disintermediate our relationship with the buyer themselves.
37:07I think it's really interesting to see. I think we don't know quite yet like where it's going to end up. I think what we can do in the meantime is, again, just give people tools to support the decisions that they want. And so like, you know, all the agentic readiness tools that we have to sort of like make your content more palatable, easy to read for machines, et cetera, you know, using the AUO tools, like you can at least know how you're showing up in those things and then make decisions based on it as a retailer. Or if you want to sort of decide that you really, that owning that direct relationship with your customer is important to you, you can sort of toggle it off and sort of like give them still a reason to go back to your site exclusively.
37:58Have you personally found that AI is changing the way you discover stuff and make buying decisions? You know, I was at like a, I was at, you know, living, living in San Francisco and Silicon Valley. I feel like sometimes there's a little bit of Valley brain that happens. I was at like a, an agentic commerce dinner, a couple of months. Yeah. And, you know, there was like this animated conversation about like, would you let an agent do like, you know, a large scale financial make, you know, investment for you without you touch, like whatever. And there was this, I think that he was the CEO and founder of Chime was there.
38:41And he was just like, I just want to remind people that like, you know, not all of America, even online shops, you know, like, I think, you know, So there's like, you know, I don't know if I'm going to – I personally like picking out my own shoes. Like that's just me. Like I enjoy that, you know. Like I enjoy the hunt of things. You know, maybe I would like an agent to tell me like if I have identified a pair of shoes, let me know when they go on sale somewhere on the internet. Like sure. But like actually letting an agent perform all these personalized tasks for you, I think, is really up to the individual.
39:22And I don't think it's necessarily something that everyone is going to do. You know, I could be wrong, but like that that's sort of like my position about it. I think that like some people, some people really love like booking their own reservations, you know, going through Resi and seeing like what's available for, you know, for dinner instead of having someone, you know, an agent do that for you. So I think it's always going to be somewhat personalized is my guess. And, you know, personally, like I think that like I would love to see the models really focused on the promises that they've made in terms of like actually delivering like the human value back.
40:07I think that like part of the perception problem the models are having in terms of like, you know, consumer opinion about large language models is that they seem maybe overly focused on problems that people don't actually have instead of the problems they actually do. You know, I'm like, instead of like buying shoes, like, you know, let's, let's work on all the science advancements that, that have been promised, you know, because that, I think is like, that's going to be the thing that's going to change, you know, user perception about AI, you know, it's not going to be like having an agent buy shoes for me.
40:50That's my personal opinion. Yeah. Yeah. I agree with that. There is a lot of promise, maybe reading radiology, you know, stuff better, like just, I don't know, just scientific insights. But like things that really make my life better. Like, you know, if you can make it so like, I don't have to spend an hour on the phone with like customer support for something and an agent can do that, sure. Like that is something that I feel like, you know, will qualitatively and quantitatively make my life better. I think that like focusing on those real life problems rather than, you know, perception, like rather than perceived inconveniences when some people like the things is like strategically would be my direction if I had a magic wand for these folks.
41:43So we're winding down and we only touched on Google here and there. So I'm going to talk about the Google problem more specifically because it is simultaneously the most important source of traffic for publishers, which we've already noted and everyone knows, but also one of the biggest threats to publisher traffic because publishers can't really say no to Google. Like if you block Google's AI crawlers, like you do lose search. Like I know there's a lot of work being done to try and make it more nuanced, but like search is still just the lifeblood for a lot of these businesses. So in this interesting time, it's an interesting time to be alive.
42:25How do publishers navigate that just very awkward dynamic? Yeah, it's funny. Like when we first did, you know, the content independence day announcement and we sort of drew this line in the sand, And I would say that like, you know, the biggest beef all the other LLMs had with us was less about us sort of like putting a wall around people who wanted it. But they were like, hey, your wall has a giant Google sized hole in it because publishers and everyone else can't sort of like say, you know, say no to Google because, you know, they still are reliant on that traffic. And so like basically you were giving them the choice of like disappearing from the internet as we know it, we're giving over everything for Google to train with for Gemini.
43:09And Google historically has not separated their crawler, unlike, you know, OpenAI. So they don't, they have one crawler that is used for both search and training. So what we said this past Content Independence Day was that, like, you know, we were taking a stand on these mixed use crawlers, not just Google, it's Bing and others. And subsequently, you know, we've worked with them to, you know, do a lot of work that is helpful to publishers in terms of like, you know, make it really easy with like one button where you can like basically tell Google your preferences and they can, and then we'll also like provide you with the receipts that will show that Google is, you know, only allowing your content for traditional search and not for training.
44:00Because I think that, yeah, they've taken advantage of, frankly, in some ways, you know, being sort of the incumbent in the space. They're so much better at crawling, you know, even than the rest of the large language models, because they've been crawling the internet, you know, for 30 years. So they definitely had sort of a competitive advantage here. And they had, you know, this leverage over publishers who couldn't, you know, who didn't, who are still reliant, you know, couldn't just block Google altogether because they need that search traffic as their business. The penultimate question, and it's also about Google, not that you're, you can be in their head, but Google has to know that if it hollows out the open web, things like AI overviews like won't work as well.
44:51Like it's also hollowing out one of its most important products by moving ahead with Gemini and AIOs. But of course, that's the direction everything is going. But like we were talking about before, the Ouroboros, you know, if there's no open web left to crawl, like what does Google's AI have left to work with? And what do others have left to work with? Yeah, it's interesting. It's like Google's in an interesting spot because it's like their core business is SEO and ad traffic, you know, and monetizing that. And so like, yes, they were competitively advantaged because they, you know, have been in the space for a long time.
45:33But to win would sort of mean, you know, getting rid, you know, shifting away from their core business too. But don't worry about Google. I feel like they will figure out a way to monetize in this new world as well. I'm confident of that. And yeah, I mean, And I think that like I'm very confident they'll figure it out. I think the interesting thing is that like there's historically been, you know, perception wise, that's my dog. But historically there's been, you know, an allergy around ads in some ways. Like no one likes to think about ads as, you know, sexy in the new world order. But like ads, the ad model like is, you know, has been around for a long time.
46:36And I'm sort of hard pressed to think that it's going to go away forever because, you know, the good side of ads is that it's like democratized content, you know, for everyone. And I don't want a world where everything, all great content on the internet is paywalled because I think it should, you know, I want people to have access to really great content. And so I think what I would look for from Google is I would bet that they are going to capitalize on, you know, whatever agentic ads look like in the future and probably probably figure that out pretty quickly. And yet to be clear, I'm not sending the collection plate around.
47:17Oh, yeah. No, no, no. Don't do that. So I won't hold you to this, but I am curious what you think. And I don't know if five years is so far in the future that it's like just maybe I'll make it two years. I mean, honestly, like it feels like in this space, it's like two weeks. It's like things. Yeah. Anyway, five years. God help us. But OK, yes. I'll change it to two years. So fast forward to two years from now. Does the open web survive in a form that would still be recognizable to us today? And is there what to be optimistic about? And before you answer, I'll just bring up like Napster. Yeah, like Napster kind of destroyed the music industry's old model, but a new one eventually emerged that paid creators and it's not perfect, but it exists.
48:05Yeah, I think I actually really do think that we're getting there. And I'll leave you with like this tidbit of good news, I think, actually, that I that actually like makes me happy is like I'll give you the example of like local news. Local news is like something that I'm really passionate about. And I, you know, that is an industry that is like very much struggled, obviously, you know, you know, since basically like want ads were taken out of newspapers. first. But what's really great about local news and why I'm bullish on local news in the future is that like if you are asking, you know, chat GBT, like, hey, what's happening in my town today?
48:42Or like, what is the local ordinance that I need if I'm getting a permit for, you know, I want to do this renovation in my house? There's probably only one or two sources of information where they can pull from for that. You know, it's not like there's a million different, you know, headline chasing sort of like outlets online. There's probably like one or two local news sources where that comes from. And that inherent scarcity that happens with local news makes it really valuable in, you know, this new world order. Because if there's only one or two places where that information can come from, like that means that like, you know, the LLMs are reliant on those one or two sources and like will need to pay for that content.
49:26And so like, I'm really bullish that in this new business model, perhaps we can sort of reorganize around content that's truly valuable to the web rather than sort of like SEO, which had devolved a lot into follow-on, headline chasing, things that you're doing for clicks. What if truly the most valuable content was what was being optimized for. And that gives me some hope for the open rep in the future for sure. I would love that to be true. And I'm trying. We're out here trying, Allison.
50:16This episode is sponsored by Basis, the leading intelligent operating system for autonomous advertising. Its enterprise AI solution transforms campaign briefs into strategies and media plans that integrate directly into omni-channel activation. Learn more at basis.com.
What if AI companies had to pay publishers every time they used their content? Cloudflare's Lara Cohen explains the new playbook for dealing with AI bots, from blocking them to billing them.