231: A Paywall Stopped ChatGPT From Reading The Article. It Confidently Summarised It For Me Anyway.

0:00A reporter at Variety magazine wrote a story a few months ago and then Variety published it. Kevin Roos and Casey Newton, the two people who host the New York Times tech podcast Hard Fork, had signed with a Hollywood talent agency on their way out of the Times. Them going independent was already public. Roos had posted about it on his ex-account about a week before Variety published the story. The agency deal was the new part. variety puts its name on it and then publishes a story and then a machine goes to read it and the website doesn't hide the story doesn't show the story doesn't ask the machine to log in it just sends the bill there's not a please subscribe to read we the humans get a bill with a number on it that just had been sitting in the web standards since 1992 waiting

1:02This is NoHacks with me, Slobodan Amanic. Nobody asked any of us if we wanted the internet changed, so I checked what AI is actually doing to it. The good, the bad, and the pointless. I only found out any of this happened because a machine told me. I had an assistant reading some things for me, doing some research, and halfway through, unprompted, it mentioned that one website would not let it in. I never sent it to Variety website. It just followed the link it found. So I went and looked. This is what I assumed was going on. I assumed a website that charges robots either gets paid or just keeps them out.

1:42Those felt like the only two outcomes available. You put up a wall and either money comes in or nobody goes through. Not what happened. what I found was stranger than a wall that doesn't work, that's broken. Let me show you what really happened. But first of all, I want to thank you for listening. And when I'm done covering the story of this episode, just stick around. I want to share some thoughts about the podcast and just where the internet is headed. So when I'm done with the variety story, with the pay me to read the story, stick around. So the website I said was Variety. It's the magazine that writes about Hollywood and that claims to have invented the word sitcom, I think decades ago.

2:29The message the assistant got that I sent was word for word, I'm reading this from my screen. You're not authorized to access this content without a valid tollbit token. There was no price, there was nothing. The number that the website gave the assistant was 402. now every time anything asks a web server for a page the server answers with a three-digit code that's how it starts it can be either okay you'll get this or it's not here look elsewhere that's 301 302 redirects the 4 xx so the code that started with four means that they mean that it's your fault that you're not getting this so maybe it's not found you're at the wrong address You've seen those, I'm sure.

3:18Five means the server messed up. The number was 402. That means payment required. That's a status quo that has been a part of web standards since 1992, and almost nothing has ever used it. Nobody was using it to charge an agent, an AI assistant, or anyone for reading a web page. Now somebody is. Tollbit is a company that sells websites payable for robots. And if you look at their website, it's either on 6 ,000 websites or 8 ,000 or 9 ,500, depending on which page you read. And some of the websites listed are Forbes, Newsweek, USA Today, The Associated Press. The first thing I wanted to know was what this wall is actually looking at.

4:10so hear me when anything asks a server for a page it sends a short message that says this path this url give me whatever you have for that another line of that message that you get sends a user agent it sends an identity of whoever's asking so chrome will say i'm chrome google scroller will just say i'm google bot that line is called the user agent and it's a text string you're writing yourself. You can say whatever you want. It's a fake ID. It's McLovin from Superbad. That's what it is. That's what user agent, that's what this thing is. The reason it works or worked until we got these new crawlers on the internet is exactly the reason McLovin got away with buying booze with fake ID.

5:01Nobody bought a check. Nobody really looked into it. Not the cops, It's not the lady selling him the booth. I've seen Superbad way too many times. And I'll see it again. So I tried. I wrote a few things in it. I tried the same page over and over, the page from Variety, and I was just changing that one simple line. Saying I'm a browser, I got the page. Gaining it no name at all for user agent, I got the page. Saying I'm Googlebot, I got the whole page. Saying I'm OpenAI's bot, we're in Tropics, nope straight to the payment page or toll pit now the McLovin part is funny this is where I tried to make my own McLovin my own fake ID I gave the robot a clearly fake name that belongs to nobody on earth and asked if I can have the page I got the page so that's the first thing that I wanted you to know from this story and the wall that they put up It's not checking whether you're a robot or not.

6:07It's checking whether the name you gave it is on a list. That's weird. If you say who you are and you're honest about it, you get charged. Make something up and you walk straight into the website. You get the pages. And let me be clear. It's not just variety. I tried the same name at five more magazines, four owned by the same parrot company and one random one, and it got pretty much the same results every time. So this is a product. This is not variety's preference. And that made me want to know what these websites say out loud about who they let in. There's a file that almost every website has.

6:50It's called robots.dxt. It's been a standard since 1994. You go to the root of any domain slash robots.dxt. There's a plain text file at the same address on every website where the website writes down which robots it doesn't want to see. It can say, you bot are not allowed in these pages, you're not allowed in these pages. Great concept. But it's a note, it's not a lock, and nothing makes a robot obey it. Variety's file names 20 plus robots. So it checked it line by line against the robots, its server actually charged or tried to charge. And if you don't match, see, Variety names all three bots open AI publishes, and then one of Entropics 3 and one of Perplexities 2, and then the three it missed got charged anyway.

7:47By Variety's own file, the three bots that were blocked were supposed to be allowed. But the server tried to build them regardless. Because most people who run a website treat robots.txt as their policy. It's not. It's the server that says the policy. And Variety is two things. What robots.txt says and what the server says, they contradict each other. And then there's the visitor. The big G elephant in the room that nobody charges. The box at the top of Google's results. I'm not talking about old school Google results. I'm talking about the new AI overview. It's built from what Googlebot patches.

8:36And the crazy thing is, while not blocking Google and Googlebot, Variety's owner is suing Google. So Penske Media, the parent company, filed last September. And the claim is that by early 2025, the box was appearing on about a fifth of the searches that returned their websites. Their organic affiliate revenue was down by more than a third. And the only way out would be leaving Google search altogether, which would be, in their own words, devastating. That's the bad part. That's not my opinion. The judge in this trial of Penscape versus Google, it's the same person that ruled Google an illegal monopolist in search a few years ago.

9:18This is what he wrote. Publishers cannot opt out of Google's use of their content for display in AI overfuse. And the only way to try was to stop being crawled at all. And the judge called that, quote, not a tenable choice. And you can see exactly this in Variety's RobotsDxt. There's a line in it telling Google Extended to stay out. Google Extended is Google's AI opt-out. it has never covered the answers at the top of the search results and google bot is not on that list so variety is doing things that look like opting out of google's ai but that's not really what's really happening and that leaves me with a question is anybody in the world paying for this tolbit the company they have a page aimed at ai companies the ones that are you know supposed to be doing the paying.

10:19And the whole page is just a list of publishers. It doesn't name a single AI company. In two years, from what I could find, the only paying customer they ever got and they ever named is a newsreader app. And there's a company that, if you're in tech, you've probably heard of, Stack Overflow. They describe what happened when they switched the paywall on. Stack Overflow is the programmers question and answer website. This happened earlier this year. I don't think Stack Overflow is doing really well. That's what AI is doing to the internet, among other things. And they've been adding bad crawlers to block lists by hand until they got crazy.

11:01So they just started answering with a 402 instead. Their website reliability engineer said that some of the traffic from these bots that used to just get a block of 403, not allowed, they just stopped sending traffic our way. So they left rather than pay. Now, an executive from Tolbit, also earlier this year, said that nearly a fifth of its publishers have made money from the paywall. That could mean anything. That could mean$10. That could mean a thousand euros. That could mean a billion whatevers. that's the vendor talking about his product. There are no names. There are no totals. This is not really reliable information.

11:53Just a little bit after that, the co-founder from the company, from Tolbit, in an interview said, it's not going to make their year this year in terms of revenue that's flowing through the pipe. So should I say negligible here? I always wanted to use that word in a sentence. English is not my first language. Let's go back to my story with two assistants, with my AI assistants. I tried two of them. I asked two assistants, chatbots, LLMs, to read the Variety article and tell me what it's about. This was not the research that caught this. This was me testing how this actually works. Both of them got told that they need to pay.

12:40Claude told me that it's blocked and before I asked any details for any details, this is what it put on the page and the response. The page blocked direct access, so let me search for coverage of the story instead. Two things there. It said it had been refused and then it said it was going to look around instead. You know, like, when you want to watch a movie, but you don't want to pay for a movie, you look around and you find it. And the copy's not great. It's not identical, but, you know, you didn't pay. I knew the answer it was going to build. It was of other people's coverage rather than news.

13:30And then I asked ChatGPT. And Sam Altman, God bless his soul, has built a crazy product. So I gave ChatGPT the link and asked, what does this article say? And I know that's not how people use ChatGPT, but I wanted to see if it can fetch it. And it came back with part of the response was, this is a clear, no fluff summary of that variety article. It was confident, didn't say anything about not being able to reach the page or read the page. Nothing, nothing about a paywall. And then I look at the source tags in that answer, and one said Google, one said Metacast, not a single one said Variety.

14:24So I asked a follow-up question. Were you able to read the article I gave you? The answer was this. Short answer, yes, but not in the way you might think. And then it explained what happened. And then it said something like, I wasn't able to directly open and read that exact page like a human browsing a paywall article. Instead, I used web search to retrieve reliable excerpts and summaries of that specific variety story, then reconstructed the key points from those sources. I want it to be fair in this investigation. This is an honest representation of what happened. He didn't pretend. He told me when I asked.

15:07But what actually happened there? Variety paid someone to write that, to find out details of that deal, and then write the story. Then he spent money to stop the machines from reading the story. And then a few seconds later, after I asked, I have two machines telling me the story anyway. And then they created a search engine and a podcast directory. The paywall, the toll bit, the whole thing, it's a good concept, but it doesn't stop the story from spreading. It couldn't. Because a deal like this, that's a fact about the world. Once it's out, it's out. And then three other outlets ran the same news that day about Kevin Roos and his co-host signing that new deal.

16:01I'm blanking on the name. Kevin Roos and Casey Newton, of course. So both Chad GPT and Claude got their response elsewhere and pretended that they knew what they were talking about. what the paywall decided was who is going to get named when a machine repeats the story. And look, this is, if you run a website, you should think about this because you're not choosing between being read and being paid. You're choosing between being read and being cited. So, you know, blocking, blocking these machines, it could lead to a lot of downstream problems. Number one, somebody else is going to give a version of the story, and what if it doesn't agree with what you want to say?

16:50Now, there's a good part. There's a good part, and I think this is real. The money is starting to move from charging per visit towards paying for access, just paying for use. Microsoft has a pilot of a marketplace that pays publishers when AI uses their work. not just once they read it, but every time they use it. It's built with Associated Press, Conte Nast, and their CEO had some very harsh words about what AI search is doing to their business. It's a pilot, but it's a better direction than just paying for a page at the time you read it because you read it and then you're done. Why pay for it again?

17:29You pay five cents to read everything on the internet you care about and that's it. Pay every time you use the information. This is what AI should do and AI companies should do. So yeah, that reported at Variety. They wrote a story, Variety published a story, and I have these two machines just in seconds repeat that story or a version of that story. And neither one of those had ever read the original version.

18:06I'm not sure I care about whether the bots pay but I want them to disclose what happened I couldn't get Claude and ChatGPT to get in and read the page and I couldn't when I was trying to fetch from my computer the right thing to do in that response is nope this doesn't work, let's try this and the chat bots don't always do that one of them did when I tried the other one well had to be pushed so ask your chatbots where they're getting the facts that's what i'm saying ask your chatbots and then follow the trail if you were responsible enough to check the links on google when you search for something and do your research that's not over it's worse just ask that's the episode for this week if you want this in writing with every link and every of the checks I ran.

19:03I also have a newsletter. I send it every Wednesday. Not today because I was out all day. Go to nohacks.co slash subscribe. This is No Hacks. Thank you for listening. Now let's talk about the future of the internet and the podcast. I try not to be too serious, but this is just me riffing at 10 p.m. on a Wednesday.

19:33I took a month off this summer. I talked about this in a podcast. I was away from all the madness in the digital world, well, the world in general, and it felt good. Coming back, I was fresh. My sleep scores were 95 plus every single day. I wasn't looking at screens. I'm guessing there's a causation. And yeah, when I came back, it wasn't, a lot of the things I was doing before my break, it wasn't easy to give back to them and to be passionate about doing them again because a lot of the stuff online is pointless.

20:18And I spent the first six months of the year just trying to cover every relevant story about the agentic web and all that and some of you who are listening have pushed on that because they don't agree that agents will change the internet I think they will but not in a way that the AI bros would like us to think but anyway the internet is what gave me everything like I grew up offline but I got online very early in my life everything not everything most things i have in life are because i had an internet connection i mean the podcast and my my first online job and all that like my web dev career none of that would happen i would not move country two times if it wasn't for internet i would not be living where i live now i don't i don't like what what's happening to it i don't like what AI is doing.

21:20AI is like, in these LLM companies, yeah, the best analogy I have is they're like the aliens from Independence Day. Big tech, AI, private equity, whatever you want. That's what they are. I think Independence Day, the original one where Will Smith practices for the Chris Rock slap 30 years before. I think that that's a movie about private equity, about big tech, about AI companies. I don't like what they're doing is what I'm saying. However, I do think that technology is genuinely good for some things. Absolutely not for everything. So what I want to talk about on the podcast in the future, and that I really haven't, is what the fuck are you doing to the internet?

22:06Why is this change now necessary? We had the internet for 30 years, 30 plus years. It worked fine. And then these companies show up. And now everything has to be different. there was never a more non-consensual tech and this is not my phrase I heard this on a podcast I just don't like what's happening I think if you think it's not good you have to speak up and I don't think I have been speaking up enough in the past so yeah this podcast is going to be more about I just try to focus on the positive things of this while ignoring not ignoring but refusing to talk about the negatives, I think it's time.

22:50I think I need to talk about that. If you stuck around till the end of the episode, this is not even the end, this is after the end of the episode, I appreciate you. I really, really, really do. I want to hear your feedback. Anything that's bothering you, anything about the internet, anything at all, hi at nohacks.co. Oh. That's it for this one. Talk to you next week.