Analysing Robots.txt at scale with HTTP Archive and BigQuery

0:10Martin Splitt:Hello, it is I, Gary, from Google Search, and you're hearing me today because we have yet another episode of Search Off the Record, you know, the podcast from the Google Search team discussing all things digital and shedding some light on how Google search works sometimes or how the internet works sometimes lots of things um let's see am I alone to I'm not alone

0:42Gary Illyes:I'm Martin hello hello did you almost forget about me am I that forgettable to you yes ouch

0:51Martin Splitt:Wow. Well, don't ask questions if you're not prepared to be heard by the answers. Okay, fine. Hello.

0:58Gary Illyes:Hello.

1:00Martin Splitt:It's been a while I've seen you. Like, less than 24 hours.

1:05Gary Illyes:Oh, true. Yeah, we've seen each other yesterday. That's true. Yeah.

1:08Martin Splitt:Do you have anything exciting coming up?

1:11Gary Illyes:Search Central Lives are coming up. We are visiting the world once again. Yes. Where are you going first? My first one is going to be Brazil.

1:22Martin Splitt:Oh, please eat some acai for me. Oh, I will. Oh, God, yes. Thank you. Oh, yes. It's so good. I very much appreciate that. It's so good.

1:31Gary Illyes:But I guess you don't want to talk about acai today.

1:34Martin Splitt:Oh, I could talk about acai. I read so much about acai the past few years. The first time I went to Brazil for a conference, it was an external conference, someone introduced me to acai. I never had that before. And it was basically just from the hotel, like walking one block. I think it was with Pedro Diaz, former Googler now, I think he's developer something, where he has a company where they are developing stuff and also some SEO. Anyway, and he just took me to this very small corner shop. And he was like, you have to try Asahi.

2:11Gary Illyes:And I'm like, oh, no, leave me alone.

2:14Martin Splitt:I mean, no, no, no, you have to try it. And then I tried it. And then I kept eating acai. And then I kept eating acai. And then I kept eating acai. On one day, I had like five bowls of acai because it was so good. And I kept ordering via room service. And at one point, the room service person was like, yeah, we think you should stop. And I'm like, no. And the person was like, yeah, it's already like 7 p.m. or something. Like, you should really stop because you're not going to sleep. then I'm like what do you mean you are not going to sleep like it's full of not caffeine but like some other one of those components that just make you not sleep great I haven't slept for two days but you're right I'm not here to talk about acai today I had a saga and you helped me a little bit with that saga and I thought that we talk about that saga oh you know what I'm talking about I

3:15Gary Illyes:I think the SQL stuff we did for WebAmonog?

3:19Martin Splitt:Well, yeah, but the project was bigger. So to give some background, we received a pull request on the official robots.txt repository to add two new rules slash directives to the unsupported tags list. Basically, Search Console would report that it recognizes these tags, but Google doesn't support them. And the pull request was great. Like it was a very good idea. The person who sent it, well, the username is 3x10 raised to 8. I don't know who that is, but the pull request was great. And it was a good idea. But I don't know about other companies, but at Google, we try to not do things arbitrarily, but rather like collect data and then say that, yes, this makes sense based on the data.

4:13Martin Splitt:And John Mueller, our manager, had this idea that how about we don't just add this one tag, but look through the, let's say, top 10 or top 15 tags and add the ones that we don't have in that list yet. Because that would give us a decent starting point, a decent baseline, and be able to say that, okay, we are documenting the top 10 of these tags that we don't support, right? And fast forward two days, I'm struggling finding a public repository of robots.txt files that we could use to identify these tags. And he suggests the HTTP archive. I have never used the HTTP archive before, other than looking at the reports that they do.

5:06Martin Splitt:I think it's called the Almanac or something. Yep, the web Almanac. So I don't know how it works. I don't know where the data lives. I don't know anything about it. In fact, I still don't. Do you? Yeah, I do. I used to contribute to the SEO chapter a few times. Okay. Wanna teach me how it works? Okay, yeah, sure. So you know nothing about it, I'm assuming. I honestly have like literally nothing. Like we did some code for it the past few days, but I don't know how it's used or why it's used.

5:39Gary Illyes:I don't know the data sets. I don't, literally nothing. Okay, so it has been running for the last, I don't know how many years. It's definitely been around since 2019, it must have been, because I think I was involved in the 2019 edition. I believe that it has been running before that as well, but I'm not sure about that. But the idea is that you basically look at a large number of websites or web pages more specifically, and look at how the web changes or like things that you can learn from looking at large quantities of websites. For instance, what language are they in? Are they mobile friendly?

6:22Gary Illyes:Are they using HTTPS? Are they, I don't know, using canonicals? These kind of things, like all sorts of stuff that you can basically infer from looking at the source code of a website or from things that you can infer from the behavior of a website. So it has to do so. It has to do a crawl. So it has to basically like know all these things. And then it also has to kind of quote unquote render them. So it has to do some sort of analysis on them to get some additional data. For instance, performance, like how fast is this loading? How do the core web vitals look like? And so on and so forth. You can't get that from just a crawl.

6:59Gary Illyes:You have to actually sort of run the website in a browser to get this kind of data. And these two things can be combined. And then a bunch of people set out every year to ask questions about the big data set of information they have. So there's like two stages. In stage one, they're like, hmm, I wonder, I don't know, for instance, I wonder how many words per page there are for all the websites that we will look at. And then they write some script that gets this data from either the crawl. So with words from a page, you probably can get some of that information from a crawl. But if it uses JavaScript, you would have to get that also from the rendered version.

7:42Gary Illyes:So when you say a crawl, is that what? Like who's crawling what? Okay, so we start with a bunch of URLs that we know exist. And I believe that these URLs are coming from the Chrome UX report, if I remember correctly. Okay. So if you opt into it, your Chrome browser sends data to an aggregate report, basically, that says like, hey, here is what we've seen in terms of performance data, for instance, from real users opening this website. It doesn't say like Martin or Gary have seen these numbers, but it basically aggregates it. So on average, across all the people who have visited this website until now or in the last year, I'm actually not exactly sure how Chrome UX report segments the data.

8:29Gary Illyes:But basically, all these URLs, all these websites have been visited by someone who sends the data into the Chrome UX report. And then you can query the Chrome UX report. So it's a public data set of aggregated user experience metrics for websites. And in this set are millions of URLs. I believe it's like 16 point something million. That's a huge data set. Okay. Historically, that have mostly been homepages. So they kind of filtered it out to only get like homepage data, kind of arguing like oh you know it's probably the more popular part of every website to go to the home page like you go to ebay.com or to amazon.com or to google.com rather than to like google.com slash how search works that is a page on this website but it's probably not one that we have a lot of data on so historically it has been focusing on the home pages but in the recent couple of years, and I'm not sure when they started this, but at some point they expanded to what they call secondary pages.

9:37Gary Illyes:So you can kind of say like, oh, we are only interested in homepages or we are also interested in how do homepages perform or look like compared to quote-unquote secondary pages, right? Because usually we have this kind of stuff in the Chrome UX reports, and for some websites, they might also be much more popular than the homepage, for instance. And then the homepage gets a bit neglected, and then whatever secondary page you have is more popular, so you put more effort into it. And then they basically run a crawl. I'm not exactly sure how they're crawling it, but they're basically doing a bigger run.

10:15Gary Illyes:I think what they do is they run through web page tests. I'm not sure. Are you familiar with web page tests? That's some service. Yes. Yes, webpagetest.org. You can go there, you can type in a URL, and it runs your website in an actual browser. And I believe what they do is they do that. They have their own instance. They're probably paying for that, I'm not sure, or have some sort of collaboration with webpagetest.org. And then they put these URLs that they got from the list from URLs or the list of URLs from Chrome UX report, and they basically run these through a browser instance on a server hosted by web page test.

10:57Gary Illyes:Okay. And that's what they do to crawl, I believe. Cool. But as you run it in a browser, you get a bunch of information that you don't get if you were basically using curl or wget or whatever on the command line to kind of just download the HTML. Like for instance, you can tell what amount of CSS has been actually used and how much is unused. You can run a lighthouse test on it. And you can run some JavaScript that you can control. And that's what we wrote. Remember? That's the JavaScript that we created. Oh, I remember. Yes. I remember. Yes, of course you do, because you love JavaScript so much.

11:36Martin Splitt:I love JavaScript. So that was also weird to me, because I didn't realize that you can use JavaScript for this kind of stuff. But anyway, the way I discovered this whole thing works, well, not how it works, but how the data is stored, I don't know if you can download it or not, but there's also a BigQuery dataset or datasets. Yes. And then you can query, write basically SQL queries to query those datasets.

12:06Gary Illyes:Yeah, that's the second step.

12:07Martin Splitt:Yeah. Which can be very harsh on your wallet, as I learned. That is true because the data is relatively large, I guess. I literally remember that Daniel Weisberg, our teammate, he wrote a blog post about how to avoid large charges on BigQuery when you are digging into Search Console data. And when I got the charge, I ran one query, like one large query. and I got like hundreds of dollars worth of charge for that one particular query. And yes, it was running for quite a while, but still it's like hundreds of dollars. So yeah, yay.

12:58Martin Splitt:It happens and it's an open source project. So yeah, I will just absorb it, I guess, but it was painful. Anyway, what?

13:07Gary Illyes:It reminds me of the, what can a banana cost, Michael? $10? Oh, yeah, yeah. I love that. It's so good. It's a great show as well. Yeah.

13:16Martin Splitt:That was a coffee, no? From Mean Girls or something?

13:20Gary Illyes:Anyway. Arrested Development. It was Arrested Development.

13:24Martin Splitt:Yeah. Anyway, and we quickly figured out that no one is actually requesting robots.txt files. So the data sets don't typically have robots.txt files in it, which was also very painful because I already paid like hundreds of dollars for that one particular query. It's great, great. But stop laughing.

13:45Gary Illyes:I'm so sorry.

13:47Martin Splitt:You're not. No. And then more internal discussions, and then we realized that why don't we just put this in the custom metrics dataset, which, again, is not something that I knew of. If I knew of that thing, then probably I wouldn't have run that initial query that cost me so much. Do you know about the custom metrics?

14:09Gary Illyes:Yeah, so step number one is kind of exactly what you then did, the custom metrics bit, where we take the URLs and we run them through web page test. And as web page test says, okay, this page is now done, there's nothing happening anymore. In this test browser, you can run some JavaScript on whatever you got. And I believe that there are some URLs in there that are robots.txt URLs, because I think in the SEO chapter, there is a robots.txt analysis. I'm not exactly sure how you can filter for robots.txt specifically from all the URLs that we have. But basically, that's step one. Like you gather these metrics and they are custom metrics because they're not by default exposed.

14:52Gary Illyes:For instance, if you run a Lighthouse test, you have certain things like, I don't know, I think Lighthouse tests for the Core Web Vitals. So you can basically say like, hey, from this database that is created from all the things that we run through WebPageTest, I want to see from each of the pages the lighthouse.corewebvitals. I don't know, largest contentfulpaint. And then you get the numbers. And then you can do things like you You can tell like, hey, so what's the average? What's the maximum? What's the minimum? Blah, blah, blah. What's the 90th percentile? You can do these kind of things. But that's not a custom metric because these metrics are kind of like default and you can get them just by running the page through the browser.

15:33Gary Illyes:But then you can run these extra JavaScripts that are looking at the content. And you can do things like, for instance, you can say, hey, give me all the children elements that are in the head. and then later on in the queries part like you get a list of all the things that are in the head let's say like you call it custom.head-invalid-elements or something and then you can say like okay so for all the things that we have in this database of these head elements which are ones that don't belong there or which is the most likely head element that we are seeing or I don't know what's the char set that people are setting in their meta char set element that they have in the head and to have any sort of metric that isn't by default available to a browser or Lighthouse or whatever other tools we are running.

16:21Gary Illyes:I think there's another one, Web App Analyzer or something like that that gives you information like what framework has this been built with or what content management system is this using. So if it's not in these kind of default tools that are running, then you can add custom code to get out what you need and that's what you did, right? Yeah, I mean, we did it. Okay, fair enough. Like we did, yeah. You wrote the code, I looked at it and cried only a little.

16:49Martin Splitt:So that's the suggestion that we got from Barry Pollard. So he pointed us to their GitHub repository for custom metrics. And then there we found this weirdo JavaScript function or class. I don't remember what it is. Anyway, that is actually extracting some limited number of rules, but they were hard-coded, right? So basically, it was a noindex, a norhive, I don't know, crawl delay, whatever. Basically, just counting those that they knew of already. And we needed the exact opposite. We wanted to learn of all the rules that people are using, not just the ones that we know about. So we twisted it around and we got some really good comments from Barry and some other folk in the GitHub community.