How to read the Indexing Report

0:10Hello,

0:15my name is Martin Splitt and I welcome you all to a new episode of Search of the Record, the podcast where we, the Google Search Relations team, are taking you behind the scenes of Google Search and hopefully have some fun along the way. With me today is John Mueller, who's the mastermind of our team. Hi, John. Hi, Martin. So good to see you or hear you. Yes, I have something I want to talk to you about and double check that I got it right. Okay. I don't know if you remember, but recently at a Search Central Life event, there was a question about the indexing report, the coverage report in Search Console, where someone's like, well, there's this page that I have and it's indexed, but the Search Console report shows it's not indexed.

0:55And why is it wrong was basically the question. But they were very friendly. They were friendly. You make them sound very angry, but I think they were friendly. That's because I'm German. I think like, that's just nine, nine. Anyway, so I thought about this a little bit and I think I know what happened here. I recently saw another example of that with someone on Reddit who thought they screwed up because they have a bunch of non-indexed pages. Yeah. I think it's something that a lot of people, especially in the beginning when they start using Search Console, they run into this kind of thing where they look at the page indexing report and they see, oh, there's like hundreds of pages that are not indexed.

1:39This is an error, clearly, on my side, something that I have to fix. Yeah. And then they go off and try to do things to understand it or fix it. I mean, you even get emails about it and that kind of urges you to go into action. And I think the mechanism we have for people to acknowledge that they understand what's happening is this like fix, confirm fixed or mark is fixed or something. And I think while generally a great thing, I think in this case, it's a little it's triggering the wrong response or the wrong expectation. I think would you say that's maybe the case or. I think maybe. Wow, maybe.

2:24It depends. I have to do It Depends. Otherwise, this can't be a search podcast. But it's something where we try to use a similar UI across Search Console for a variety of different features. And that includes this marked as fixed button in a number of places. And in some cases, there are things that you can fix. Like if you have a bunch of 404 pages and you realize your server was set up wrong, you can tell us you fixed this issue and we'll try to fix it. On the other hand, if something else went wrong in indexing and it's more like our systems decided we weren't as curious as you might be, then it's not something that you necessarily fix yourself.

3:11It's not something purely technical. Yeah, I would generally agree. And I think I had a talk with someone from Search Console, with Hilaire from the Search Console team. Brilliant guy. Lovely person. Very helpful. And I think that opened my eyes to how we should probably talk about this and how people should probably look at this feature. And it's slightly different from what I think people are looking or how they are looking at it now. Okay. Tell me more. So, all right. So he said, oh, I wish people would not look at it as a list of things to fix. That's what we kind of have told people. And also not a static inventory that you need to monitor.

3:54Like, oh, is this page still in the index? Is this page out of the index? What's happening here? instead he says it's great to look for patterns and to look for things that are unexpected and then dig deeper into that okay it's an interesting way of looking at it i think because it's not like is this index is this not index it's a different thought it's like is the site in the index doing what I expect to do. And I think it can help you to spot things that haven't happened the way you intend them to happen. So for instance, if you're migrating and you're setting up redirects, then indexing report shows you, ta-da, a steep rise in redirects, page with a redirect.

4:42It's not indexed because it's redirecting somewhere else. And if you look at it as an inventory, as a monitoring tool only, then you're like, ah, what's happening? Why is there pages that are not indexed because they're now redirecting? When in reality, that's exactly what you want them to do. True. Because you just set it up. And on the other hand, a problem would be if that wasn't showing up. Well, a problem, I say a problem, but like it takes a while for search to see this change and actually process this change. And you know that once it's reflected in the data, Aha, search has understood this.

5:19As long as that hasn't happened in the index report, then you know, ah, okay, interesting. We have to wait a little longer because the processing hasn't completed yet. Yeah. And I think that's a really powerful tool. And it also then doesn't matter as much that the data is not always real time. That's the other thing. Like people publish a new section or a new category on their website. And then they're like, ah, they're not indexed. And in reality, it might just be that we have a data delay or it might take a few hours longer. We sometimes do have longer delays, but we normally communicate these in Search Console, right?

5:56We have these like little notifications that we send out. But yeah, so I think that's really, really handy. The other thing is that sometimes you see changes that you haven't triggered. I mean, with a redirect or like if you remove a whole category of stuff on your website, then you expect to see a steep rise in 404s because you removed stuff from your website. And again, if that hasn't happened, then that tells you something has gone wrong, maybe, or something hasn't happened yet or hasn't been processed yet. Yeah. I also think, especially with 404s, that's something that sometimes throws people off.

6:31Yeah, yeah, yeah. Because they're labeled as an error, and it's almost like your website is returning an error. And then a lot of people see that and say, well, I have to fix this error. Yeah. And I have to make sure that Google doesn't see a 404 because if Google thinks my website has errors, surely Google will think my website is not as useful. Yeah, it's not good. We need to fix this. It's an error. You want to fix an error, right? But it's an error that is expected. It's actually a good thing. The problem is that, and then I get the question like, so, but why does Search Console show it as an error?

7:07Because it is an error. It's just an expected one. Yeah, I think that part is always tricky for people in the beginning. And especially tricky if you have, I think, management that doesn't understand it so much. And management tells you, it's like, you should make sure Search Console is always green. Everything is okay. and then people might be worried that's like oh I have this error report and this number is not going down yeah what what do I need to do to make this error number go down so that my boss thinks I'm doing a good job and actually if you're removing things from the web or if these pages never existed yeah they're supposed to return an error that's actually the right thing yeah Yeah, absolutely.

7:56And the other thing is, so it can't judge for you if this is a problem or not. You have to do the judging yourself. And the other thing is sometimes it's changes that you haven't caused, like page with a different canonical. That's something that makes sense to look into, but it's not something that should alarm you because that just means that the report as is, is now reporting for different URLs. And I know that the numbers then sometimes look like, ah, it's going down, but then that usually means it's going up somewhere else. And I find that useful because it tells you how Google thinks about your site and your URLs.

8:36And if you have like, I don't know, www.example.com slash blah, it might be that external people are all linking to a different version of that, the non-dubdubdub, just example.com slash blah. And maybe Google thinks, hey, you know what? This is shorter and everyone uses this. So maybe we use that as the canonical. And that can help you change your canonicals to something that everyone else would prefer anyway. And I think that's a cleaner way of doing things, but it's not a problem that you need to fix, right? Yeah, I think fundamentally the question of whether you should change your canonical to what Google thinks your canonical is or not?

9:18That's up to you. That's a very different question. Ah, okay. But the aspect of Google picking another URL within your website as a canonical, that happens all the time. And that's normal. And it's not something that you need to worry about because it's still in the index. People can still find it. Yeah, exactly. That's something, by the way, that is a little bit easier if you have a domain property set up in Search Console. Yeah, that's true. Because then you don't have to worry about dub, dub, dub, non-dub, dub, dub, and whether you should say W or dub or whatever. All of these things are a little bit easier.

9:55And basically, all of that is shown in the performance report. That's true. I do think it is useful to see the site from a kind of like bird's eye view. And I really, really enjoy the report for that. So I wouldn't use it to look specifically after like, oh, this page needs to be in the index and this page isn't in the index. And what do we do now? But more like, are we seeing some pattern moves? And why are we seeing them? And for that, it's great because you can see like, oh, picked a different canonical for a lot of pages. You can click on that. You get a bunch of examples. And then you can analyze one or two of these examples and then basically get a feeling for, ah, okay, so that's happening on the page.

10:41And sometimes you can basically just conclude, well everything is fine there's an a shift of something but the shift was to be expected or at least the shift isn't malicious right yeah so i think the way that hillel presented this makes a lot of sense to me because it's more like uh don't think of it as inventory and we have a similar problem with the site query as well yeah where people are like it's not showing up for the site query so it must not be in the index but google search console says it's in the index so who's right. Yeah. I think site query is an artificial type of query. So that's something where partially you can use it to double-check if a URL is indexed, but Search Console is really kind of the source of truth there because sometimes we just don't show things in site queries and sometimes we do show things in site queries that are not actually indexed like that.

11:37For example, if you do a move from one domain to another and you do the move properly with redirects and all of that, then basically if you ask Google about the old URL, the old domain with a site query, Google will tell you. It's like, oh, yeah, I know about this. And it's like, here's a bunch of URLs. And this will happen maybe a year or two or even three afterwards, after you do the move. And if you look at purely the site query for that, you might conclude that, oh my, the site move actually didn't work. Like, Google is still indexing all of these old URLs. And sometimes when you look at those results, you'll see in the title maybe the new domain name or something like that.

12:19So you can tell Google saw the move, but it still knows about those old URLs. And if you explicitly ask it about something, then Google will be like, sure, I can give you what you're asking for. It's like, I'm not saying that this is the current source of truth. but it's like, you're asking for this site and here it is. And it might be like, if you're a normal user, you might have remembered the old domain. You're like, what happened to this old website that I really liked? And if you look it up on Google, Google will tell you, it was like, here's the content that was there. And if you click on it, you'll of course be redirected to the new site.

12:57Does that sometimes also happen if you have Ahrefs length set up and have different language versions? probably because the way hreflang works is the individual versions have to be indexed but then when they're served in the search results they're swapped out against the local version and i don't know for example what happens with the site query if you do i don't know site colon something.ch and you're in germany maybe you see the german version maybe you see the swiss version I don't know. Interesting. Someone should try it out. Please try it out and write it in the comments. We're curious to see how that happens.

13:37It would be interesting. I think maybe taking a step back to some of the errors that you mentioned, one of the things that I think is worth watching out for is whether or not there are any systemic issues with regards to your hosting setup. And this is something that you do see in the page indexing report as well. And it's the kind of thing where you look at and you're like, well, this is unexpected. Why are so many URLs suddenly considered 404 or 403 or, I don't know, dropped for other reasons or even canonicalized to some other page? And sometimes we've seen CDNs, for example, or hosting providers, they have some kind of bot protection, and maybe they turn it on when there's a lot of crawling happening.

14:27And depending on how this bot protection is set up, it can happen that suddenly your CDN or your website host serves 404 or 410 or 403 or something instead of something reasonable, at least from what I would consider reasonable, something like a 503 that basically says, oh, I can't deal with you right now. Come back later. Maybe if they think Googlebot is a malicious bot as well, maybe a 404 or something makes sense. But that's something I think worth watching out for. Like every now and then taking a look at that report to see, has there been a significant change in the number of crawlers or in the number of pages that are dropped from indexing?

15:12And if so, what are some examples that I can follow up on? The other one, which I think is a bit harder for sites to diagnose, is when the hosting provider shows something like an interstitial. Something like, are you a bot or not? And it serves that with a 200 response code. And you can guess what happens. It's like, we'll try to index that on the one hand, which will result in your page's content kind of disappearing and being replaced with this, are you a bot page? But what also happens is because this page is shared across a lot of other pages on your site and maybe other pages across other websites, it can happen that we pick a canonical of someone else who has the same error page.

16:00And that's really challenging to kind of debug because you basically have to look at those pages and look at the canonical or look at the page that Google chose as a canonical when you look that up. And sometimes you have to almost guess a little bit of what would happen when Google accesses this page, because as a user, you might not see that interstitial at all. But if you see a lot of pages on your site being duplicate eliminated in favor of some random other page, oftentimes that's a sign that you have some kind of a soft error on your page, some kind of soft block that tries to block crawlers and bots and basically just ruins your SEO.

16:47So that's something where, of course, that's the kind of thing where I would say you should go off and try to get that fixed with your provider or your providers. If you have a CDN and hosting, figure out where it's coming from. And then using something like marked as fixed in that report makes a lot of sense. Because then you're really saying, like, there was a significant issue on my website. Like you found all these pages and you canonicalize them somewhere else or you call them 404, but actually they exist. How dare you take away my pages? And then we will try a sample of those. So the way the mark this fixed works is we try a sample of the pages that you're basically telling us are fixed.

17:32And if we see that they're actually fixed, then in most cases, we will trigger a faster recrawl of the other pages. So it's not so much that we wait and like, oh, well, we'll see if this is actually working better, but we'll try to recrawl that a little bit faster. That makes sense. And again, this is about patterns. So if you see like a bunch of errors, then you can figure out, ah, so this is all the products because X, Y, Z or this is all the sites. So this is not just like a specific template that is misbehaving. helps you troubleshoot a little bit faster. Because if it's just one specific template where pages are experiencing some specific problem, then it's probably something within your hands, so to speak.

18:16But if it's the entire site or an entire part of the site that is behind the CDN, then it's probably something to do with the CDN. And again, these patterns are worthwhile. The same with the trend lines. I think we haven't really talked about the trend lines in the report yet. So not only do you get the reason for a problem, but you also get this line. Because sometimes a server has a hiccup. There's some sort of problem. We are seeing some 500-something error, which can or cannot be a problem. That depends. If you're seeing a little blip, then maybe look into the server logs, like what happened here.

18:51And then someone tells you, oh yeah, we upgraded the host. We moved between different computers in the data center, whatever. And then there's not much you can do about it. And it's kind of fine because it's going to work itself out. But if it's a steep line going up, maybe look into it. Yeah, yeah. On the other hand, you might also see that and it's like a horizontal line and it's like a few blips and it doesn't, yeah. Yeah. I think it's important that people realize that computers don't always work. And for the most part, the internet is built in a way that is supposed to be a bit resilient against that and just retry.

19:29Yeah. So it's like when we look at the crawl stats of a website, every now and then you'll see things like, oh, I don't know, a handful of requests failed or a handful of DNS lookups failed. And of course, we want to report that because maybe you care about those handful. But for most sites, it's something where it's like, oh, this didn't work this one time when Google tried it, but the next time it worked and everything kind of worked itself out. And it's not going to be the case that if Google runs across an error and it just exists for a couple minutes or whatever, that it's going to cause any issues across the site.

20:07Even for the most part, when we run across errors and they exist maybe for a day, which is like a really long outage in internet time, that's something that we try to kind of look over and say like, oh, maybe it's something temporary. We'll just kind of like keep things stable and come back and check again. But if it's longer than a day, then of course our systems are going to react to that. Definitely. And you want to have a look at it. And also, if you add or change your site or if your site is very new, then you can actually also use this report to see a little bit how your site goes through the different stages.

20:45Because at some point, you're going to see pages in discovered, currently not indexed, which tells you we know they exist, but we haven't actually visited them. And if we haven't visited them, we can't put them in the index. Versus crawled, currently not indexed, which means we visited them and we didn't put them in the index. And that can have all sorts of different reasons. Would you say that that is often or only sometimes a sign of a quality issue? Sometimes. So it's definitely the case if our systems are seriously worried about the quality of a website, that they will reduce the number of pages that they index.

21:25Because if we have strong concerns about the overall quality, then it doesn't make much sense for our systems to spend a lot of time on the website. So we'll probably crawl a lot less. We'll index a lot less. And then you'll see things like crawled not indexed or discovered not indexed, which from our point of view is basically our system saying we know about this. And once we're happy, we looked at it. Once we're happy, we will take another look and see if we can index it. It's not so much that I would say you should take these situations and try to fix them. Yeah. Like from a technical point of view, it's not that you need to fix this technical issue, that Google is not indexing this page at the moment.

22:12But rather, you almost need to, when you recognize a bigger pattern like this, like Google is not indexing a lot of your pages, and there's no technical reason, you almost need to take a step back and think about the quality overall. And thinking about quality is really challenging. Because a lot of times, it's your website, and it's your baby. And of course, it's the best baby ever. But taking a step back and trying to look at it with the eyes of someone who is not directly involved with your website, sometimes that opens up some ideas for areas where you can improve. Where maybe if most of your website is AI generated and it worked for a while, it might be that people look at this AI generated site and they're like, well, I can tell this is AI generated.

23:00There's nothing unique or valuable that is available here for me. That's not to say that all AI generated content is bad, but sometimes you just run across websites where you're like, anyone could have written this. This tells me nothing. Yeah, that's true. And I think what makes this difficult is not only the fact that obviously the way you wrote it is the way you thought is best. And that's why you think it's high quality, of course. So that's really, really hard to kind of step out of your own perspective. but sometimes it's also there's so much other stuff that is just as good so why would we add it to the index and then that can tell you like maybe this content isn't as valuable as i thought it is because other people are covering the same thing and then what's the value of this version of it being in the index yeah that's true i feel we we could have a whole podcast about quality i think Maybe one other thing that is worth mentioning with regards to quality is it's not just the text.

24:01So a lot of times people will say like, well, my text is unique or my articles are good. And they're packaged in a page that is terrible to access where anyone who, when they try to load it, like their computer fan spins up and they're like, oh my gosh, I have to run away to make sure my computer doesn't explode. So maybe that's an extreme case. But you've all seen these pages where basically the text is there, but it's almost hidden away, hidden behind ads, hidden behind interstitials, hidden behind other things that are moving and coming and going, maybe hidden below a bunch of filler content, which we sometimes see, for example, with recipes where there's this really long story on top that maybe most people don't really care about.

24:48and then the recipe comes. These are all the kind of things where the overall quality is much more than just that piece of text that you say, like, this is my main content. This is what Google should be counting for my site. And from our point of view, we almost have to take into account the full experience on a page because that's what users see. It's not that users go to a web page and turn on some magic mode that just pulls out the text, but rather they have the full experience of this website with all of the 3D, 4D animations and everything. I agree. I very much agree. Oh my God. The other thing is there's no such thing as a ratio between indexed and non-indexed pages.

25:34I mean, there is, but it doesn't matter. I've seen so many healthy websites that have a million non-indexed pages and half a million that are in the index, and they are doing perfectly fine. So I think that's the other thing. Like a lot of people are seeing this and they're like, ah, Google probably thinks my website is low quality because only, I don't know, 20 % of my pages are indexed. No, that's not something that you need to worry about. That's not something that tells you like this is good or this is bad. Lots of it depends. Lots of it depends. I think that's something that people don't normally know.

26:08Yeah. When I take a look at the search console for our developer documentation, for example, I think it's something like 5 % of the pages are indexed. Wow. Which, when you look at that report, you're like, oh my gosh, something must be seriously wrong here. But when you look at the details, you see it's like, oh, a lot of them are no index. There's a bunch of 404s, like canonical things. things like the important content is all indexed. We see that important content also in the performance report. So all of that is OK. But if you just look at the page indexing report, you're like, whoa, this looks like something is seriously broken.

26:52MARK MANDELMANNICKI I wonder how this looked when we did the big migration from the old structure to the new structure that we did a few years ago. That must have looked really chaotic. MARK MANDELMANNICKI Yeah. Yeah, I imagine some of that, but also probably a lot of code samples and probably, I don't know, from the other different product areas that are hosted on the same site. Maybe they have different setups where they say like a lot of this content should be no index. Maybe, I don't know, if you have APIs with older versions, maybe you say all the older versions are no index, which is a decision that you can make and is fine from our point of view.

27:29It's just in the reports, you'll see that reflected, or you usually will see that because we look at these pages and initially go and say, well, maybe we should index this. And then, well, maybe not. Maybe. Hmm. Interesting, but I think we're good. Yeah. Yeah. Anyway, I think that's summing it up roughly what I wanted to bring out there. So don't think of it as an inventory that you need to fix. don't think that everything on the website needs to be indexed don't think that the ratio between index and non-indexed is some sort of like quality measure or some sort of score that you need to look at i think as long as everything you care for is indexed with some url and again domain properties make this easier because you can see if canonicals just change and don't have to really like, oh, this dropped out because of canonical.