Should you block your Search result pages?

0:10Well, hello, hello, hello, everybody, and welcome to a new episode of Search of the Record, the podcast where we from the Google Search Relations team take you behind the scenes of Google Search and hopefully have some fun along the way. My name is Martin Splitt. I am from said Google Search Relations team. And with me is John Miller. Hello. Hi, Martin. So good to see you or hear you. Hear you. Yes. Hear me. Hear me. Hear ye. Hear ye. So what did you bring us today, John? You have a topic in mind? Yes. I thought we could talk about search results pages. What do you mean? Our search results. Well, search results search.

0:58also our search results, but not the normal. Oh, gosh. How do I position this? So basically, if you have a website, do you have a website, Martin? Yes, I do. Yes. Fantastic. More than one. Oh, my gosh. OK. Wow. Big domain farm. It also has search results because you can search on the website and find things there. Exactly. That's what I wanted. Yes. Ah, those search results. My search results, not Google search results. Your search results. My search results. OK. Or anyone else's search results pages. So if you have a website and you have a little search feature, maybe on the top or on the bottom, where people can say, oh, let me search for Swiss cheese on this website because I know this is from a Swiss person.

1:45It must have Swiss cheese. Where is the article or the photo or whatever about Swiss cheese? This is something that a lot of sites do. I think it makes sense for a lot of sites, especially if you have a non-trivial number of pages. If you have maybe five pages on your website, of course, people can just click through to those five pages. But if you have a bunch more pages or if you have a lot of different kinds of content, then having something like a search feature on a website generally makes sense. And pretty much all content management systems provide some kind of functionality around that line where it depends a bit on how you're set up, of course, where maybe they have something like a live index where they search through your database and show the results to the user and bring that up directly.

2:41Or they have some kind of other setup, maybe even a third party that they're using to perform this kind of search service across their website. And the reason that I bring that up is not so much that your search results and Google's web search results should be the same, but rather because Google's systems, like Googlebot, goes off and crawls your website. and sometimes it will find your search functionality or sometimes it will find some links somewhere that go into your search functionality. So it might be, for example, that, I don't know, someone links to your search page for Swiss cheese and then Google doesn't know that this is any kind of a special page, but rather it looks at that URL and says, oh, martinsplit.com slash search slash whatever Swiss cheese.

3:42And then it goes off and tries to crawl your pages. OK. Which can be fine. Do you have Swiss cheese on your website? I do not have Swiss cheese on my website, John. Oh, my gosh. No, I'm sorry. How can you have a website in Switzerland and not have Swiss cheese? This is forbidden. Oh, my gosh. Google will count that against you, I'm sure. Yeah, it's a quality strike or something, right? Yeah, sure, sure. But is there something on the technical side that I need to do if I have search result pages on my website? MARK MIRCHANDANI - So there are two things. One is more of a technical thing, which is if Google finds these search results pages within your website, it can happen that it crawls those pages.

4:30Maybe you have related search results linked on the bottom. maybe someone else links to a bunch of search results. And that can result in Google basically finding an infinite number of URLs on your website. So if you have this search feature, someone could link to every word in the universe and say, it's like, check Martin's website for this word, which is unlikely, but it could happen. And from a practical point of view, we consider this a so-called infinite space because it's an infinite number of URLs that potentially exist here. Depending on how your website kind of deals with these search results, it can be that you also have maybe some filtering mechanism for different categories or different types of pages, or it's like alphabetical order or date order or time order or I don't know, like any other kind of ordering.

5:26And all of that basically means we find or we could potentially find an infinite number of pages on your site. And I know some people are like, wow, if I had an infinite number of pages in Google, then I would be king. But having an infinite number of pages known to Google is not a good thing because Google will try to crawl all of those pages. And you can imagine what happens when we see, I don't know, 100 million pages that are new from Martin Split will go off and try to crawl those. OK. So that's something where then suddenly crawl budget becomes a question. And there's your server load. And your server is like, oh my gosh, this Googlebot is crawling infinitely.

6:11I am going to block it because who knows what they have planned. Could happen. I mean, it's not unreasonable if you suddenly see millions of requests coming in. And especially search pages tend to be something that are not cached. So that means that every time someone accesses one of these search URLs, the system has to go off to the database and do a lookup and figure out, like, what is the best matching thing? And then it has to run a ranking algorithm over your pages and figure out, like, which ones should be shown on top, what is a snippet, and all of these things. So that's something which I think from a technical point of view can easily cause problems.

6:56Of course, because the internet is so big and so many websites do search pages in different ways, we try to recognize this. So when our crawler systems come across a website and see, it's like, oh, this looks like a search results pattern. This looks like an infinite space. like we could potentially find tons of URLs here, then we will try to deprioritize that. So it's like we try to do some things, but it would be easier if we didn't have to do these kind of things, right? Ah, yeah, true. Plus it might actually, like depending on how efficient the search algorithm is on your page, it might actually cause your server quite some work and actually might make the site even slower for other people, right?

7:46So you want to avoid that. Yeah, okay. That's kind of annoying. I mean, it can be really annoying depending on what actually happens and how your site is configured. I've seen sites that link from one search results page to maybe 10, 20 other pages. We're like, oh, have you tried this query? What about if you add an attribute to your query? And then we're like, okay, wow, 10 more pages. is, wow, let me get those two. MARK MANDELSKI - Oh, no. MARK MANDELSKI - So that's something which we try to encourage people to avoid. And there are basically two ways that we recommend dealing with that. One is with robots.txt, where you can say, it's like, oh, by the way, the search page, you don't have to crawl it.

8:32It's like, my site is completely crawlable. It has clean categories. It has maybe a sitemap file, maybe an HTML sitemap, maybe an llms.txt. you don't have to do. Sorry, I had to.

8:47Okay. Back to serious face. So it's like maybe your site has all of these features to make it crawlable in a normal way. And you can just use robots.txt and say everything in search is blocked. Like you don't have to go in and look at these at all. I think that's probably the easiest approach. That's also something that a lot of people can add, even if they're using some kind of content management system. If you're using WordPress or Drupal or Joomla or whatever else is out there nowadays, you can easily, usually, edit your robots.txt file and say, by the way, search, disallow. So that's one approach.

9:30The other is to serve the search results pages with no index in either the HTTP header or with a robots meta tag, which is slightly different in that Google can still look at those search results pages, but then it sees like, oh, there's nothing for me to do here. I will stop at one. It's like, I might find a bunch of these, but I'm not going to crawl infinitely through this pile of pages. and that's also something that our systems, when they do scheduling, they try to learn. Say like, oh, this batch of pages is all noindex. Like, you don't have to deal with that. Okay, that makes sense. But if I don't have full access to the source code and I don't have robots.txt, then I just have to hope for the best?

10:22Or is Google like detecting this kind of stuff? Or is it even like treating my site differently if it finds one of these infinite crawl spaces? It's surprisingly common to run across these kind of infinite spaces. So we do try to recognize it, but I don't think we have any information in Search Console where we tell you, it's like, by the way, we found this and we fixed it for you. And it's always, if our systems have to automatically recognize it, maybe it takes a while for them to recognize that. And depending on your hosting setup, maybe that it takes a while is unproblematic. Maybe it's something where you see like, oh my gosh, my server is going down.

11:11I need to do something immediately. One thing that I've seen people do or try to do to fix that is to use the removal tool in Search Console. Because it feels, or it could feel, like no index and removal tool is kind of the same thing. but the difference is the removal tool basically filters it out in the search results pages. It doesn't prevent crawling. So if Google has gone off and crawled all of your search results pages and keeps crawling them using the removal tool will hide them in search, which is maybe okay, but it won't prevent crawling. Yeah, it doesn't solve the actual issue, right?

12:05It's just hides the symptom. Exactly. Which I could imagine if you're working for some site and someone's paying attention and seeing this, maybe they think it's fixed if you hide it like this. But definitely not the way that you should handle it. I mean, other things that you could do is maybe serve a 404 result code for these search pages. I don't think it's very clean because then also some users might have issues with caching and things like that. What you definitely shouldn't do is serve something like a 500 result code, which theoretically you could. Also, it's like, oh, by the way, this page is not working for you.

12:48But what happens there is that our systems will recognize this and say, oh, we tried to crawl and the server basically broke, which means maybe we're crawling too much. And then we will reduce crawling overall for the whole website, including all of the pages that are not in your search results pages. So definitely don't do that. Oh, OK. That makes sense. OK, so now I guess robots.txt is the best way to go about this? I think it's the easiest way. Noindex, I think, is almost cleaner because you avoid kind of this potential state of a roboted URL being shown in search. I think it's very unlikely that if someone searches for something where you actually have content, that we would show a robot URL because we actually have better content for those queries.

13:44But theoretically, that could show up. And with a no index, that's prevented. With a no index, though, I was like, we would still crawl the page. So it's pros and cons. Oh, maybe another thing worth mentioning. I've seen some CMSs, in particular Blogger does this. I think I've seen some others, which basically take search results pages and use them as tag landing pages. So if you have different categories of blog posts, for example, and if you click on that category name, then what Blogger does is basically a search for that category. Oh. And usually, you would want a search engine to be able to crawl and index category pages.

14:32You wouldn't want them to be treated the same as a normal search results page. So that's something where I'm, now that I think about it, not 100 % sure how you would manage this on Blogger. Maybe there is a URL pattern that you can differentiate between this kind of tag, a category page, link and a normal search result, then you could just block those search results pages and leave the category pages. Because I think category pages in general are very valuable. They make it a lot easier for search engines to crawl a website, to understand the context within a website, because then you really have this cleaner understanding of here's this category of things and here are the individual things that belong within this category.

15:20You almost have that hierarchy within a website a little bit clearer. Okay, interesting. Now, I can't just have robots rules for every possible search term because I don't know what people will be typing in. And we know that we keep seeing new things people type into search boxes. But I'm always nervous whenever there's patterns or wildcards being used in robots rules because you have to be a little careful. How would you go about that? Would you try to have one rule as broad as possible to filter out search results? Or do you split them up? Or how do you do that? I would try to have one rule that covers everything.

16:01From my point of view, if you have one clean rule that covers the whole pattern, then it's easier to maintain. It's easier to recognize which rule applies. I'm sure you've looked at robots.txt files and someone has come to you and is like, why is this one page blocked by robots? And you're like, let me look at your labyrinth of text, which most websites have very short robots.txt files, but some really try to be nuanced. And that makes it a lot harder. So for example, if you have something like slash search and then question mark, Q equals whatever for the different queries, I would just block slash search and then question mark.

16:43And that way, you're not blocking things like search for cheese.php, but rather like anything that is just in this slash search. And then that follows with some kind of URL parameter. OK, but you would try to get one rule that is trying to cover as much ground as possible. Definitely. Yeah. Maybe another thing worth mentioning around search results pages is way back in the early days in our Webmaster Guidelines, when they existed as Webmaster Guidelines, I think we also listed search results pages as something that you have to block from Google. And we listed it, I think, either as automatically generated content or something like these infinite spaces type things.

17:36Because theoretically, it's an automatically generated content. It's also infinite space. And we recommended blocking those pages in the Webmaster Guidelines. Nowadays, we don't have that listed in the search policies, but it's still something that I think, just purely for technical reasons, makes sense. So it's not necessarily something where we would say, we think your site is spam if you allow your search results pages to be indexed. It's just like you're being very inefficient, and that's up to you. It's like, go for it if you want. But it's very inefficient. MARK MANDELAVICIUSSKI - Interesting.

18:17OK, so we are not seeing this as a quality indication or a quality issue. It's more like, well, why are you doing this? What's the value behind this, basically? MARK MANDELAVICIUSSKI - Yeah, it's almost like, well, if you have this under control, it's like, that's up to you. Just like we don't have any quality guidelines for calendars. It's like, no, you should not make your calendar accessible to the year 2099. And it's like, if you do, it's like, whatever. But it's up to you what you serve there. MARK MIRCHANDANI - But that would mean that if I have a few searches that I know are super important for me to show up in search results, I don't have to block them for reasons like, OK, so I can.

19:04MARK MIRCHANDANI - Sure, sure, exactly. MARK MIRCHANDANI - So you could theoretically do something like a nuanced robots.txt, where you say, I'm blocking all of search, and I'm allowing this more nuanced pattern within the search results page. If you link to that URL from within your website, we'll probably pick that up and show that. And that could be kind of a replacement for category pages for a handful of things. but it's like that's almost something where I would say probably you could do a better job if you just created a real category page rather than creating like this I don't know like weird mix of search results page category page that is kind of like when someone lands there they think it's more like a search results page and if you made a category page of your own you could provide more context, like why is this term important, and which of these pages are actually the best on your website for this term.

20:04Yeah, I was wondering, like, maybe there are some people who have good reasons. Actually, I'm interested, if you have a good reason for having your search result pages in the index, let us know in the comments, because I, besides the scenario, it's interesting, I thought of the exact same scenarios, like, what if my system doesn't have category pages, but I can use or abuse search results to do that. But then I mean like some websites probably have some legitimate reason for indexing their search results, but I'm not so sure what that would look like. Yeah. It's good to know that it's not like a penalty thing.

20:40There is one place where you could run into quality issues with search results pages. Oh, tell me more. namely if you let people search for things that are totally irrelevant to your website and your search results page includes those terms on those search results page and is indexable so for example like you said you didn't have any swiss cheese on your website but maybe you have photos so if i go to your search page and search for a Swiss cheese photos, then perhaps your search page will be like, oh, I have photos. Here are some photos. And you search for Swiss cheese photos and puts it in a big heading on top.

21:27Because suddenly now, if that page is indexable, then it can happen that your page shows up for the term Swiss cheese in the search results. Because suddenly you have this page on your website where it mentions Swiss cheese. And it's like, Swiss cheese is pretty, I think, innocent. It's not so much that you'd be like, oh my gosh, someone hacked my website to show up for Swiss cheese. That's terrible. I'm lactose intolerant. How dare they? I don't know if you are. I'm not. Okay. But you could imagine someone doing that and saying, for Swiss cheese, call this number. And then photos. And then your search page is like, oh, photos.

22:17I have some photos. And it has a page with a big heading on top that says, for Swiss cheese, call John with this number. And that could be problematic. So of course, with Swiss cheese, probably nobody will care. like if someone basically abuses your website to show Swiss cheese, but you could imagine casinos, pharmaceuticals, any kind of restricted content, any adult-related content. It could be suddenly like all of these long adult terms and then a phone number and then photos in the end. And your website basically has a page for these terms now with that phone number. then suddenly your pages are showing up in search for these problematic queries and it's not so much that someone has hacked your website to do this because your website is doing that freely and basically saying like oh you search for photos here's a photo but because it's accessible for any search term that comes up it's suddenly a liability it's more like a vector for other people to spam And we've seen that happen, that people do that at scale.

23:30They will try to recognize common CMSs that don't have their search results pages blocked and go off and link to thousands of sites with millions of pages, all with maybe some adult terms and a phone number or pharmaceuticals and a phone number or something else and a telegram address or some other kind of contact mechanism where the goal is not so much that people go to your site and kind of see your photos, but rather that in the search results, they'll see, for these pharmaceuticals, call this number on martinsplit.com. And sometimes that does show up for these kind of queries. And when we see that happen, we might flag that as hacked.

24:16So in Search Console, you might see that as something that is flagged as hacked. It can also be that algorithmically, we recognize that and say, this is clearly content that doesn't belong on this website. We will try to block it, which works a bit in your favor, where it's like, oh, well, Google took care of the problem. But maybe Google only took care of part of the problem, of part of these kind of problematic queries. So it's a lot better to take care of that at the source and say, everything in search, I'm going to disallow by robots.txt, or all my search results pages have a no index tag on them so that like it can be seen but like clearly not indexed like that so that kind of plays into the same pattern it's interesting i didn't anticipate people would do that but it makes perfect sense like your services show up in search riding the wave of another website basically that's an interesting strategy and yeah i don't think relying on automatic systems in general is a good idea in terms of like, oh, Google will fix it for me.

25:27Yeah, but we do the best we can. But that doesn't mean that it's like reliable and always 100 % working in your favor. I think that's definitely a case. It's also not guaranteed that we recognize it immediately. Maybe it takes a month. And maybe during that month, some normal customer of your website looks at your website or does a site query or something and sees all of these pages where it's like, I've been searching for Martin Split. And then it's like, for pharmaceuticals, call this number, martinsplit.com. It's like, that's not great. And it's just so easy to take care of this from the start.

26:09And if you're working on multiple websites for other people, just putting that in robots.txt by default makes it so much easier. True. OK. All right. So we learned a bunch of stuff about search results on websites. So they can be infinite crawl spaces because we can basically generate pages upon pages of these. And maybe we even link to like, did you mean? And then we create even more that the crawler sinks into. And it sounds like they're relatively easy to get rid of with robots.txt and or noindex. Yeah. OK. All right. So, well, I think that sums up that. I didn't know where you were going with search results.

26:52I'm like, oh, yeah, search result pages are looking different every day. Maybe he wants to talk about a new feature that I haven't even seen. But OK, now somehow my website needs to take care of this. Yeah. Interesting. I fixed it differently on my website in a really lazy way. Basically, I have the search query box on top. And if you click the search button, it redirects to Google and does a site query. MARK MANDELMANNIKERIERIENKOVICHER That is brilliant. That's one way to do it. I might just remove the search page, because my pages are relatively easy to find. I don't know why people would search.

27:30Anyway, so cool. Thank you so much. That was really, really useful and insightful. And I thank you all out there for listening in to this episode. And of course, John, for joining me. Thanks for being here. Yeah, thanks for having me. So we hope that you all out there had fun and learned something today. And if you're interested in more content like this, then definitely subscribe to the podcast and leave us a like. And if you have to tell us something, then please do use the comments. We're looking forward to hear from you all. Also, what kind of topics you're interested in in future episodes, leave that in the comments.

28:04We are looking forward to hear that from you. And with that, I'd like to say thank you all for listening and goodbye. Bye.

28:14we've been having fun with these podcast episodes i hope you the listener have found them both entertaining and insightful too feel free to drop us a note on linkedin or chat with us at one of our next events we go to if you have any thoughts let us know and of course do not forget to like and subscribe thank you so much for listening and goodbye