1. Podcasts
  2. Search Off the Record
  3. Do sitemaps still matter?
Search Off the Record

1 Oct 2026 · 26m · 9 chapters

Episode artwork

Do sitemaps still matter?

In short

Whether sitemaps still matter, what Google uses them for, and how XML sitemaps compare to RSS, HTML sitemaps, and LLMS.txt; also why Search Console may show “couldn’t fetch.”

Guests

Martin Splitt (Google Search Relations) and John Mueller (Google Search Relations). Mueller describes early involvement in creating a Windows-based sitemap generator that crawled sites and helped discover messy, parameter-heavy URLs.

Key claims

Sitemaps are XML URL lists (mainly URL + lastmod). Google dropped sitemap priority and changefreq as they were often unhelpful. Sitemaps can help discovery and timely recrawling for frequently changing content (e.g., news, ecommerce price/product updates). “Couldn’t fetch” can be due to host load or crawl-demand decisions, not necessarily invalid XML. Sitemap dates are used only if reasonable; canonical URLs should be listed.

Notable examples

News sitemaps limit to last 1,000 changed pages; RSS lists recently changed URLs (often capped) and can be submitted similarly; HTML sitemaps are for users and aren’t structured like XML; LLMS.txt is hopeful but not a substitute today.

Written by AI. May contain mistakes.

Chapters

0:45

The Historical Context of Sitemaps

Discussion on the origins and development of sitemaps in web technology.

“And with that, I'd like to say hello to my guests today.”
3:32

Understanding Sitemaps and Their Purpose

Exploration of what sitemaps are, their purpose, and how they assist search engines.

“But seriously, this must be like 20-ish years.”
6:15

Do All Websites Need Sitemaps?

An analysis of which types of websites benefit from having sitemaps.

“But this is not like a spam thing where our spam systems would say, this is a bad sitemap.”
7:22

The Importance of Sitemaps for News and E-commerce

Examining the specific advantages of sitemaps for content-rich websites like news and e-commerce.

“If the site stays small forever, that's also fine.”
8:34

Key Features and Best Practices for Sitemaps

Discussion of useful fields in sitemaps, including last modified dates and canonical URLs.

“And by the time we notice with normal crawling, maybe that's a bit late.”
11:23

Sitemaps vs. RSS Feeds

Comparison of sitemaps and RSS feeds in relation to website indexing and updates.

“I mean, if you ask those who made the RSS standard, they will say it's very different.”
14:00

Understanding Sitemaps and Their Importance

Learn about the different types of sitemaps, their uses, and how they interact with crawlers.

“that you probably shouldn't be doing with lots of URLs.”
19:23

Trends in Sitemap Technology and Crawling

Discuss the potential future of sitemaps with AI technologies and crawling behaviors.

“So we've talked about LLMS.txt for a few times, and I'm sure we're going to continue talking about it for a long time.”
24:19

Closing Remarks and Listener Engagement

Wrap up the episode with thanks, listener interaction, and a reminder to subscribe.

“And thanks a lot for running through the entire history of sitemaps, basically.”
A man walking through an airport, asking his phone a question

Ask about this episode

Get the gist, explore an idea or ask about something said.

Claude

On Search Off the Record, what are the key points of this episode?

Searching your listening

Transcript

Automatic transcript. May contain errors.

0:11Ahoi, grüzi, hello. My name is Martin Splitt from the Google Search Relations team, and I welcome you all to a new episode of Search of the Record, the podcast where we, the Google Search Relations team, are taking you behind the scenes of Google Search and hopefully have some fun along the way. Today's topic is sitemaps. We will talk a little bit about if you need one, what they are, what they aren't, the XML format, the RSS differences, HTML, maybe LLM's TXT, and who knows, maybe we'll look into some of your questions around sitemaps as well. And with that, I'd like to say hello to my guests today.

0:53Hi, John. Hi, Martin. How are you doing, John? How am I doing? I'm okay. I'm standing here talking into my microphone. I think that's okay. I think that's good. Yeah, I do the same. And I actually have a question for you. Okay. I get emails and people seem to be struggling with sitemaps. Okay. Yeah, they are seeing something in Search Console that makes them worried. And I wonder if we should clear these things up. Do you think so? Okay. Can we do that? I think we can. Awesome. Sitemaps goes a long way back. It was basically how I made my way to Google. What? If you believe it or not. What? How so?

1:40So at some point, I forgot who it was, but someone came up with this idea of sitemaps and they shared it with the folks at Microsoft and Yahoo at the time. And this came out as kind of this semi-standard thing that everyone agreed upon, or at least like three companies agreed upon. And right about that time, I started doing a little bit more on the web. So I created a Windows-based sitemaps generator. And it basically crawls your website like a normal crawler and finds all the URLs and puts them in a sitemap file properly, which at the time was kind of unique in that a lot of people didn't crawl their own website because they had no idea they had to do that.

2:30So it ended up being, I think, reasonably popular, not like super, super popular kind of thing. But it's basically something where I got involved a little bit more with the SEO community, the early folks in SEO at the time, and with general people who are making websites, and started helping them out with regards to all of these questions that came up from sitemaps. Because if you never crawl your website and you have this interesting PHP or whatever-based website, then you end up with a lot of URLs that can be found, which don't make a lot of sense. So this was before something like WordPress or any of the mainstream CMSs were kind of clean.

3:16It's like lots of weird loops where crawlers found lots of parameters and lots of stuff to clean up. So that was kind of interesting. And then someone from Google noticed and we got into talking and the rest is history. How long ago is this? 500 years ago. Yeah. But seriously, this must be like 20-ish years. I think that's how old sitemaps are, no? I don't know. I'm afraid to look. Okay. I mean, you just joined Google yesterday. So yes, of course, all of this is fresh and new. Okay. So sitemaps are XML files. And search engines use them for what exactly? The goal at the time, I think, was just to discover all of the URLs on your website.

4:05Where I think, like I mentioned in the beginning, websites were really messy and hard to crawl. So if the site owner gave you a list of URLs and told you, like, these changed on this day and these have different priorities and whatever, then that's a lot easier for a search engine to say, like, OK, I will just focus on these. I will still crawl to make sure I don't miss anything, but I will try to focus on these. Okay, but with all the changes and all the improvements and all the developments we've done in crawling, do we still need them? Can't we just figure this out? Can't we just use AI? Oh, yeah, or use an AI because, you know, it's 2026.

4:44Let's use an AI. I think so. So over the years, some things went away with sitemaps. Oh, my gosh, this is going to be a very nostalgic episode. I didn't realize. Some things went away. So there used to be a priority field in sitemaps. And you can imagine if you tell SEOs and ask them, it's like, what is your priority for your pages? Everything. Everything is number one. Yes, of course. Every URL is a maximum priority, yeah. That ended up not being very useful. There was also a field for change frequency. Like, how often does this page change? And what do you think SEOs did? It changes all the time.

5:25Changes all the time. Super important. Always fresh. Always fresh. Always fresh. I mean, partially, I think it's not because they're trying to be misleading, but it's just if you have a server-side dynamic website, the PHP file that runs is going to be fresh all the time anyway. So theoretically, it's correct, but it's not very useful. I think those were the two fields that we dropped. Maybe there are some other fields that I forgot in the meantime. But basically, what we primarily focus on is the URL and the date, the change date. And a change date should be like when you last made major changes on this page.

6:08If you basically give us today as a change date for all of your URLs, we will realize that it doesn't make much sense to focus on that. But this is not like a spam thing where our spam systems would say, this is a bad sitemap. Okay. And do all of the websites need a sitemap still? Or is it more useful for some types of websites and not so useful for others? I think smaller websites probably don't need a sitemap because we can just crawl them. but I know it's really hard to make this threshold decision as a site owner. You're like, oh, I have 10 pages. That's small. I have 50 pages. Is that big?

6:54Is that small? Is that medium? I don't know. So my recommendation usually is just turn it on because in the meantime, almost all content management systems, even the static hosting systems, they generate sitemaps files by default unless you turn them off. So I would generally recommend just keeping it turned on. Even if you have a small site, if the site grows over time, it's fine. If the site stays small forever, that's also fine. There's no harm in having a sitemap file if your site is small. Do websites with a lot of content that changes a lot, like let's say news websites, do they benefit from a sitemap specifically?

7:38or is it just like any other website? Definitely. Definitely news websites. There's even a special kind of sitemap for news sites specifically where I think you're only supposed to list the last 1 ,000 pages that changed on a website so that it's easier for systems who want to keep up with a news website to actually keep up. I think it's news sitemap. I haven't looked at it in a while, but basically for any website that has a non-trivial amount of content, and if that content changes regularly, the site map file is super helpful. Because maybe, like in some category section, you change something on your site.

8:24With traditional crawling, we would have to crawl all the way and kind of find that page and be like, oh, look, I just looked at this page again and it's new. If you have an e-commerce site, that's pretty common, like the price change. And by the time we notice with normal crawling, maybe that's a bit late. Whereas if you can tell us, hey, this product page changed, it's a lot easier to go there directly. And we've talked about some things that we are not using anymore, like change frequency and priority. But there's also last mod. So you can tell when the URL last changed. Is that still? That's a date.

9:04Yeah. Yeah. Is that still something that we use, or should I ignore that as well? My understanding is we want to use it, but a lot of sites get it wrong. So we have this weird love-hate relationship, I think, with it. In that, again, my understanding, maybe this has changed in the meantime, my understanding is that if we can tell that within a sitemap file, the dates are kind of reasonable, then we will try to take that into account. Whereas if we can't tell that the dates are reasonable, then we will focus more on the individual URLs. And then we'll be like, oh, a new URL popped up. We will process it.

9:46MARK MANDELAVYSCHENKOVICHER But I could use timestamped URLs in the sitemaps file to make sure that, no? JOHN MUELLER, so in the sitemap file, you're supposed to. No, supposed to. It's not like a law. There's no police. Sitemaps police. You get a ticket. The idea is that you include the URLs that you want to be canonical. So if you have URLs that are tagged with specific things that you use internally for tracking or whatever, you would kind of submit the URLs in the sitemap file the way that you want to have them indexed, not the way that they're kind of like temporarily on your website. So if you have a, I don't know, a static looking URL and you add a date stamp to it because you want to make sure that the newest version is always used, then we will try to index the non-date stamp version as a canonical.

10:44So if you can make sure that canonical version is in a sitemap file, we can take that into account. My understanding is we also use sitemap files with regards to choosing the canonical. So if we kind of have a variety of different URLs that we discover, and they're all kind of similar, but if you list one of them in a sitemap file, it's a little bit more likely that we will pick that as canonical. Okay, that doesn't sound too bad. Yeah. But if I want to make sure, like if I have a blog, sometimes some CMS has created an RSS sheet. Is that kind of the same as a sitemap, or is it different? It's very similar.

11:30I mean, if you ask those who made the RSS standard, they will say it's very different. But with regards to focusing purely on the URL and maybe a date, it's very similar in that the RSS feed will list the URLs that recently changed. And often it has a cap, maybe, I don't know, 10 or 20 of the last URLs that changed. It includes a date stamp of this page, which I don't know exactly how it's stored in RSS, probably some obscure standard, but it has a date and a time associated with it. So if we look at the RSS file, we can find those URLs, we can find those dates as well. So in a sense, an RSS file can be used as a sitemap file.

12:20You can also submit it in Search Search Console as a sitemap, it's very similar. The main difference is that it focuses on the last URLs there. So if you have, I don't know, 1 ,000 sitemap files because they're kind of limited in size, but you have a lot of pages, then a system would have to look at those 1 ,000 sitemap files first to find out what has changed. Or if it has the RSS feed, it can just look at the RSS feed and be like, oh, these 50 pages have changed recently. I can focus on those. So they both have usefulness with regards to finding new and updated pages. OK, but wait. So sitemap files do have a limit as to how much they can contain.

13:05Yes. What's the limit? Oh, my gosh. Ah, I know. I'm testing you. I should look it up before telling you. But my memory is 50 ,000 URLs and 50 megabytes. Yes. So something like that. And the 50 megabytes is non-compressed. You can gzip sitemap files as well. But you can't target a gzip size. So the size limit is based on the non-compressed site. You can submit as many sitemap files as you want in Search Console. You can link them in your robots.txt file. You can also create a so-called sitemap index file, which is a kind of a sitemap file that links to other sitemap files. You can only nest them once, but you can submit as many sitemap index files as you want.

13:57So you can kind of do crazy stuff that you probably shouldn't be doing with lots of URLs. Yeah, I think if you're at the point where you have multiple sitemap index files because they get too large, then you definitely have enough URLs to warn the sitemap in the first place. Yeah. Yeah. Now, maybe one thing that's also kind of unique with sitemap files is you can submit them with robots.txt. You can tell everyone basically where they are. You can just call them sitemap.xml, and a lot of systems will guess and take a look. Or you can call them anything. You can call them martinscat.xml. You probably don't even need to use.xml, but that's pretty common for XML files.

14:42If you call it something unique and you don't list it in robots.txt, then you need to tell us about it. And you can do that in Search Console and say, like, here's my sitemap file. So that's kind of helpful. RSS is a bit different in that usually RSS is linked from the head of HTML pages. So if you know an HTML page, you can find the RSS feed, usually, or in many cases, which means that connection is a little bit clearer. Some people don't want their sitemap files to be known to other people, and that's perfectly fine. That means you would use a different file name, for example, and you would tell Google about it directly.

15:28The downside is that other systems that might want to use sitemap files wouldn't be able to find it. That means if you're using Bing and you assumed that Bing would figure it out too, you probably have to tell Bing as well. That also means if you care about AI training crawlers, if you want to make sure that your content is in these AI systems, then they usually don't have any kind of a console or any setup where you can submit a sitemap file. So either you stick to kind of the generic naming, call it sitemap.xml, or you focus on RSS feeds, and then these AI crawlers can sometimes find those files and process them.

16:15I don't know if they document this anywhere, but I've seen that happen in my server logs where some AI crawler accesses my sitemap file and they like it. I don't know. I mean, I don't know what they do with it, But they read it. They do something with it. I've also seen that for my RSS files, which, again, is something that most websites have. So it's easy for these AI crawlers to notice and be like, oh, here's some more content that I could use for AI training. And that's perhaps something that the site owner wants. OK. But do I need to use an XML-based format? Because I remember there were HTML sitemaps.

16:59at some point. Oh, yes. HTML sitemaps. That, I think, is very confusing to the average person because you probably know what XML is and what HTML is. But a lot of people, when they open an XML file, they see those weird brackets and they're like, oh, this is HTML or very similar, right? I mean, yeah, sure. It is very similar. It looks like this weird programming stuff. And they're called the same. But an HTML sitemap is basically almost like a map of your website for users. It's not something that replaces an XML sitemap file. OK. So, of course, if someone is crawling your website and they find an HTML sitemap file, they can crawl those links like any other link on a website.

17:55So that's helpful. but it's not the case that the HTML sitemap can be submitted as a sitemap file. It can't be processed in one-to-one the same way as a kind of XML sitemap file because it doesn't have this strict structure to it. It's basically a collection of links, and collection of links can be any kind of thing. But that still would help crawling because from this place we can find all the links as well. It's just not as structured and it's not as clear and clean cut as an XML sitemap, I guess. Yes, absolutely. And it's something I think that a lot of HTML sitemaps, they don't include everything.

18:45So if you have an e-commerce site, you wouldn't list all of your products in an HTML sitemap file. you would list maybe the categories so that people can go to the right place, but then pick out the product individually. So it's similar, but not exactly. Would that need for such a structured sitemap as an XML file, would that go away with, let's say, like inclusion of an LLM's TXT? Oh. Yeah. There we have it. Ha ha ha ha ha. LLMS.txt, the new sitemap file for AI crawlers. So we've talked about LLMS.txt for a few times, and I'm sure we're going to continue talking about it for a long time. Basically, LLMS.txt is a markdown file, which sometimes includes links to different pieces of content.

19:41And it's almost like an HTML sitemap, but the hope is that an AI system uses it to understand your website a little bit better. I think the hope is bigger than the reality but maybe that will change at some point, who knows? But it's basically a text file that has various links to parts of your website which is kind of similar to an HTML sitemap. It probably wouldn't include all of the pages on your website but rather more the categories. And I think, from a functional point of view, you can't use those as a sitemap file in our systems because it's just not the strict format. But it could be at some point that our systems or search systems in general process markdown files, and maybe we will also find those links there and then be able to follow them.

20:37So I would consider it more as, on the one hand, it's hopeful, like currently none of this happens, But it could be the case in the future that it's more like an HTML sitemap file, which has a lot of links, but it's not meant to be comprehensive. All right. So it's not going away in favor of LLM's TXT, I think. I mean, I don't think so. We don't know. But who knows what the future brings. Like next week, the sitemaps team will send us an email and be like, hey, LLM's text is a new sitemap. I don't know. I think from a practical point of view, for anyone who's making websites, you should not kind of focus your energy on this hopeful state that maybe in the future it will be different.

21:28You should focus on what actually works now, what is documented now. So it's fine if you want to play around or if your CMS makes LLM's text files automatically, but I would not rely on it. I would not treat that as a priority. I would kind of see it as like, well, who knows what will happen in five years or whatever. I don't know. Internet time is weird. Who knows what will happen? But it's also something where I think the usefulness of sitemap files is not going to go away completely just because of that comprehensive nature of these sitemap files. That makes sense. All right. Now, to the question from the beginning, why does Search Console sometimes show couldn't fetch, even though the sitemap XML is valid, because you can validate that.

22:17That's the nice thing about such a structured format. So it's valid, it's publicly accessible, and it's linked from the robots.txt, and yet Search Console says can't fetch sometimes. Can't fetch, yes. Yes. We actually see this a lot in the forums. And there are primarily two reasons for that. One could be with regards to host load. We've talked about that in the past, like how much Google systems are able to crawl from a website. And it could be the case that we don't have any time to crawl the site file because we're too busy with other things. That can happen. Then we would also positioned that as couldn't fetch because we didn't have time to actually fetch it.

23:06The other, also related to host load, is kind of the crawl demand side, where if our system say, we don't actually have any need to crawl a lot from this website, we're just going to skip the sitemap file because we got enough already. And the crawl demand is very often based on the perceived quality of a website. And that can have a really large impact on how much we crawl and index from a website. So it's not purely a technical thing. Sometimes it's that our systems assume that the overall quality of this website is not fantastic. Therefore, we're not going to spend a lot of time crawling and indexing the content.

23:46Therefore, we're not going to bother with the sitemap file at the moment. And if we see over time that the quality of the website improves significantly, then yes, we will go off and kind of use that sitemap file. but maybe we just don't want to. Okay, so it is not just a technical thing from my perspective or my site's perspective. It's also, do we actually need the sitemap file and do we actually want the sitemap file as well? All right. Definitely. I think that answers that question. And thanks a lot for running through the entire history of sitemaps, basically. I think there's one thing that we didn't mention and that's you can also do hreflang in it, right?

24:33We had an hreflang episode at some point. I think that's also an aspect, but I don't think, I'm wondering actually how many people are using that. I would love to hear some comments from our listeners out there if they're using sitemaps to also link to different language versions or if they are using other mechanisms to do so. Cool. Yeah. There's also image and video sitemap files that you can do for the images and videos. I don't know how important those are nowadays because it's easier for us to recognize images and videos on pages in the meantime. But it is something that is still supported.

25:12So you can do that. Nice. All right. So we talked about HTML sitemaps, XML sitemaps. We talked about the difference between sitemaps and RSS feeds. And we talked a little bit about this funky error in Search Console. So, okay, I think my sitemap needs are met. Cool. And with that, I'd like to say thank you so, so much, John, for being here with me and talking about sitemaps. And also thanks a lot for all of you who are listening. Let us know in the comments below if you have any other questions regarding sitemaps or other formats. And just leave us a like and subscribe to the podcast if you want to hear more from us.

25:51Thank you so much and have a great day. Goodbye. Bye.

25:57We've been having fun with these podcast episodes. I hope you, the listener, have found them both entertaining and insightful too. Feel free to drop us a note on LinkedIn or chat with us at one of our next events we go to. If you have any thoughts, let us know. And of course, do not forget to like and subscribe. Thank you so much for listening and goodbye.

From the publisher

Sitemaps have been a core element of web development and search engine optimization for two decades, yet Search Console warnings like "Couldn't fetch" continue to generate concern among site owners. Are XML sitemaps still necessary with modern search crawlers, or can search engines and AI bots figure out site structure on their own? In this episode of Search off the Record, Google Search Relations team members Martin Splitt and John Mueller break down the history, technical specifications, and some misconceptions surrounding sitemaps, RSS feeds, LLMs.txt, canonicalization signals, and Search Console indexing behaviors.

In this episode, you'll learn:

  • The Origins of Sitemaps: How early web crawling challenges led to the sitemap standard and how building a sitemap generator started John Mueller's career at Google.

  • Obsolete vs. Active XML Fields: Why Google dropped support for the priority and changefreq tags, and how the lastmod date is actually evaluated.

  • XML Sitemaps vs. RSS Feeds: How RSS feeds function as lightweight sitemaps for recent updates and why AI training crawlers utilize both.

  • Technical Limits & Index Files: The exact URL (50,000) and file size (50MB uncompressed) limits, alongside best practices for sitemap index files and robots.txt declarations.

  • HTML Sitemaps & LLMs.txt: Why user-facing HTML maps and Markdown-based LLMs.txt files cannot replace structured XML sitemaps for search engines.

  • Troubleshooting "Couldn't Fetch": How host load management and perceived site quality (crawl demand) trigger non-technical fetching errors in Search Console.

Key takeaways for SEOs & developers:

  • Keep Sitemaps Enabled by Default: Even for small websites or dynamic CMS setups, keeping auto-generated sitemaps active carries no downside and assists with discovery as sites grow.

  • Provide Accurate lastmod Timestamps: Avoid setting all page timestamps to the current date; Google evaluates date reliability and will ignore lastmod signals if they are inaccurate or abused.

  • Submit Canonical URLs Only: Always place clean, canonical versions of URLs inside your sitemap file rather than parameter-tagged tracking URLs to help guide Google's canonical selection.

  • "Couldn't Fetch" Isn't Always a Syntax Error: If Search Console reports "Couldn't fetch" on a valid, accessible XML file, the cause is often host load throttling or low crawl demand linked to perceived site quality.

 

Chapters

  • 00:00 – Introduction & Welcome

  • 00:57 – Sitemaps History & John Mueller's Path to Google

  • 03:42 – Why Search Engines Started Using XML Sitemaps

  • 04:46 – Deprecated Fields: Priority, Change Frequency, and lastmod

  • 06:36 – Sitemaps for Small Sites vs. High-Volume News Outlets

  • 08:50 – Canonical Selection & Parameterized URLs in Sitemaps

  • 11:13 – Comparing RSS Feeds and XML Sitemaps

  • 12:58 – Size Limits, Compression, Index Files, and robots.txt

  • 14:15 – Custom File Naming, Privacy, and AI Crawlers

  • 16:54 – Why HTML Sitemaps Don't Replace XML Sitemaps

  • 19:06 – Will LLMs.txt Replace Sitemaps for AI?

  • 22:05 – Decoding Search Console's "Couldn't Fetch" Error

  • 24:16 – Hreflang, Image, and Video Sitemap Extensions

  • 25:14 – Key Takeaways & Outro

Resources Mentioned:

 

Don't forget to like and subscribe to the podcast on your favorite platform to catch every behind-the-scenes episode from the Search Relations team!

Episode transcript →  https://goo.gle/sotr114-transcript

Listen to more Search Off the Record → https://goo.gle/sotr-yt  

Subscribe to Google Search Channel → https://goo.gle/SearchCentral

Search Off the Record is a podcast series that takes you behind the scenes of Google Search with the Search Relations team.

 #SOTRpodcast #SEO #GoogleSearch #SearchConsol#SEO #WebDevelopment #GoogleSearchConsole #Sitemaps #SearchEngineOptimization #GoogleSearchRelations #LLMstxt #TechnicalSEO #SearchOffTheRecord

Speakers: Martin Splitt, John Mueller

More from Search Off the Record

All episodes