Are websites getting "fat"? Page weight, HTML size & Googlebot limits explained

13:36Martin Splitt:Wild. I mean, different kind of compression.

13:39Gary Illyes:Yeah, yeah, yeah. I think it's interesting to see, like, the PDF version of web pages oftentimes are either a lot smaller or a lot larger. and this one is like, you can tell like this is really dense content.

13:52Martin Splitt:Do you think stuff like the reader mode alleviate anything on the user's side?

13:58Gary Illyes:Ooh, I don't really know what the reader mode, I know what it does from a user's perspective, but I don't know what it does in the background.

14:05Martin Splitt:I was thinking about it a little bit a while ago and my conclusion was that it doesn't change much because like from a loading perspective at least because you still have to first get the resources and then the phone or the app does some magic locally to remove fluff and present you with a more readable version of the page, but at least makes the internet kind of sort of more bearable to enjoy. Wow, I used two weird words together. But like trying to think about it from like holistically, do you think that this even matters? like what the weight of a web page or website is, which we still haven't clarified, by the way.

14:49Martin Splitt:Correct, yes. Because, like, for example, for me at home, I have a 10 gigabit connection. Yeah. Right? I have a 10 gigabit fiber here as well. Yeah. And I don't feel the pain. Like, for example, if something is, I don't know, what did you say, like 96 megabytes or whatever, for a PDF file, it will still come down in... relatively short period of time. And at that point, it's dependent on the server that's sending the stuff, not on my connection. So to me, it really doesn't matter that websites are getting bloated. On my phone also, I have a permanent 5G from Swisscom, my phone company. So again, it is super fast.

15:33Martin Splitt:It really doesn't matter. It just comes down like that in a snap.

15:38Gary Illyes:Ah, that's nice. Ah, sweet summer child. I'm so happy for you. Yeah, right.

15:43Martin Splitt:But like I live in a bubble, right? Yes. But I do travel a lot. And then if I go to less developed countries, for example, then suddenly it becomes problematic because like the cell phone towers would advertise that they have 5G, but in reality they have like a really weird, I don't know, 3G or something. Not even 3G. Yeah, I guess it depends on where you are living and what kind of connection you have as well.

16:08Gary Illyes:It's wild. But I remember when we went diving in Antarctica, we were on the ship where we had a metered satellite connection. And I believe, if I'm not mistaken, what did I pay for? 100 megabytes. 100 megabytes, which, again, the spec for HTML is 15. Yeah. So just to keep that in mind, I think like$20 or something. Yeah, yeah. So it was expensive. Yeah. And that made me think about things. Like I activated data saver mode on my phone. I set the connection to metered. I did not open things like most of the social media apps. I just never opened because they would suck this 100 megabyte dry in minutes.

16:50Gary Illyes:So I think it still matters. But it begs the question, at what level is what kind of weight acceptable? Right. And I don't have a good answer to that. And then on Blue Sky, I got into this conversation about HTML size. So in this case, there's a document that is 15 megabytes. And we both agree that pretty much most of these 15 megabytes are actually useful content. Because there is not much in terms of images or style sheets or whatever. It's just the pure HTML and it's minimal markup around maximum content. And then people were like, yeah, well, in that case, it's kind of okay to have 15 megabytes of HTML that you need to deal with.

17:31Gary Illyes:And we both know that our browsers, even on our relatively latest technology kind of laptops and fast internet connections, take a while to process that 15 megabytes of HTML. Actually, I wonder how much goes over the wire, because I'm pretty sure there's compression involved. Oh, God. Okay. I tried to also open this page in reader mode, and that kind of crashed the browser tab. Great. So in this case, it's kind of acceptable. but then what if the markup is only overhead and i mean like what do you mean it's like well you know if it's like five megabytes but it's only very little content is that bad is that worse as in this case the 15 megabytes and i'm like that's tricky because then we come into this weird territory of the ratio between content and markup yeah and i said well but what if a lot of it is markup that is metadata for some third-party tool or for some service or for regulatory reasons or licensing reasons or whatever, then that's useful content, but not necessarily for the end user, but you still kind of have to have it.

18:36Gary Illyes:It would be weird to say that that is worse than the page where the weight is mostly content.

18:43Martin Splitt:You know, this is interesting because historically I have some beef with structured data, for example. because, I mean, this is a very long thing or long-running thing because at one point early in Google's life, Sergey Brin, one of our funders, said that computers should be able to figure out anything from the text that they receive so site owners slash back then webmasters wouldn't need to provide anything extra and the context was something around spam. Anyway, and then StructureData comes in And structured data is specifically not for users. It is specifically for machines. And depending how much structured data you add to a page, it can increase the bloat considerably.

19:32Martin Splitt:Like if you look at in our documentation, like what kind of structured data Google supports, it's a lot. Yeah. Like you can fill a page easily with just the links to the documentation. And then if you keep piling on structured data on your pages, basically stuff that users will never actually see, then what? Like, is that good? Is that bad? For the site, it's probably good, or it can be good, because it can bring in additional traffic through whatever magical stuff search engines might be able to show based on the structured data. But for the user, I don't know. Yeah. Well, this begs the question, and it came up in discussions at IETF, IETF being the Internet Engineering Task Force, basically who create the standards for the internet, whether we should think about separating, like holistically in general, separating metadata stuff from user visible content.

20:33Martin Splitt:Like what if we would have something like for clients, HTTP clients that you trust, you would expose an API sort of endpoint on your site where you just give the metadata that they would want, right?

20:51Gary Illyes:Yeah.

20:53Martin Splitt:Like in an ideal world, that is not the most insane idea. Like for example, if you had the HTML standard, like all the text for an LLM is irrelevant. Like it will process the stuff differently than you and I. True. Wait, can you read? I can read sometimes. You can read, right? Okay, just checking. And you could have something like append parameters, like question mark output equals JSON-LD, let's say. Yeah. And then if the client is trusted, say through cryptographic authentication or something like that, then you would serve them the machine, purely machine-readable version, even in binary format, doesn't matter.

21:40Martin Splitt:But then, unfortunately, this is a utopistic thing because not everyone on the internet is playing nice. That's true. We know how much spam we have to deal with. On our blog, we say somewhere that we catch like 40 billion URLs per day that's spam or some insane number. I don't remember exactly, but it's some insane number and definitely billions. will that just exuberate the amount of spam that search engines receive and other machines receive? Maybe. Like I would bet$1.05 that will actually increase the amount of spam that search engines and LLMs and others ingest. So, yeah.

22:31Gary Illyes:Yeah. I think that's an interesting, like there was this principle that was quite popular in the early 2000s, early 2010s, hypermedia as the driver of application state or something like that, where the idea is to kind of like have separate URLs with different representations, depending on who consumes it. Yeah. And I think generally that makes sense and you can do this to whatever degree, but I'm not seeing it happening as much anymore.

23:00Martin Splitt:I don't know how much sense it makes because one of the things that we had when, what is it called? Mobile first indexing. One of the things that came out of that was lots of documentation based on stuff that we observed when we were trying to switch over sites. And one of the, actually, do we still have the documentation for mobile first indexing? That might actually have this still if it's there. But one of the things that we noticed is that there is no parity between the mobile version of a website and the desktop version of a website if there's different URLs for the two. So mobile would be like martinsplit.com slash m slash content.

23:42Martin Splitt:And then the desktop version would be martinsplit.com slash content. And for a very large number of pages that we observed, there was no parity between the content or no exact parity. Content was missing. That was the worst. Because then you're not going to rank for the stuff that your desktop version was ranking before. and then we had outreach campaigns about it. But then links were also missing. To me, that's harder to explain. That's bad. And the navigation was different. The whatever was different. Metadata was missing. Ahreflang was missing. Link elements were missing. And yeah, it was a pain in the...

24:23Martin Splitt:Lower back. Lower back. Thank you. Otherwise, Lisa, our producer, makes me re-record stuff. And yeah, so even for trivial stuff like a mobile version and the desktop version of a web page serve under different URLs, even for trivial stuff like that, there can be large differences. Yeah. So wait, I want to go back to the first question. Okay. The question that got us here, basically in this really, really deep and dirty, dirty hole. Are websites getting fat?

24:59Gary Illyes:I think if we rephrase it to our web pages getting larger, I think that is true. Yeah. The answer is yes. Is that a problem? I think that depends on the context because we identified that some people are constrained by either slow internet connections and that can hit us here in Switzerland as well. I know in some parking garage, I have very bad reception.

25:27Martin Splitt:Oh, yeah. You are in an atomic bunker and then you don't have reception. It's like, oh, I will cry your river, Martin.

25:33Gary Illyes:Correct. I was very surprised and not very happy with it. Correct.

25:39Gary Illyes:So internet connection speed is still playing a role. And I think in the last couple of years, the increase in transferred byte size has outgrown the increase in transfer speeds. if you look at the median transfer speed increase for mobile connections.

25:58Martin Splitt:Oh. I think so.

25:59Gary Illyes:As a hypothesis, I would have to check the data, but I feel like that's the case. The other thing is there are mechanisms to alleviate this, like caching. So of course, you get the full painful hit in the first place, the first time you visit the website. But for instance, the HTML specification uses caching. So when I reload, I don't have to download all the 15 megabytes again. Yeah. compression on the transport level, on the network level, helps as well because then you have less data that you need to transfer. You still have as much data to store on the user's device or on the crawler's end.

26:36Gary Illyes:So yeah, that's only helping you so far, but there are ways to make this less painful. There's also this lazy loading thing where you can say like, okay, so there's a lot of heavy content, like lots of images, for instance. We're only loading those that you actually scroll to or go to and possibly see rather than loading all of them in the first place. But I still think it's worthwhile thinking about how much data are we throwing around. Yeah.

27:03Martin Splitt:But the website versus web page distinction, do you think or do you agree with me now that the original question is meaningless? Yes. Okay. That's good.

27:16Gary Illyes:You caught me there. That's a very good catch.

27:19Martin Splitt:Okay. Well, it's Friday today, so this just made my day because someone finally agreed with me. Do you think that we need to do anything to reduce the size of pages on the internet?

27:33Gary Illyes:That's a tricky one.

27:35Martin Splitt:And I mean you and I as people who like the internet and work on standards and stuff.

27:43Gary Illyes:I think, yes. I think we are wasting a lot of resources. and I mean we had that in another episode where we said that we know that there are studies that show that websites that are faster have better retention and better conversion rates. And speed is in part also based on size because the more data I ship, the longer it takes for the network to actually transfer that data and the longer it takes for the processor of whatever device you're on to actually process it and display it to you. So and again like if storage is a problem, then you're excluding some people there as well. I think yes. I think that makes sense.

28:20Gary Illyes:I also think that not everyone is, and I myself know this, I'm now building like a new website and I'm looking specifically for a workflow that makes it easier for me to deal with images, for instance, because so far I've just been like, here is this 59 megapixel image. I know you're on a smartphone, but I don't care. Like here is the five megabyte image. Good luck.

28:44Martin Splitt:which is not the right way to do it. May the odds ever be in your favor. Yeah, exactly. You know, for that specifically, internally we have a linter that prevents submission to the developer sites that we are using for the search documentation if the image is over one megabyte.

29:01Gary Illyes:Yeah. Interesting. Arbitrary limit, in my opinion, but yeah. Well, I like to think

29:07Martin Splitt:that they came up with that number using some methodology, not just like I woke up and chose violence. Yes. And like I was thinking about it a lot because it's like, should I ask for an exception? Because our images often are like seven megabytes or so. And then I was looking at the compressed versus the less compressed versions of the images. And my non blue eyes could barely tell the difference. I know what you mean. Yeah. And at that point on a big screen and at that point, does it matter? that we lost some pixels or some shades in pixels. Yeah. Because our images are not fine art, right? No.

29:49Martin Splitt:For fine art, sure. So what's our takeaways? Are websites slash webpages getting fat? Are they? Yes. I think pages, yes. Connections are getting faster. So does it matter? I think it still does. Okay. How do we fix it? Maybe we talk about it in another episode. Yeah, sounds like a plan. Because there's a lot that you can do. And I'm pretty sure that every little bit helps. Not just with search engines, and we shall talk about that a little bit later, but also with your users. Yes. Because users definitely do like snappy websites, and bloaty pages don't help with that. agreed?

30:41Gary Illyes:I agree.

30:42Martin Splitt:Your honor. You have to say your honor. I agree your honor. Thank you. Objection.

30:49Gary Illyes:Objection. I'm not honorable. All right.

30:53Martin Splitt:Well, Martin, thank you for the chat. Thanks a lot for you. This was our second chat today because we recorded two episodes. And I will leave you for two days, three days because it's the weekend. And on Monday I'm not going to the office because I don't want to. Yeah, especially if there's still snow. So I would like you to remember this face and this voice and have nightmares, please. I thank you for the lovely conversation. But thank you nonetheless for chatting with me. Thanks. If you like this episode, please like and subscribe wherever you get your podcasts. And have a very nice day, please.

31:38Martin Splitt:And thank Thank you.

31:39Gary Illyes:Bye bye.

31:43Martin Splitt:We've been having fun with these podcast episodes. I hope you, the listener, have found them both entertaining and insightful too. Feel free to drop us a note on Twitter at Google search C or chat with us at one of the next events we go to if you have any thoughts. And of course, don't forget to like and subscribe. Thank you and goodbye.

32:09you