Optimizing login-page content for Google Search

0:09Hello and welcome to a new episode of Search of the Record, a podcast coming to you from the Google Search team where we will talk all about search and maybe have some fun along the way. My name is Martin, and I am a developer advocate or search advocate at the search relations team here at Google. And with me is John. Hi, John. Hi, Martin. John, I have a question. Uh-oh. No, I read something and I thought about it, and I'm not sure what to think about it. So someone said online somewhere that they don't need to do SEO or they don't need to worry about SEO because what they do, like the stuff they have on their website is behind the login.

0:52And I'm not sure if that means that really they don't have to do SEO because I think they might still have to do some amount of SEO. What do you think? I think the real answer is it depends. All right. Okay. I'm so sorry. I'm going to step in here as Barry Schwartz. What does it depend on? I think if you really don't care about what is indexed, then do whatever you want kind of thing. Maybe something will be indexed. Maybe it won't. But I have zero care about what is visible in search. And nobody can access my content anyways. so it probably doesn't matter. If you care a little bit about what is visible in search, then maybe you should think about how you set things up.

1:46Okay. How would I know how to set it up if I want? I mean, I guess I want my website to show up in search somehow, but I guess I don't want to just show the login page, right? Yeah. I mean, there are a variety of different directions that you could go there. The variations I usually see are things like paywalled content, where basically you do want Google to index things, but the content itself might be behind a paywall or a login page or something like that. So when a user comes, they would see the kind of the interstitial to log in. We have a bit of documentation on how to set up paywalled content.

2:35So perhaps that's not really what the person that was asking you was asking about. Because it also sounds like they don't want to show the content to Google either, which is fine. Yeah. So with paywalled content, what usually happens is you try to recognize when Google is crawling, and you serve Google the content that you want to make available. And you add the paywalled structured data to the page to make it clear to Google that, hey, actually, this content is not available to everyone. There are some limitations. And that could be maybe you require a login. Maybe you require payment. Maybe after a certain number of iterations, you're like, oh, this is enough free content.

3:28Now you have to pay for it. Like there are lots of variations with regards to paywalled content. It also doesn't have to be something that's behind like a clear payment thing. It can just be something like a login or some other mechanism that basically limits the visibility of the content. Ah, so for instance, if I have to watch a video or click on an ad or something to get to the rest of the content, that's kind of also fine? I don't know about fine, but that kind of falls into the category of, well, there's something that needs to be done before this content is actually visible. And then you would use a paywall structure data.

4:10Okay. Also, if you have something like different thresholds where you say it's like some people get to view five pages for free and others have the whole content available for free because you're doing A-B testing, maybe about the prices or things like that. Then you'd want to use the paywall structured data just to make sure that when Google is looking at it, they realize that sometimes this content is not available. Okay. Got it. Got it. And the paywalled structured data helps us to understand that users might see something different. And that's totally fine. OK. I think there's one thing maybe to watch out for with paywalled content is that when a user looks at your page, you don't load the content into the HTML, but rather you make sure that it's really not loaded into the page's DOM.

5:05so that if a browser has something like a, what is it? The text reader, speech reader. Screen reader, yeah. Screen reader, that the screen reader doesn't go off and read all of this text that you're trying to hide. Those kind of things. So that would be kind of the thing that I would watch out for. That if it's really paywalled content or limited content, make sure you don't load it into the browser and use JavaScript to turn it on. but rather that it's really only served to the user when you want to make it available. Okay. But that's like one specific kind of content that you are hiding away or like making not immediately accessible.

5:48But what if I have, I don't know, a website where I share apartment ads, for instance, or like apartments to rent, and I want people to log in to see the apartment, how or to interact with the apartment or to apply for the apartment, how would I go about that? Is that just like, immediately show a login page? Or are there better ways of doing that? I guess the question would then be, do you want this content to be visible in Google or not? If it's visible in Google, then that would be kind of the model of paywalled content. If you don't want it visible in Google, maybe this is a private forum or a private community where you're sharing things.

6:36Or maybe you have something like, I don't know, a private service where people who have a subscription, they have access to this content, but it's not shown in Google or specific tools or something. like I don't know you have a spreadsheet that runs in a browser kind of thing where everyone has their own private content and they all have urls okay fine all right how are how are like bigger services doing it do they just show the do you know like if they just show the login page or do they how do they do this dude I don't think they use all paywall uh structured data now I I think if you're looking at a service like, I don't know, Search Console or Google Drive, where you have kind of this private content that is hosted online with a specific URL, then fundamentally, in order for someone to see that content, they have to log in.

7:41And usually, I guess it depends on how they set it up. But oftentimes, when you try to access a page like that, but it'll redirect you to a login page. And I think how they deal with the login page determines a little bit how things could potentially end up being indexed. So for example, with Search Console, one of the things that they do is that they have a set of marketing pages that are freely available. And if you try to access a Search Console URL directly without being logged in. It'll redirect you to a marketing page, which has a link that says you can sign in here to actually get the full information.

8:29That makes sense. And I think from an SEO point of view, that's fantastic because you search for Search Console and you can find these marketing pages. And if anyone accidentally links to their private Search Console URL, like, oh, I want to look at the performance report for my site and they share that with people, then that URL will redirect to the marketing page so that, on the one hand, users who find that link randomly, they end up on the marketing page. They know what it's about. And for search engines, they will find this marketing page and they'll be like, oh, okay, this has indexable content.

9:10We will just index this. Okay, so we're seeing it depends a little bit on what kind of content you're hiding. And like there are in between bits and pieces, like you don't have to just completely direct to a login page, I guess. Okay, interesting. I think one of the things we noticed over the years, specifically around login pages, is that if you have a very generic login page, we will see all of these URLs that show that login page, that redirect to that login page as being duplicates. Like if whenever you access a private URL, it just says username and password, then we will think all of these individual private URLs are actually the same.

9:55So we'll fold them together as duplicates and we'll focus on indexing the login page because that's kind of what you give us to index. And that means in the search results, that login page is going to be very popular because all of these random links, they keep redirecting to it, or they keep showing the same login page. So if someone is searching for your service and want to know more about your service, and the only thing or the primary thing they find in search is like, here is how to log in, that might be a kind of a weird experience for them. Okay, that is, yeah, I mean, yeah, that's great.

10:36So should they then, for instance, check if it's a legitimate Google bot and then just give like the actual content of the URL or at least like some sample content? Or how would you fix that specific problem where everything gets de-dubbed? I think if this is private content, you don't want to share that with Google bot. Well, okay, if it's private content, sure. Or if it's private content, you don't. But then how do you keep Googlebot from putting it in the index? You just put a no index on it or robot the URL away so that we are not even crawling it? What's the idea? JOHN MUELLER OK. So I think for these situations where you want to show a login page, it's good to have some context on the login page.

11:24So the Search Console model is basically or show a marketing page instead of the login page. But if you have a generic login page, put some information about what your service is on that login page, which could be enough to just have a sample of text like, oh, you're accessing Martin's furniture lookup site, or I don't know, some intranet thing where maybe some private content is. And then if you have some information on that login page, then we can index that. that information. And if you have different types of services that use the same login page, then those different services will have slightly adjusted login pages.

12:06So if you're searching for, I don't know, maybe we'll just stick with Google Drive. Like if you're searching for Google Docs, you'll find a login page maybe for Google Docs. If you're searching for Google Sheets, you'll find a login page maybe for Google Sheets. So having a little bit of information on there is important. The other thing you mentioned is whether all of this should just be blocked by robots.text, which is another common strategy for dealing with things that you don't want to have indexed. The problem, I think, with doing that is the URLs could become indexable. So we wouldn't see the contents of the login page, but rather we would just see, like, oh, it's like People are linking to this specific Google Doc, and we can't access it.

12:59But maybe we should show it in the search results if someone is searching for something similar. And also, this could be visible if someone does something like a site query for your site. And it's like, oh, tell me all of the URLs that are indexed for this hidden section of a website. And then Google and other search engines might be like, oh, it's like, I know about all of these URLs. I don't have any information on what's on there, but feel free to try them out, essentially, which is probably a bad idea. And if you have random hashes in the URL, so a collection of random characters, it's not a bad thing or not a terrible thing.

13:42But if you have things like usernames or email addresses in the URL, then, of course, all of those could become indexable. So if it's private content, serve it with a no index or redirect it to a login page somewhere. Don't use robots.txt. And optimally don't leak private details in URLs. Sure. Yeah, of course. Yeah, I think that's always a good practice. But sometimes you have things like you form submission parameters in a URL somewhere and it gets stuck there. Okay. Any other common problems you're seeing with login pages specifically or with content that is behind some sort of login? Yeah, I think the other question that I sometimes run across is whether or not the login page should be indexable by itself.

14:41And I think that depends a bit on the nature of the content that you have behind the login page. For example, if you have a kind of an intranet that is available publicly where your employees can only access it, then you probably don't need that login page index in search because your employees should be able to find the URLs for your private content on their own, hopefully. So in a case like that, you probably can just serve, I don't know, the login page with an error code or use server-side authentication or put a noindex on the login page so that if it does get found, then at least it won't be indexable like that.

15:30So that's, I think, one aspect as well, which I've seen in the past every now and then, that people's intranets end up getting indexed. There's a login form, but you probably don't want people to accidentally run across your intranet URLs. Now, I think those are kind of the primary aspects. And showing a login page is generally fine. Whether or not you redirect to a login page or show the login page directly, ultimately, I think is more a technical decision on your side. Sometimes there are security implications around that. Oh, security implications? Well, I think like cookies, for example, right?

16:14Ah, okay, fair. So maybe you have something like a login.yourdomain.com and like everything gets routed through there. Then you want to redirect to a login page there. That makes sense, yeah. Okay, yeah, sure. Okay, I see what you mean with security implications. Okay. And I think this is a problem pretty much for any site that has kind of private sections on the site, which are accessible through individual URLs. But definitely a problem for sites like Google Drive or all of the various Google services where you end up kind of like having a lot of content that is private to yourself, to the user, and where you have a lot of different login pages.

17:05And specifically, if you have multiple services that go through the same login page, then it's worthwhile to kind of think about how you actually want your service to be foundable in the search results. Yeah. And for the most part, you do want things findable. And if people link to something private, you do want something smart to happen there. So it's kind of good to think about how you should combine things. and we regularly see Google services getting this wrong or getting, I mean, not necessarily getting it wrong and that you can access the private content, but wrong in the sense that we index things that probably we shouldn't be indexing like that.

17:54And then all you get is a login page. Yeah, that's not great. Yeah, yeah. I think Search Console used to have that problem before they move to having the marketing pages as a redirect target, where you would search for Search Console and you would find someone's Search Console URL in the search results and its indexes like sign in here kind of thing, which is like, it's a login page. Of course, you can reach Search Console that way, but it's not really the best way to show Search Console in the search results. and because Google has so many different services and so many different teams working on these services, you invariably run across situations like that.

18:40I mean, for some of the services, it's also tricky. If I have a Google Doc that I make public, kind of like a non-website website, so to speak, and then it gets indexed and then it is visible and people actually can use the content and then I delete the file or if I make it private again, then it is indexed. It will take a while until it falls out of the index. So there will be, yeah, surprises, let's put it that way. MARK MANDELAVYSCHENKOVICHER I mean, surprises in the sense that if you're not prepared, sure. But I think it just makes it hard for search engines to go and actually index or find content on Google Docs, where it's like, oh, maybe there's something here.

19:23Maybe all of this is private. Probably it's private. But maybe I should check anyway. But, you know, I think the other aspect that's kind of interesting is that internally we don't give SEO advice on these kind of things. So every now and then someone with a public service will ping us internally at Google and be like, oh, how do I make sure my service is indexed properly? And essentially we have to point them at our public documentation. Maybe we'll point them at this podcast in the future. Yeah, but it's something that just comes up every now and then. I think larger websites, especially those that have private content, they probably have similar things.

20:08Even e-commerce sites where you have something like you can look up your account or the orders that you had in the past. They will have a specific URL and maybe someone will link to that and a search engine will try to index it. and how you handle that kind of depends on what is actually shown in the search results. Yeah. And whatever makes sense for a user who might land on that or want to land on that. Yeah, that makes sense. All right. Okay. Would you say there's something that people should do to make sure they are doing this right for them? Is there the top tip that you want everyone to take away from who has to do with logins?

20:53JOHN MUELLER - I think the most important part is that you understand how things are currently working for your site. So the way I would do that is I'd open an incognito window in a browser where you're basically not logged in to any of the services that you usually use, and then you search for something associated with your site. That could be like if the primary content is behind a login page, then you could search for your name, like the name of the service. So you could search for, I don't know, Search Console or Google Docs or something like that. And then you click on maybe the top couple of results to see what actually comes up there.

21:39And if the top result is something like a login page and there's no information on this page at all otherwise, then probably that's something that you can improve. Whereas if the top results are kind of reasonable marketing content for people who are not logged in yet, then that seems OK. And I think with regards to more specific sections of a site, that gets a little bit harder because you have to search for those parts specifically. You almost have to know that there is something that could be found. For example, on an e-commerce site, if you have a page that shows your orders, you could search for that URL pattern or specific words that might be on a page like that and see what comes up.

22:25And just kind of from there, while you're not logged in in an incognito window, see, is there actually reasonable content that comes up? Does it do a reasonable kind of a redirect to a login page or, I don't know, login experience if you want to add more content to those pages? Or is this kind of jarring for the user that is like, what am I doing here? And why did I end up on this page that's asking for a password now? I'm not trying to hack this website kind of thing. So that's kind of the direction I would go there. And it's like if you see that things are OK in the search results, then probably you're already doing things properly.

23:09If you see that things are not going OK, then I would recommend digging into those specific URLs, trying to figure out where are they coming from, what happens when you use Search Console's URL testing tool to look at those pages? Does it show like what you see or does it show something different? And based on that, you would try to make a plan for improving things. Okay, that sounds pretty good. And I think that's pretty actionable advice, especially with like checking how your service currently presents in search to someone who's not logged in is probably a very, very good first stage to make sure that you have a good customer experience in the end.

23:52So that makes sense. All right. I think that's pretty much that sorted. Do you have anything else you want to say about login pages? I could tell you more, but you have to log in first, Martin. How does that show up in the index? Give me some sample content first. I want to know if I want to... Is that a paywall? Do you want me to pay for this information? No, but you should subscribe to this podcast and then we'll tell you more. And that's actually free. So definitely do subscribe. Leave us a comment. Have you seen any services that are screwing this up in the search results or do you have more questions regarding login pages or paywalls?

24:40Let us know in the comments below and we'll probably be talking about these specific issues more in depth. You can also submit to the office hours as well if you have a specific question. But if it's a broader thing, then we might discuss it here in the podcast. Awesome. Well, John, it has been a pleasure. Thank you so, so much for being here. And I think I've never thought this much about login pages. I don't know. For me, they're always just like username, password, or email and password and then like a button and that's it. But yeah, there's more to it. Thanks a lot for joining me. Thanks to all the listeners out there.

25:16That's it for this episode. I do hope people enjoyed that a lot. John, if they want to talk to you. Where do you hang out online these days, behind or not behind a login? Where can people reach out to you? I don't know. Sometimes it's hard. I'm mostly active on Blue Sky nowadays, so people can drop me a note there or send me a private message if they log in. So that would potentially be a good place. Okay. So everyone, follow John on Blue Sky. and thanks a lot for listening. Please do like and subscribe if you enjoyed this episode and goodbye. Bye.

25:57We've been having fun with these podcast episodes and we hope that you, the listener, have found them both entertaining and insightful too. Feel free to drop us a note on LinkedIn or chat with us at one of the next events that we go to if you have any thoughts. And of course, don't forget to like and subscribe. Thank you and goodbye!