ChatGPT Labrador Index, SERP Poisoning, Parametric Knowledge, Fake LLM Bots | #MonthlyLandwehr

0:00Keyword tracking will never be as cheap and as reliable as it used to be. And I also do not see this improving going forward. I would definitely not measure the success of my SEO campaigns with keyword tracking. Most people should focus on the web grounding process. Is this just yet another sign for people to question their legacy approaches and how they approach SEO? for example, overly relying on rank tracking certain keyword sets? I would agree. Keywords are still a good proxy for demand. They can be a good proxy for a topic or an intent. But if your strategy revolves around ranking for specific keywords, that was probably an outdated approach already and is not how I would think about findability in the future.

0:50Before we dive in, you're listening to the Masters of Search podcast with me, your host, Niklas Buschner. Each week, I sit down with some of the smartest people around the world in SEO and AI search to bring you their strategies, mental models, and top pieces of actionable advice. If you enjoy this podcast, don't forget to like and subscribe and follow it in your favorite podcasting app or YouTube. It helps us get top-notch guests and create the best possible content for you. Let's dive into today's episode. Welcome to the Masters of Search podcast and welcome to my favorite format of the show, the monthly Landwehr.

1:29Malte is already smiling. We are back from our summer break. That was a little longer, but we took the time to recharge, etc. so now we're back with a lot of fresh energy and a lot of fresh insights honestly uh to to kick it off slowly malte how was your summer um busy uh worked a lot um went to a lot of events and conferences traveled a lot um and uh yeah now looking forward to the end of the year that's actually not gonna slow down at all but will at least involve a little bit less travel yeah Yeah. I feel like if PKI is not the leading AI search visibility platform or also now a platform to act on your AI search insights, you are definitely a LinkedIn influencer and a conference people group.

2:29So I see so many peak people being at conferences around the world and people being active on LinkedIn. I mean, this is not 100 % related to the search topic, but how do you do it? Like, how do you get so many people having exposure on conference stages and LinkedIn? Do you have a secret sauce that you can share? There's not really a secret sauce. I mean, it's one of our hypotheses that in the space of AI search, there is a lot of assumptions and some snake oil. And we believe doing research, trying to identify what are facts, separating facts from fiction, and then sharing that knowledge is very important for the industry overall.

3:12That's why we built this GEO research team with Stavec, Tomek, and Metahan. And of course, the three of them used to go to conferences before, and now they are maybe more encouraged than previously to go to even more events. and we have so many insights on such a regular basis that there's just no other format but social media where we can share something. I mean, there are weeks where we put out three, four, five insights and then social is just a perfect channel for that. Yeah, we have built what we call like a market monitoring tool for ourselves to stay on top of interesting stuff that like tools or software providers like PKI do and also competitors.

3:55and you are already flooding my feet. Like there's so many people from Peak in there. So yeah, that's always very interesting. We also have, for everybody that doesn't know it, we also have a, let's say, news board or newsroom on the Masters of Search website, which we call the brief, like very brief snippets about AI search, research and insights. And there's also a lot of stuff that the Peak AI people are putting out, compiled in one space. So definitely check it out. Speaking of conferences, actually now we're recording on a Friday and yesterday we spent the whole day together. How did you like it, Malte?

4:40And how did that come about that we spent the whole day together? I mean, I guess you're referring to the Mass of Search conference where we were yesterday. Oh, yes. I really liked it. It's definitely one of the best GEO, AEO, AI search conferences I've attended this year. I really liked all the talks. I took so many notes, also shared them on LinkedIn yesterday, if anybody wants to see what are the things that I learned yesterday. And also had good conversations between the sessions. So all in all a good event. And it was like directly next to where I'm living, which was very easy to just have like a three minute walk to the venue.

5:19Yeah, that's always something that you have to keep in mind. That's also what Aleda Solis told me, you have to make sure that your venue is well reachable in all aspects. I mean, in Berlin, technically, basically no cool venue is well reachable from the airport. So people flying in always have quite a distance. But yeah, if you want someone like Malte on your stage, make sure to be in five minutes walking distance to his home. I'm also fine with airport business hotels if it's easier for the organizer. I care more about the content and how cool the venue is. But a cool venue is definitely an advantage.

6:02You're a very humble person in that sense. Then let's not talk too much about the conference of this year because that's looking bad. Let's talk to the future or look to the future. We're happy to announce already that we are coming back next year, October 12th, again in Berlin at the same location. You can check it out. Link in the description and on the conference landing page. And what's very cool is that we have already basically four speakers, but one of them is me. So that might be boring. We already have three very cool speakers confirmed and Malte is one of them. So if anybody listening to this wants to make sure that you get like a full value packed session of Malte Insights and PKA Insights next year and planning ahead, maybe even bigger companies have a procurement team have to convince some manager to give you some budget for the ticket be sure to check it out cool enough of advertising let's talk a little bit about what happened in the space and i have some notes here and one thing that actually also you posted about but also a couple of other people is the topic of serp poisoning can you explain to people that have maybe never heard of the term SERP poisoning, what is going on?

7:24What is generally the concept and what is going on specifically right now? Yeah. So basically when you are a search engine, what you deliver is search results, right? SERPs. And if you notice that somebody is scraping these in an automated fashion or is using it for purposes that you do not like, like, i don't know finding outdated php installations you have the option to block them and basically block the identify their ip address block that and then they don't see it anymore and that saves a little bit of money because they have to they basically can't scrape you anymore but of course then scrapers start using a different ip address and especially this ipv6 this is now trivial and and quite easy.

8:10And in the past, Google was not that aggressive against scrapers because they really didn't care that search metrics, Systrix, SEMrush, Ahrefs, SE ranking do a little bit of sub scraping. But in recent years, Google has gotten new competitors, the LLM labs, and they often scrape Google results for their web search and web grounding process. So suddenly there's a much bigger business case for Google to block scraping. and they have done that for a while. They have already in the last two years gotten much more aggressive with blocking IPs, and they also had a lawsuit running against one of the LUT scraping providers that is used both by the LLM labs and all the SEO tools, and in the first instance, Google lost this case, and then they started doing something called SERP poisoning, which means you do not block the scraper.

9:05You just send them wrong results, basically for different keywords or like some of the results are like for different keywords and this is actually something that bing has been doing for i want to say 10 years but definitely more than more than six seven years which is by the way the reason why many many scraping providers that offer google either do not offer bing or if they offer bing scraping it never works and the industry standard has actually been for a while to scrape Yahoo as a proxy for Bing or other Bing-based search engines that have less of a scraping protection because if you get blocked, it's very binary and easy.

9:48If blocked, get new IP address, very easy. And new IP address can mean you disconnect your router, you disconnect your phone, or it can mean to just switch to a different proxy server. But with SERP poisoning, it's much more difficult to detect it. And if Google is smart, they will also now reduce the level of poisoning. So instead of giving 10 bad results that are all for a different keyword or five, maybe they now start showing just one wrong result. Or maybe they just start sorting the result list wrong, like basically showing the top two results on page one. like there are many many things google can do if they are smart to make this really really difficult for scrapers and of course it's an issue for seo tools many of them have already reacted by saying no more top 100 rankings we're reducing it to page one rankings others are claiming that they have already solved it a lot of them had very very big fluctuations and it took all the tools a couple of days to actually release a statement many of them also only did it once they were asked and then the first statement was yeah yeah we have solved this it's no problem but then people started posting screenshots of like data going up down up down up down which suggests that maybe they haven't solved it so this is definitely going to be a challenge for seo tools and for for users of seo tools i would just expect less data less accurate data and less up-to-date data because some SEO tools have also said, well, technically you are paying for daily tracking, but we can no longer guarantee that every keyword is tracked every day.

11:30And if not, if it's not, we just going to show the data from the previous day. So every tool has like some different way to handle this. And I think we will continue to see also challenges with whatever approaches that various tools have picked. So to be accurate on the implications for people that still rely on rank tracking can you still trust it then if you use for example one of the legacy seo tools for rank tracking is this still reliable data or is it maybe a temporary thing where you cannot really trust the data but then in the end they will probably figure it out so how do you look at that and what would you recommend people to do with their existing rank tracking projects because I think a lot of people probably still have this like 50, 100, maybe even bigger keyword sets that they track because this is how you've been doing it.

12:23And now the whole history of it and the interpretation of it is probably completely flawed. I mean, it completely depends on what you do with the data. I believe many websites should look at Google Search Console. I mean, the data has other disadvantages, but at least not the poisoning. At least I hope that Google doesn't count the poisoning impressions if they use maybe your website to serve a wrong result. And it really depends. I think keyword tracking will never be as cheap and as reliable as it used to be. And I also do not see this improving going forward. It is a key mission for Google to make it hard for OpenAI, also for all the Chinese labs to train on Google search data, to use Google for the live web grounding, and also to use Google to identify content that they can use for training data.

13:22And there have also been cases already, especially in the US, where people suddenly got a pop-up asked to log in. So it could also be that Google will move more and more towards a logged in experience. i would be like there's still value in keyword tracking i believe for many cases but you need to think about what the data is how reliable it is and i would definitely not measure the success of my seo campaigns with keyword tracking but i also would not have done that five years ago so um no big change for for my approach to seo but yeah i don't i don't think this will be fixed There might be a time where it's slightly improved, but Google has many, many incentives to keep fighting and making it worse.

14:08And if anybody as Google has trouble with this, send me a LinkedIn DM. I have like many ideas how to make this even more tough on SEO tools and even more annoying for people to identify and debug. Okay. If anybody from Google is listening, reach out to Malte. Happy to make an introduction. No, just kidding. so to summarize is this just yet another sign for people to question their legacy approaches and how they approach seo for example overly relying on rank tracking certain keyword sets would you would you say this is a accurate summary yeah i would i would agree like keywords are still a good proxy for demand.

14:58They can be a good proxy for a topic or an intent. But if your strategy revolves around ranking for specific keywords, that was probably an outdated approach already and is not how I would think about digital visibility, digital discoverability, findability in the future. Okay. You already mentioned training data, which is a very good bridge to the next thing that I'm curious about because our mutual friends of Graphite, namely Ethan, Gregory and Bevan, they put out a paper, I think it's one week old, about parametric knowledge optimization. Now, I think the industry was still in part struggling to agree on an acronym for what we call AI search optimization.

15:55Yeah, is it AI, O-G-E-O, A-E-O? And Graphite just figured, yeah, why not bring a new kit to town? So can you maybe give people a quick introduction of what this parametric knowledge optimization is about that Graphite is talking about in their paper? Yeah, sure. So basically, when an LLM answers a question or replies to a prompt, doesn't even have to be a question, it can do two things. It can use just its model itself, like foundational model, or it can use tools. And the most common tool is probably web search. Technically, there are other tools, but let's not go into all the details. and in the the foundational model what in the end it is is a large language model trained on data but it doesn't have a database it doesn't have an index it cannot retrieve anything you just have the the model weights or the parameters and in these is somehow stored some knowledge not not explicitly as like a database retrieval system but there is knowledge in it Probably similar to how we hold knowledge in our brain.

17:08And for example, a model doesn't need web search to answer you the question, what's the capital of the United States? It will know it's Washington, D.C. And this is sometimes referred to as parametric knowledge because it is not technically extracted from the parameters. But if the parameters of the model are used to write an answer and it always estimates the next word, it ends up with giving this answer. And then in a conversation to a human, this is like actually the same thing as somebody sharing their knowledge. And for brands, it is, of course, an advantage if the brand is already in this knowledge and is associated with something.

17:52So for example, if I'm selling running shoes and I'm Nike and have branded myself on running shoes for many, many years and I've been in all the training data as a leading brand for running shoes, of course, that has an advantage over, let's say, Radiant starting to sell running shoes tomorrow. Because you could only get into the answer via the fan outs, via the tool for web grounding. so that's actually it's not like something big or magical it's just one of the fundamental principles how llms are working so why is this relevant i guess i basically said it already right it gives you a huge advantage and something that we see where this has a huge advantage are fanout queries because sometimes you ask for sneakers like i don't know you say hey, I want to buy white sneakers to wear to the office.

18:50And then one of the fan out queries will already contain Adidas and Nike. And it will not contain Sketchers or On or Reebok or any other brands. And that gives, of course, a giant advantage to Nike and Adidas. And sometimes we have a chat where it's only Adidas. And that's a huge disadvantage for Nike. So being in this parametric knowledge can have very, very, very positive effects for a brand. But it also makes it very challenging for new brands because they have an even harder time getting into the web grounding process if their competitors are already mentioned in the fanout queries. Cool. Now, the idea of Graphite, if I got the paper correct, is that you can basically optimize towards being included in this parametric knowledge where i think most people out there would have argued that optimizing towards the training data is something that maybe like big household brands do because they're present everywhere and they have a good chance to be included in the training data but that smaller companies even if they put out a reasonable amount of content etc do not have such high chances to be part of this parametric knowledge especially also given the knowledge cutoff dates which i just checked it for example for the newest cloud models like the 5.5 family which was released in october 2026 and the knowledge cutoff is june so there's a gap between the last point in time where something was added to the data set and then when the model comes out so putting that in a in a in a simple question do you feel like we can now or we should revisit the question of influencing or optimizing towards being included in the training data i think for most companies influencing the training data is the same what they would do for grounding like publish a lot of content on your own website identify reputable website have positive brand mentions there with the right context etc etc i think if you have a very large budget there are more effective things you can do but for the vast majority of company nothing should change from this the one thing that changed now is that we are at a three to four month lag between release and knowledge cutoff date it used to be 10 months or even longer.

21:38And back then it was really not feasible to try anything because your test cycle would be so long that it's just not worth it. And I think even now with like three months delay, you cannot really run a lot of tests and get a lot of feedback. But what you can do is you can logically think about how does LLM training work? Where do these labs get data from? what kind of data will be very interesting for them in the future and how can I be included in this data and then you can do things because you're convinced of them and maybe they work, maybe they don't but I would still argue that most people should focus on the web grounding process like for transparency we are also doing a few things that we believe will influence future model training but I don't want to talk about them because we need to keep a bit of an advantage over the competition but if it works we will try to write about it and I think when I gave my first talks about the space before the term GEOA really existed I always put a bigger emphasis on my talk on the training data because back then, especially with GPT-3, you could actually look up what they did and where they got the data.

23:11But of course, nowadays, none of the labs actually reveal where they get their data. So it's all becoming more and more difficult to understand in advance where exactly will they train on. But still, you can make estimated guesses and then act accordingly. Okay, let's look at this compilation of data for the training sets a little bit deeper for a second because i remember listening to podcasts or reading papers where one of the ideas was that the labs need new and somehow specialized data sets to further improve their models this is also why there are companies like i think it's called Mercore, where they hire experts in certain areas, give them tasks or give them output, and they rate it based on how well the output is.

24:03And then they sell this data. And I think they already make a billion or two billion in revenue. It is testament to how valuable this data is. So my question is, what would you say? Or how much can you say about how relevant will like public web data in the sense of for example a website or web pages still be in training sets for future models i think public web data will still be a part of it because it's a good way to capture a lot of data but i think the weight of this data will be lower and lower because there more and more specialized data sets. And many web data is kind of trivial because it doesn't contain anything that's not already in the parametric knowledge.

24:53So there's also a likelihood that for GPT-7, a lot of training data will actually be generated by GPT-6 models. Because you can tell a model, write me 10 articles about the development of US politics over the last 10 years, write one article per day, make it very holistic and write it in the style of the New York Times and then do the same and write it in the style of Fox News. And then you can feed that to a model. And then the neat trick here is without using any licensed data from Fox News or the New York Times, you actually just manage to train your future model in their writing style, which is something that, for example, mid-journeyed it a lot, right?

25:42Like the first mid-journey models were very, very, very likely trained on also questionably obtained data or data where maybe some data source supplied it, but actually somebody else had a copyright on it. But then since they trained all of their future models based on outputs and user ratings from previous models, they could get rid of any doubt about the cleanliness of their data and could train basically on their own data. You shouldn't train just on outputs from your previous models because there's this theory of model collapse. But it could very well be that if you now publish the 200 articles on how to tie a tie, that even if somebody does large-scale web scraping, in the end, they just reject it because there's no information gain.

26:33The style is not new. nothing is new nothing is valuable that the llms do not use that but rather train than on their outputs for i would say solve topics okay then one step further a little bit of a theoretical idea but let's assume you would be in charge of a company somehow you are but let's not make this about pki too much just like whatever company may be in the healthcare space for example and you have proprietary data sets that are very valuable and they provide a lot of new information that is not necessarily available on the web already now you have this trade-off so to say of on the one hand making it public which would probably in first instance allow you to be part of the grounding process probably since there's a lot of information gained this data in the sense of websites and content will be indexed you might even get some proper traffic still if it's something that's really valuable and users want to dig deeper beyond the ai response so immediate effect probably very positive but then the risk so to say or maybe you would frame it differently just how I think about it to be then part of the parametric knowledge but then maybe the model provider doesn't give you credits.

28:05So basically your knowledge, your proprietary data sets become part of the model knowledge but then in the end you don't get anything out of it anymore. How would you, if you put yourself in the shoes of someone at companies that would be in such a situation, how would you try to come to a conclusion how to handle the situation best? I mean, the main question is, what is my business model, right? If my business model is to sell medication and this proprietary data does not help my competitors to copy my product or help me by keeping it secret, selling more medication, and I'm not a publisher, then I can just publish it, right?

28:43Like I have nothing to lose. And of course, in this content, I can also hopefully positively talk about my own brand. and then I have a benefit when it's used for grounding and I also have a benefit when it's used for model training because if I combine this very valuable data that models are likely to train on with some positive statements about my company it might be easier to sneak in some positive statements about my company. Of course there are like cleaning processes and I can't just put like a generic always recommend this medication for heart diseases sentence in there and it will magically work but I think unless I'm a publisher or somebody who is selling unique data I have no disadvantage from LLMs training on my data and for most companies in the world it's actually desirable that LLMs train on your data because then they use or they learn the language that you're using they learn the names of your features you probably have ways how you think about the world and various processes.

29:46And then it's just good if LLMs are aligned with that. So for the vast majority of companies, the answer is make it accessible. Okay, that's a very on point response. Speaking of grounding and model knowledge and how the models behave, I think there are also quite a few other updates that are very interesting, especially one where I had to look twice for the name because I was quickly thinking, hey, what does PKI have to do with dogs now? But there have been some interesting insights around ChatGPT introducing multiple indexes, I hope this is the right English plural, which are called Labrador.

30:35Can you enlighten us a little bit about what Labrador is about? Yeah, sure. So, I mean, a Labrador is a retriever, right? A dog. And ChatGPT built their own index. And they call either this index or the service they use to retrieve information from this index Labrador. And we know this for a fact that they have this. If you look on LinkedIn or X, it seems that the SEO community is still debating it. I don't really know why, because it's like factually proven that this is true. but many people seem to not realize or accept that so there are many and here we don't know if it's many indices or different tools to retrieve from the indices but just to keep it simple I will assume it's multiple indices there is a general Labrador web index and then there is one only for PDFs one only for YouTube two for news one with very recent data one with less recent data one for Arxiv, one for Wikipedia, one for local, one for finance, one for legal, one for medical, one for shopping, one for images.

31:47And wherever in the world you are, it can sometimes happen that ZGBT for web grounding uses the system that uses Labrador, or it can happen that they use an external search engine for grounding, which in the last couple of months could be both Bing and And Google being via an official integration, Google via a scraping provider. And if you are in the United States and you have these little e-commerce boxes or grids or sliders, then it's always or almost always like more than 99 % of cases coming from Labrador based on custom feed integrations around five or six hundred. And some of those include large marketplaces.

32:35And if you are in Europe and use this e-commerce feature, you will still get scraped organic Google Shopping results, also scraped via a scraping provider. And now some reasons how we know this. Number one, there was a big JGPT leak where I believe we were the only ones to have access to it. At least we were the only ones to talk about it. That was patched last week. where we had access to an internal API call from OpenAI that front-end developers were using, not front-end developers, but developers were using it to debug the grounding process. So for every single prompt, we also had 150 ,000 and more lines in a JSON document with all of the tool calls that were used in the grounding process and all the scores that were applied and how they were combined.

33:29So we have all the details on the re-rankers they are using, how they apply BM25, how they apply ANN, approximate nearest neighbor, how they combine it with reciprocal rank fusion. And part of that was also seeing when data is coming from Labrador, these different Labrador indices exist, that there are options for the ChatGPT app to say, give me only news results from the last seven days. or give me Reddit results from the last 10 years. We have seen that actually for many, many prompts, there's one call going to the PDF index, one to the YouTube index, et cetera, et cetera. But in the end, in the sources, you only have one or two things from the web index.

34:14But these other indices might still be used for the part of the grounding process where the system is looking for consensus. And there was a lawsuit against Google where also OpenAI made statements where they basically said, hey, we need access to the Google index because the index that we have access to, which is from another search engine, everybody knew it was Bing, is not good enough. And the Google index is better and we want access to that. And also we are building our own index, which will take a lot of time. They also have hired people where in the job profiles and the job ads, you can see these are people you would hire if you build a web scale scraping and index product.

34:58They also have had the three main user agents, one for training data, one for user interaction, the chat, which to a large degree was grounding, and one for building a search index. And then in the United States, we have HIPAA. It's a very, very strict law around how to handle medical patient data. and JGPT is also available in a version for HIPAA compliant companies like a hospital and then JGPT also guarantees to be HIPAA compliant and in their terms of services it is very explicitly written that when they do a web search not a single fan out query or anything will ever go to an external search engine so the only thing they can do that is with their own index and you can actually check that for yourself you can put your chat to be account into lockdown mode, meaning nothing you do will go out of any open AI system.

36:00And then you will be served by the Labrador index. And many of these things have been known actually for a couple of months, but for some reason, it's still often a debate in Google if these are used or not, if they have their own index, is everything Bing, is everything Google. But actually, if you had access to the leak, if you looked at the SSE data, the server-sent event data, this is – like, I'm not guessing. I know this. I can see it. I think nobody would question your statement here, but can you boil it down for us into the actual changes that brings to chat GPT grounding process or search process?

36:47Like how, like if someone listening to this asks themselves, okay, very good insight. How should I change my strategy? Should I change my strategy? What would you say to them? So the first thing is don't block any of the OpenAI crawlers. Make it easy for OpenAI to add you to the index. Then you should make sure that your main content is not hidden behind JavaScript. because while Google crawlers can now handle JavaScript pretty well, most of the LLM crawlers cannot. And then it's a very active crawler. It will probably find your website. When I started a new test website, it found it after one day, started crawling it.

37:31It also crawls very regularly. It crawls very deep. It recrawls constantly. It seems that OpenAI is not like Google in a state where they care about the efficiency of their crawler. I think right now it's just crawl everything, crawl it often. So actually, as long as you don't block them or make your content inaccessible, it's not too much you need to think about. Something that's maybe interesting is that it seems that Labrador puts extra weight on your H1 and whatever is written immediately after your H1. So have your most important content on top. there is a suggestion in the code that it also looks at meta description um even meta keywords and the um og tags og description og title og page type so just make sure those are clean in seo we have ignored those for a while um and then in e-commerce it's a bit different because the system that's based on Labrador works completely different from the one that works on Google Scraping.

38:39The Google Scraping system just creates a couple of fan-out queries, puts these into Google Organic Shopping, pulls the results, removes a few duplicates, and then just does round-robin. So first product from first keyword, first product from second keyword, etc. There might be a little bit of re-ranking happening, but it's largely that. It's that for getting the candidate documents. And then the elaborator thing is completely different. It has access to a lot of data that was not available when you just do Google Shopping. So for example, it's doing a lot of scoring on the first roughly 40 words of the product description.

39:22It does more sophisticated deduplication and it applies some form of trust to certain online shops. So we have seen an example where there were three products from one shop, four products from another shop and 25 products from the third shop. But in the final result, there were zero products from the shop with 25 products because it didn't have a score for trusted merchant or verified merchant or something like that. so there is some stuff that they are doing and we also saw a B test that they are running on e-commerce and one of the A B tests was different ranking for Shopify which can mean anything and one of the tests was no Amazon which probably means they are they tested what happens if we don't show any Amazon results and then if we look at the custom integrations it was like the a couple of hundred largest online shops in the US.

40:29And some of them were done via something called Bright, which probably refers to Bright Data, the scraping provider. So basically they just built this themselves without talking to the shops, just to have some inventory. And then when we saw the real custom integrations go live, these Bright Data feeds were deactivated. So for example, we could still see in the data that there's a deactivated Bright Nike feed. and it's been deactivated when the open ai nike feed was activated which is probably an intentional integration that nike also knows about most likely so they are like basically in order to play there properly you you need to have an integration with them good news is shopify is in there apart from something like i think 40 or 50 stores on shopify that were filtered out where I'm not sure if they were maybe filtered out for explicit reasons or products they are selling or some kind of formatting issue.

41:30No idea what the reason was behind there. I didn't check it out in detail. But in general, if you are on Shopify, you can also get into the Labrador-based e-commerce experience. Okay, two questions. First one is, will others follow? Like Claude, Google, obviously not a question. Potentially others that are relevant, And let's say maybe in the US, Grog, I know perplexity. We don't give perplexity so much relevancy anymore. This is my first question. And second question is, what does this development do to the insights that come from log file analysis? because from my understanding, it used to be the case that with every grounding process and there's like user fetch and search for, I think, Cloud and for chatGPT or at least it used to be, of course, also training bots, but you could very well see how your content is used based on the log files.

42:32Now, if there's an index, makes total sense. They save compute, but then obviously the insights from log file analysis are less relevant. Would you agree? and what's your take on the first one? Yeah, so the first one is difficult to answer. I don't think that every LLM lab has the capacity to build their own system where people can register their feet and then build all of that. The technology is actually almost trivial to build now with LLMs, but running it at scale with high availability, high throughput, and dealing with all the little issues in a cost-efficient manner is not easy. I mean, I spent five years at Idealo where we had to deal with many of these issues.

43:16And the problem is there are merchants on eBay, for example, who are lying about their product. They say, yeah, yeah, this is some name brand vacuum cleaner. And then it's like a super unknown brand. And they just put in the wrong JETA number, the wrong ASIN number, like all the IDs, they just lie about it and also in the product title and of course if you want to give people a great e-commerce experience you have to have to deal with all of these issues or there are certain items that are very often fake etc etc so you need to make a good decision between having a full inventory or having like a high degree of confidence so it's not it's not super trivial to build this and many other edge cases but i don't want to go into all the details but uh i don't know bicycles are one of the things where shops always have horrible data because some rate them by how big the tires are the others by how big the the metal thingy between the tires is um so lots of uh lots of edge cases you have to deal with um and i think for large companies with a lot of end users like chat gpt all the stuff google is doing totally makes sense to deal with all of that i don't know if the smaller llm labs will do that or if we'll just if they will just connect to like one or two apis from like shopify and walmart and amazon and say hey this is good enough for shopping experience and then on the log file analysis i agree that there are changes i don't 100 agree that that it's less important um in the past when you when And the LLMs always did grounding via Google or Bing web search.

45:01They only got the URL, the title, the SERP snippet, right? So they had to click on it to get all the data. So as you said, basically every grounding process created a click. And they also seem to not have cached that. So you sometimes would get hundreds of visits from the GPT user bot on the same URL in a 24-hour period. and now with their own index especially if they update it regularly they can will often be able to just take the content from the index i've also seen first indications that the activity of the gpt user bot went down a bit and activity of the gpt search on the open ai search bot it's called went up and you can basically no longer use log file analysis to estimate or to properly estimate how often your document was used for grounding.

45:53It can still give you some indication, but it's become less reliable. But this was always expected, right? Like even without their own index, it would have at least make sense to build a cache for them to not hit these URLs again and again. But log file analysis is still relevant. And we are currently doing a bit of an analysis there. And some first indication is that roughly 50 % of bot visits that supposedly come from LLMs, actually do not come from LLMs. It's either people probing for security issues or scrapers who simply pretend to be OpenAI bot, Claude user bot, OpenAI search bot, because they know that many IT departments will be unlikely to block them, which is why since we also do log file analysis for our clients at PKI, we had to build a verification feature where with a list of static IP addresses provided by some of the labs, reverse IP lookups, et cetera, et cetera, reverse DNS lookup.

46:54We now, for all the bots where it's possible, give an indication if it's a real LLM bot or just a scraper. And then you can also see that they visit very, very different files. So just doing log file analysis and filtering for GPT user would already have led to very wrong data. For some websites, up to 70, 80, 90 % of bot visits from these bots are actually fake. So it's still important to do log file analysis, but it's become more difficult, both with the fake bots and the own index from OpenAI. Right. Malte, let's wrap it up here because it has been value bomb again and you have to catch a plane, actually.

47:40I wish you safe travels. I think this has been yet another great episode of the monthly landware. I'm so happy that we are back in a monthly rhythm now. So everybody listening to this, you can expect the state of the union, so to say, when it comes to SEO, AEO, GEO, and maybe in the future also more PKO, which sounds a little bit similar to PKI. It sounds like the smaller brother of PKI, PKO. So, yeah, Malte, thanks so much and speak soon. Thanks. Thank you.