AI Agents, LLM Crawlers, and the Future of Findability

0:00You gotta imagine there are people in this world right now that have algorithmic systems that are going AI and just mass producing content and just publishing it by the millions. So that's the baseline. When you go from GPT-4 to GPT-5, all that content gets better. So what are you doing to get better than that content? Welcome to the Talking AI Podcast, where we talk AI with both experts in the field and early adopters. I'm your host, Matt Page, and we're here to demystify AI for you so you can get some value from it. Let's talk some AI. Two years ago, Google sent a visit for every two pages. It scraped.

0:38Six months ago, that number was one visit for every six pages. And today it's one for every 18. And with OpenAI, that drop is even more staggering from one visit for every 250 pages six months ago to one for every 1 ,500 today. That's some data from Cloudflare's CEO. And AI isn't just changing search, it's completely rewriting the rules of SEO and how things get discovered. Clicks are disappearing and AI-generated answers are the new front door to the internet. And if your brand isn't showing up in those answers, cited, trusted, and referenced, in some cases, you might as well not exist. And in this episode, I'm joined by Seth Bessmerton, CEO and co-founder of Conductor, an enterprise platform for SEO content and AI visibility.

1:25Unpacked exactly what this shift from SEO to GEO, or whatever term you want to call it. There's several emerging means for companies. If you've ever relied on Google for traffic leads or visibility, this conversation might just change how you think about search forever. Seth, welcome to Talking AI. Seth Pyshenko, CEO, CEO, and CEO of Google. Very excited to be here. Thanks for having me. This is going to be fun. I was going through the product earlier. A lot of interesting stuff. We're in this crazy space right now. I know everybody is in terms of what AI is doing and impacting companies, but you're in the heart of it.

1:56It is completely transforming. But I'm curious, you've been in search for over a decade. You've been doing this for a very long time. If you zoom out, how has the way people discover information evolved? What's the biggest change that AI, specifically generative AI, has introduced? Sure. You said AI is becoming the new front door to the internet. And what I thought about when you said that was it's becoming the front door. It's becoming the house, becoming the garage. It's becoming the backyard. It's becoming the whole block. It's becoming the house. And you can generate it with AI if you want to, get a nice image of it.

2:34It's really that if you, the physics of, let's go back in time a little bit more. Yeah. Long time ago, a hundred years ago, you had a question, you waited online, talked to your village elder, got some information. Yeah. Last 15 years, you became really dependent on technology. You asked Google, the number of searches has consistently grown over 15 years, but also the kinds of things people search for. Used to be in the very beginning, people would type in these various STEM words like business cards or look for this. And then over time, you started asking lots of longer questions. How can I do this?

3:10Or what happens if I would do that? A lot of health-related stuff. We've become very dependent on technology. But the physics of the web had been generally where you go to a search engine and then you go to a website. And for many years, the director for all the engineers at Google was get them in and get them out. If you have a good search engine, people aren't hanging out in the search engine. You're sending them off and they don't come back. In fact, a ranking factor used to be that if people went to your website and then went back to Google, then your site got punished for that. Which is really, it couldn't be more of the opposite of the world we're in now where AI is becoming the entire experience on the web.

3:53You said it in the stats, which were, I think, published by Cloudflare, but it's the directive is very different they don't want to send visits to your website which means people are going to be spending almost all of their time on the internet and i'd say this outside of watching videos on youtube or spending time on social media that's kind of like the entertainment arm of the internet but for everything else it's going to be it's going to be largely in an ai interface and if you think that going from the village elder to google was like a massive exponential change in the kinds of questions we ask and how dependent we are we're going to only be exponentially, once again, more dependent on AI to help us with even more nuance and even more things, every aspect of our lives.

4:33And ultimately it's going to be our biggest advisor, consigliere, shopping partner, travel advisor. It's really, it's profound how much of an impact it's going to have. I like how you described it as the physics of the internet. I think one thing too, like I feel like a lot of people feel like, oh my God, our clicks are going away, Search is going away, but I think the point you mentioned earlier, I think we're asking more questions now. We're just asking them to AI in a more nuanced fashion and we're getting the answer directly back or better yet, having a conversation with it, which is not the user experience with Google.

5:09It's okay, let me get something close and then I'm going to go do my own research. It's a completely different user experience. I think that's starting to emerge with these new tools. Quick break in the pod. If you're listening to this podcast, chances are you've been thinking about how to actually use AI inside your business. And that's exactly why we built the AI Opportunity Finder. It's a free tool that helps you uncover high impact, tailored AI use cases based on your business, your goals, your pain points, and your industry. No fluff, no generic use cases, just real ideas that fit your business and the rank by ROI potential.

5:44It takes about three minutes to run and it's like having your own personal AI strategist for free. If you want to try it for free, check out the link in the show notes or go to hatchworks.com backslash AI dash opportunity dash finder. Yeah, the nuances is, you know, you think about a business, businesses have had these like kind of content maps or content strategies, which is like research, consider, buy. This would be like, I was just on the phone with that email marketing software company. So it would be like, I'm starting a business. What's the best way to communicate with my customers? That's early stage.

6:17middle stages, I'm deciding whether I should do more email or paid ads. Late stages, I'm deciding between MailChimp and Salesforce. That was like your content strategy to come up with those three zones. Now, when you're an AI, it's like at the end of every results, it'll tease you and say, well, maybe you can go a little bit in this direction or you can go a little bit in this direction. Do you want me to go and look into those things for you? Okay, sure let's go down that and now it'll take you from the end it'll take you all the way across the journey but there's so much more information and you're so much more informed along the way which means from a business's perspective that then the volume of content that you have to create is only more exponential because the nuance of the kinds of questions people are asking is so much more than it ever was before and that transition from i can't remember the terms you the buckets you mentioned, but like this unaware to product aware state is not happening on your blog anymore, where you're trying to make that transition.

7:18It's happening in the user interface. That's a great point around like how AI is actually kind of nudging you along. It's almost like this self-discovery thing. It's thinking ahead of you in a sense. It also, it teaches you how to make decisions. It gives you the framework in which you should make the decision by. It then lets you say, well, where do you weight these things? Do you care more about this or do you care more about that. And then if you care more about that, then it takes you down a whole different stem of questions and evaluation opportunities. So all of this, again, it's interesting.

7:50One may have said that in the last couple of years, there's only so much content Google can handle. And that's because if people are only typing in things like best running shoes, how many pieces of content can show up for best running shoes? But in a world of AI, I actually think it's like suffocating right now because there's so few pieces of content that line up to all the specific nuances. So just take the running shoes example. Someone goes to AI, I go to AI, I'm going to say I'm 42. I have flat feet. I've had this problem before. I've tried these shoes and they don't work for me. One foot is wider than the other.

8:30I'm in the streets of New York and, you know, and this is the kind of running I do. that's a lot more stuff like who has their who has content for that why is reddit the number one most cited source of information for ai right now is it because reddit is the ultimate sort of gospel of truth for information no it's because it's really the only place where a lot of this stuff exists because there's no other content that backstops all these nuanced queries that people have in a yeah that's the key you hit on it's nuanced and ai is able to take that into consideration I'll throw another variable for you.

9:03You can't tell right now, but I'm six foot eight. So I got size 15 shoes. Throw that into the mix while you're at it too. And we're going to get to this in a second. Like how do you actually tactically start to adapt? So you do show up in AI search. We're going to get to that in a bit. So everybody listening, I know that's probably the top thing on your mind right now. But I'm curious, like, as you were talking earlier, this like experience came back to my mind. I was in middle school, maybe sixth grade, something like that. And I remember experiencing Google for the first time. It was like Google and dog pile and things like that.

9:34Where in the past you had to go to the library and go research something and all of that. Do you remember your first experience with both Google and then also ChatGPT or whatever tool you had? I'm curious, like what were those inflection points or how are they the same or different if you remember it all? Well, I could share my first experience in remembering like the beginnings of SEO, if that's helpful. Yeah. Yeah. So I had a roommate and we lived on top of this bike shop on 34th street in Manhattan. And I was in college and his name was Shermie. And he initially when he was living with me was on unemployment.

10:17He like lost his job. He was getting checks effectively from the government. And he was starting a business. Fast forward, maybe a year and a half. He was making like thousands of dollars a day. And we had all this cool stuff in our apartment that he'd be like, what are you doing? Electric drums and this crazy couch. We had one of these like giant TVs. Like it was like the first like black screen TV. And when you went to Google and you typed in the word business cards, he had a website that showed up number one. He had another website that showed up number two. He had another website that showed up number three.

10:54Now at that time I was starting a business while going to college that was giving therapy over the internet. I remember no matter who I talked to about that business, nobody was interested in it. And then I remember looking at what he was doing and I was like, well, how do other companies figure out how to do what you do? And that became the very beginnings of getting involved in search and digital marketing and the business grew really quickly once we started it from that point. Yeah, it's interesting. I think so many people. Go ahead. But I was saying on ChatGPT, I don't remember the first time looking at it, but I do remember the first time.

11:26I'm I remember the feeling when I used it, which was this is gonna take over the world. Was that a good feeling or a bad feeling or mixed? See, I was more, I think scared probably in the beginning, I mean, no, I were scared. Like I, I wasn't sure what it all meant. Yeah. Natural kind of human reaction there. But there's so many terms I think emerging right now you have, I can think of three off the top of my head, GEO, AEO, AIO. What are these? Do you have a favorite? And I'm curious too, like you're so deep in the SEO world. Where is it the same and where is it different? Because I'm of the belief there's a lot of core principles that are still the same that exist in SEO that still pertain to this new world we're going to.

12:21Sure. Great, great question. with. There's something like 50 or 60 or 70 startups that have emerged in the last six or months or so that are offering AEOG and software. Most of them, we have a slide that has a picture of all of them and it's, you know, is this really innovation or is this, or is this not, right? Because they're all kind of do the same thing. And it also shows that it's not that hard to do the same thing if there's so many companies that all did this so quickly. I think that companies certainly are scrambling right now to get answers to this. It's not settled in. Do I have a preference on AEO versus GEO?

13:05Not really at the moment. I don't really love any of them. Answer engine optimization, like AI is much more than an answer engine. It's your partner. It is your... Yeah. I put stuff in there and coaching for like my parent, like parenting coaching. You know, I was doing over the weekend language training, learning Spanish and using like the voice version of Chachipiti. So it's so much more than that. And then GEO, like GEO is already a thing, but the bigger thing is that it's about getting, it's about being and owning and showing up and performing well in AI. I think there will be a category that settles in on that.

13:43I think we're identifying how we want to go with that. Right now, Conductor is extremely differentiated because unlike these 50 or 60 startups that just do how many times did someone mention you in ChatGPT, we have a full cycle solution where we do measure, but then we also help you identify which topics you should be building content for. We help you identify how to build good content for that. And we actually build the content for that. We measure it and show you the full end-to-end performance of the content you create. We monitor your site to make sure that you're technically going to be healthy and that your content stays healthy.

14:14So if everyone else is AEO, that just does one thing. Does that make us AEO because we do everything? These are some of the small problems that software companies have to figure out. Yeah, really good points there. And I think too, with this whole move, they think of SEO like authority, right? Relevance, the technical optimization, the user experience, all those things still apply in the same world. Where is it the same? Where is it different? And then I think we can start to get into like, where is the nuance on how you start to show up in some of these, like what is showing up? Right. Cause it's a different type of thing.

14:53I think then it used to be with Tim blue links. It's actually, it was a great question. So there's a lot, it's a part of why I brought up, yeah. Why I brought up these small companies is because part of their positioning is like, if you're doing SEO, that's bad. And you gotta do the new stuff. Yeah. And there's a lot of falseness or misperception in that. So my view of SEO for the last five years, it was not stuffing keywords in your pages. That didn't work. That hasn't worked for a long time. That already wasn't working. I mean, I always say when Larry and Sergey started Google, were they ever, you know what?

15:31When people search in Google, we're going to show them the companies that are really good at SEO. No, they're going to show them the companies that have really good content. really good digital experiences. So how did you do SEO for the last five years? Well, you had to be really good at knowing what people want and you had to be really good at delivering it to them in the form of better content and better digital websites, which meant technically strong websites and really good content and products. So all of that is the fundamental formula for showing up in AI. The thing with AI though, is it's times 10.

16:08If you had three personas before, now you have 300 personas. You have three buyer journey stages, you now have 10. If you were making 10 pieces of content a week, you now need to make 100. Everything is exponentially greater because of the thirst for my eye to have so much more nuanced content. Now, what is showing up? That's a really good question. So the simple versions of that are things like, is your brand being mentioned? is your content being a source of citation yeah and then there's what's the sentiment when people talk about your brand is it positive or is it negative why is it positive why is it negative when you show up are you showing up what's your relative are you first on the list or are you last on the list right these are also things so i think that from a brand perspective what you want is, well, firstly, you want to know what people are looking for.

17:06So what's their customer journey look like? And if you were to simulate that customer journey, is it favorable to you? Are you in the conversation? And are you being referenced in the conversation favorably and ideally better than your competition? That's really showing up. Now, maybe there'll be some opportunities where there'll be some link backs to your website and you'll get some visitors that will be few and far between, but will convert really well. But I even think that's going to be very limited going forward. It's going to be about having somebody advocate on your behalf. Yeah. I'll give you a story too.

17:40Is that AI conference or something? I was talking about talking with a leader at a very large bank. I'm not going to mention the name, but I think if somebody on his team just asked, is, you know, X bank a good bank, right? And got back the results. And what it was referencing was a lot of forums, right, that were specific to individual branches and a lot of bad experiences that might have happened was what was getting returned. So the persona perception of this brand was what was happening in this like random forum about some experience they had at the branch level, which I think is a completely different thing from if you search that on Google, you probably get, you know, maybe it's the branded term that pops up or something very intentional from the brand.

18:28And so that was, I think, kind of an aha moment for them, which then you get to, okay, well, how do you account for that? Maybe you're more present in these forums and, you know, maybe you're using AI to help there in some way. But I feel like everything just got a lot more real in a sense, more, more nuanced back to your point earlier. Absolutely. Absolutely. You know, there's still a lot of unknown. ChatGPT just rolled out like their agent mode not too long ago, a couple of weeks ago. So this is interesting because even though you're still on AI, AI now is like going out to websites and doing stuff on your behalf.

19:05The interesting how much of the internet is activated through an agent mode type experience versus like people syndicate. Does Amazon sync all of its products catalogs directly into ChatGPT or does ChatGPT go and shop for you? Either way, you're still doing it on ChatGPT. You're not doing it on Amazon's website. Exactly, yeah. Yeah. In one point it was Dharmesh CTO HubSpot had this article is really good. We can put it in the show notes, but he mentioned people's available attention is increasingly scarce and an AI's attention, AI agent, whatever you want to clarify it as is effectively infinite.

19:41So it's almost, how are you optimizing and serving not only for humans now, but also the AI agents that will be, like you said, going out and searching your website. And I've seen some stuff there as well. Like you have the robots.txt file that kind of gives these crawlers context for how to use your site. Curious your thoughts there. I've heard of a new form of llm.txt where you're kind of giving context for the agent in terms of how to use your website, what is available, or even maybe an MCP connection. Any kind of early thoughts into that? There's not really a standard Yeah. What we advise our customers is that short term, your primary way of communicating with AI is going to be through content.

20:27Longer term, there will be some place where you could put content or data or tables or a version of meta information that will be just for the AI. Now, whether that is like a feed, whether that becomes redirecting the user agent when the AI bot comes to your website versus the human, that's sort of undetermined. There's not a standard protocol right now, but it is definitely in our future that there will be, there has to be just a gentic way of feeding information to AI. Yeah. I mean, you could even see where there's an agent on your side, your website or product catalog or whatnot, where the search agent and them can kind of interact on behalf of the user and the brand.

21:11Right. So then it becomes even more nuanced in a sense, which is kind of cool. Yeah, it's crazy to think about. Tracking our product is like how often are the AI bots coming to your site? Like we're, our company is looking at, we publish a piece of content and in the next day or two, like we could have hundreds of AI bots coming to look at that content. Wow. And if we make a good piece of content, we know because the AI bots are consuming it so, so fast. And then content that's not great, AI bots don't really come that often. So that's a new first class citizen. He's an AI bot. I've heard things too.

21:50FAQ type of concept potentially works well because it's like question and answer. Well-format, matto comment, a content where you say in summary of it's in bullets, it's easily consumable. Any hot takes on that would maybe working or maybe it's just kind of a little bit of hype right now. Yeah. If you think about that world of an AI only feed, that's going to look like some kind of table. Could look like a table. Now, when you look at a big piece of content, like a blog piece of content, like it's a big chunk of stuff. But what does AI do? Like it throws it into a vector database. It gets rid of all the junk and it takes out, it puts it into a table in its own way, right?

22:32A little more complicated than that. So an FAQ, I feel is somewhere in between. Yeah. It's very structured. what you know and i think that's exactly how ai likes to make content so you know when i talk about you have 100 personas now you need to build content for that doesn't necessarily mean that now you have to make 100 pieces of content around the same topic here's a good example i like to play pickleball so if you were doing a search for like how to play pickleball someone who's 18 is gonna to have a very different set of questions than someone who's 95, you know, 85, 75. Yeah. You know, like my dad plays, he's 78.

23:10So it's going to be like, is it dangerous? Am I going to get hurt? What are the ways to avoid getting hurt? Yeah. And then someone who's 18 is going to be like, is this like old person's work? You know, people under 25 actually play this game. Yes. Champion of the world is 21 years old and the best female is 18. So that would all be like questions that you can have in there that are all persona specific, but that doesn't necessarily need to be five different versions of how to play pickleball. It doesn't all have on there. So that's a way to solve some of that problem. So yeah, short, long answer of saying thumbs up on the FAQ content, glossary content, all that stuff's really good.

23:51So let's get into like kind of a thought experiment. This may be even in some of the platform as well. But I think folks can start to kind of reverse engineer some of this, right? Because you're hitting at the core of the awareness journey and jobs to be done for a specific persona. And in the past, you do a lot of user research, talking to customers. All of that's still great. But you can almost simulate some of that now with AI talking to ChatGPT, where you can have a conversation and figure some of this stuff out. And you can get into, okay, well, what, what questions may this person, uh, to your point, ask about pickleball when they're chatting with AI.

24:29So you can start to almost get this framing of the types of questions, conversations people are going to have, which can then influence your content in a sense. I think that's one thing too, like people don't need to feel like, oh God, AI's, you know, taken over, we're screwed. You can use it on your side too, to kind of help combat it as well in a sense. Curious, any of those kind of like tactics that you can use AI. Up to win in AI, you need to use AI. Yeah, exactly. What you described is almost exactly what our product does. So, for example. Just much easier, I'm sure, than talking to ChatGPT.

25:11It does a lot of the synthesis for you. So you give in a couple of prompts or topics that you want to make a piece of content about. What we go and do is we go and find all the best pieces of content on the web for that given topic area. So we'll look at what shows up on the Google search page. We'll look at content that's used as cited sources in AI search, and then we'll analyze it and we'll analyze it all. And then we will then use AI as part of the analysis to then say, well, what are the kinds of questions and things that are areas of interest that people would have that this content supports and then take that one degree further what are the kinds of questions and topics that are related to this that are like one degree further one degree behind in the customer journey and then we produce a whole list of questions that become the basis for a content brief so then it's like here's all the questions that you should address in order of priority for this particular audience and then when When you make the content, we actually write the content and structure it that way.

26:16Or if you take a piece of content you already have and you want to score it, you just drop it in the product. We generate all the questions and then we do analysis to see how good does that content do against performing for all these questions. And then we, and then you, and then we give them recommendations about how to make the content better. You know what? People ask a lot about how this tire works in these kinds of conditions. You kind of addressed it, but it would really good if you were more specific over this scenario. And then some content creator can go in and add that elements and make the content more a far taken.