AI is Eating Search

25:58And Google's been around for 25 plus years. So no surprise there. A lot of things that work the way they work in AI search today, I would definitely say work that way sort of by accident. And so that's part of the challenge, I think, of being in the space, whether you're us or one of our customers that's trying to figure out how to make this work is like everything changes super quickly. Of course, that's always true in AI. It's not just about like models getting better. It's really about like all of the underlying kind of glue that's connecting these models to search and to the internet, just like being pretty rudimentary and like getting more sophisticated over time.

26:32I don't know if that answers your question, but that's the way I think about it. What's the name of the category in your mind? So there's AISEO, AEO, people that come out with, than the GEO generative engine optimization that A16Z posted about. Do you think any of these are winners? Do you think we need something new? I'm terrible at naming things, so I'm the last person who should name the category. I would say people ask us, like, are we a GEO company? And I say, yes, right? Because you can only fight so many battles. But again, I mean, I think if you think about, people have different definitions of what SEO means, but I think most common understanding, right, is it's basically about, like, showing up higher in search results.

27:10It's about showing up more frequently. And that is an important part of optimizing how your business and your products and services perform in these tools like Chat2PT. But again, I think that's only the tip of the spear because it's not just replacing discovery. It's actually replacing how people actually consume content about your business and increasingly how they interact with your business. It's not just the entry point. It's more of the full journey of how somebody engages with you or engages with your website. And so I think SEO, AEO, GEO, that's generally been more focused on that entry point.

27:46And the real prize is, again, I think more and more traditional web browsing is going away. How do you optimize the complete customer experience for a world where most people are doing most things in a tool like ChatGPT instead of browsing around your website? So I don't know what a good category name for that is, but I think that's the one we're in. the way we think about it is like agent experience right so like by analogy to the customer experience there's tons and tons of technology around making sure that customers have a good experience when they do come to your website and like go down funnels and like have a high nps score and ultimately convert and are happy with the service they get you know i think it's going to be really important to be that data driven about how these like ai systems are interacting with your content with your website, with your infrastructure, because ultimately they are serving your ultimate customer, serving the user, and you have to do a good job with the AI systems if you want to do a good job for the user.

28:44So maybe agent experience is what I would call it. Let's go forward to a feature where, like Sam Allman says, JGBD is kind of like the all-encompassing personal assistant. I have all my memories, I have all my preferences. Do you see that playing a big part in kind of like how the re-ranking and the selection of these websites gets done. Is it used at all today? It definitely is, right? So ChatsPD personalization, memories, just explicit preferences, if you set them up, do definitely affect the results you get. Now, again, obviously, they're still using traditional search. So the search index itself is not necessarily personalized.

29:18But you can definitely tell ChatsPD, right? Like, I don't want to read anything from these sources, and it will try to obey. That can affect both, like, what it's searching for and also which sources it prefers and ultimately how it presents information from those sources to you, right? I mean, I have the classic set of ChatDB instructions in my ChatDB config, which is like, I'm a senior engineer, like be concise, like don't over-explain things and don't glaze me too much. And I certainly get different results using AI search with that profile that I do if I'm like in an incognito window with like the default consumer experience of ChatDB.

29:54So I think like, again, understanding your audience matters a lot. Ultimately, maybe where this ends up is like no single chat to me. Every single person kind of has their own, right, their own fully personalized experience. And I think if you're like a marketer, who's like, how do I measure this stuff? That's kind of a nightmare. But I just think that's the reality you may be living in. So like, again, understanding like your audience, understanding your personas, like your ideal customer persona, thinking about how they want to use these tools, what they're trying to accomplish, and sort of modeling that out and measuring it is like a really important thing you should start doing.

30:26And that's something, again, that we try to help with in terms of what we're doing with Monitoring and Scrunch, not just monitoring prompts, but trying to group them into customer personas and actually monitoring the complete experience for somebody who is a senior engineer, is a product manager, is whatever ICP makes sense for your business. Speaking of clusters, I'm always curious to mine for data. You don't have to talk in specifics, but what are the major clusters? Anything in there that surprises people? Clusters in terms of? prompts, usage, things your customers really care about. For example, shopping, big cluster.

31:03But if you don't use ChatGPT for shopping, you don't really care. Coding, big cluster. Again, if you're not a coder, you don't care. What else is like that? In terms of what we see as important, there's clusters of how people use LMs that are definitely a little bit less commercially relevant. So for example, there are a lot of people who are very high volume users of ChatGPT who are like doing role-playing, for example, and then there's some other systems that have been more of that. And if you're us, that's maybe a little bit less interesting. The biggest thing I would say is like, if you expand from coding a little bit, what I would really say is it's about problem solving.

31:41I don't know if you guys have seen this, but there was a study that's gone around. There's been like a couple of versions of it around like what is basically the prompt intent of what people type into ChatBD, right? Are they using it to find information? Is it navigational, which doesn't really make sense? Was it generative? like they're trying to create some text or create an artifact are they doing research and you know they mined all this data from like like panel data like clickstream data like a bunch of consumers who had opted into having their data collected which is like that classically this stuff happens in marketing research one of the takeaways that the study had was like people use chat to pt a lot for these sort of generative tasks right they're trying to accomplish something they're not necessarily trying they're not necessarily trying to do a search in the sense that somebody's you know trying to go to Google and search for a topic.

32:24And their takeaway was like, maybe Google's safe. People are not necessarily using Chat2BD that much for these search tasks. So Google's still the king for search. You don't have to worry. Like Google, you know, nobody's moving your cheese too much. But actually what I would say is when you look deeper and like study how people use the tools, we've done some like qualitative studies and things like that. And then we have the same data everybody else has as well. People go to Chat2BD, especially in a business context, to solve problems. They're like, I need help with something. I need to accomplish something.

32:53I need to write this proposal. I need to come up with this financial model. For coders, it's really, really potent. If you've used AI coding tools, they write the code for you. You get a solution. You can run the solution. Problem solved, you're done. So that's very tactile for most of us. But even outside of coding, I think people do a lot of that. And so what I would say is, you think about search, again, in a commercial context, it's usually somebody being like, how should I start to think about solving a problem? How should I go find options I can consider to solve this problem I have? How can I use Excel to build my financial model?

33:30How can I write a proposal that'll look good to my boss? These are search intent, informational queries. In chat search people, people don't really do that. People just say, solve the problem for me. What I think is really interesting about that is it's actually the highest intent. It's the most valuable thing you could be doing. Because you're going from being like, I think I need to solve this problem and maybe I will to being like I'm actually in the middle of trying to solve it inside you know chat to VT or cursor or what have you right depending on what you're doing so I'd say like that's kind of the biggest cluster if you want to get a little bit more abstract one specific example that takes people a little bit by surprise is like there are a huge number of people doing like b2b software bake-offs in chat to VT you know if you're at a big company and your boss is like hey we need to buy something to solve this problem but like procurement makes us get three examples.

34:21Like we can't just go buy something. We have to like, you know, like run a bake off. People are outsourcing a lot of that type of work to ChatGPT and particularly to deep research, but even just with regular search. And it's really good at it, to be honest, right? Like it's really good at being like, here's your comparison table of like different CRM systems or something like that. Yeah, make the check boxes, focus on what I'm looking for, not what they're selling, you know? Right, right. And so for B2B software companies, you can imagine. As a quick note, how do you know? Because you don't directly see the traffic, but how do you know?

34:54Yeah. So there's two parts to it, right? So there is looking at panel data, which I think is what everybody who's publishing these studies mostly is doing. And I, in previous lives, worked in ad tech. So to the extent the web is maybe less pleasing of a place to be, we're all looking for the guy who did this. It was me. you know and we used just a ton of panel data in that pursuit to do like consumer modeling and stuff like that so i mean that's sort of the the secret of all of these studies of like what people are doing the internet is they generally come from like clickstream panel data which is valuable but it also could be misleading because the type of people who are in panels isn't necessarily representative of the whole population and specifically it's not necessarily representative of people who are doing like high value things like being you know staff software engineers at strike or something like that.

35:43So the other thing we've done is we've just done sort of qualitative research. Like we've just, you know, like followed people home, watched them use products, asked them, you know, asked to like post survey follow-ups where somebody's been like, I found you through Chatsby T. We've been like, what were you doing when you found us? Things of that nature. And so that's where a lot of those B2B software examples come from. Just to double click on the B2B software thing. I'm sure people ask you this, but I don't think you've covered it yet. What if I try a prompt injection inside of my, you know, feature list page.

36:10Yeah. I mean, it often works right now. Right? Like, I don't think this is a secret. So everyone should do it. I think it works until it stops working, right? And I would say like, there's not a lot of stories of people getting banned for like ChattoBD search so far, but it's coming. Right? So I would always say... Yeah, so there's a window, right? I don't know. Do you get banned? I mean, you're just trying things. I think you know again like talking my own book what I would say is as things get more sophisticated like black hat stuff tends to get kind of pushed out and penalized and there's negative consequences to doing it and then there's some gray hat stuff which like typically you know people do and don't talk about really what I would say though about most AI search things in terms of like just optimizing how you show up in these answers if you're a business is like I think we're actually in one of these rare windows of time where it's positive sum all the way around.

37:09What I mean by that is when you look at improving how you show up in chat for media answers, most of the time, the problem is not that you were insufficiently glazing yourself in your product description page. A lot of the time, the problem is you're just not being very descriptive about your product in general on your product description page. And so if you provide more information, more context, it helps the model do a better job. It's going to have a more accurate comparison table. It's going to guide the user to a better solution. And so assuming you do a good job providing that context, you're going to be happy, the user's going to be happy, and ultimately the platform provider's going to be happy.

37:46I think at Google and SEO, things often feel pretty zero-sum. Everything's competitive and everybody's looking for a trick. There's so much low-hanging fruit in this space that I would just say before you go to that, I would focus on how do you actually focus on serving the user, serving the user through the AI platform they're using. And just like clear writing, good structured content, adding lots of helpful examples and facts and FAQs makes a huge difference. But if you want to prompt inject chat to be needed through search, it's definitely achievable. And again, I think like these systems are like 100th of the sophistication they'll eventually be.

38:25Again, not even talking about the LMs, just talking about sort of like the glue code, right? Between like the search and web pages and AI. And when you say tweaking the content, do you still mean doing that in the traditional formats or what about things like, you know, LLMs.txt, are these like AI-native ways to alternately serve content? ChattoPT today, right, is using your existing web pages. So it's using HTML you're serving off your web server. I know that there's debate about this, but I'd say the evidence is like, ChattoPT is not indexing and it's not retrieving content from LLMs.txt by default.

38:59And I think a lot of people who are a little bit further removed from the original audience, like the fast AI guys for LLMs.txt sort of like have mistaken what it's for. It's great for documentation. Like it's great if you've actually already written a ton of prose and you're like, how do I load this into my context window more efficiently? As a discoverability tool, I would say it's not moving the needle for most people yet. But these things can always change, right? Like everything's changing day by day. Anything else that you think people think it's good, but it actually doesn't make a difference.

39:28But what else would you put in that list? Oh man, embeddings. You know, I think a lot of people are very focused on, And especially in some of the SEO optimization community, there's a lot of focus on understanding embeddings and similarity search and things like that with the idea that, again, maybe the search technology is changing. And I think, obviously, embeddings are very, very useful in general. But I think as a tool to understand how these AI platforms are consuming your content, embeddings aren't that relevant. You're better off focusing on just, again, having a good structured set of content that makes sense to a human with like clean HTML and things like that, trying to like super optimize like, you know, topic similarity and things like that, I don't think makes much of a difference and may harm things in some cases.

40:14I just want to open up the space to any other practices you see that may be super effective or super ineffective that people have got into their heads that, oh, we got to do this for our GEO, but like it doesn't matter. I mean, I feel like we've covered quite a bit of it. I would just, you know, again, I would say like clear writing matters a ton. And then this is, I guess we haven't said this on the pod yet. So this is one thing definitely people should take away, which is like, ChatSuite doesn't execute JavaScript. Most of the AI search indexers and retrievers don't execute JavaScript. So if your content is not rendered on the server side, it's typically not going to be available, right?

40:52And we see that it trips people up all the time. And I would just say like, I'm perhaps like an old fashioned, like SSG enthusiast, right? But even if you're using Next.js and things like that, we've seen lots of examples with customers where they've got a use effect in some page somewhere and it breaks server-side rendering. And then all of a sudden, they're like, none of this content's available. Turn JavaScript off, check your page. If it looks good, you're fine. If it's not, fix it. Every developer here knows how to do that. I think that's really helpful. I think the one for me is I have been involved with a company that put a lot of effort into this programmatic SEO, but like augmented by AI.

41:35So you get into this really terrible, horrible situation where you're generating a whole bunch of pages for using LLMs in order to be read by LLMs in order to rank higher. Right. The infinite chain. Maybe it works. And if it works, like, okay, there's a number where it makes sense. I don't know. I think there's good, bad versions like everything, right? So what I would say is, first off, like it's just a fact of life that a lot of content, especially in marketing websites, is being produced using AI. Like that's just, like it's already so prevalent. You know, when you get to like scaled programmatic SEO, which is a little bit more controversial, I think that it can be super helpful in some cases if it's done well.

42:15And really when I say done well, what I mean is like if it's bringing some kind of like insight that actually is particular to you and your company and what you do and making it easier to consume for AI, right? Or for search in general. So if you're just taking kind of like content off the public web, that's there already. that's already super well represented in AI and you're remixing it and maybe they'll cite my page instead of this other page. I mean, it works sometimes, but I wouldn't say it's a durable strategy. We have customers who are doing things where they take, for example, support tickets that are coming.

42:45They're looking at their support tickets and they're doing programmatic SEO generation of how-tos from support tickets and putting that on their website. And number one, it's super helpful for users. It's helpful for the support team because they get fewer support questions. and that content is like the almost the exact ideal content you could give to an LLM right because people are going to chat to be like how do I solve this problem and it's like here you go I think it can be done well and it can be really helpful in those cases I think there's a lot of things that work right now in terms of just like using AI to remix content and get more scale but like everybody has access to these tools right so eventually you know it's no longer I think we just had a couple more things so do you have any sense on difference between chatgpd search versus deep research and how they leverage search read content is it just using the same tool or do you see very different results yeah i mean the ingestion pipeline is is similar right so you know all the practices i mentioned like deep research also doesn't read javascript right um it's just quantity right it's doing more searches and it follows up um the thing that makes deep research really powerful.

43:56So maybe it's good to have a taxonomy. There's regular search, which is like, it does one search, it gives you the results, it ranks them, it summarizes them, you get an answer. There's multi-search, which we're starting to see more in regular ChatGPT4, and also in AI mood, where it just does a bunch of searches simultaneously. And then deep research, what's different about it is that it's sequentially following up with a reasoning model to be like, hmm, these sources didn't answer the question, let me try something else. And that's really the game changer about how deep search works. I think from an optimization perspective, there's not necessarily a ton of fundamental differences.

44:29But what I would say is like, sometimes more content isn't better. And that's true for me as a user using deep search. Like I definitely have experiences where I'm like, deep research gives me results that are inferior to just using regular chat duty search because it's adjusting more content, but the content isn't necessarily like contributing to the understanding I'm looking for. And then as a business who is publishing content that's being consumed by this thing, it's like how consistent is your content right do you have outdated things like is it getting information that contributes to like the user being helped and having a helpful understanding what you do or is there like outdated and conflicting stuff and like classic example and that would be like pricing um we see cases all the time where like people have tons of pages on their site that mention pricing and some of them are out of date and like the more content chat is consuming off of your website, the more likely it is to get conflicting answers.

45:23And sometimes people end up with the wrong prices. So I'd say that's something to be mindful of, but I don't think it changes the game a ton in terms of your strategy as a business. One thing I might think about, just trying to think this through, I've never thought about this problem, but if I were trying to optimize my websites for deep research, I would tell them what to search next. So you really need that next link or here's related links and you obviously want to make it favorable to yourself? I don't know. You can certainly do that. And you see that that works to some extent. Like it is obviously, because it's serialized, it's influenced by the previous results.

45:58But what I would also say in that situation is like, why not just include the content in the first place? Right? Rather than having it do a follow-up. Well, you just can't include everything, right? Like maybe there's just different branches you could take. So the branching factor is high. For sure. There's limits on context window and things like that. So I mean, I think it is reasonable. But what I would also say is actually that's a perfect case for something like programmatic SEO, where you might also benefit from saying, let's have more focused pages that describe a complete solution or the complete piece of information for some particular version of a query or version of a persona, rather than it being choose your own adventure.

46:37And I think that's where having more technical sophistication and how you manage content and what you show maybe to AI versus to Google or to regular people can be interesting. And so, you know, we have some folks doing that. Some websites have infinite scroll. I guess if you're not rendering JavaScript, that doesn't matter. But like, yeah, I mean, this is this whole thing about lms.txt, like you just cat everything into one giant file. I mean, I can do that. It's just, yeah, it actually matters. Yeah, I mean, there's tons of like lms.fulltxts that are, you know, well over the context window of common models.

47:11So you're like, is this helping? I don't know. Again, to me, I'm like, for some of these things, search works pretty well. Like regular search over your regular website works pretty well and naturally sort of solves that chunk size problem. But you do need to have content on your website that's like, that's helpful and targeted to the questions people are trying to ask. Awesome. Any case studies on companies that are doing this amazingly well that people should learn from or are people still keeping it under wraps? We definitely have case studies. And I think the ones that are most interesting to me personally, again, are the ones where we're seeing people who are not just getting more traffic, right?

47:47Like traffic is kind of like the first approximation everybody uses in SEO and then also in this AI search phase. But people who are actually getting more conversions, more actual business. And so like two that I would mention, like two really great customers who are both in kind of the dev infrastructure space. One is Clerk, you know, the user authentication company. And they have like, they have great docs in general. Like they're well set up to succeed in AI because of that. But they've seen a huge lift in AI traffic from sort of like targeted optimizations and like looking at the type of content that AI wants to use and generating more of it.

48:20But they've actually seen an even bigger uplift in conversions coming from ChattoBD. So I think the stat is like, they've seen like a 6x growth in ChattoBD traffic or traffic from the ChattoBD perplexity, but the common set. But they've seen a 9x lifted conversions. And again, I think that goes back to like when people are looking, like if people are asking like, you know, how do I implement enterprises? So in my app, for example, right? It's because they actually have that problem. Like they're looking for a solution and they're ready to implement, right? So the closer you can get them to actually being able to solve the problem, the higher propensity they have to convert.

48:49And that's the dream, in my opinion. And are you guys helping with that? Just for people, they sign up, Clerks signs up for Scrunch. You guys kind of do look at how it performs right now. And then are you helping them generate this content? How much of it do they do on their own? I think where we're at right now is like, we're the feedback and experimentation system. So I can't take credit, basically, is what I'm saying. The team at Clerk is great and has been really thoughtful about the types of content they need to create. They know their audience really well, right? They're developers, they have a Discord, they're super engaged with the people who actually use Clerk every day.

49:26So we're providing, I think, the supporting role of helping them understand what's working and double down on it. We're not like a content generation company, right? We don't know their business as well as they do. they can do a better job generating content. We are, I think, working on helping them put that process more on Rails. You know, so creating more structure, being able to run multiple experiences at a time, and ultimately helping them try to figure out how to deliver more tactically of that value to the AI platforms. And that's where maybe some of these other things, like we didn't get into it, but, you know, obviously MCTs, NL Web, everybody is wondering what the tactical mechanism is going to be for getting this content into these AI platforms if it's not just traditional AI search in the future is.

50:11So starting to do more work there, which I'd also put under that sort of like agent experience bucket. But they're the star of the show. And I'd say that's true for all of our customers. So, you know, we are a solution people can use to solve this problem. We're not an agency that comes in and solves it directly. But we do have, I'll give a plug here, we do have tons of really great agency customers. So if you're looking for an agency to do it for you, we can definitely steer you in the right direction. And can you give a range of like, just literally, okay, like, you know, I'm a company, I want to improve my rankings or optimization.

50:43How long and what kind of uplift is typical? Just to give an idea of what's on the table for these, like, you know. I mean, I think it varies a little bit by vertical. Like DevTools is a very good fit for AI. So I'm not going to promise that everybody's going to see a 6x uplift in traffic, but there's a lot of low-hanging fruit. So I would say we typically see people get double-digit improvements in traffic within a month or two if they, you know, if they are actively working to improve, right? Updating their website, publishing content. Some folks are more in wait and see mode and, you know, that makes sense for some businesses.

51:17But if you're actively working, it's achievable. Those things on like higher conversion, but also just raw higher traffic is not something that I was thinking about or watching. But it's really like, it's starting to flip for some people where like it's actually more than normal Google. Yeah. And then like, okay, like while your budget has to shift over, basically. Yeah. Just a last wrapper, two last wrapper questions. A quick one is, which is, do you have a sense of market share of, let's say, ChatGPT versus Google AI overviews? I assume like Claude is a lot smaller. What is the market share?

51:50What are the rankings in your mind of what people care about? ChatGPT, I would say, by far, in a way, is the thing that people think about as being an AI platform that has the strongest consumer presence and the most durable relationship with consumers. right so everything else is kind of a distant second now ai overviews and ai mode which is brand new so i don't have great stats on that obviously are like they're putting being put right in front of your face so huge huge traffic and then meta ai right like sleeper has published stats on you know having 700 million i'm sure more than that now active users because they're you know if you do an instagram search you end up in that ai you haven't mentioned perplexity which I think is interesting.

52:31Yeah. Yeah. Perplexity is pretty big. So I would actually say that Perplexity is the second biggest AI native search platform after ChattoVT. And it's got, I think more importantly, it's got people who are really passionate about using it. You know what I mean? I would say that people who are really into Perplexity are probably more into it than many of the other options. But it is smaller in terms of raw volume. I think what's actually really interesting, and I didn't pull... You guys can pull this up the same as I can. But if you go look at App Store rankings, for example, I think what's actually more interesting than just looking at how is Chat2PT the number one app in the App Store in the category, which mostly has been, what's actually interesting is to look at the volatility of it.

53:13and if you look like go look back at deep seek look at when grok first launched a mobile app things like that and you can look at similar you can look at similar data from like similar web for example chat suite has pretty consistently been very high traffic obviously the growth has been insane but in terms of like relative market share it's been consistently one of the best and in contrast like we've seen more like peaks and troughs with things like gemini and deep seek and rock and things like that so they may well become firmly established but nobody has this sort of durable relationship with tons of people like Chatsby does in the space so far.

53:47And perplexity, though, if you look at it again, in absolute terms, it's lower. But if you look at the consistency of people using it over time, it's really, really strong. So I would say, especially if you are in a category that has an affinity for perplexity, if you're in tech, if you're in maybe some segments of finance, things like that. Early adopters. Yeah. You need to pay attention to it. And same thing for Claude, right? Like Claude famously, way fewer users. Revenue, actually a different story in terms of being compared to OpenAI. But people who use Claude are very, very passionate about it for the most part.

54:22That's an interesting thing to think about from a strategy perspective. And I will say, I don't think it's... Chatfix is durable, but not infinitely so. If you look at the reaction to BlazeGate, the reason people I think stick with it is because they actually just really like the product, which is sometimes underappreciated, I think, in tech. right? We always talk about like platform wars and politics and stuff like that. But like, people just really like it. But when people feel maybe like a little bit betrayed, like the product is going in the wrong direction, there's a lot of pushback. And so it's going to be an interesting time to be alive over the next couple of years as it has been so far.

54:56But I think it's here to say. Awesome, Robert. This was great. Anything else we missed or any call to action for people? Are you hiring? Are you obviously want to more people should use it? That's kind of obvious. yeah i mean i i would say i think this is the future like of the web right like the future of the web is like you need to be sort of ai compatible and understand how the things you're publishing shows up in these systems and so just like as somebody who has been a web enthusiast for 30 years because i'm old uh don't get caught up in snake oil and investigate what works and like do things that make sense and you know don't ignore it either i would also just say like for especially for the latent space audience, is we are definitely hiring.

55:41And I think a lot of what we're doing is we're doing research into exactly what I just said, which is understanding how AI and the web interoperate in the future and what the future of the web should look like if you're a business who's trying to be in front of customers. And so if you're a type of person who's interested in helping us figure that out, we are definitely hiring. We have a lot of open roles and we would love to talk to you. Awesome. Thank you, Robert. Thank you so much. Nice to see you guys. Thank you.