

7 Aug 2026 · 2m · 1 chapter

Retailers should invest in infrastructure to serve non-human (bot/crawler) traffic effectively, treating it as part of delivering a better shopping experience.
Joe Dorian is Chief Product Officer at Botify, a company focused on serving search and crawler needs for websites.
Non-human traffic has different requirements than human users; retailers shouldn’t view bot handling as a chore, but as an essential component of customer experience.
The episode mentions using a robust CDN/content delivery network and coordinating multiple parties to deliver the best human experience, then extending that approach to efficiently and effectively serve crawlers with high-quality delivery.
Written by AI. May contain mistakes.
Discussion on the importance of catering to non-human traffic for optimal shopping experiences.
“Those bots and non-humans have very different needs.”

Get the gist, explore an idea or ask about something said.
On Voices of Search // A Search Engine Optimization (SEO) & Content Marketing Podcast, what are the key points of this episode?
Automatic transcript. May contain errors.
0:00Joe Doran: The Voices of Search podcast is a proud member of the I Hear Everything podcast network. Looking to launch or scale your podcast? I Hear Everything delivers podcast production, growth, and monetization solutions that transform your words into profit. Ready to give your brand a voice? Then visit IHearEverything.com. Welcome to the Voices of Search podcast and I Hear Everything production. In this podcast, we'll share the news, knowledge, and strategies you need to navigate the ever-changing world of SEO. Ready to expedite your company's organic growth efforts? Sit back, relax, and get ready for your daily dose of search engine optimization wisdom.
0:41Joe Doran: Here's today's host of the Voices of Search podcast, Tyson Stockton. My name is Tyson, and today I'm joined with Joe Dorian, Chief Product Officer at Botify. Hi, what's one infrastructure investment every retailer should make today? How they're specifically serving non-human traffic. How do I serve the bots? Right now, they have a very robust CDN or content delivery network play. They usually kind of like have multiple parties that are actually doing that to make sure that they deliver the best human experience possible. Time for a one minute break to hear from our sponsor, Previsible. So you're looking for SEO help, and you got a couple of options.
1:23You could start replying to spam from agencies that claim they can get you to rank number one on Google. You can pay an hourly rate for a consultant who will inevitably nickel and dime you with hourly charges. Or you can work with a cookie cutter agency to quickly launch a strategy-less project with low success rate. None of those sound very good, now do they? Well, that's where Previsible's integrated consulting model comes in. Previsible draws from a collective 40 years of SEO and digital marketing experience to unlock your organic growth opportunities. They build custom solutions that combine strategy, technical expertise, content, and reporting to effectively operationalize SEO for your business.
2:02Previsible's four-stage approach ensures that your SEO programs thrive by starting off with a strategy-first approach. Then they support you in your efforts to create quality content, help you identify technical issues, and most importantly, they'll work with your cross-functional teams to integrate your SEO strategies to make sure that your SEO budget actually drives results, not just your agency's bottom line. So join brands like Yelp, eBay, Canva, Atlassian, Square, all who rely on the SEO consultants at Previsible. For more information, go to previsible.io. That's previsible, P-R-E-V-I-S-I-B-L-E dot I-O.
2:44They're not making that investment to do it in the bot traffic. Those bots and non-humans have very different needs. The shift has to be from thinking of non-human traffic as something that I kind of have to do. to have the best shopping experience, I have to service my non-human traffic, those crawlers very effectively, very efficiently with a high quality.
Enterprise retailers index only 40-50% of their product pages. Joe Doran, Chief Product Officer at Botify, breaks down why heavy JavaScript rendering and fragmented product data leave nearly half of retail catalogs invisible to Google and AI crawlers alike. He details the crawl budget compounding effect that shrinks indexation, the confirmation-crawl behavior LLMs use to verify and cite pages, and why structured product feeds—not visible page content—now determine agentic commerce visibility. He also makes the case for rethinking attribution beyond session-based tracking to measure multi-touch contribution across AI search, paid media, and traditional organic channels.
See Privacy Policy at https://art19.com/privacy and California Privacy Notice at https://art19.com/privacy#do-not-sell-my-info.