How 57% of Website Traffic Is Now Bots

9 Jun 2026 · 25 min · 12 chapters

Ask about this episode

Ask anything about it. ChatGPT or Claude reads this page and answers with the times it was said.

Connect VO and ask about every podcast you hear, including the moments you saved. Add to ChatGPT · Add to Claude

In short

The episode argues that AI is not just replacing jobs; it’s changing hiring and boosting output. It also claims bots now make up most website traffic (Cloudflare data: 57% bots, surpassing humans months earlier than expected), so companies must optimize for “agents” (B2A) with better documentation, APIs, and structured, agent-friendly content.

Guest backgrounds

No guest is clearly identified in the transcript. Speakers mention Aaron Levy (Box CEO) discussing AI disruption in categories like engineering, and Matt (CEO of Legion protein) discussing engineering productivity.

Key claims

Job openings surged by 731,000 (JOLTS) despite AI fears; AI reduces engineering cost, enabling more projects/features and more marketing/product roles. Bots/agents require documentation and easier product access.

Notable examples

Ubersuggest and AnswerThePublic free tools; Legion’s engineering output; Uber suggests MCP; Cloudflare bot-traffic graph; CEO “knee-deep in AI” discussion referencing Paul Graham.

Written by AI. May contain mistakes. Listen to the episode to check what was said.

Chapters

Tap a time to open that second in VO

AI's Impact on Job Creation

0:00 to 1:30

Explore how AI is expected to create more jobs than it displaces.

“So what if AI is actually creating more jobs than it is replacing?”

Historical Context of Job Trends

1:30 to 3:00

Understand historical job market trends and their relevance today.

“And we think the new categories and the new jobs it creates outweighs the jobs that it displaces, at least in marketing.”

Shifting Job Categories and Roles

3:00 to 4:30

Discuss the new job categories emerging due to AI advancements.

“more so talking about backlinks, getting more branded searches.”

Marketing Strategies Using Free Tools

4:30 to 6:00

Learn how giving away free tools can enhance marketing efforts.

“or 10 engineers and pay all these expenses, they're hiring one engineer or a part-time engineer and using AI and then getting something that will get them initial traction.”

AI's Role in Efficiency and Cost Reduction

6:00 to 7:30

Discover how AI-driven strategies are making marketing more cost-effective.

“It's not like them trying to poke holes in how your campaign's going or whatever, or talking about the results.”

The Demand for Engineers in a Tech-Driven World

7:30 to 9:00

Examine the increasing demand for engineers amidst evolving tech landscapes.

“I would say it's probably our biggest portion is B2B.”

B2B Software Companies and Competition

9:00 to 10:30

Understand the competitive landscape for B2B software companies with AI.

“but to just move quicker, which is causing people to have to increase the amount of employees they have for different divisions, including marketing or engineering.”

The Importance of Rapid Feature Launches

10:30 to 12:00

Learn why software companies need to launch features quickly to stay relevant.

“it's starting to become hard for me to keep up.”

Challenges in Marketing Due to Rapid Changes

12:00 to 13:30

Explore the challenges marketers face as product features rapidly evolve.

“I remember Eric used to make me take surveys or quizzes.”

Data-Driven Decision Making

14:00 to 16:30

Learn the importance of data in influencing decisions and feedback styles.

“if they have data and facts to back it up.”
Show all 12 chapters

CEO's Role in AI Development

16:30 to 19:48

Understand the balance for CEOs between being involved in AI and managing their business.

“So, Paul Graham is the founder of Y Combinator.”

Impact of Bot Traffic on Marketing

19:48 to 22:28

Discover how to optimize for increasing bot traffic in digital marketing.

“We've been talking about this on this podcast since a few months ago.”
Hear the part that matters, and keep it.Open this episode in VO. Double tap your headphones to save a moment as you listen.
Get VO free

Transcript

Automatic transcript. May contain errors.

0:00Eric Siu:So what if AI is actually creating more jobs than it is replacing? So the latest JOLTS data showed that U.S. job openings surged by a massive 731 ,000 in April. So that's the green line, right? The job openings. So when you look at 2008, this is when the GFC was happening over here. You can see the number of job openings really dropped a lot.

0:17Neil Patel:What's GFC?

0:18Eric Siu:Great financial crisis. Oh, okay. 2008? Yeah. Yeah, the financial boom, the mortgage. And then what happened is job openings percentage changed from a year ago. like it starts going up but the number of actual job openings green is what you should be paying attention to so this um this you know uh let's call it the great financial crisis really dropped job openings down a lot and then you can just see it starting to rise rise rise rise rise and then 2020 okay um the pandemic hits and there's a sharp uh you know v and then they call this a v-shaped recovery so boom you know the stimulus comes in and the number of jobs shoots up but then what happens is oh you know things uh interest rates start going up right and then there's a lot more that goes in that the war happens, right?

0:57Eric Siu:All these things. So jobs start going down again. But guess what? It's starting to come up because everyone was like, oh, AI is going to start to really, you know, destroy the job markets. I think there's going to be job displacement, but I think net-net, there's going to be a lot more jobs available. So remember, we've been talking about this recently. AI, at least my stance on this is that AI is actually going to create more jobs long-term than, you know, destroy jobs.

1:21Neil Patel:So we see it in marketing too. We believe AI is actually going to create more jobs than it displaces. It doesn't mean AI won't get rid of jobs, but it'll create new jobs, new categories. And we think the new categories and the new jobs it creates outweighs the jobs that it displaces, at least in marketing. Here's how I see it, Neil.

1:41Eric Siu:I think, you know, if we go back 100 years, right, or 150 years, you and I might be farmers, right? Because that was the most common job. And we probably fail at that, right? I'll speak for myself. I don't think I'll speak for you too. I don't think you'd be a good farmer either. But my point is you have what it was like 92 or 98 % of the population were farmers. And so these are jobs nobody wanted to do. Okay, people are like, oh, what's going to happen to all like the Uber drivers, for example, the taxi drivers? I think what's going to happen is, one, I don't really think they really enjoy their jobs per se.

2:09Eric Siu:And that's just my hunch. My hunch is they're going to be able to do something that they really want to do and be able to take care of their friends and family a lot better.

2:17Neil Patel:That's what I think. Yeah. And speaking of this jobs data, what Aaron Levy here, he's a box CEO, is talking about is AI has disrupted many categories. And one of the most obvious ones that is disrupted is engineering. And everyone's like, oh, you won't need as many engineers. But if you look at the data that he's sharing, more people are running engineering products or software products. So it's causing people to actually want to hire more engineers because they're able to do a lot more with less. and we're seeing this in marketing. Eric knows this for probably eight to 10 years now, I've been pitching people in marketing how they should release free tools because free tools is a great way to get brand mentions, backlinks.

2:59Neil Patel:Although back in the day, I wasn't talking about brand mentions, more so talking about backlinks, getting more branded searches. And it's one of the most effective strategies. And I think in many categories, it's actually more effective than just paying for advertising. That's why I like the concept of just giving away tools for free. And we've done this concept with Ubersuggest. I tested it out with Ants of the Public as well. And we have quite a few other tools on our websites. Some are just simple like A-B testing calculators and the list goes on and on. What we found is when I've given talks on how to build a big company, giving away free tools, collecting leads from those free tools and then selling them on something else.

3:38Neil Patel:So for us at NP Digital, we give away a lot of free tools, collect leads, sell those leads on marketing services. yes we do have paid versions of our tools but some of our tools are purely for free what we found is when i give that pitch or speech at events and share the data very few people did it i want to guess the reason why very few people did it eric because it takes work takes work and more importantly that work costs a lot of money now i'm finding so many people to hit me up asking me questions about how to convert tool traffic into revenue. Like no joke, dude, I'm getting two to three inquiries a day.

4:17Neil Patel:I mean, not a day, a week about it. And when I talk to those individuals, the reason they're willing to do it now is because AI makes it way more cost effective. So they still hire an engineer that they may have not had before, but instead of having to hire five or 10 engineers and pay all these expenses, they're hiring one engineer or a part-time engineer and using AI and then getting something that will get them initial traction. Now, most of them still aren't looking to go full blown. They're just looking to get something out there and see what traction it gets and then spend more over time.

4:48Neil Patel:But you're seeing more companies or at least more marketing departments willing to invest in engineering products to get their brand out there, to get the backlinks, to get the brand mentions. And they wouldn't have been willing to do this two, three years ago because the cost was ridiculously high compared to now. so you're seeing companies just willing to do a lot more and another example of this is legion you know the protein powder that i take and the protein bars you've tried oh so i was talking to their ceo and he was telling me how matt matt's a mutual friend of ours i'm the one who actually introduced him to him uh and he hired him he's been there probably for like eight to ten years now matt runs his engineering he's like dude you know our engineering guys get way more done and he's like people would say that we don't need any more engineers but he's like because it's so much more cost efficient we've actually increased the number of projects and the output that we've been uh you know we've increased the output as well we're now because it's so affordable we're actually thinking about hiring a few more engineers and just exploding our product pipeline and just doing way more because it's so much more cost effective and the economics make more sense now

5:57Eric Siu:you know it's funny let me just go gonna go on a little tangent here we'll come back to this we hired a software engineer um from oracle okay and from based in india okay because of that point neil i think because this guy's so good um they did everything they could to retain this person um and which is interesting to me because it's like it's people are like oh we need to keep the really good uh the really good engineers and i think you're starting to see the counter offers come out now right uh so i'll just leave that one at that because we we ended up coming to another arrangement with this guy um but i think we're going to need to hire more because because this because at this point over here so again job openings soar by 731 000 which kind of it changes the narrative that ai is actually creating new jobs right so ai is going to cause companies to hire more in sales because agents can let them process more leads and do more customer research ai will cause an explosion of new marketing roles because of how much more efficient it is to launch campaigns and target the list goes on so the thing is i'm noticing in these single brain conversations that we're having with, for just kind of this AI revenue agent implementation.

7:05Eric Siu:Everyone's like, can you do this? It's always every single call. It's not like them trying to poke holes in how your campaign's going or whatever, or talking about the results. It's like, how about this? How about this? How about this? It's constantly more, more, more, more, more, right? And so going back to the initial topic that we had where the guy wants everything, wants to build a trillion dollar company. I think that's gonna be much more, it's gonna be hard still. It's gonna be much more accessible. is my point here.

7:28Neil Patel:Yeah, and dude, I don't know what chunk of your clients is B2B, but I'm assuming a good portion. Good chunk, yeah. Yeah, same with us. I would say it's probably our biggest portion is B2B. Is that for you as well? Yep. So what we're seeing in marketing for B2B is a lot of our clients are software companies in B2B. We have security in other sectors as well. But let's just stick with software for now. Software companies would produce products. You then go market the features and you go market how that company has new solutions to specific problems, right? The whole purpose of features are the features are supposed to solve problems that their ideal customer is facing.

8:07Neil Patel:And what we're finding now is, and we've been seeing this for more than a year now, because AI has drastically increased the output for B2B software companies, they're cranking out so many more features and doing more things with their products than they ever did before that now companies are like, we need to hire more and we need to do more or it's going to be too hard to keep up on the marketing side. The main reason being is they wouldn't have the feature set that their competition would have, which is going to make it harder to market a product. So no matter how good you are marketing a product, if someone's looking to get A, B, and C done and your competitors can do all three A, B, and C and you can only do A, well, it doesn't matter how good you are marketing.

8:48Neil Patel:If you don't do B and C, you're going to start losing customers. Yeah, right. And so actually, by the way, go ahead, finish it up. No, but we pretty much ended there, but it goes to people have to hire. They're now in a rat race and you have no choice but to just move quicker, which is causing people to have to increase the amount of employees they have for different divisions, including marketing or engineering.

9:11Eric Siu:Yeah, I think, by the way, it's because you're forced to launch new features. Guess what? The role of product marketers is becoming more and more important because now it's like every, not even every month or so, you might have every week or every two weeks or so you're launching something. You look at how fast Anthropic or OpenAI are shipping new products all the time, or even Google shipping new product features all the time. It's happening at lightning speed. And so Neil's point here, by the way, it's maybe it's becoming kind of the status quo now to have to move that fast. And that might burn people out, but I don't know, man.

9:43Eric Siu:Like if it's the status quo, like it's someone else is more than happy to take your spot if you don't want to move that quickly. and maybe that's just a reality right now. I don't know.

9:52Neil Patel:We'll see. I think it is a status quo. We're also seeing engineers being offered a lot more money than they used to, at least the good ones. We're seeing companies trying to poach really hard. I bet you you're going to experience this if you already haven't. We've experienced this as well. And at least for us internally, when I look at our projects like Answer the Public and Ubersess, we should try to release our goal was at least one major feature a month or one major change. It's gotten to the point where the team's releasing so much on a weekly basis, it's hard for me to keep track of because I'm traveling or doing other things.

10:25Neil Patel:So it's like, dude, if they're messaging me on WhatsApp on four new things that they did this week, it's starting to become hard for me to keep up. And I think other marketers are going to have the similar issues unless you have someone focused on, hey, I'm going to focus on product marketing and just create material around these new features and videos and social posts and do SEO campaigns around it and paid advertising campaigns, but it's becoming overwhelming because of how fast people are moving within organizations now.

10:53Eric Siu:Just a quick break. If you want to run LinkedIn personalized ads and landing pages, you can't do it on LinkedIn ads right now, but you can do it with Carrot. It's www.carrot.ai.

11:05Neil Patel:Again, karrot.ai. And we'll see you on the other side.

11:09Eric Siu:Yep. I'm going to show you something real quick. Because we're talking about how fast AI is moving right now, I'm going to show you how we've been communicating faster because of the speed of AI. And basically what we're having people do now is create artifacts, right? And what I mean by an artifact, if you see this on the screen right now, is we're having people create HTML documents. So imagine Neil and I were recording this podcast right now. I can take this transcript or maybe I'm doing granola. I can take the transcript from that. And then I can use our design.md to design kind of in whatever format I want and make an HTML file that then I can share with my entire team.

11:43Eric Siu:Maybe I can share with one of the teams. but the whole idea is that we can communicate a lot faster with you know one or two three three slides or so in this case someone's showing me the type of ads that they want to run for for chat gpt ads um but this is how we're talking now because it's like instead of having to read the entire transcript it's like that's it's hard to parse that information right so if you have the ai parse it for you make it into a nice little html file it makes it way easier for you to get information across and if everyone is aligned with where you're trying to go let's say you're communicating to 10 people or 100 people or so if you can align those people that quickly it causes less problems in your organization so i'd highly recommend that whenever you're talking to people now one have your recorders on and then make these artifacts and make as many artifacts as you can because it just makes life a lot easier when it comes to communication at least for us so yeah no

12:31Neil Patel:i'm with you communication is also really huge in marketing and business i think not enough organizations spend enough time fine tuning their communication processes and working together as a team and figuring out how different people communicate with each other and how they comprehend what you're communicating. I remember Eric used to make me take surveys or quizzes. I don't know what they were called. Personality tests. There you go. And he would make friends take them and other entrepreneurs that he knows and it would compare. Other people take them too. Yes. I don't mention that, but I still remember that.

13:11And with these tests, it's really good.

13:15Neil Patel:Even though I wasn't a fan of them and Eric would make me take them, he's like, can you just please take it? And he would pester me until I finished it. And then he would send me the link so he would get access to the data. It was great because he would understand how my personality is and the way I think. And he would share how he thinks. And it would allow us to better communicate with each other because we understood what made the other person tick or move or what they didn't like to hear. Like Eric knew that I love data. So before, years and years ago, Eric used to be like, you got to try this, you got to try this.

13:48Neil Patel:This is a better way. And he noticed that I would barely do anything with any of the information that he passed me. Then after the personality test, He quickly learned that I'm very data-driven and data-oriented, and someone can change my mind really easily if they have data and facts to back it up. Like, I'm not stuck in my way, as long as you have something to prove me wrong. And I'm happy to be proven wrong, especially if what you show me can make me better or make my business grow. So then when Eric would come to me, he would start using facts, stats, and data, even if it was a small sample set, because he knew that it would be more likely to get me to take action.

14:23Eric Siu:You know what's funny? The best data is like, hey, do you see this actually makes money? And you're like, oh. It's a little different when you have the money there. But on the note, Neil got smart with that. You actually removed it, so I can't pull it up anymore. So Neil got smart and deleted it. No, I didn't remove it. But you're not on my thing anymore. So it must have done something. But what I will say is the personality test thing is huge too because you know why? Once you start adding it into whatever AI harness that you're using, whatever agent harness you're using, you can then say oh hey you have the entire repository hey today i'm going to be giving feedback to neil how does he prefer to receive his feedback because neil and i the reason why one reason why we get along so well is because i prefer to take my feedback direct no sugarcoating but the vast majority of people don't like that they take a lot of personal offense to it right and so but the thing is when i looked at neil's personality test on the on the tough meter he's like 99 right so he's very direct um but also his composure 99 meaning that and i've actually never seen i very rarely see neil lose his poo poo right only when he's getting screwed on something will he lose it um but it's very rare so all that to say is i know where i can push i know where i need to kind of uh let go sometimes and that's why i think it's worth it to have everyone in your company do it and now that you have ai it's good to have all that stored so you can work better

15:44Neil Patel:with your team yeah although what i've been doing and this actually could be why my personality testing is shared with you anymore um i don't know when it was but let's call it a year ago or two years ago, there was some time where we started sending out a mass email to everyone that my information was subscribed to because you get these emails to erase and delete the data. Oh, okay. That's what happened.

16:04Eric Siu:That's what happened.

16:05Neil Patel:I don't know if they actually did it or not. I knew I was getting a lot of follow-up emails from that, but I just kind of ignored the follow-ups. It was just more so. That was two years ago.

16:12Eric Siu:So I think you disappeared two years ago. That's what it was. We're just like, delete.

16:16Neil Patel:Because we don't want people having it because what we saw is when your data is out there, a lot of these organizations are getting hacked. Might as well just remove it from any place that it doesn't need to be.

16:25Eric Siu:Yeah. Although, I mean, that data is like not that bad. No, it's not that valuable. So, okay. Let me pull up this Paul Graham post. So, Paul Graham is the founder of Y Combinator. So, Neil, I want to get your thoughts on this. So, he says, the only thing worse than having the CEO knee-deep in building stuff with AI is not having the CEO knee-deep in building stuff with AI.

16:48Neil Patel:I look at that as it depends on the type of company you are. Paul Graham typically deals with startups and product people and engineers. So in that case, I totally agree. But for all companies, I would not say it's a good fit to have someone knee deep just building things in AI. Example of this is, and this is an extreme example, let's say you're a cold storage trucking company. I know someone who actually owns one of those things where, you know, fruits, vegetables, when it gets transported or food gets transported. A lot of it has to be in temperature-controlled storage units, and then it has to be delivered to facilities that are temperature-controlled, and then from there it gets distributed to, let's say, grocery stores.

17:31Neil Patel:Having the CEO of that company being knee-deep in AI, I'm not saying AI can't help them. I don't know if them focusing the majority of their time on AI is right for that industry or that job. But there are cases where it is, and I would say there's also cases where it isn't.

17:50Eric Siu:Dude, I got to show you this graph shortly. But my take on the Paul Graham post is this. So one, I think he's saying that, okay, it's really bad to have the CEO knee-deep in building stuff with AI because it caused a lot of distraction. It caused a lot of thrash in the organization, too. Because now that someone like me, especially I have all these ideas all the time, I'm maybe starting to pry into other people's GitHub repos. Hey, how's this going? Hey, I'm going to ship this PR today. So it does start to cause a lot of chaos. But I think he's also saying for tech startups, but if also the CEO is not super AI pilled, he's not going to know how to really steer the ship, right?

18:26Eric Siu:At least in a tech startup environment. Now, look, if you're a trucking company, or we're talking about like an agricultural company, maybe not so much. But my take on it is I think that if you're a CEO and you're driving the ship on the strategy here, you should at least be spending some time. but not i don't think it's like uh it also depends on the business to neil's point so

Read the full transcript

18:47Neil Patel:dude what's funny on agriculture is actually not to say the ceo but they're really heavy in ai right now i believe it yeah agriculture is just so funny and because we work with some of the biggest agriculture companies out there like cargill and we work with i think the number two and number three player as well um cargo most people won't know what type of company is but it's It's an agriculture company. They're privately owned. They do around$160 billion a year in revenue. At least that's what Google says. I think it's one of the largest privately owned companies in the world. There's not even that many publicly traded companies in the world that are generating$160 billion in annual revenue.

19:26Eric Siu:Yeah, I think you kind of need to. So going back to this, as I was saying the agriculture thing, I was like, I'd imagine they'd be pretty big with AI. But my thing is this. If you're a CEO, you should really understand your space well. but you should also understand how to use the augmentation power of ai and that's why paul graham is saying this so i i do think if you're listening to this um like my stance is yes you should be you should be knee deep in it but you shouldn't be uh disrupting your company that's what i would say so by the way speaking of because we're talking about bots right now did you see the uh cloud flare traffic over time how bots have basically surpassed humans here check this out yeah 51 of now website traffic is bot traffic exactly 57 okay so if i look at the let me just put this to the last seven days or so so he the ceo i think uh his name is matt he said that he didn't see bot traffic surpassing um human traffic until maybe like 2027 but it's already happened six months into 2026 and so okay the the question now is like okay what do you do about that?

20:30Eric Siu:We've been talking about this on this podcast since a few months ago. You have to start optimizing for agents, whether that means having really good documentation, whether that means for your products, like Uber suggests as an MCP, right? ClickFlow has an MCP and an API. You want to make it easy for agents to find things and you want to make it easy for agents to use things as well. Because we are now optimizing not just B2B, B2C, we're doing B2A, business to agent.

20:56Neil Patel:Yes. We did a webinar on that. I don't know when, maybe three, four months ago on how bot traffic has over-surpassed human traffic. And I think the title of the webinar was how to optimize for bots or like in the new world. I'm messing and butchering the title. We had something that was much more catchy, but the webinar did really well because people aren't thinking about optimizing for bots. But here's a caveat. As marketers, you've been optimizing for bots for a very long time and for the majority of your career. You just don't know it. You optimize for Googlebot. You optimize for Meta and Instagram and their algorithms.

21:31Neil Patel:Yes, you did also optimize for humans, but you've been working on optimizing for bots for a long time. There's not much of a difference when it comes to optimizing for agents. They look for some of the same things that humans do. They look for some things that may be different. So you got to think in structured data, right? So if you've already been optimizing for robots like the Googlebot or the chat GPT bot, I forgot what it was called. It's something like that. You should be fine. Just think about what an agent would want to see and how to make the experience delightful for them. Still think about traditional platforms as well and what their bots would want to see.

22:10Neil Patel:And also think about humans as well and what they want to see. So now you're really optimizing for three things. The platform bots, agents, and the third one is humans.

22:21Eric Siu:Anyway, thank you all for listening. Hope you enjoyed it. Please don't forget to rate, view, and subscribe, and we will see you tomorrow.

From the publisher

Neil and Eric discuss why AI may be creating more jobs than it replaces, breaking down new JOLTS job openings data, AI’s impact on engineering and marketing teams, the rise of AI-powered product development, and why businesses are hiring more to keep up with faster innovation. They also explore AI agents, bot traffic surpassing human traffic, and what marketers must do to optimize for the future of search and growth.

Key Takeaways:

• AI is creating new job categories while increasing productivity across teams.

• Faster product development is driving demand for more engineers and marketers.

• Businesses must optimize for AI agents, bots, and humans simultaneously.

Chapters:

(00:00) AI and the Job Market

(01:40) AI Creating More Jobs

(03:02) Free Tools Growth Strategy

(05:04) Engineering Productivity with AI

(06:34) Why AI Drives Hiring

(07:35) AI and B2B Software Growth

(09:11) Product Marketing Becomes Critical

(11:09) AI-Powered Team Communication

(12:33) Personality Tests and Team Alignment

(16:30) Should CEOs Be Deep in AI?

(20:03) Bot Traffic Surpasses Humans

(21:07) Optimizing for AI Agents

More from Marketing School - Digital Marketing and Online Marketing Tips

All 935 episodes
How 57% of Website Traffic Is Now BotsMarketing School - Digital Marketing and Online Marketing Tips · 25 min
Listen in VO