AI bots are increasingly affecting website performance, analytics, infrastructure costs, and content visibility. New research and infrastructure data suggest that the challenge is no longer simply scraping, but managing how automated traffic interacts with websites and the businesses that depend on them.
Scraping Is The Least Of The Problems
Many discussions among SEOs and site owners center on AI bots scraping. It’s a valid concern that AI systems harvest content for LLM training with virtually zero attribution when the content is remixed into an AI answer.
- Site owners worry about intellectual property.
- Search marketers worry about how AI systems use their content.
But infrastructure teams are increasingly seeing different and equally consequential problems.
The Banality Of Bots Getting Lost And Scraping Things
The issue is increasingly that many bots are creating unnecessary load, consuming resources, and sometimes becoming trapped in inefficient loops.
According to the report, one recurring pattern involved Meta’s meta-externalagent crawler following URL variations for days on end before mitigation systems caught on.
This kind of behavior is not malicious. It is automation operating with poor coding practices or insufficient guardrails.
Cloudflare’s David Belson illustrated the banality of lost bots draining resources:
“There’s the person who didn’t know what the hell they were doing yesterday, but vibe coded a bot today and let it loose. They’re not even bothering to check robots.txt.”
That observation captures an important reality. Today’s infrastructure problems now derive from poorly designed automation operating at scale.
Bots Are Consuming Resources Without Creating Value
The consequence of this behavior is that websites spend resources serving automated traffic that may provide little or no business value in return.
This is a big problem for ecommerce sites. Unlike requests for static pages, cart-related requests typically bypass caching and require the server to use resources. Depending on the site’s architecture, those requests can trigger PHP execution, database queries, session handling, and other resource-intensive processes.
Seen in this light, scraping is the least of a website’s problems. A crawler that repeatedly triggers expensive application logic and consumes server resources degrades performance for legitimate visitors.
The economic impact should not be ignored. According to the report, roughly 80% of AI crawling activity is associated with model training, eclipsing search or user-driven crawls.
For many businesses, the question is: Is there value returned by that traffic to justify the resources being consumed?
Businesses Are Trapped Between Visibility And Cost
If the solution were simply blocking bots, the problem would be solved. Unfortunately, many automated systems consuming resources are also connected to discoverability and visibility.
Some bots help search engines discover content. Some may contribute to AI citations and visibility in AI-generated answers. Others may simply consume content and resources without producing directly measurable business benefits.
Businesses are being asked to absorb the costs of automated traffic while simultaneously evaluating whether that traffic contributes enough visibility to justify those costs.
The Question Now: Which Bots Are Worth Paying For?
The report argues that site owners should ask this question:
Which bots, on which parts of my site, under what conditions?
Bot management affects visibility, infrastructure costs, and site performance. The goal is aligning automated traffic with business objectives.
Traffic Numbers May Already Be Affected
Automated traffic also affects website analytics. According to the report, AI bot traffic increased 300% over the past year. By the end of 2025, approximately one in every 31 visits on TollBit’s network originated from an AI bot.
As automated traffic grows, traffic volume alone becomes a less reliable indicator of audience growth.
A site can show rising visit counts while experiencing no corresponding increase in customers, subscribers, conversions, or revenue. In some cases, the additional traffic may be automated.
The report argues that the most meaningful signals come from metrics tied to actual business outcomes, including branded search demand, direct traffic, engagement quality, and revenue.
As automated systems account for a larger share of overall traffic, raw visit counts become less useful as a standalone measure of success.
Solutions And Mitigation Tactics
The report advocates a deliberate approach to bot management.
The first step is visibility.
Before making changes, site owners should understand what automated traffic is actually doing. The goal is not identifying every individual bot but identifying patterns such as repeated requests, loops, and activity focused on dynamic endpoints.
The second step is protecting high-cost site functions.
Cart URLs, checkout paths, internal search pages, filtered product pages, and parameter-heavy URLs often consume significantly more resources than standard content pages. Restricting unnecessary crawler access to those areas can reduce waste without affecting important content.
The report also recommends separating search crawlers from AI crawlers.
Not every bot provides the same value. Search crawlers contribute directly to discoverability and deserve broader access than AI training crawlers or unknown scrapers.
A single policy applied to every automated system can no longer be justified as the ecosystem grows more complex. That’s why the report advocates targeted changes rather than broad restrictions.
The goal is not eliminating automated traffic. The goal is managing it in a way that supports business objectives while reducing unnecessary costs. One way is to decide which bots can access specific parts of a site and under what circumstances.
Takeaways
Bot traffic is no longer primarily a scraping issue. The data suggests it has become an infrastructure, visibility, analytics, and business-management issue.
The biggest challenge is that many bots are consuming resources, triggering expensive functionality, inflating traffic metrics, and creating costs that site owners must absorb.
Bot management is not about blocking the most bots. It’s about managing bots according to what the site is optimizing for by distinguishing between valuable and wasteful automated traffic.
Read Kinsta’s data-backed report:
The AI & bot traffic reality check
Featured Image by Shutterstock/DC Studio
