Jan-Willem Bobbink shared a take on X, that AI visibility trackers are quietly breaking the analytics of brands who are paying them to track for them. It’s time we put more focus on this issue, as it is causing misalignment, misreporting, and misspending of resources and marketing budget in the clamor to be more visible in AI.
Jan-Willem hits on the issue of the lack of attribution in RAG loops. When a tracker triggers a prompt, and that prompt triggers a fetch, the brand is essentially paying a tool to generate its own AI visibility, and it begins to report on itself.
This is known as being ouroboros, which is a word you will likely see appearing more and more in the SEO industry as we describe AI/LLMs.
The ouroboros effect of how AI starts to quote itself, something that Pedro Dias has covered recently.
A large number of AI visibility tools have received significant amounts of funding in recent months, and some of them charge brands tens of thousands of dollars to “track” visibility, but this looping effect is beginning to become a reality, and how third-party tools track AI visibility will have a knock-on effect.
One example I point back to a lot is the drop in citations that ChatGPT produced when it released the 5.0 model in August 2025.
A number of tools that provide ChatGPT visibility saw the graphs decline, not because websites had violated spam policies or their short-termist tactics had run their course, but because of how the tools tracked citations, and the model produced less. This isn’t a measure of visibility, but a rehashed version of rank tracking, and these graphs can cost vendor contracts, incorrectly inform budget spending, and create false panic (or false celebration).
The Dangers Of The Observer Effect
In physics, the observer effect states that the act of monitoring a phenomenon changes it. This is happening in real-time for the SEO industry.
Most LLM trackers use a headless browser or a specialized API. When Perplexity or ChatGPT “searches” for fresh info to answer your tracker’s prompt, it doesn’t just hit your homepage; it performs a RAG fetch and can hit multiple URLs.
Because these bots often rotate IPs/proxies or use “stealth” headers to avoid being blocked by anti-scraping walls, they look like legitimate organic discovery crawls. This is how a number of rank tracking tools have operated for a number of years.
Because of this, you might report to a client, or other stakeholders, that “AI interest in our product pages is up 40%,” when in reality, 35% of that was just your own tracking tool refreshing its cache, or other tracking tools looking for you as a competitor of their brand.
AI Tracking Noise Is Worse Than Rank Tracking Noise
As Jan-Willem noted, we used to ignore rank tracker noise in Google Search Console because impressions were a “soft” metric. But log file data is hard data used for infrastructure, understanding how bots are accessing your website (server log file analysis), and now, in the age of AI, understanding how AI platforms are interacting with your site.
When you present a report to your client, peers, or your chief marketing officer, you are trying to prove brand preference within a large language model. If your data is polluted by your own tracking (and other people’s tracking), you risk a “false positive” strategy.
You might double down on content that isn’t actually popular with real AI users, but is simply the content your tracking tool happens to trigger most often.
What To Do Right Now
Until a vendor builds the “Clean Log” API Jan-Willem is calling for, you have to treat log files with skepticism.
Run your tracking tools on a “quiet” staging environment or a specific set of sacrificial URLs to measure the “noise floor” created by the tool itself.
Look for specific patterns (user-agent fingerprinting) in the logs that correlate with your tool’s scan times. Even if IPs rotate, the timing often shows patterns that can be identified easily.
And stop reporting “total AI fetches” as a success metric. Focus on how often your brand is mentioned relative to competitors, which is a metric derived from the LLM output, not your server logs.
More Resources:
Featured Image: Master1305/Shutterstock
