When ChatGPT, Claude, or Google’s AI get asked for brand or product recommendations, they almost never return the same list twice — and almost never in the same order.
That’s the big finding from a new study from Rand Fishkin, CEO and co-founder of SparkToro, and Patrick O’Donnell, CTO and co-founder of Gumshoe.ai. They investigated whether generative AI recommendations are sufficiently consistent to be measured.
What they tested. Six hundred volunteers ran 12 identical prompts through ChatGPT, Claude, and Google’s AI nearly 3,000 times.
- Each response was normalized into an ordered list of brands or products. The team then compared those lists for overlap, order, and repetition.
- The goal was to see how often the same answers actually appeared.
The short answer: almost never. Across tools and prompts, the odds of getting the same list twice were under 1 in 100. The odds of getting the same list in the same order were closer to 1 in 1,000.
- Even list length varied wildly. Some responses named two or three options. Others named 10 or more.
- If you don’t like the result, the data suggests a simple fix: ask again.
Why we care. We’ve heard that personalization drives AI answers. This is the first research that puts real numbers behind that claim — and the implications are massive. If you’re looking for a concrete way SEO and GEO diverge, this is it.
Random by design. This isn’t a flaw. It’s how these systems work.
- Large language models are probability engines. They’re designed to generate variation, not to return a stable, ordered set of results.
- Treating them like Google’s blue links misses the point and produces bad metrics.
One thing that works. While rankings collapsed under scrutiny, one metric held up better than expected: visibility percentage.
- Some brands appeared again and again across dozens of runs, even though their position jumped around. In some cases — hospitals, agencies, consumer brands — names showed up in 60% to 90% of responses for a given intent.
- Repeat presence means something. Exact rank does not.
Size matters. The smaller the market, the more stable the results.
- In tight spaces — like regional service providers or niche B2B tools — AI answers clustered around a few familiar names. In massive categories — like novels or creative agencies — results scattered into chaos.
- More options create more randomness.
Prompts are chaos. The team also tested real human prompts, and they were a mess — in a very human way.
- Almost no two prompts looked alike, even when people wanted the same thing. Semantic similarity was extremely low.
- Here’s the surprise: despite wildly different phrasing, AI tools still returned similar brand sets for the same underlying intent.
Intent survives. For headphone recommendations, hundreds of unique prompts still surfaced leaders like Bose, Sony, Apple, and Sennheiser most of the time.
- Change the intent — gaming, podcasting, noise canceling — and the brand set changed with it.
- That suggests AI tools do capture intent, even when prompts are strange.
What’s useless. Tracking “position” in AI answers.
- The study is blunt: ranking positions are so unstable they’re effectively meaningless. Any product selling AI rank movement is selling fiction.
What might work. Track how often your brand appears across many prompts, run many times. It’s imperfect. It’s messy. But it’s closer to reality than pretending AI answers behave like search rankings.
Open questions. Fishkin points to gaps that still need answers.
- How many runs are needed to make visibility numbers reliable?
- Do APIs behave like real users?
- How many prompts accurately represent a market?
Bottom line. AI recommendation lists are inherently random. Visibility — measured carefully and at scale — may still tell you something real. Just don’t confuse it with ranking.
Search Engine Land is owned by Semrush. We remain committed to providing high-quality coverage of marketing topics. Unless otherwise noted, this page’s content was written by either an employee or a paid contractor of Semrush Inc.
