How to sanity check an AI visibility score
An AI visibility score is only as reliable as the questions and answers behind it. Before you trust one, ask which questions were tested, how many answers were collected, on which engines and in which country, what counted as visible, and how far the number has to move before the change is real.
In this guide
What is an AI visibility score?
An AI visibility score is the share of AI answers, to a set of questions, that name your brand. No engine publishes one, so every tool builds its own from the answers it collects.
The sum fits on one line: answers that name the brand, divided by answers collected. If a tool collects 200 answers and 24 of them name you, your score is 12%.
Some tools blend in more, such as where you appeared in the answer, how you were described or whether your website was cited, to make one score out of 100. That can be useful, but it’s no longer a count you can check by hand, so ask how the blend is weighted.
The answers can come from any of the engines people use to research a purchase: ChatGPT, Claude, Perplexity, Gemini, Microsoft Copilot, Google AI Overviews and AI Mode. A score only covers the engines its answers came from, so a ChatGPT score says nothing about Google.
Why do two tools give the same brand different scores?
Because they ask different questions, on different engines, in different countries, and count different things. A score describes the answers one tool collected, not every conversation a customer might have.
Definitions differ even at the source. ChatGPT’s Business Hub, OpenAI’s own report for businesses, counts a mention for an online business only when ChatGPT references its website, according to OpenAI’s help centre. A tool that counts every answer naming you is counting something broader, so the two will never match.
The gap between tools can be wide. Writing for Data-Mania, one author ran the same site through two tools in the same week and got scores of 22 and 4.
We publish two numbers ourselves, and they measure different things. The Vantage Index, our benchmark of UK brands, counts how often each brand is named in live AI answers to a fixed set of questions, as how the Vantage Index is measured explains. Our free AI checker gives a readiness score: how ready a website’s content and technical set-up are for AI search. The checker doesn’t put a single question to an AI engine, so its score says nothing about how often you’re named.
How many questions and answers does a score need?
More than most people expect: around 70 different questions to get within 5 points, and around 200 to get within 3. One question checked again and again tells you very little.
Those figures come from Sistrix, which studied 10,826 daily series across four AI models in eight countries. One question checked for 20 days read anywhere from 0% to 59%, around a true rate of 25%. A hundred different questions on a single day read from 21% to 29%. To avoid misleading results from a small sample, widen the questions before you repeat them.
Our own score shows the same pattern. We track how often Tilio is named in UK answers from ChatGPT and Google AI Overviews, on questions that don’t mention Tilio, and on 10 November 2026 we grew the questions we track from 41 to 1,041.
| Period | Questions tracked | Answers a day | Daily score, lowest to highest |
|---|---|---|---|
| 21 October to 9 November 2026 | 41 | 82 | 3.7% to 11.0% |
| 10 to 19 November 2026 | 1,041 | 2,082 | 4.0% to 5.9% |
On 82 answers a day, our daily score ranged over more than seven points, and a sample that size can swing that far by chance alone. On 2,082 answers a day, it stayed within two. The second range is the one worth reading.
Single answers vary even more. SparkToro, working with Gumshoe, collected 2,961 runs of 12 questions across ChatGPT, Claude and Google’s AI from 600 volunteers. On ChatGPT and Google’s AI, the same list of brands came back less than once in 100 runs.
Two research papers put it more formally. Single-run measures give “a misleadingly precise picture of domain performance”, according to one, and the other says visibility should be treated “as a distribution rather than a single-point outcome”.
How much does a score need to move before it means anything?
On around 100 questions, treat a move of less than 4 or 5 points as noise unless it holds for several weeks. Compare weekly averages, never single days.
Weekly averages work because they pool more answers. Sistrix found that with 20 questions, a weekly average carries about 1.8 points of uncertainty, against 4.8 for a single day. To track changes when AI answers vary from day to day, read the week, not the day.
A score can move a long way with nothing behind it. Data-Mania reported a score of 12 on 2 September 2026 and 22 four days later, on the same tool and the same 25 questions, with nothing published in between. Over 40 runs it averaged 32.2 and ranged from 12 to 54.
A score can move for three innocent reasons:
- Chance. Every answer is generated fresh, so a small sample swings on its own.
- A model update. Engines change their models and the sources they search, often without notice.
- A change to the questions. Adding, removing or rewording questions changes what’s measured. It’s why we show our two periods apart.
To tell campaign progress from normal AI answer variation, look for a move that’s bigger than the noise, holds for four weeks or more, and shows up on the questions the work was aimed at.
What does a single score hide?
It hides which engines name you, what they say and whether your own site is the source. Two brands can share a score and need opposite fixes.
Take one example from our Vantage Index for UK supermarkets. In the scan of 19 November 2026, across 124 questions and 372 answers, one supermarket scored 38.7% overall. By engine, that was 54.8% in ChatGPT, 22.6% in Google AI Overviews and 38.7% in AI Mode. The headline matches AI Mode by coincidence and hides a gap of more than 32 points between the other two, and closing it in AI Overviews may take different work from ChatGPT.
Two more things pad a total. A mention isn’t a recommendation: an answer can name you in a long list, or to warn against you, and it still counts. And questions that already name your brand, such as “is [your company] any good?”, nearly always mention you, so a score that includes them flatters you. Ask for branded and unbranded questions to be reported apart.
Three things are worth asking of any answer behind the score: are you there, how are you described, and where did the engine get it? The last one tells you what to fix, and our guide to mentions and citations explains why.
What should you ask whoever gives you the score?
Ask seven things, and expect a straight answer to each. They work on a tool, an agency report or one built in-house.
- Which questions were asked, and who chose them? You should be able to read the list, and it should be written the way your customers ask.
- How many answers sit behind the number, over how many days? Around 70 different questions, run over weeks, is what it takes to get within 5 points.
- Which engines, and which country? A UK score should be collected in the UK, and reported engine by engine as well as in total.
- Do questions that already name our brand count towards it? They should be reported apart, or the total flatters you.
- What counts as visible: named, cited or recommended? Each is a different result, and the score should say which one it counts.
- How far does the number have to move before you’d call it a real change? Anyone who knows their sample size can give you a figure, such as 4 or 5 points on 100 questions.
- Can I read the answers behind it? The dated answers are the evidence. Without them, a score can’t be checked.
Our guide to choosing the questions to track shows what a good list looks like.
None of these favours one kind of provider. The engines vary in the same way for everyone, so the method decides how reliable a score is, whoever runs it. Google’s own guidance makes a related point, advising caution with third-party tools that promise ranking success or claim access to its internal metrics.
Is an AI visibility score worth tracking?
Yes, as a direction and not a verdict. A well-built score tells you where you stand and whether you’re moving, and the answers behind it tell you what to do.
SparkToro’s study showed how inconsistent single answers are, and still concluded that visibility measured across dozens to hundreds of questions, run many times, “is a reasonable metric”. The IAB, the advertising industry’s standards body, draws a similar line in its framework for measuring visibility in AI answers. It separates “Decision-Grade” measurement from “Directional” measurement, and asks measurement providers to disclose how theirs works.
Use a score as you would any sampled figure. Watch the weekly trend, compare yourself with competitors measured on the same questions, and read the answers whenever the number moves. Our guides to what good AI visibility reporting looks like and what AI visibility platforms can and can’t measure go further on each.
Tilio, a UK answer engine optimisation (AEO) agency, tracks an agreed set of customer questions daily, keeps branded and unbranded questions apart, and reports mentions, recommendations and citations separately. Every percentage comes with its basis: the questions, the engines, the period and the number of answers. Our page on how we measure AI visibility sets out the method and its limits.
Frequently asked questions
What’s a good AI visibility score?
There isn’t a universal one. A score only compares with others built on the same questions, engines and country, so judge yours against competitors measured on the same set. The trend in your own score over several weeks tells you more than its size.
What’s a fair way to sanity check an agency’s claim about AI Overviews visibility?
Ask for the questions, the number of answers, the dates and the country, then read some of the answers yourself. Search Console’s Generative AI performance report gives a second view of how often your pages appeared in AI Overviews and AI Mode, though it only shows impressions.
How reliable is AI citation tracking data from an agency versus doing it in-house?
It’s as reliable as the method, whoever runs it. The engines vary in the same way for everyone. What counts is a fixed set of questions, daily runs, enough answers to narrow the range, and evidence you can read.
What are the limitations of testing AI visibility with a handful of questions?
A handful swings too much to trust. Sistrix found one question checked for 20 days read anywhere from 0% to 59% when the true rate was 25%. It takes around 70 different questions to get a score within 5 points.
Why did my AI visibility score change when nothing on my site changed?
Usually chance, a model update or a change to the questions being tracked. One documented score went from 12 to 22 in four days on the same tool and the same 25 questions, with nothing published in between.
How do I report AI visibility to a board?
Give one headline figure with its basis: the questions, the number of answers, the engines and the period. Add the trend as weekly averages, competitors measured on the same questions, and two or three real answers that show what customers see.
Written by Jack F, co-founder of Tilio, a UK AEO agency that tracks how businesses appear in AI answers and works to improve it. More about Jack F