Measuring AI visibility statistically: sampling, confidence intervals, position bias and multiple testing
By the SEOCells team, OmCells R&D Labs LLP · Published
AI answers are stochastic, so visibility must be estimated, not read off one query. Treat each run as a trial, report the citation rate with a score confidence interval, separate whether you are in the candidate set from whether a given run shows you, randomise order in head-to-head tests, and control false discoveries when testing many factors.
Key points
- 0 citations in 41 answers is not “0%”: the 95% score interval is 0%–8.6%.
- Separate “is my page ever a candidate?” from “how often does a run cite it?” — they need different fixes.
- Both people and LLM judges favour the first position; swap order and average.
- Testing hundreds of factors needs false discovery rate control (Benjamini–Hochberg).
- Clicks undercount visibility: with an AI summary, Pew observed result clicks in 8% of visits vs 15% without.
Why is a single query not a measurement?
Answer engines sample their output (decoding temperature), may issue different sub-queries on each run, and retrieve from indexes that change. The same question can therefore cite different sources minutes apart. A single observation is one Bernoulli trial; any rate claimed from it has an interval of almost the whole unit range.
How should a citation rate be estimated?
With n independent runs of a question and x runs citing you, the citation rate is a binomial proportion p̂ = x/n. The textbook Wald interval p̂ ± z√(p̂(1−p̂)/n) collapses to zero width when x = 0 or x = n and has poor coverage for small n. Agresti and Coull (1998) showed that the score interval (Wilson, 1927) behaves far better:
centre = ( p̂ + z²/(2n) ) / ( 1 + z²/n )
half = z / ( 1 + z²/n ) · √( p̂(1 − p̂)/n + z²/(4n²) )
Example: x = 0, n = 41, z = 1.96 → 95% interval 0% to 8.6%
x = 0, n = 10 → 95% interval 0% to 27.8%The example is real: an AEO Report on seocells.ai itself (1 November 2026) found 0 citations in 41 sampled answers. The honest statement is “cited in fewer than about 9% of answers, with 95% confidence”, not “never cited”.
Is it a retrieval problem or a selection problem?
A low citation rate has two different causes that call for different fixes. Either the page is rarely among the candidates an answer engine retrieves for the question — a relevance, crawlability or authority problem — or it is retrieved but usually loses to other sources when the answer is written — a passage-quality, evidence or position problem. Repeated sampling across runs, days and engines is what lets the two be told apart: a page that is never seen differs from one that appears in some runs but not others.
How do position effects distort comparisons?
In web search, the probability of a click depends strongly on rank, partly because users examine top results more (Craswell et al., 2008); Joachims et al. (2017) showed how to correct learning-to-rank for this bias by re-weighting each click by how likely its position was to be examined. Language models acting as judges show an analogous order effect: Zheng et al. (2023) documented position bias in LLM-as-a-judge comparisons and recommended swapping positions. Any head-to-head test of your passage against a competitor's should therefore run both orders, blind the source names, and report the rate at which slot one wins regardless of content — a direct estimate of the bias.
How should rank-weighted visibility be summarised?
Being cited first is worth more than being cited fifth. Discounted cumulative gain (Järvelin & Kekäläinen, 2002) weights a result at rank i by 1/log₂(i + 1); normalising by the ideal ordering gives nDCG in [0, 1]. Applied to answer citations, it yields one position-aware visibility figure per question that can be averaged across a question set.
DCG@k = Σ_{i=1..k} rel_i / log2(i + 1) nDCG@k = DCG@k / IDCG@kWhy do many-factor analyses need false discovery control?
An audit that tests hundreds of page traits against rankings will, at α = 0.05, flag about 5% of truly irrelevant traits as “significant”. Benjamini and Hochberg (1995) control the expected proportion of false discoveries instead: sort the m p-values ascending, find the largest k with p_(k) ≤ (k/m)·q, and reject hypotheses 1…k. With q = 0.1, at most about one in ten reported factors is expected to be a false lead. Recommendations derived without such control should be read as hypotheses.
Why are clicks not enough?
Pew Research Center analysed the browsing of 900 U.S. adults in March 2025: users who saw a Google AI summary clicked a traditional result in 8% of visits, against 15% for users who did not, and clicked a link inside the summary in just 1% of visits. A visibility programme measured only by referral clicks will undercount its effect; brand mentions in answers, citation rates and later branded searches belong in the measurement model.
How does SEOCells apply this?
The SEOCells AEO Report asks each question several times per engine, reports rates with 95% intervals and a stability figure, compares your page fairly with the pages that are cited, and scores commercial impact separately from citations. Every estimate is shown with its range, and expected lifts are labelled as measured or not yet validated.
Questions answered
How many times should I ask an AI assistant the same question?
Enough to make the interval useful. With 10 runs and no citation, the 95% score interval still reaches about 28%; with 40 runs it narrows to about 9%. Repeat across days, because indexes and models change.
What is AI share of voice?
The share of citations or brand mentions in AI answers, across a defined question set, that belong to you rather than competitors. It should be reported with an interval and a fixed, documented question set.
Why report a range instead of one score?
Because AI answers vary between runs. A single number hides that variation and makes normal noise look like progress or decline.
References
- Wilson, E. B. (1927). Probable Inference, the Law of Succession, and Statistical Inference. JASA 22(158). doi:10.1080/01621459.1927.10502953
- Agresti, A. & Coull, B. A. (1998). Approximate is Better than “Exact” for Interval Estimation of Binomial Proportions. The American Statistician 52(2). doi:10.1080/00031305.1998.10480550
- Craswell, N., Zoeter, O., Taylor, M. & Ramsey, B. (2008). An Experimental Comparison of Click Position-Bias Models. WSDM 2008. doi:10.1145/1341531.1341545
- Joachims, T., Swaminathan, A. & Schnabel, T. (2017). Unbiased Learning-to-Rank with Biased Feedback. WSDM 2017. arXiv:1608.04468
- Zheng, L. et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS 2023 Datasets and Benchmarks. arXiv:2306.05685
- Järvelin, K. & Kekäläinen, J. (2002). Cumulated Gain-Based Evaluation of IR Techniques. ACM TOIS 20(4). doi:10.1145/582415.582418
- Benjamini, Y. & Hochberg, Y. (1995). Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing. JRSS-B 57(1). doi:10.1111/j.2517-6161.1995.tb02031.x
- Pew Research Center (22 July 2025). Google users are less likely to click on links when an AI summary appears in the results
Search engines and AI assistants change often; check the linked documentation for the latest details. Nothing in this guide is a promise of rankings, citations or results.
Get your citation rate with a range, and see who is cited instead.
See how the AEO Report measures