More customers from Google, AI and social
Sign inStart free
AI-native DeepTechDeepSEODeepAEOAI agentsGenerative searchHuman-approved
Advanced · with research citations

How AI search engines retrieve, rank and cite sources: a technical primer

By the SEOCells team, OmCells R&D Labs LLP · Published

AI search engines answer in stages: they rewrite the question into searches, retrieve candidate passages with lexical (BM25) and dense (embedding) retrievers, re-rank them with a stronger model, and generate an answer conditioned on the top passages, attaching citations. A page is cited only if it survives every stage, so each stage implies different content requirements.

Key points

  • Lexical matching still matters: BM25 remains a robust baseline across domains (BEIR, 2021).
  • Dense retrievers match meaning, so clear, unambiguous passages that name their subject retrieve better.
  • Re-rankers judge a query and passage together: the passage itself must answer the question.
  • Language models use information at the start and end of their context best (“lost in the middle”).
  • Citation quality in generative search is imperfect: in one audit only 51.5% of sentences were fully supported.

What is the pipeline behind an AI answer?

Production systems differ and their internals are not fully published, but the research literature and the engines' own documentation describe a common architecture: retrieval-augmented generation (RAG). Lewis et al. (2020) formalised RAG as a model that combines knowledge stored in its weights (a pre-trained sequence-to-sequence model) with an external index of documents that a neural retriever searches. WebGPT (Nakano et al., 2021) showed the browsing variant: a model that issues searches, reads pages and must collect references to support its answer.

  1. Query understanding: the question is rewritten or expanded into one or more search queries (sometimes several in parallel, often called query fan-out).
  2. Candidate retrieval: a first-stage retriever returns hundreds to thousands of candidate documents or passages cheaply.
  3. Re-ranking: a more expensive model scores each query–passage pair and keeps the best few.
  4. Generation: the language model writes the answer with the top passages in its context window.
  5. Attribution: sentences are linked to the passages that support them, shown as citations.

How does lexical retrieval (BM25) score a page?

BM25, from the probabilistic relevance framework (Robertson & Zaragoza, 2009), scores a document D for a query Q by summing, over query terms, an inverse document frequency weight times a saturating function of the term's frequency in D, normalised by document length:

score(D, Q) = Σ_i IDF(q_i) · f(q_i, D) · (k1 + 1)
                         / ( f(q_i, D) + k1 · (1 − b + b · |D| / avgdl) )

f(q_i, D)  frequency of term q_i in D      |D|  length of D      avgdl  mean length
k1 ≈ 1.2–2.0 (term-frequency saturation)    b ≈ 0.75 (length normalisation)

Two practical consequences follow. Saturation (k1) means repeating a term adds less and less — keyword stuffing buys almost nothing. IDF means rare, specific terms (product names, places, technical terms) discriminate far more than common words. In the BEIR benchmark of 18 datasets, Thakur et al. (2021) found BM25 a robust zero-shot baseline, with re-ranking and late-interaction models best on average but at high computational cost.

What changes with dense (embedding) retrieval?

Dense Passage Retrieval (Karpukhin et al., 2020) encodes questions and passages separately with two BERT encoders and scores them by inner product, sim(q, p) = E_Q(q)ᵀE_P(p), retrieved with maximum inner product search. On open-domain QA benchmarks it beat a strong Lucene-BM25 system by 9–19 percentage points in top-20 passage retrieval accuracy. Because a whole passage is compressed into one vector, a passage that mixes several topics produces a blurred vector; one idea per passage retrieves more precisely.

ColBERT (Khattab & Zaharia, 2020) keeps one vector per token and scores with late interaction, Σ_i max_j q_i·d_j (MaxSim), which preserves term-level matching while staying efficient. HyDE (Gao et al., 2022) shows another pattern: a model first writes a hypothetical answer, and the system retrieves real documents near that answer's embedding. Pages that read like a good answer to the question sit close to such vectors.

How does re-ranking decide the final sources?

Cross-encoders such as monoBERT (Nogueira & Cho, 2019) read the query and a candidate passage together and output a relevance score; on MS MARCO this re-ranking of BM25 candidates was a large step over the previous state of the art. A cross-encoder sees exactly the passage you wrote, in context of the question — so the passage must contain the answer explicitly, not imply it across several paragraphs.

Does position on the page or in the context matter?

Liu et al. (2023) showed that language models use information best when it appears at the beginning or end of a long input context and that performance degrades when the relevant information sits in the middle, even for long-context models. Exactly how a given engine orders passages is not public, but the finding supports an editorial rule that costs nothing: state the answer first, in the opening passage of the page and of each section.

How reliable are AI citations?

Liu, Zhang and Liang (2023) audited four generative search engines with human evaluation: on average only 51.5% of generated sentences were fully supported by their citations, and only 74.5% of citations supported the sentence they were attached to. The ALCE benchmark (Gao et al., 2023) found that even the best models lacked complete citation support about half of the time on ELI5. For publishers this cuts both ways: your page may be used without being cited, or cited for a claim it does not make. Monitoring the actual answers, not only clicks, is necessary.

What does this imply for content?

Pipeline stage, what it rewards, and the corresponding editorial practice
StageWhat it rewardsPractice
Crawling and indexingReachable, renderable textAllow search crawlers; server-render the main text
Lexical retrievalThe query's discriminative terms presentUse the customer's words and specific names, without repetition
Dense retrievalPassages semantically close to the questionOne idea per passage; name the subject; answer-like phrasing
Re-rankingThe answer explicitly in the passageDirect 40–60 word answers under question headings
GenerationEvidence early in context; quotable factsAnswer first; numbers, dates, entities, sources
AttributionSentences traceable to one passageSelf-contained claims with their evidence beside them

Aggarwal et al. (2024) tested such content edits on a benchmark of generative-engine queries and reported visibility gains of up to 40%, varying by domain. Treat that as evidence that presentation matters, not as an expected effect for any one site.

Questions answered

Do AI search engines use keywords or meaning?

Both. First-stage retrieval typically combines lexical methods such as BM25, which reward the presence of specific query terms, with dense retrieval, which matches meaning through embeddings. A good page uses the customer's words and is semantically clear.

Why is my page indexed but never cited?

Indexing only makes a page eligible. It must also be retrieved for the question, survive re-ranking against competing passages, and contain a passage that states the answer plainly enough to be quoted.

Does putting the answer at the top really matter?

Research on long contexts (Liu et al., 2023) shows models use information at the beginning or end of their input best. Engine internals are not public, but answering first is a low-cost practice consistent with that evidence.

References

  1. Lewis, P. et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. NeurIPS 2020. arXiv:2005.11401
  2. Nakano, R. et al. (2021). WebGPT: Browser-assisted question-answering with human feedback. arXiv:2112.09332
  3. Robertson, S. & Zaragoza, H. (2009). The Probabilistic Relevance Framework: BM25 and Beyond. Foundations and Trends in Information Retrieval 3(4). doi:10.1561/1500000019
  4. Thakur, N. et al. (2021). BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models. NeurIPS 2021 Datasets and Benchmarks. arXiv:2104.08663
  5. Karpukhin, V. et al. (2020). Dense Passage Retrieval for Open-Domain Question Answering. EMNLP 2020. arXiv:2004.04906
  6. Khattab, O. & Zaharia, M. (2020). ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT. SIGIR 2020. arXiv:2004.12832
  7. Gao, L. et al. (2022). Precise Zero-Shot Dense Retrieval without Relevance Labels (HyDE). arXiv:2212.10496
  8. Nogueira, R. & Cho, K. (2019). Passage Re-ranking with BERT. arXiv:1901.04085
  9. Liu, N. F. et al. (2023). Lost in the Middle: How Language Models Use Long Contexts. Transactions of the ACL (2024). arXiv:2307.03172
  10. Liu, N. F., Zhang, T. & Liang, P. (2023). Evaluating Verifiability in Generative Search Engines. Findings of EMNLP 2023. arXiv:2304.09848
  11. Gao, T. et al. (2023). Enabling Large Language Models to Generate Text with Citations (ALCE). EMNLP 2023. arXiv:2305.14627
  12. Aggarwal, P. et al. (2024). The GEO paper. KDD 2024. arXiv:2311.09735
  13. Google Search Central — AI features and your website

Search engines and AI assistants change often; check the linked documentation for the latest details. Nothing in this guide is a promise of rankings, citations or results.

See which passages of your page AI answers actually use, and who they cite instead.

See the AEO Report