AI search
How AI Search Engines Select Sources: What's Documented
AI search engines select sources by first retrieving pages from a search index they built by crawling, then citing the pages whose content supports the answer they generate. If a page can't be crawled or indexed, it can't be picked.
By SEORecheck Research Team, SEO auditorsPublished 9 min read

AI search engines select sources in two steps. First they retrieve candidate pages from a search index they built by crawling the web, often by running several related searches. Then the model writes an answer from those pages and cites the ones that support what it says. If a page isn't crawled and indexed, it can't be picked.
That's the short version, and it holds across Google, ChatGPT search, Copilot, Perplexity and Claude. The details differ by platform, and much of what gets sold as "AI ranking factors" isn't documented anywhere. This guide covers what each platform actually publishes, what it doesn't, and what that means for your site.
How do AI search engines select sources?
They don't search the whole web live and then pick a favorite. They pull pages from a search index using their ranking systems, read what those pages say, and attach links to the pages that support each part of the answer. Selection is retrieval first and citation second.
Google describes this most clearly. Its guide to optimizing for generative AI features on Google Search says its AI features use retrieval-augmented generation (RAG), grounding responses "by relying on our core Search ranking systems to retrieve relevant, up-to-date web pages." The systems then review the information on those retrieved pages and show "prominent, clickable links to relevant web pages that support the information."
So to be cited by AI search, a page first has to be a strong search result for at least one of the queries the system runs. After that it needs to contain a specific statement the model can use. A page that ranks but never states anything clearly gives the model nothing to quote. A page with a great answer that isn't indexed never gets retrieved. Both steps matter, and the first one is plain SEO. We cover that overlap in GEO vs SEO.
What happens before an AI answer cites a page?
Before any citation appears, a page goes through a pipeline that looks a lot like classic search, plus one extra step at the end:
- Crawling. A platform-specific bot fetches the page. If robots.txt blocks that bot, or a firewall blocks it, the pipeline stops here.
- Indexing. The page is processed and stored in the platform's (or a partner's) search index.
- Query rewriting and fan-out. The user's question is turned into one or more search queries.
- Retrieval and ranking. The index returns the most relevant pages for those queries.
- Generation and grounding. The model writes the answer from the retrieved content and links the pages that back up its claims.
Steps 1, 2 and 4 are the ones you can check and fix. Step 3 explains why pages that don't rank for the main query still get cited. Step 5 is where clear, specific writing helps.
Which index and crawler does each AI search engine use?
Each platform documents a different crawler and a different way to opt out. The table below sticks to what each vendor's own documentation says. Anything a vendor doesn't publish is marked as not documented.
| Platform | What the docs say about retrieval | Crawler that controls search visibility | Where to measure |
|---|---|---|---|
| Google AI Overviews / AI Mode | RAG on Google's core Search ranking systems and Search index (Google) | Googlebot (normal Search crawling) | Search Console Generative AI performance report |
| ChatGPT search | Sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers" (OpenAI) | OAI-SearchBot | Not documented on that page |
| Microsoft Copilot / Bing AI summaries | Grounding queries are "the key phrases the AI used when retrieving content" (Bing) | Bing's crawling and index | Bing Webmaster Tools AI Performance |
| Perplexity | PerplexityBot is "designed to surface and link websites in search results" (Perplexity) | PerplexityBot (index), Perplexity-User (live visits) | Not documented on that page |
| Claude | Blocking Claude-SearchBot "may reduce your site's visibility and accuracy in user search results" (Anthropic) | Claude-SearchBot, Claude-User | Not documented on that page |
Two patterns stand out. First, each vendor keeps search crawlers separate from training crawlers. OpenAI says GPTBot is for model training and doesn't decide search appearance. Anthropic says the same about ClaudeBot, and Perplexity says its bots aren't used to train foundation models. Blocking a training bot doesn't take you out of AI search. Blocking a search bot does.
Second, the "user" agents (ChatGPT-User, Perplexity-User, Claude-User) fetch pages live when someone asks a question. OpenAI states that ChatGPT-User "is not used to determine whether content may appear in Search." For the robots.txt rules by bot, see our robots.txt guide. For platform-specific steps, see the guides for ChatGPT, Perplexity and Copilot.
What is query fan-out, and why does it change which sources are picked?
Query fan-out means the AI system breaks one question into several related searches and draws on the results of all of them. A page can be cited because it ranks for a narrow sub-question, even if it never ranks for the user's original wording.
Google's AI features documentation says AI Overviews and AI Mode "may use a 'query fan-out' technique — issuing multiple related searches across subtopics and data sources — to develop a response." Its optimization guide gives an example: for "how to fix a lawn that's full of weeds," the fan-out queries might cover herbicide recommendations and ways to prevent weeds.
Here's what that looks like for a small business. Say someone asks, "Is a heat pump worth it for an old house in a cold climate?" The system might search separately for heat pump performance in low temperatures, insulation requirements, running costs and installation in older buildings. An installer's page that clearly explains insulation requirements for older homes could be cited for that one part, even if a national publisher wins the main query.
Microsoft's "grounding queries" in Bing Webmaster Tools are the same idea seen from the publisher's side: the phrases the AI actually searched for when it retrieved your page. That's also why Google says AI features show "a wider and more diverse set of helpful links" than classic results.
Does the AI choose the best page or the best passage?
It's both, but no platform publishes how it weighs them. What Google does document is that its systems review "the specific information from those retrieved pages" and link pages that support the answer. In practice, a page earns a citation when it contains a specific, checkable statement that matches part of the answer.
A few things follow from that, without relying on any secret formula:
- Direct answers help. A sentence like "Most residential heat pumps need X to work well in older homes, because…" is easier to ground an answer on than a vague introduction.
- Specifics beat generic text. Google's guide asks for "non-commodity content" with "a unique point of view." If your page repeats what every other page says, the model has no particular reason to pick yours.
- You don't need special formatting for AI. Google says there's "no requirement to break your content into tiny pieces for AI to understand it." Normal headings and paragraphs work fine.
Bing's AI Performance announcement gives similar advice: deeper coverage of the topics you're cited for, clear structure such as headings and tables, claims backed by evidence and examples, and information kept current.
Does freshness affect which sources AI search selects?
Yes, at least for Google and Bing. Both name freshness in their documentation. Google says its RAG approach uses "up-to-date web pages from our Search index" to improve accuracy and freshness. Microsoft recommends IndexNow so AI systems reference "the most current version" of your pages.
What neither says is that newer pages always win. Freshness comes into play through the index: if a crawler hasn't picked up your updated price list or opening hours, the AI can only retrieve the old version. So for AI visibility, a refresh only counts once it's recrawled and reindexed. That's why notifying search engines (IndexNow for Bing, an accurate sitemap lastmod for Google) matters more than changing a date on the page. Our content refresh strategy explains how to update pages so the change is real and gets picked up.
What is not documented about AI source selection?
No AI search platform publishes a list of citation ranking factors, a weighting of trust signals, or a minimum authority threshold. Claims about exact "AI ranking factors," citation scores or guaranteed tactics are guesses or vendor marketing, not documentation.
Google goes further and rules out several popular tactics in its optimization guide:
- llms.txt: "Google Search itself doesn't use them."
- Special schema: "Structured data isn't required for generative AI search, and there's no special schema.org markup you need to add." Structured data still helps search engines understand your entities. It just isn't an AI-citation switch.
- Rewriting for AI: no need to rewrite your content specifically for AI systems.
- Inauthentic mentions: planting brand mentions across the web isn't effective.
Other platforms say less. OpenAI, Perplexity and Anthropic document their crawlers and opt-outs but not how they rank sources. Treat anything beyond that, including correlation studies of cited domains, as a hypothesis to test on your own site, not as a rule.
How do you check whether AI search engines can select your pages?
Work backwards through the pipeline. Confirm that the right bots can crawl your pages, that the pages are indexed and snippet-eligible, that the important text is in the HTML, and then measure actual citations in Search Console and Bing Webmaster Tools.
A practical checklist:
- Robots.txt and firewall. Allow Googlebot, Bingbot, OAI-SearchBot, PerplexityBot and Claude-SearchBot if you want to appear in those products. Decide separately on training bots such as GPTBot, ClaudeBot and Google-Extended. Check that your CDN or bot protection isn't blocking search bots that robots.txt allows.
- Indexing and snippets. Google requires pages to be "indexed and eligible to be shown in Google Search with a snippet."
nosnippet,max-snippet:0ornoindexremove that eligibility. Also confirm your site is included in Search generative AI features in Search Console. - Text in the HTML. If key answers only appear after JavaScript runs, crawlers that don't render JavaScript may never see them. See JavaScript SEO rendering problems.
- Clear entity information. Consistent Organization markup with
sameAslinks, and an accurate Wikidata entry if your business has one, help systems connect your site to your brand. - Answerable pages. For the questions customers actually ask about you, such as prices, service areas or comparisons, make sure a page states the answer plainly.
A free SEORecheck audit (request one here) runs these GEO checks on your site: AI crawler access for search, user and training bots, snippet controls, JavaScript-only text, Organization/sameAs and Wikidata, and how your pages answer the questions people ask AI assistants about your business. The sample report shows the format.
Then measure. Google's Generative AI performance report and Bing's AI Performance report (citations and grounding queries) are first-party data. For what to track across both classic and AI search, see SEO vs GEO performance measurement.
Key takeaways
- AI search engines select sources by retrieving from a search index first and citing supporting pages second. Being crawlable and indexed comes before everything else.
- Query fan-out means a page can be cited for a sub-question it ranks for, not just the main query.
- Search bots and training bots are separate. Blocking OAI-SearchBot, PerplexityBot or Claude-SearchBot hurts AI search visibility. Blocking GPTBot or ClaudeBot doesn't.
- Google requires indexing, snippet eligibility and inclusion in Search generative AI features. It doesn't require special schema, chunking or llms.txt.
- No platform publishes citation ranking factors. Clear, specific, current, original content is what the documentation actually asks for.
- Measure citations in Search Console's Generative AI performance report and Bing Webmaster Tools' AI Performance report.


