AI search
AI Crawler Access and SEO: Which Bots to Allow or Block
AI crawler access controls whether AI assistants can find and cite your pages. You can block training bots without hurting Google rankings, but blocking search bots like OAI-SearchBot removes you from that AI search.
By SEORecheck Research Team, SEO auditorsPublished 9 min read

AI crawler access is the set of rules (in robots.txt, your firewall and your CDN) that decides which AI bots can fetch your pages. It matters for SEO because AI search tools can only cite pages their crawlers can reach. Blocking training bots doesn't hurt your Google rankings. Blocking AI search bots removes you from that tool's answers.
The tricky part is that one company often runs several bots with different jobs. Block the wrong one and you lose AI citations you wanted. Leave the wrong one open and your content ends up in training data you meant to keep out. This guide sorts the bots by job, explains what each choice costs, and shows how to check what your site actually lets through.
What are AI crawlers, and are they all the same?
No. AI crawlers fall into three groups: search bots that index pages so an AI tool can cite them, user-triggered fetchers that load a page when someone asks an assistant about it, and training bots that collect content for future models. Each group needs its own decision, and each vendor documents its own user-agent tokens.
The vendors' own documentation draws these lines clearly:
- OpenAI. OpenAI's crawler documentation describes OAI-SearchBot as the bot that surfaces websites in ChatGPT's search features and GPTBot as the crawler for training foundation models. ChatGPT-User handles actions started by a user, and OpenAI says robots.txt rules "may not apply" to it.
- Anthropic. Anthropic's help page lists ClaudeBot (model training), Claude-User (fetches pages when Claude users ask questions) and Claude-SearchBot (indexing to improve search results).
- Perplexity. Perplexity's bot guide describes PerplexityBot as the bot that surfaces and links websites in search results, not one used for model training. Perplexity-User supports user actions and "generally ignores robots.txt rules."
- Google. In Google's list of common crawlers, Google-Extended is a robots.txt token that controls whether Google may use your content to train Gemini models and to ground answers in Gemini Apps and Vertex AI. It has no separate user-agent string.
Which AI bots should you allow or block?
For most businesses that want customers to find them, the sensible default is to allow AI search bots and user fetchers, then make a separate business decision about training bots. Blocking a training bot doesn't remove you from AI search. Blocking a search bot does.
| Bot (robots.txt token) | Operator | Job | If you block it | Obeys robots.txt? |
|---|---|---|---|---|
| Googlebot | Google Search, including AI Overviews and AI Mode | You lose Google Search and Google's AI features | Yes | |
| Google-Extended | Gemini training and grounding outside Search | No effect on Google Search ranking or visibility | Yes (token only) | |
| OAI-SearchBot | OpenAI | ChatGPT search results | Pages stop being surfaced in ChatGPT search | Yes |
| GPTBot | OpenAI | Model training | Future content is kept out of training | Yes |
| ChatGPT-User | OpenAI | User-started fetches | Robots.txt may not apply | Not always |
| Claude-SearchBot | Anthropic | Search indexing | May reduce visibility in Claude's search results | Yes |
| Claude-User | Anthropic | User-started fetches | May reduce visibility when users ask about you | Yes |
| ClaudeBot | Anthropic | Model training | Future content is kept out of training | Yes |
| PerplexityBot | Perplexity | Perplexity search results | Pages aren't surfaced and linked by Perplexity | Yes |
| Perplexity-User | Perplexity | User-started fetches | Robots.txt generally ignored | No |
Microsoft Copilot is the odd one out. It mostly relies on Bing's index, so access there comes down to Bingbot and Bing's own controls. Our Microsoft Copilot search optimization guide covers that in detail.
A robots.txt example that splits search from training
This file lets AI search and user fetchers in but opts out of training. Copy the idea, not the file blindly: check it against your own rules first.
# AI search and user fetchers: allowed
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
Allow: /
# Model training: opted out
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /
# Everyone else, including Googlebot
User-agent: *
Disallow: /cart/
Disallow: /account/
Remember that a crawler follows only the most specific group that matches it. A bot named in its own group ignores your User-agent: * rules, so repeat any private-area disallows in that group. Our robots.txt for SEO guide explains grouping and matching rules.
Does blocking AI crawlers hurt your Google rankings?
Blocking Google-Extended, GPTBot, ClaudeBot or other non-Google bots doesn't change your Google rankings. Google says Google-Extended is not a ranking signal and doesn't affect Search visibility. The one AI-related block that hurts Google is blocking Googlebot, because Google's AI features are built on the normal Search index.
This is the misunderstanding we see most often. Site owners add Google-Extended to robots.txt to "keep out of AI Overviews" and find it does nothing there. Google's page on AI features says that to appear as a supporting link in AI Overviews or AI Mode, a page must be indexed and eligible to be shown in Google Search with a snippet. It names the controls that limit this: nosnippet, data-nosnippet, max-snippet and noindex.
Google's guide to optimizing for generative AI features adds one more requirement: the site must be included in Search generative AI features in Search Console. The same guide says you don't need special AI text files or markup. Files like llms.txt "neither harm nor help" your visibility in Google Search.
In practice:
- Want to stay in Google Search but out of AI Overviews? Snippet controls on the affected pages are the documented tool. They also shrink your regular snippets, so use them sparingly.
- Want to stay out of Gemini training? Disallow Google-Extended. Your Search rankings stay the same.
- Want to be cited in ChatGPT, Claude or Perplexity? Allow their search and user bots. These tools don't read Google's index directly.
What does blocking AI search bots cost you?
Blocking an AI search bot means that tool can't index your pages, so it can't cite or link them, even when someone asks about your business by name. The assistant may still answer, but from other sites, directories or old data, and without your page as the source.
Think about who asks AI assistants questions like "who installs heat pumps near me", "is [brand] legit" or "compare [product] vs [competitor]". These are often people close to buying. If OAI-SearchBot or PerplexityBot is blocked, those answers come from review sites, competitors or your old listings. That's why we treat search-bot access as part of SEO, not as a content-licensing question.
Training bots are a different trade-off. Allowing them may help models learn about your brand over time, but no vendor documents a link between training access and being cited. If your content is your product (paid research, courses, original data), opting out of training while keeping search access is a reasonable middle ground.
A note on timing: OpenAI says it can take about 24 hours after a robots.txt update for its systems to adjust. Don't judge a change on the same day.
Why might AI crawlers be blocked without your knowledge?
Many sites block AI bots without anyone deciding to. The usual causes are CDN defaults, bot-protection features, security plugins and firewall rules that challenge any non-browser traffic. Robots.txt can say "allow" while the firewall returns a 403 or a challenge page.
The biggest example is Cloudflare. On July 1, 2025 it changed its default to block AI crawlers unless they pay creators. If your site moved to Cloudflare or turned on its bot features since then, check what's actually configured instead of assuming.
Other common silent blocks:
- Bot-fight or "block AI scrapers" toggles. Perplexity notes that Cloudflare's Bot Fight Mode can't be skipped with WAF custom rules, so allow-listing may need other settings.
- IP-based blocking. Anthropic warns that blocking by IP may not work reliably, because it can stop bots from reading your robots.txt preferences at all.
- Rate limits that throttle bursts of requests from one crawler.
- Staging rules left on after a launch, such as a blanket
Disallow: /or a password-protected path.
If you allow-list bots in a firewall, verify them properly. Perplexity recommends matching both the user-agent string and the published IP ranges, because anyone can fake a user-agent. OpenAI, Anthropic and Perplexity all publish IP lists for this.
Can AI crawlers read your content once they get in?
Not always. Access alone isn't enough: many AI crawlers read only the raw HTML and don't run JavaScript. If your main text, prices or product details appear only after scripts run, an AI bot with full access may still see a nearly empty page.
A Vercel and MERJ analysis of AI crawler traffic published in December 2024 found that "none of the major AI crawlers currently render JavaScript", including GPTBot, ClaudeBot and PerplexityBot. They downloaded JavaScript files but didn't execute them. The same study found that about a third of GPTBot and ClaudeBot fetches hit 404 pages, which is another reason to keep redirects and old asset URLs clean.
Google is different. Its AI optimization guide says Google can process JavaScript content as long as it isn't blocked. So a client-rendered site can do fine in AI Overviews and still be invisible in ChatGPT or Perplexity. The fix is server-side rendering, static generation or at least key content in the initial HTML. See JavaScript SEO rendering problems for how to test this.
How do you check your site's AI crawler access?
Check three layers in order: what robots.txt says to each AI user agent, what the server actually returns to those bots, and what the raw HTML contains. A gap between any two layers is a finding. Then confirm with server logs that real, verified bots are reaching your pages.
- Read robots.txt as each bot. Open
/robots.txtand, for each token in the table above, find the group that applies to it. Watch for a broadUser-agent: *disallow that catches bots you didn't list. - Test the response. Request a key page with the bot's user-agent string (for example with
curl -A). A 403, a 503 or a challenge page means your firewall or CDN overrides robots.txt. Spoofed tests can be treated differently from real bot IPs, so confirm in the logs too. - Compare raw HTML with the rendered page. View source, not the inspector. If headings, body text or prices are missing from the source, AI bots that don't render JavaScript won't see them.
- Check snippet controls. Look for
nosnippet,max-snippetordata-nosnippeton pages you want quoted. - Verify in logs. Filter access logs by AI user agents and match their IPs against the published lists. Our guide to server log analysis for SEO walks through it.
- Ask the assistants. Ask ChatGPT, Claude and Perplexity questions about your business and see whether your pages are cited.
If you'd rather not do this by hand, a free SEORecheck audit checks AI crawler access in robots.txt for search, user and training bots, snippet controls, text that appears only after JavaScript runs, and your Organization/sameAs markup. You can see how these checks are reported in the sample report.
After any change, recheck. A plugin update, a CDN setting or a new firewall rule can undo your access rules without notice. Keep robots.txt in version control and add an AI-bot test to your release checklist.
Key takeaways
- AI bots do different jobs: search indexing, user-started fetches and model training. Decide on each group separately.
- Blocking Google-Extended, GPTBot or ClaudeBot doesn't affect Google rankings. Blocking Googlebot removes you from Google Search and its AI features.
- To appear in AI Overviews and AI Mode, a page must be indexed and snippet-eligible, and the site must be included in Search generative AI features in Search Console. llms.txt doesn't help Google Search.
- Blocking OAI-SearchBot, Claude-SearchBot or PerplexityBot keeps your pages out of those tools' answers and citations.
- Firewalls and CDNs, especially Cloudflare's AI-blocking default since July 2025, often block bots that robots.txt allows.
- Many AI crawlers don't run JavaScript, so put key content in the initial HTML.
- Check access in robots.txt, in server responses, in the raw HTML and in logs, then check again after every deploy.


