Guide
robots.txt for AI crawlers
Who each AI crawler is, what blocking it does, and rules you can copy. Then test your live file for free.
The crawlers
| User-agent | Operator | What it does |
|---|---|---|
| GPTBot | OpenAI | Crawls content that may be used to train OpenAI models |
| OAI-SearchBot | OpenAI | Builds the search index behind ChatGPT search results |
| ChatGPT-User | OpenAI | Fetches a page when a ChatGPT user asks about it |
| ClaudeBot | Anthropic | Crawls content that may be used to train Claude models |
| PerplexityBot | Perplexity | Builds the Perplexity search index |
| Google-Extended | A control token (not a separate crawler): lets you opt out of Gemini training and grounding while staying in Google Search | |
| CCBot | Common Crawl | Builds the open Common Crawl archive many models train on |
| Applebot-Extended | Apple | A control token that lets you opt out of Apple AI training |
Operators change their crawler names and purposes; confirm against each operator's own documentation before relying on this table.
Allow AI crawlers explicitly
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /Block training crawlers, stay in AI search
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /What sites actually do
In our scan of 2,080 top WordPress sites, 81% said nothing about AI crawlers, 17% blocked at least one, and 2% explicitly allowed them.
Test it
- Open
yourdomain.com/robots.txtand read what is actually served. Caches and plugins often differ from what you expect. - Check that no earlier
User-agent: *rule contradicts your AI rules. - Run the free scan below to see how the file reads to a checker.
Pair this with an llms.txt so that what you allow and what you recommend agree.
FAQ
Will blocking GPTBot remove me from Google?
No. GPTBot is OpenAI's crawler. Google Search uses Googlebot, which is separate. Google-Extended is a control token that does not affect Search ranking.
Is robots.txt enforced?
No. It is a request that well-behaved crawlers honour. It is not access control. If content must be private, put it behind a login.
Where is robots.txt on WordPress?
WordPress serves a virtual robots.txt at /robots.txt. A physical robots.txt file in the web root overrides it, and SEO plugins such as Yoast and Rank Math include an editor.
Should I block or allow AI crawlers?
It depends on whether you want your content used for training and surfaced in AI answers. Publishers worried about reuse tend to block training crawlers; organisations that want to be found by AI search tend to allow them. Silence is a choice too, and it is the most common one.
Test your robots.txt
Paste your URL. We read your live robots.txt for AI-crawler rules, plus llms.txt and schema.org.
Scan your site