Which AI Crawlers Should I Allow in robots.txt?
Allow AI search crawlers (OAI-SearchBot, ChatGPT-User, PerplexityBot) and block training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot) — copy-paste robots.txt.
By Mubashir Hussain · CrawlReady AI · Human-reviewed technical guide
Some guides may be AI-assisted and are always human-reviewed for accuracy before publish. See our Google generative AI search guide and Google's AI content guidance.
Which AI crawlers should you allow? For most public marketing sites in 2026: allow search crawlers so you can be cited, and block training crawlers if you do not want your pages used to train models. robots.txt is a public file — not authentication. Well-behaved bots honor it; scrapers may not.
Training vs search — they are different tokens
- Googlebot — Google Search. Allow.
- OAI-SearchBot / ChatGPT-User — ChatGPT search. Allow if you want citations.
- Claude-Web / PerplexityBot — Claude and Perplexity answers. Allow if you want citations.
- GPTBot / ClaudeBot — training crawlers. Block to opt out of training.
- Google-Extended / CCBot — generative-AI training and Common Crawl. Block independently of Googlebot.
llms.txt is not a training opt-out. Use robots.txt User-agent blocks for that.
Recommended robots.txt for public sites
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-Web
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /dashboard/
Sitemap: https://example.com/sitemap.xmlGenerate from your URL
Enter your site in the AI Crawler Robots.txt Generator and choose Block training, allow AI search — we detect your sitemap and produce this starter file.
Supporting tools: GPTBot checker, OAI-SearchBot checker, ClaudeBot checker, and PerplexityBot checker.
Frequently Asked Questions
Which AI crawlers should I allow in robots.txt?
For most public sites: allow Googlebot and AI search crawlers (OAI-SearchBot, ChatGPT-User, Claude-Web, PerplexityBot). Block training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot) if you do not want content used to train models.
Can robots.txt block AI training crawlers only?
Yes. Use separate User-agent blocks for GPTBot, ClaudeBot, Google-Extended, and CCBot while keeping Googlebot and OAI-SearchBot allowed for search.
Does Allow: / override Disallow rules?
Under Google's robots.txt interpretation, the most specific matching rule wins. Put Allow: / first, then Disallow for private paths like /admin/.
Important disclaimer
This guide is for educational purposes only. No tool or technique guarantees search rankings, AI inclusion, or specific traffic results. Refer to official documentation from search engines and AI providers for current policies.
