Skip to main content
Free AI crawler robots.txt generator

AI Crawlers robots.txt Generator

Write robots.txt for AI crawlers in seconds. The recommended policy blocks training bots (GPTBot, ClaudeBot, Google-Extended, CCBot) and allows search bots (OAI-SearchBot, ChatGPT-User, Claude-Web, PerplexityBot, Googlebot).

Generate robots.txt rules for AI crawlers

Disallow GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended, and Amazonbot. Allow OAI-SearchBot, ChatGPT-User, Claude-Web, PerplexityBot, and Googlebot.

Policy options

Free AI crawler robots.txt generator

The recommended default is block training, allow AI search. Other options block GPTBot only, block all AI crawlers, or allow everything.

Block training, allow AI search (recommended)

Disallow GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended, and Amazonbot. Allow OAI-SearchBot, ChatGPT-User, Claude-Web, PerplexityBot, and Googlebot.

Allow all AI crawlers

Googlebot, Bingbot, and major AI bots can fetch public pages.

Block all AI crawlers

Disallow GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, and more.

Block only GPTBot

Keep other AI crawlers allowed; block OpenAI GPTBot only.

Block only ClaudeBot

Block ClaudeBot and Claude-Web; allow other bots.

Block only PerplexityBot

Block Perplexity's crawler only.

Block Google-Extended

Opt out of Google's generative-AI training crawler token.

Custom rules

Download a commented starter file and edit per-bot rules manually.

User-agents

Training vs search crawlers

Allow search bots if you want to be cited. Block training bots if you do not want content used to train models. They are separate robots.txt tokens.

Crawler Vendor Purpose Typical policy
GPTBot OpenAI Foundation-model training Training — usually block
OAI-SearchBot OpenAI ChatGPT search indexing Search — allow to be cited
ChatGPT-User OpenAI User-triggered fetches Search — allow live answers
ClaudeBot Anthropic Claude training crawler Training — usually block
Claude-Web Anthropic Claude user/web fetches Search — allow citations
PerplexityBot Perplexity Perplexity AI answers Search — allow citations
Google-Extended Google Generative AI training opt-out token Training — block independently of Googlebot
CCBot Common Crawl Open web corpus Training — usually block
Applebot-Extended Apple Apple AI training Training — usually block
Amazonbot Amazon Amazon AI indexing Training — usually block
Copy-paste examples

Example robots.txt rules

Block training, allow AI search

User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-Web
Allow: /
User-agent: PerplexityBot
Allow: /

User-agent: *
Allow: /

Block GPTBot only

User-agent: GPTBot
Disallow: /

User-agent: *
Allow: /

Block ClaudeBot

User-agent: ClaudeBot
Disallow: /
User-agent: Claude-Web
Disallow: /

User-agent: *
Allow: /

Block PerplexityBot

User-agent: PerplexityBot
Disallow: /

User-agent: *
Allow: /

Block multiple AI crawlers

User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: PerplexityBot
Disallow: /

User-agent: *
Allow: /
Related tools & guides

Check and learn more

GPTBot robots.txt checker → ClaudeBot robots.txt checker → PerplexityBot robots.txt checker → Robots.txt for AI crawlers → Block AI crawlers from your website →

Also try the AI Crawler Checker for a full crawl readiness report.

AI crawler robots.txt FAQ

Which AI crawlers should I allow in robots.txt?

For most public sites in 2026: allow Googlebot and AI search crawlers (OAI-SearchBot, ChatGPT-User, Claude-Web, PerplexityBot) so you can be cited. Block training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot) if you do not want content used for model training. llms.txt is not a training opt-out.

What is an AI crawler?

An AI crawler is an automated bot that fetches public web pages for AI search, answers, or model training. Examples include GPTBot, ClaudeBot, and PerplexityBot.

Should I block AI crawlers?

Block training crawlers if you want to limit model training use of your content. Allow search crawlers like OAI-SearchBot if you want ChatGPT or AI search visibility.

How do I block GPTBot in robots.txt?

Add a User-agent: GPTBot block with Disallow: / before your wildcard Allow rules. Use this generator and select Block only GPTBot.

How do I allow AI crawlers?

Use User-agent: * with Allow: / and avoid Disallow rules that block AI user-agents. Select Allow all AI crawlers in this tool to generate a permissive starter file.

What is Google-Extended?

Google-Extended is a robots.txt token to manage whether Google may use your content for generative AI training surfaces. It is separate from Googlebot search crawling.

Can robots.txt stop AI training?

robots.txt can block well-behaved crawlers that respect it, but it is not DRM. It is a public crawl policy — not authentication or legal protection.

Sponsored

Hostinger promo

Working coupon code for cheap web hosting.

All promo codes & coupons →

Sponsored links — we may earn a commission at no extra cost to you.