Skip to main content
Free AI crawler robots.txt generator

AI Crawler Robots.txt Rules Generator

Control which AI crawlers can access your website. Generate robots.txt rules for ChatGPT, Claude, Perplexity, Google AI, Common Crawl, and other AI search bots in seconds.

Generate robots.txt rules for AI crawlers

Googlebot, Bingbot, and major AI bots can fetch public pages.

Policy options

Free AI crawler robots.txt generator

Pick a policy above — we write the User-agent blocks for GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and more.

Allow all AI crawlers

Googlebot, Bingbot, and major AI bots can fetch public pages.

Block all AI crawlers

Disallow GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, and more.

Block only GPTBot

Keep other AI crawlers allowed; block OpenAI GPTBot only.

Block only ClaudeBot

Block ClaudeBot and Claude-Web; allow other bots.

Block only PerplexityBot

Block Perplexity's crawler only.

Block Google-Extended

Opt out of Google's generative-AI training crawler token.

Custom rules

Download a commented starter file and edit per-bot rules manually.

User-agents

Supported AI crawlers

Crawler Vendor Purpose
GPTBot OpenAI Training & broad crawling
OAI-SearchBot OpenAI ChatGPT search indexing
ChatGPT-User OpenAI User-triggered fetches
ClaudeBot Anthropic Claude crawler
PerplexityBot Perplexity Perplexity AI answers
Google-Extended Google Generative AI training opt-out token
CCBot Common Crawl Open web corpus
Applebot-Extended Apple Apple AI training
Amazonbot Amazon Amazon AI indexing
Copy-paste examples

Example robots.txt rules

Block GPTBot only

User-agent: GPTBot
Disallow: /

User-agent: *
Allow: /

Block ClaudeBot

User-agent: ClaudeBot
Disallow: /
User-agent: Claude-Web
Disallow: /

User-agent: *
Allow: /

Block PerplexityBot

User-agent: PerplexityBot
Disallow: /

User-agent: *
Allow: /

Block multiple AI crawlers

User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: PerplexityBot
Disallow: /

User-agent: *
Allow: /
Related tools & guides

Check and learn more

GPTBot robots.txt checker → ClaudeBot robots.txt checker → PerplexityBot robots.txt checker → Robots.txt for AI crawlers → Block AI crawlers from your website →

Also try the AI Crawler Checker for a full crawl readiness report.

AI crawler robots.txt FAQ

What is an AI crawler?

An AI crawler is an automated bot that fetches public web pages for AI search, answers, or model training. Examples include GPTBot, ClaudeBot, and PerplexityBot.

Should I block AI crawlers?

Block training crawlers if you want to limit model training use of your content. Allow search crawlers like OAI-SearchBot if you want ChatGPT or AI search visibility.

How do I block GPTBot in robots.txt?

Add a User-agent: GPTBot block with Disallow: / before your wildcard Allow rules. Use this generator and select Block only GPTBot.

How do I allow AI crawlers?

Use User-agent: * with Allow: / and avoid Disallow rules that block AI user-agents. Select Allow all AI crawlers in this tool to generate a permissive starter file.

What is Google-Extended?

Google-Extended is a robots.txt token to manage whether Google may use your content for generative AI training surfaces. It is separate from Googlebot search crawling.

Can robots.txt stop AI training?

robots.txt can block well-behaved crawlers that respect it, but it is not DRM. It is a public crawl policy — not authentication or legal protection.

Sponsored

Hostinger promo & Cursor discount

Working coupon codes for cheap web hosting and AI code editor deals.

All promo codes & coupons →

Sponsored links — we may earn a commission at no extra cost to you.