Skip to main content
Free AI crawl tool

ClaudeBot Access Checker

Anthropic's ClaudeBot and Claude-Web crawlers read robots.txt before fetching pages. Paste your URL to see if Claude can access your public content.

Recommended OAI-SearchBot: Allow / for public content GPTBot (training): Allow or block based on your policy llms.txt links: 10–30 curated pages

Website URL to analyze

About this tool

Anthropic's ClaudeBot and Claude-Web crawlers read robots.txt before fetching pages. Paste your URL to see if Claude can access your public content.

Verify ClaudeBot and Claude-Web access via robots.txt so Anthropic crawlers can discover your public content.

What we check

In your free report

  • robots.txt rules for Googlebot, Bingbot, GPTBot, OAI-SearchBot, ClaudeBot, and PerplexityBot
  • Homepage HTTP status and whether the path is allowed for each user-agent
  • llms.txt presence and basic structure at your domain root
  • XML sitemap discovery via robots.txt and common locations
  • Meta robots, canonical, and indexability signals on scanned pages
Recommended OAI-SearchBot Allow / for public content
GPTBot (training) Allow or block based on your policy
llms.txt links 10–30 curated pages
Best practices

Recommendations from official docs

Control AI bots independently

OpenAI documents GPTBot (training) and OAI-SearchBot (ChatGPT search) as separate robots.txt tokens. You can allow search indexing while opting out of training data collection.

Allow search bots on public pages

Google and Bing need crawl access to index your site. Block only paths that should never appear in search or AI answers (admin, cart, staging, private APIs).

Publish llms.txt for AI discovery

A curated llms.txt at your root helps LLMs understand your site faster than parsing your full sitemap. Keep it under ~2,000 words with 10–30 high-value links.

Verify WAF and CDN bot rules

robots.txt is advisory. Cloudflare Bot Fight Mode or aggressive WAF rules can block crawlers even when robots.txt allows them — whitelist official crawler IP ranges.

How to fix

Step-by-step action plan

  1. 1 Fetch your live robots.txt and list each User-agent block separately for AI and search bots.
  2. 2 Allow OAI-SearchBot and Googlebot on public marketing and product pages you want discovered.
  3. 3 Add Sitemap: https://yourdomain.com/sitemap.xml and ensure the sitemap returns HTTP 200.
  4. 4 Publish llms.txt with one H1, a blockquote summary, and grouped absolute URLs.
  5. 5 Re-run this checker after changes — OpenAI notes robots.txt updates can take ~24 hours to apply.
Official references

Learn more from the source

Related tools

Fix what this scan finds

AI Crawler Checker → AI Crawler Comparison → LLMs.txt Checker → GPTBot Robots.txt Checker →
Run full-site audit (up to 100 pages)

Need help fixing AI SEO issues?

Optimize your website for AI SEO, ChatGPT Search, Google AI Overviews, GPTBot, llms.txt, robots.txt, schema, and crawlability.

Hire Mubashir to fix crawl issues

Questions about this tool

What crawlers does Anthropic use?

ClaudeBot and Claude-Web are common Anthropic user-agents declared in robots.txt.

Should I block ClaudeBot?

Allow it on public marketing and docs pages if you want Claude to discover and reference your content.