Skip to main content
Free AI crawl tool

Free AI Crawler Checker

Enter your website URL to see which AI crawlers and search bots can access your site. We parse robots.txt and check OAI-SearchBot, GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, Googlebot, Bingbot, llms.txt, and sitemap.xml in one scan.

Recommended OAI-SearchBot: Allow / for public content GPTBot (training): Allow or block based on your policy llms.txt links: 10–30 curated pages

Website URL to analyze

About this tool

Enter your website URL to see which AI crawlers and search bots can access your site. We parse robots.txt and check OAI-SearchBot, GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, Googlebot, Bingbot, llms.txt, and sitemap.xml in one scan.

See which search and AI crawlers your robots.txt allows, plus llms.txt and sitemap signals that affect discovery in Google, ChatGPT, Claude, and Perplexity.

What we check

In your free report

  • robots.txt rules for Googlebot, Bingbot, GPTBot, OAI-SearchBot, ClaudeBot, and PerplexityBot
  • Homepage HTTP status and whether the path is allowed for each user-agent
  • llms.txt presence and basic structure at your domain root
  • XML sitemap discovery via robots.txt and common locations
  • Meta robots, canonical, and indexability signals on scanned pages
Recommended OAI-SearchBot Allow / for public content
GPTBot (training) Allow or block based on your policy
llms.txt links 10–30 curated pages
Best practices

Recommendations from official docs

Control AI bots independently

OpenAI documents GPTBot (training) and OAI-SearchBot (ChatGPT search) as separate robots.txt tokens. You can allow search indexing while opting out of training data collection.

Allow search bots on public pages

Google and Bing need crawl access to index your site. Block only paths that should never appear in search or AI answers (admin, cart, staging, private APIs).

Publish llms.txt for AI discovery

A curated llms.txt at your root helps LLMs understand your site faster than parsing your full sitemap. Keep it under ~2,000 words with 10–30 high-value links.

Verify WAF and CDN bot rules

robots.txt is advisory. Cloudflare Bot Fight Mode or aggressive WAF rules can block crawlers even when robots.txt allows them — whitelist official crawler IP ranges.

How to fix

Step-by-step action plan

  1. 1 Fetch your live robots.txt and list each User-agent block separately for AI and search bots.
  2. 2 Allow OAI-SearchBot and Googlebot on public marketing and product pages you want discovered.
  3. 3 Add Sitemap: https://yourdomain.com/sitemap.xml and ensure the sitemap returns HTTP 200.
  4. 4 Publish llms.txt with one H1, a blockquote summary, and grouped absolute URLs.
  5. 5 Re-run this checker after changes — OpenAI notes robots.txt updates can take ~24 hours to apply.
Official references

Learn more from the source

Related tools

Fix what this scan finds

AI Crawler Comparison → LLMs.txt Checker → GPTBot Robots.txt Checker → ChatGPT SEO Checker →
Run full-site audit (up to 100 pages)

Need help fixing AI SEO issues?

Optimize your website for AI SEO, ChatGPT Search, Google AI Overviews, GPTBot, llms.txt, robots.txt, schema, and crawlability.

Hire Mubashir to fix crawl issues

Questions about this tool

Which AI crawlers are tested?

OAI-SearchBot, GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, and major search bots like Googlebot and Bingbot.

How is bot access determined?

We parse your robots.txt and test whether your homepage path is allowed for each user-agent using standard matching rules.

Can I block some AI bots but allow others?

Yes. Use separate User-agent blocks in robots.txt for each bot you want to allow or disallow.