AI Crawler Checker — Can ChatGPT Crawl Your Site?
An AI crawler checker tests whether ChatGPT, Claude, Perplexity, and Google can fetch your pages. We parse robots.txt for OAI-SearchBot, GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, Googlebot, and Bingbot, then check llms.txt and sitemap.xml. It does not scrape live AI answers.
Website URL to analyze
Ready to publish
Generated file
Recommended actions
Prioritised fixes — highest discoverability impact first
An AI crawler checker tests whether ChatGPT, Claude, Perplexity, and Google can fetch your pages. We parse robots.txt for OAI-SearchBot, GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, Googlebot, and Bingbot, then check llms.txt and sitemap.xml. It does not scrape live AI answers.
See which search and AI crawlers your robots.txt allows, plus llms.txt and sitemap signals that affect discovery in Google, ChatGPT, Claude, and Perplexity.
In your free report
- robots.txt rules for Googlebot, Bingbot, GPTBot, OAI-SearchBot, ClaudeBot, and PerplexityBot
- Homepage HTTP status and whether the path is allowed for each user-agent
- llms.txt presence and basic structure at your domain root
- XML sitemap discovery via robots.txt and common locations
- Meta robots, canonical, and indexability signals on scanned pages
Recommendations from official docs
Control AI bots independently
OpenAI documents GPTBot (training) and OAI-SearchBot (ChatGPT search) as separate robots.txt tokens. You can allow search indexing while opting out of training data collection.
Allow search bots on public pages
Google and Bing need crawl access to index your site. Block only paths that should never appear in search or AI answers (admin, cart, staging, private APIs).
llms.txt is not a training opt-out
A curated llms.txt at your root helps LLMs understand your site faster than parsing your full sitemap. Keep it under ~2,000 words with 10–30 high-value links. To opt out of training, Disallow GPTBot (and similar training tokens) in robots.txt.
Verify WAF and CDN bot rules
robots.txt is advisory. Cloudflare Bot Fight Mode or aggressive WAF rules can block crawlers even when robots.txt allows them — whitelist official crawler IP ranges.
Step-by-step action plan
- 1 Fetch your live robots.txt and list each User-agent block separately for AI and search bots.
- 2 Allow OAI-SearchBot and Googlebot on public marketing and product pages you want discovered.
- 3 Add Sitemap: https://yourdomain.com/sitemap.xml and ensure the sitemap returns HTTP 200.
- 4 Publish llms.txt with one H1, a blockquote summary, and grouped absolute URLs.
- 5 Re-run this checker after changes — OpenAI notes robots.txt updates can take ~24 hours to apply.
Learn more from the source
Learn how to improve these signals
Need help fixing AI SEO issues?
Optimize your website for AI SEO, ChatGPT Search, Google AI Overviews, GPTBot, llms.txt, robots.txt, schema, and crawlability.
Questions about this tool
Is this a free SEO crawler?
Yes. Paste a public URL and we crawl robots.txt, AI bot access, llms.txt, and sitemap signals. No account is required.
Which AI crawlers are tested?
OAI-SearchBot, GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, and major search bots like Googlebot and Bingbot.
How is bot access determined?
We parse your robots.txt and test whether your homepage path is allowed for each user-agent using standard matching rules.
Can I block some AI bots but allow others?
Yes. Use separate User-agent blocks. The usual 2026 split is: allow OAI-SearchBot and ChatGPT-User for ChatGPT search, and disallow GPTBot if you want to opt out of training.
Which AI crawlers should I allow in robots.txt?
Allow search crawlers (Googlebot, OAI-SearchBot, ChatGPT-User, Claude-Web, PerplexityBot) if you want to be cited. Block training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot) if you do not want content used to train models. Generate that file in the AI crawlers robots.txt generator.