Which Claude user-agents matter?

Anthropic documents ClaudeBot and Claude-Web in robots.txt. We check both against your homepage path.

How do I block ClaudeBot only?

Select Block only ClaudeBot in the AI Crawler Robots.txt Generator and publish the file at your domain root.

Free robots.txt checker

Anthropic uses ClaudeBot and Claude-Web user-agents in robots.txt. Paste your URL to verify whether Claude can access your public pages.

Website URL to analyze

Recommended actions

Prioritised fixes — highest discoverability impact first

Need to fix ClaudeBot rules? Generate an updated robots.txt in seconds.