Skip to main content
Free AI crawl tool

AI SEO Audit for Small Websites

Small sites still need AI-ready crawl access, clear metadata, and a valid sitemap. Scan up to 100 pages free — built for founders and small teams who cannot afford Ahrefs or Semrush yet.

Recommended OAI-SearchBot: Allow / for public content GPTBot (training): Allow or block based on your policy llms.txt links: 10–30 curated pages

Website URL to analyze

About this tool

Small sites still need AI-ready crawl access, clear metadata, and a valid sitemap. Scan up to 100 pages free — built for founders and small teams who cannot afford Ahrefs or Semrush yet.

Free AI SEO audit tailored for small business sites — crawler access, metadata, sitemap, and llms.txt without paid enterprise tools.

What we check

In your free report

  • robots.txt rules for Googlebot, Bingbot, GPTBot, OAI-SearchBot, ClaudeBot, and PerplexityBot
  • Homepage HTTP status and whether the path is allowed for each user-agent
  • llms.txt presence and basic structure at your domain root
  • XML sitemap discovery via robots.txt and common locations
  • Meta robots, canonical, and indexability signals on scanned pages
Recommended OAI-SearchBot Allow / for public content
GPTBot (training) Allow or block based on your policy
llms.txt links 10–30 curated pages
Best practices

Recommendations from official docs

Control AI bots independently

OpenAI documents GPTBot (training) and OAI-SearchBot (ChatGPT search) as separate robots.txt tokens. You can allow search indexing while opting out of training data collection.

Allow search bots on public pages

Google and Bing need crawl access to index your site. Block only paths that should never appear in search or AI answers (admin, cart, staging, private APIs).

Publish llms.txt for AI discovery

A curated llms.txt at your root helps LLMs understand your site faster than parsing your full sitemap. Keep it under ~2,000 words with 10–30 high-value links.

Verify WAF and CDN bot rules

robots.txt is advisory. Cloudflare Bot Fight Mode or aggressive WAF rules can block crawlers even when robots.txt allows them — whitelist official crawler IP ranges.

How to fix

Step-by-step action plan

  1. 1 Fetch your live robots.txt and list each User-agent block separately for AI and search bots.
  2. 2 Allow OAI-SearchBot and Googlebot on public marketing and product pages you want discovered.
  3. 3 Add Sitemap: https://yourdomain.com/sitemap.xml and ensure the sitemap returns HTTP 200.
  4. 4 Publish llms.txt with one H1, a blockquote summary, and grouped absolute URLs.
  5. 5 Re-run this checker after changes — OpenAI notes robots.txt updates can take ~24 hours to apply.
Official references

Learn more from the source

Related tools

Fix what this scan finds

AI Crawler Checker → AI Crawler Comparison → LLMs.txt Checker → GPTBot Robots.txt Checker →
Run full-site audit (up to 100 pages)

Need help fixing AI SEO issues?

Optimize your website for AI SEO, ChatGPT Search, Google AI Overviews, GPTBot, llms.txt, robots.txt, schema, and crawlability.

Hire Mubashir to fix crawl issues

Questions about this tool

Is this built for small websites?

Yes. The free scan covers up to 100 pages — enough for most small business, portfolio, and startup sites.

Do I need Ahrefs or Semrush first?

No. Start here for crawl and AI readiness, then add paid keyword tools when you are ready to scale content.