Skip to main content
Free SEO tool

Free Technical SEO Audit Tool

Run a free multi-page technical SEO audit: HTTPS, redirects, canonical tags, meta titles, schema markup, XML sitemap, robots.txt, and internal link structure — the same engine as our full-site AI visibility audit.

Recommended OAI-SearchBot: Allow / for public content GPTBot (training): Allow or block based on your policy llms.txt links: 10–30 curated pages

Website URL to analyze

About this tool

Run a free multi-page technical SEO audit: HTTPS, redirects, canonical tags, meta titles, schema markup, XML sitemap, robots.txt, and internal link structure — the same engine as our full-site AI visibility audit.

Full technical SEO audit across up to 100 pages — HTTPS, indexability, metadata, schema, sitemap, robots.txt, and internal links.

What we check

In your free report

  • robots.txt rules for Googlebot, Bingbot, GPTBot, OAI-SearchBot, ClaudeBot, and PerplexityBot
  • Homepage HTTP status and whether the path is allowed for each user-agent
  • llms.txt presence and basic structure at your domain root
  • XML sitemap discovery via robots.txt and common locations
  • Meta robots, canonical, and indexability signals on scanned pages
Recommended OAI-SearchBot Allow / for public content
GPTBot (training) Allow or block based on your policy
llms.txt links 10–30 curated pages
Best practices

Recommendations from official docs

Control AI bots independently

OpenAI documents GPTBot (training) and OAI-SearchBot (ChatGPT search) as separate robots.txt tokens. You can allow search indexing while opting out of training data collection.

Allow search bots on public pages

Google and Bing need crawl access to index your site. Block only paths that should never appear in search or AI answers (admin, cart, staging, private APIs).

Publish llms.txt for AI discovery

A curated llms.txt at your root helps LLMs understand your site faster than parsing your full sitemap. Keep it under ~2,000 words with 10–30 high-value links.

Verify WAF and CDN bot rules

robots.txt is advisory. Cloudflare Bot Fight Mode or aggressive WAF rules can block crawlers even when robots.txt allows them — whitelist official crawler IP ranges.

How to fix

Step-by-step action plan

  1. 1 Fetch your live robots.txt and list each User-agent block separately for AI and search bots.
  2. 2 Allow OAI-SearchBot and Googlebot on public marketing and product pages you want discovered.
  3. 3 Add Sitemap: https://yourdomain.com/sitemap.xml and ensure the sitemap returns HTTP 200.
  4. 4 Publish llms.txt with one H1, a blockquote summary, and grouped absolute URLs.
  5. 5 Re-run this checker after changes — OpenAI notes robots.txt updates can take ~24 hours to apply.
Official references

Learn more from the source

Related tools

Fix what this scan finds

AI Crawler Checker → Sitemap Checker → Indexability Checker → Meta Title Checker →
Run full-site audit (up to 100 pages)

Need help fixing AI SEO issues?

Optimize your website for AI SEO, ChatGPT Search, Google AI Overviews, GPTBot, llms.txt, robots.txt, schema, and crawlability.

Hire Mubashir to fix crawl issues

Questions about this tool

How many pages does the technical audit scan?

Up to 100 public pages discovered from your sitemap and navigation, plus a scored homepage report.

Is this the same as the /audit page?

Yes — same scan engine. This landing page targets the technical SEO audit keyword.