What Is OAI-SearchBot? How to Whitelist It in robots.txt
OAI-SearchBot is ChatGPT's search crawler. Copy the robots.txt to allow it, whitelist OpenAI IPs on Cloudflare, then test if you're blocked — free.
By Mubashir Hussain · CrawlReady AI · Human-reviewed technical guide
Some guides may be AI-assisted and are always human-reviewed for accuracy before publish. See our Google generative AI search guide and Google's AI content guidance.
OAI-SearchBot is OpenAI's crawler for ChatGPT search. If you want pages cited in ChatGPT answers, whitelist it in robots.txt — and on your CDN if Bot Fight Mode or a WAF is blocking the user-agent. Blocking it (or forgetting a rule while User-agent: * disallows everything) keeps ChatGPT search off your site even when Google still indexes you.
OAI-SearchBot vs GPTBot vs ChatGPT-User
| User-agent | What it does | Whitelist if you want |
|---|---|---|
| OAI-SearchBot | ChatGPT search indexing and live retrieval | Citations in ChatGPT search |
| GPTBot | Broader crawling, including training-related fetches | Content in OpenAI's general knowledge pipeline |
| ChatGPT-User | A person asked ChatGPT to open a page | User-triggered fetches of your URLs |
Whitelisting OAI-SearchBot does not automatically allow GPTBot. Add each user-agent you actually want.
How to whitelist OAI-SearchBot in robots.txt
Put this at https://yourdomain.com/robots.txt. The dedicated group is what OpenAI reads for that bot — it overrides a blanket User-agent: * block.
User-agent: OAI-SearchBot
Allow: /
Disallow: /admin/
Disallow: /dashboard/
Sitemap: https://yourdomain.com/sitemap.xmlNeed the shortest allow-everything rule?
User-agent: OAI-SearchBot
Allow: /If your file already has User-agent: * with Disallow: /, keep that for other bots if you want — just add the OAI-SearchBot group above (or below) it. Do not put Disallow: / under User-agent: OAI-SearchBot unless you intend to hide the whole site from ChatGPT search.
Generate a full file with the AI Crawler Robots.txt Generator if you also need GPTBot, ClaudeBot, and PerplexityBot rules.
Whitelist OAI-SearchBot on Cloudflare and other firewalls
robots.txt is only a request. Cloudflare Bot Fight Mode, a WAF challenge, or a host security list can still return 403 or a JS challenge. OpenAI then never sees your HTML.
- Download the current CIDRs from openai.com/searchbot.json.
- Allow those IPs in Cloudflare (IP access rules or a WAF skip for that list). Skip Bot Fight Mode for them if it is enabled.
- If you use AWS, Oracle, or another cloud firewall, allow HTTPS (443) from those ranges — or from the internet if you already do for Googlebot.
- Confirm the homepage returns HTTP 200 for a normal GET, not a challenge page.
OpenAI documents crawler verification and user-agents in its bots documentation. Re-check the JSON file when you rotate WAF rules; IP ranges change.
How to check if OAI-SearchBot is allowed
Paste your URL in the OAI-SearchBot Checker. It parses live robots.txt and tells you whether ChatGPT's search crawler is allowed, blocked, or missing a specific rule (so it falls back to *).
Also grep access logs after you whitelist:
grep -i 'OAI-SearchBot' /var/log/nginx/access.log | tailNo hits yet is normal on a small site. A 403 with that user-agent means the firewall, not robots.txt, is still blocking.
Common whitelist mistakes
- Allowing
GPTBotbut notOAI-SearchBot— ChatGPT search still cannot index you - A plugin that writes
Disallow: /for*on staging, then shipping that file to production - Whitelisting in robots.txt while Cloudflare “I'm Under Attack” or Bot Fight Mode challenges unknown bots
- Blocking
/llms.txtor the XML sitemap for that user-agent
Official references
Policies change. Confirm current tokens and IP lists in OpenAI's bots documentation and the OpenAI crawler guide before you publish production rules.
Frequently Asked Questions
How do I whitelist OAI-SearchBot?
Add a User-agent: OAI-SearchBot group in robots.txt with Allow: / for public pages. If Cloudflare or a WAF still blocks the bot, allow the IP ranges OpenAI publishes at openai.com/searchbot.json.
What is OAI-SearchBot used for?
OAI-SearchBot is OpenAI's crawler for ChatGPT search. It fetches public pages so ChatGPT can cite them in live answers. It is not the same as GPTBot, which is used for training and broader crawling.
Is OAI-SearchBot the same as GPTBot?
No. They are separate robots.txt user-agents. You can allow OAI-SearchBot for ChatGPT search citations and still block GPTBot if you do not want training crawls.
Does User-agent: * Disallow: / block OAI-SearchBot?
Yes, unless you add a more specific User-agent: OAI-SearchBot group. OpenAI follows that dedicated group instead of the * rules.
Important disclaimer
This guide is for educational purposes only. No tool or technique guarantees search rankings, AI inclusion, or specific traffic results. Refer to official documentation from search engines and AI providers for current policies.
