OpenAI · Search indexes
OAI-SearchBot
OpenAI’s search-index crawler. This is the bot that makes a site eligible to appear in ChatGPT search results and citations.
- Operator
- OpenAI
- Traffic type
- Search indexes
- Verification
- User-agent + IP range
- robots.txt
- Honors robots.txt
Usually means: Crawlers that build the indexes ChatGPT search, Claude search, Perplexity, Google, and Bing draw from. Blocking them is how brands disappear from AI answers without noticing.
User-agent
robots.txt and most WAF rules match the token, not the full Mozilla string. Operators often wrap OAI-SearchBot in extra product or version text.
OAI-SearchBot
Example full string: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot
What OAI-SearchBot does
OAI-SearchBot discovers and refreshes pages so ChatGPT search can surface them. It is not GPTBot (training) and not ChatGPT-User (live browse).
How to get discovered
Allow OAI-SearchBot, add a Sitemap line, serve content in HTML, and publish pages worth citing. Then verify with Rankealo’s crawler checker — a robots Allow is useless if Cloudflare still 403s the UA.
Why OAI-SearchBot might skip you
Disallow: / on OAI-SearchBot, a “Block AI bots” CDN toggle, JS-only apps, or sitemaps that omit the URLs you actually want cited.
robots.txt rule
Allow this token on public pages you want retrieved or cited. A CDN “Block AI bots” toggle can still 403 it after robots.txt says Allow.
# Keep OAI-SearchBot eligible to fetch this site
User-agent: OAI-SearchBot
Allow: /Allowlist so OAI-SearchBot can reach you
A user-agent is spoofable. OpenAI publishes current CIDR ranges as JSON — fetch that file for WAF/CDN allowlists instead of copying ranges from a blog. Ranges rotate; a screenshot of ten prefixes will go stale. If Cloudflare “Block AI bots” is on, this is the list that gets the real crawler through.
https://openai.com/searchbot.json ↗
Sources
JavaScript: reads raw HTML only.
Other OpenAI bots
Training, search, and user-fetch tokens from the same operator are not interchangeable. Allow the discovery path even when you opt out of training.
OAI-SearchBot FAQ
What is OAI-SearchBot?
OAI-SearchBot discovers and refreshes pages so ChatGPT search can surface them. It is not GPTBot (training) and not ChatGPT-User (live browse).
Should I allow OAI-SearchBot in robots.txt?
Yes if you want OpenAI to be able to fetch and cite this site. Blocking OAI-SearchBot is how pages stay invisible to that product even when Googlebot still crawls them.
How do I know a request is really OAI-SearchBot?
A user-agent is a claim anyone can send. Match the OAI-SearchBot token, then check the source IP against OpenAI’s published ranges. Use those ranges as a WAF allowlist so a “Block AI bots” rule does not 403 the real crawler.
Letting OAI-SearchBot in is the start. Getting cited is the job.
Rankealo checks whether AI crawlers can reach you, then publishes pages built to be retrieved and quoted in ChatGPT, Claude, Perplexity, and Gemini.
