OpenAI · Training crawlers
GPTBot
OpenAI’s training crawler. Blocking GPTBot opts you out of future model training. It does not remove you from ChatGPT search if OAI-SearchBot stays allowed.
- Operator
- OpenAI
- Traffic type
- Training crawlers
- Verification
- User-agent + IP range
- robots.txt
- Honors robots.txt
Usually means: Bots that collect public pages for future models. Blocking them opts you out of training. It does not, by itself, remove you from live AI search citations.
User-agent
robots.txt and most WAF rules match the token, not the full Mozilla string. Operators often wrap GPTBot in extra product or version text.
GPTBot
Example full string: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
What GPTBot does
GPTBot collects public pages that may improve future OpenAI models. It is not the citation path. Confusing GPTBot with OAI-SearchBot is the most expensive robots.txt mistake in GEO.
How to get discovered
You do not get “discovered” in ChatGPT search via GPTBot. Allow OAI-SearchBot and ChatGPT-User for citations; treat GPTBot as a separate training choice.
Why GPTBot might skip you
An explicit GPTBot Disallow (intentional), or a CDN AI-bot block that also hits search bots you meant to keep. Check the UA-differential, not just robots.txt.
robots.txt rule
This snippet opts out of training-style collection. Keep the search and user-fetch tokens from the same operator allowed if you still want citations.
# Optional training opt-out — does not remove you from AI search by itself
User-agent: GPTBot
Disallow: /Allowlist so GPTBot can reach you
A user-agent is spoofable. OpenAI publishes current CIDR ranges as JSON — fetch that file for WAF/CDN allowlists instead of copying ranges from a blog. Ranges rotate; a screenshot of ten prefixes will go stale. If Cloudflare “Block AI bots” is on, this is the list that gets the real crawler through.
https://openai.com/gptbot.json ↗
Sources
JavaScript: reads raw HTML only.
Other OpenAI bots
Training, search, and user-fetch tokens from the same operator are not interchangeable. Allow the discovery path even when you opt out of training.
GPTBot FAQ
What is GPTBot?
GPTBot collects public pages that may improve future OpenAI models. It is not the citation path. Confusing GPTBot with OAI-SearchBot is the most expensive robots.txt mistake in GEO.
Should I allow GPTBot in robots.txt?
Only if you want to opt out of training-style collection. Blocking GPTBot does not, by itself, remove you from live AI search citations — those use the search and user-fetch tokens from OpenAI.
How do I know a request is really GPTBot?
A user-agent is a claim anyone can send. Match the GPTBot token, then check the source IP against OpenAI’s published ranges. Use those ranges as a WAF allowlist so a “Block AI bots” rule does not 403 the real crawler.
Letting GPTBot in is the start. Getting cited is the job.
Rankealo checks whether AI crawlers can reach you, then publishes pages built to be retrieved and quoted in ChatGPT, Claude, Perplexity, and Gemini.
