Anthropic · Training crawlers
ClaudeBot
Anthropic’s training crawler for Claude. Blocking ClaudeBot is a training opt-out. Keep Claude-SearchBot allowed if you still want to be cited.
- Operator
- Anthropic
- Traffic type
- Training crawlers
- Verification
- User-agent + IP range
- robots.txt
- Honors robots.txt
Usually means: Bots that collect public pages for future models. Blocking them opts you out of training. It does not, by itself, remove you from live AI search citations.
User-agent
robots.txt and most WAF rules match the token, not the full Mozilla string. Operators often wrap ClaudeBot in extra product or version text.
ClaudeBot
Example full string: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)
What ClaudeBot does
ClaudeBot collects public content that may be used to improve Claude. Claude-SearchBot and Claude-User are the retrieval tokens.
How to get discovered
Do not use ClaudeBot as your Claude visibility plan. Allow Claude-SearchBot and Claude-User, and write pages Claude can quote. Optionally Disallow ClaudeBot if you do not want training use.
Why ClaudeBot might skip you
Intentional Disallow, or a Claude* wildcard that also killed Claude-SearchBot. Prefix mistakes in robots.txt are common here.
robots.txt rule
This snippet opts out of training-style collection. Keep the search and user-fetch tokens from the same operator allowed if you still want citations.
# Optional training opt-out — does not remove you from AI search by itself
User-agent: ClaudeBot
Disallow: /Allowlist so ClaudeBot can reach you
A user-agent is spoofable. Anthropic publishes current CIDR ranges as JSON — fetch that file for WAF/CDN allowlists instead of copying ranges from a blog. Ranges rotate; a screenshot of ten prefixes will go stale. If Cloudflare “Block AI bots” is on, this is the list that gets the real crawler through.
https://claude.com/crawling/bots.json ↗
Sources
JavaScript: reads raw HTML only.
Other Anthropic bots
Training, search, and user-fetch tokens from the same operator are not interchangeable. Allow the discovery path even when you opt out of training.
ClaudeBot FAQ
What is ClaudeBot?
ClaudeBot collects public content that may be used to improve Claude. Claude-SearchBot and Claude-User are the retrieval tokens.
Should I allow ClaudeBot in robots.txt?
Only if you want to opt out of training-style collection. Blocking ClaudeBot does not, by itself, remove you from live AI search citations — those use the search and user-fetch tokens from Anthropic.
How do I know a request is really ClaudeBot?
A user-agent is a claim anyone can send. Match the ClaudeBot token, then check the source IP against Anthropic’s published ranges. Use those ranges as a WAF allowlist so a “Block AI bots” rule does not 403 the real crawler.
Letting ClaudeBot in is the start. Getting cited is the job.
Rankealo checks whether AI crawlers can reach you, then publishes pages built to be retrieved and quoted in ChatGPT, Claude, Perplexity, and Gemini.
