Anthropic · Training crawlers
anthropic-ai
Anthropic’s legacy training token. Rankealo still tracks it because older robots.txt files and crawlers still emit anthropic-ai.
- Operator
- Anthropic
- Traffic type
- Training crawlers
- Verification
- User-agent + IP range
- robots.txt
- Honors robots.txt
Usually means: Bots that collect public pages for future models. Blocking them opts you out of training. It does not, by itself, remove you from live AI search citations.
User-agent
robots.txt and most WAF rules match the token, not the full Mozilla string. Operators often wrap anthropic-ai in extra product or version text.
anthropic-ai
What anthropic-ai does
anthropic-ai is the older Anthropic training identifier. ClaudeBot is the current training crawler; Claude-SearchBot is the citation path.
How to get discovered
For Claude citations, allow Claude-SearchBot and Claude-User. Include anthropic-ai in a training Disallow group so legacy hits follow the same policy as ClaudeBot.
Why anthropic-ai might skip you
You only Disallowed ClaudeBot and forgot anthropic-ai — or the reverse. Keep both tokens in the same training group.
robots.txt rule
This snippet opts out of training-style collection. Keep the search and user-fetch tokens from the same operator allowed if you still want citations.
# Optional training opt-out — does not remove you from AI search by itself
User-agent: anthropic-ai
Disallow: /Allowlist so anthropic-ai can reach you
A user-agent is spoofable. Anthropic publishes current CIDR ranges as JSON — fetch that file for WAF/CDN allowlists instead of copying ranges from a blog. Ranges rotate; a screenshot of ten prefixes will go stale. If Cloudflare “Block AI bots” is on, this is the list that gets the real crawler through.
https://claude.com/crawling/bots.json ↗
Sources
JavaScript: reads raw HTML only.
Other Anthropic bots
Training, search, and user-fetch tokens from the same operator are not interchangeable. Allow the discovery path even when you opt out of training.
anthropic-ai FAQ
What is anthropic-ai?
anthropic-ai is the older Anthropic training identifier. ClaudeBot is the current training crawler; Claude-SearchBot is the citation path.
Should I allow anthropic-ai in robots.txt?
Only if you want to opt out of training-style collection. Blocking anthropic-ai does not, by itself, remove you from live AI search citations — those use the search and user-fetch tokens from Anthropic.
How do I know a request is really anthropic-ai?
A user-agent is a claim anyone can send. Match the anthropic-ai token, then check the source IP against Anthropic’s published ranges. Use those ranges as a WAF allowlist so a “Block AI bots” rule does not 403 the real crawler.
Letting anthropic-ai in is the start. Getting cited is the job.
Rankealo checks whether AI crawlers can reach you, then publishes pages built to be retrieved and quoted in ChatGPT, Claude, Perplexity, and Gemini.
