Anthropic · AI answers
Claude-User
Claude’s on-demand fetcher when a person needs a live page for an answer. Blocking it hides you from in-product browsing and citations.
- Operator
- Anthropic
- Traffic type
- AI answers
- Verification
- User-agent + IP range
- robots.txt
- Honors robots.txt
Usually means: Live fetches when someone asks an assistant a question. If these bots cannot reach the page, you cannot be the citation in that answer.
User-agent
robots.txt and most WAF rules match the token, not the full Mozilla string. Operators often wrap Claude-User in extra product or version text.
Claude-User
What Claude-User does
Claude-User retrieves a URL because a Claude conversation needs it now — to summarize, quote, or check a source. It is separate from ClaudeBot (training) and Claude-SearchBot (index).
How to get discovered
Allow Claude-User, keep public docs and product pages in server-rendered HTML, and put a self-contained answer near the top. Claude quotes passages it can extract without running your frontend.
Why Claude-User might skip you
robots.txt Disallow, an edge bot fight, or HTML that is empty until React hydrates. If the same URL works in a browser and 403s as Claude-User, the block is at the CDN, not in content quality.
robots.txt rule
Allow this token on public pages you want retrieved or cited. A CDN “Block AI bots” toggle can still 403 it after robots.txt says Allow.
# Keep Claude-User eligible to fetch this site
User-agent: Claude-User
Allow: /Allowlist so Claude-User can reach you
A user-agent is spoofable. Anthropic publishes current CIDR ranges as JSON — fetch that file for WAF/CDN allowlists instead of copying ranges from a blog. Ranges rotate; a screenshot of ten prefixes will go stale. If Cloudflare “Block AI bots” is on, this is the list that gets the real crawler through.
https://claude.com/crawling/bots.json ↗
Sources
JavaScript: reads raw HTML only.
Other Anthropic bots
Training, search, and user-fetch tokens from the same operator are not interchangeable. Allow the discovery path even when you opt out of training.
Claude-User FAQ
What is Claude-User?
Claude-User retrieves a URL because a Claude conversation needs it now — to summarize, quote, or check a source. It is separate from ClaudeBot (training) and Claude-SearchBot (index).
Should I allow Claude-User in robots.txt?
Yes if you want Anthropic to be able to fetch and cite this site. Blocking Claude-User is how pages stay invisible to that product even when Googlebot still crawls them.
How do I know a request is really Claude-User?
A user-agent is a claim anyone can send. Match the Claude-User token, then check the source IP against Anthropic’s published ranges. Use those ranges as a WAF allowlist so a “Block AI bots” rule does not 403 the real crawler.
Letting Claude-User in is the start. Getting cited is the job.
Rankealo checks whether AI crawlers can reach you, then publishes pages built to be retrieved and quoted in ChatGPT, Claude, Perplexity, and Gemini.
