Anthropic · Search indexes
Claude-SearchBot
Anthropic’s crawler for Claude search discovery. If Claude-SearchBot cannot index you, Claude search has no page to retrieve.
- Operator
- Anthropic
- Traffic type
- Search indexes
- Verification
- User-agent + IP range
- robots.txt
- Honors robots.txt
Usually means: Crawlers that build the indexes ChatGPT search, Claude search, Perplexity, Google, and Bing draw from. Blocking them is how brands disappear from AI answers without noticing.
User-agent
robots.txt and most WAF rules match the token, not the full Mozilla string. Operators often wrap Claude-SearchBot in extra product or version text.
Claude-SearchBot
Example full string: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-SearchBot/1.0; +claude-searchbot@anthropic.com)
What Claude-SearchBot does
Claude-SearchBot crawls to discover and refresh content for Claude’s in-product search. Blocking it is the fastest way to vanish from Claude citations while still allowing ClaudeBot by accident — or the reverse.
How to get discovered
Allow Claude-SearchBot explicitly. Keep llms.txt and a sitemap current, and put canonical facts in HTML Anthropic does not need to execute JavaScript to see.
Why Claude-SearchBot might skip you
A robots group that Disallows ClaudeBot and accidentally uses a prefix that also matches Claude-SearchBot, or an edge rule that blocks all Claude* UAs.
robots.txt rule
Allow this token on public pages you want retrieved or cited. A CDN “Block AI bots” toggle can still 403 it after robots.txt says Allow.
# Keep Claude-SearchBot eligible to fetch this site
User-agent: Claude-SearchBot
Allow: /Allowlist so Claude-SearchBot can reach you
A user-agent is spoofable. Anthropic publishes current CIDR ranges as JSON — fetch that file for WAF/CDN allowlists instead of copying ranges from a blog. Ranges rotate; a screenshot of ten prefixes will go stale. If Cloudflare “Block AI bots” is on, this is the list that gets the real crawler through.
https://claude.com/crawling/bots.json ↗
Sources
JavaScript: reads raw HTML only.
Other Anthropic bots
Training, search, and user-fetch tokens from the same operator are not interchangeable. Allow the discovery path even when you opt out of training.
Claude-SearchBot FAQ
What is Claude-SearchBot?
Claude-SearchBot crawls to discover and refresh content for Claude’s in-product search. Blocking it is the fastest way to vanish from Claude citations while still allowing ClaudeBot by accident — or the reverse.
Should I allow Claude-SearchBot in robots.txt?
Yes if you want Anthropic to be able to fetch and cite this site. Blocking Claude-SearchBot is how pages stay invisible to that product even when Googlebot still crawls them.
How do I know a request is really Claude-SearchBot?
A user-agent is a claim anyone can send. Match the Claude-SearchBot token, then check the source IP against Anthropic’s published ranges. Use those ranges as a WAF allowlist so a “Block AI bots” rule does not 403 the real crawler.
Letting Claude-SearchBot in is the start. Getting cited is the job.
Rankealo checks whether AI crawlers can reach you, then publishes pages built to be retrieved and quoted in ChatGPT, Claude, Perplexity, and Gemini.
