xAI · AI answers
xAI-SearchBot
xAI’s user-triggered search fetcher for Grok answers. Allow it if you want Grok to be able to retrieve and cite the page.
- Operator
- xAI
- Traffic type
- AI answers
- Verification
- Not independently verifiable
- robots.txt
- Usually honors robots.txt
Usually means: Live fetches when someone asks an assistant a question. If these bots cannot reach the page, you cannot be the citation in that answer.
User-agent
robots.txt and most WAF rules match the token, not the full Mozilla string. Operators often wrap xAI-SearchBot in extra product or version text.
xAI-SearchBot
What xAI-SearchBot does
xAI-SearchBot fetches pages in service of Grok search-style answers. It is closer to ChatGPT-User / OAI-SearchBot than to a bulk training crawl.
How to get discovered
Allow xAI-SearchBot (and related Grok UAs), serve HTML Grok can parse, and avoid blocking the whole xAI family because one training token exists.
Why xAI-SearchBot might skip you
Blanket xAI/Grok Disallows, Cloudflare AI-bot rules, or pages that only exist behind an app download interstitial.
robots.txt rule
Allow this token on public pages you want retrieved or cited. A CDN “Block AI bots” toggle can still 403 it after robots.txt says Allow.
# Keep xAI-SearchBot eligible to fetch this site
User-agent: xAI-SearchBot
Allow: /xAI has not published a machine-readable range file for this token. Do not treat the user-agent alone as proof of identity.
JavaScript: reads raw HTML only.
Other xAI bots
Training, search, and user-fetch tokens from the same operator are not interchangeable. Allow the discovery path even when you opt out of training.
xAI-SearchBot FAQ
What is xAI-SearchBot?
xAI-SearchBot fetches pages in service of Grok search-style answers. It is closer to ChatGPT-User / OAI-SearchBot than to a bulk training crawl.
Should I allow xAI-SearchBot in robots.txt?
Yes if you want xAI to be able to fetch and cite this site. Blocking xAI-SearchBot is how pages stay invisible to that product even when Googlebot still crawls them.
How do I know a request is really xAI-SearchBot?
xAI has not published a range file for this token. Treat the user-agent as a claimed identity, and do not build a hard IP allowlist from blog posts or screenshots.
Letting xAI-SearchBot in is the start. Getting cited is the job.
Rankealo checks whether AI crawlers can reach you, then publishes pages built to be retrieved and quoted in ChatGPT, Claude, Perplexity, and Gemini.
