xAI · AI answers
Grok-DeepSearch
Grok’s deeper user-triggered research fetch. This is how a page gets pulled into a Grok DeepSearch-style answer.
- Operator
- xAI
- Traffic type
- AI answers
- Verification
- Not independently verifiable
- robots.txt
- Usually honors robots.txt
Usually means: Live fetches when someone asks an assistant a question. If these bots cannot reach the page, you cannot be the citation in that answer.
User-agent
robots.txt and most WAF rules match the token, not the full Mozilla string. Operators often wrap Grok-DeepSearch in extra product or version text.
Grok-DeepSearch
What Grok-DeepSearch does
Grok-DeepSearch retrieves source pages while Grok runs a deeper search pass. If the fetch fails, that URL cannot appear as a grounded source.
How to get discovered
Allow Grok-DeepSearch, keep long-form evidence on crawlable URLs, and make claims stand alone in a paragraph a model can quote.
Why Grok-DeepSearch might skip you
Rate-limit pages, JS-only docs, or WAF rules that flag “DeepSearch” / Grok as scrapers.
robots.txt rule
Allow this token on public pages you want retrieved or cited. A CDN “Block AI bots” toggle can still 403 it after robots.txt says Allow.
# Keep Grok-DeepSearch eligible to fetch this site
User-agent: Grok-DeepSearch
Allow: /xAI has not published a machine-readable range file for this token. Do not treat the user-agent alone as proof of identity.
JavaScript: reads raw HTML only.
Other xAI bots
Training, search, and user-fetch tokens from the same operator are not interchangeable. Allow the discovery path even when you opt out of training.
Grok-DeepSearch FAQ
What is Grok-DeepSearch?
Grok-DeepSearch retrieves source pages while Grok runs a deeper search pass. If the fetch fails, that URL cannot appear as a grounded source.
Should I allow Grok-DeepSearch in robots.txt?
Yes if you want xAI to be able to fetch and cite this site. Blocking Grok-DeepSearch is how pages stay invisible to that product even when Googlebot still crawls them.
How do I know a request is really Grok-DeepSearch?
xAI has not published a range file for this token. Treat the user-agent as a claimed identity, and do not build a hard IP allowlist from blog posts or screenshots.
Letting Grok-DeepSearch in is the start. Getting cited is the job.
Rankealo checks whether AI crawlers can reach you, then publishes pages built to be retrieved and quoted in ChatGPT, Claude, Perplexity, and Gemini.
