Google · Search indexes
Google-InspectionTool
The UA Search Console’s URL Inspection uses. If it cannot fetch you, you cannot debug Google’s view of the page — including AI-related features that ride on Search.
- Operator
- Traffic type
- Search indexes
- Verification
- User-agent + IP range
- robots.txt
- Honors robots.txt
Usually means: Crawlers that build the indexes ChatGPT search, Claude search, Perplexity, Google, and Bing draw from. Blocking them is how brands disappear from AI answers without noticing.
User-agent
robots.txt and most WAF rules match the token, not the full Mozilla string. Operators often wrap Google-InspectionTool in extra product or version text.
Google-InspectionTool
What Google-InspectionTool does
Google-InspectionTool fetches a URL when a site owner inspects it in Search Console. It is a diagnostic crawler, not the everyday Googlebot crawl, but it must be allowed for you to see what Google sees.
How to get discovered
Allow Google-InspectionTool and Googlebot. Use inspection to confirm rendered HTML, canonicals, and whether AI-overview-eligible content is even in the document.
Why Google-InspectionTool might skip you
IP allowlists that forgot Google’s inspection ranges, basic-auth on staging that you thought was production, or WAF rules that treat InspectionTool as a scraper.
robots.txt rule
Allow this token on public pages you want retrieved or cited. A CDN “Block AI bots” toggle can still 403 it after robots.txt says Allow.
# Keep Google-InspectionTool eligible to fetch this site
User-agent: Google-InspectionTool
Allow: /Allowlist so Google-InspectionTool can reach you
A user-agent is spoofable. Google publishes current CIDR ranges as JSON — fetch that file for WAF/CDN allowlists instead of copying ranges from a blog. Ranges rotate; a screenshot of ten prefixes will go stale. If Cloudflare “Block AI bots” is on, this is the list that gets the real crawler through.
https://developers.google.com/static/search/apis/ipranges/special-crawlers.json ↗
Sources
JavaScript: can render JS (often a second pass).
Other Google bots
Training, search, and user-fetch tokens from the same operator are not interchangeable. Allow the discovery path even when you opt out of training.
Google-InspectionTool FAQ
What is Google-InspectionTool?
Google-InspectionTool fetches a URL when a site owner inspects it in Search Console. It is a diagnostic crawler, not the everyday Googlebot crawl, but it must be allowed for you to see what Google sees.
Should I allow Google-InspectionTool in robots.txt?
Yes if you want Google to be able to fetch and cite this site. Blocking Google-InspectionTool is how pages stay invisible to that product even when Googlebot still crawls them.
How do I know a request is really Google-InspectionTool?
A user-agent is a claim anyone can send. Match the Google-InspectionTool token, then check the source IP against Google’s published ranges. Use those ranges as a WAF allowlist so a “Block AI bots” rule does not 403 the real crawler.
Letting Google-InspectionTool in is the start. Getting cited is the job.
Rankealo checks whether AI crawlers can reach you, then publishes pages built to be retrieved and quoted in ChatGPT, Claude, Perplexity, and Gemini.
