Google · Search indexes
Googlebot
Google’s Search crawler. Cloudflare notes Googlebot traffic can also support AI features such as AI Overviews — blocking it is not an AI-only decision.
- Operator
- Traffic type
- Search indexes
- Verification
- User-agent + IP range
- robots.txt
- Honors robots.txt
Usually means: Crawlers that build the indexes ChatGPT search, Claude search, Perplexity, Google, and Bing draw from. Blocking them is how brands disappear from AI answers without noticing.
User-agent
robots.txt and most WAF rules match the token, not the full Mozilla string. Operators often wrap Googlebot in extra product or version text.
Googlebot
Example full string: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
What Googlebot does
Googlebot discovers and indexes the open web for Google Search. That index also feeds AI Overviews and AI Mode. It renders JavaScript on a second pass; most AI crawlers do not.
How to get discovered
Never Disallow Googlebot on public marketing pages. Keep a sitemap, fix crawl errors, and still serve meaningful HTML in the first response so non-Google AI bots can use the same page.
Why Googlebot might skip you
noindex, accidental Disallow, canonical loops, or a site that only exists after login. Googlebot skipping a URL also starves AI Overviews of that source.
robots.txt rule
Allow this token on public pages you want retrieved or cited. A CDN “Block AI bots” toggle can still 403 it after robots.txt says Allow.
# Keep Googlebot eligible to fetch this site
User-agent: Googlebot
Allow: /Allowlist so Googlebot can reach you
A user-agent is spoofable. Google publishes current CIDR ranges as JSON — fetch that file for WAF/CDN allowlists instead of copying ranges from a blog. Ranges rotate; a screenshot of ten prefixes will go stale. If Cloudflare “Block AI bots” is on, this is the list that gets the real crawler through.
https://developers.google.com/static/search/apis/ipranges/googlebot.json ↗
Sources
JavaScript: can render JS (often a second pass).
Other Google bots
Training, search, and user-fetch tokens from the same operator are not interchangeable. Allow the discovery path even when you opt out of training.
Googlebot FAQ
What is Googlebot?
Googlebot discovers and indexes the open web for Google Search. That index also feeds AI Overviews and AI Mode. It renders JavaScript on a second pass; most AI crawlers do not.
Should I allow Googlebot in robots.txt?
Yes if you want Google to be able to fetch and cite this site. Blocking Googlebot is how pages stay invisible to that product even when Googlebot still crawls them.
How do I know a request is really Googlebot?
A user-agent is a claim anyone can send. Match the Googlebot token, then check the source IP against Google’s published ranges. Use those ranges as a WAF allowlist so a “Block AI bots” rule does not 403 the real crawler.
Letting Googlebot in is the start. Getting cited is the job.
Rankealo checks whether AI crawlers can reach you, then publishes pages built to be retrieved and quoted in ChatGPT, Claude, Perplexity, and Gemini.
