Common Crawl · Training crawlers
CCBot
Common Crawl’s CCBot builds open web datasets that many labs reuse for pretraining. Blocking it is the widest training opt-out you can make in one token.
- Operator
- Common Crawl
- Traffic type
- Training crawlers
- Verification
- Not independently verifiable
- robots.txt
- Honors robots.txt
Usually means: Bots that collect public pages for future models. Blocking them opts you out of training. It does not, by itself, remove you from live AI search citations.
User-agent
robots.txt and most WAF rules match the token, not the full Mozilla string. Operators often wrap CCBot in extra product or version text.
CCBot
What CCBot does
CCBot crawls the public web into Common Crawl. Those snapshots are reused by multiple model builders. It does not power ChatGPT search.
How to get discovered
Do not rely on CCBot for citations. Allow search/answer bots for GEO; Disallow CCBot if you do not want to be in open crawl dumps.
Why CCBot might skip you
Disallow: / for CCBot (honored), or the crawl simply not sampling your host this month.
robots.txt rule
This snippet opts out of training-style collection. Keep the search and user-fetch tokens from the same operator allowed if you still want citations.
# Optional training opt-out — does not remove you from AI search by itself
User-agent: CCBot
Disallow: /Common Crawl has not published a machine-readable range file for this token. Do not treat the user-agent alone as proof of identity.
Sources
JavaScript: reads raw HTML only.
CCBot FAQ
What is CCBot?
CCBot crawls the public web into Common Crawl. Those snapshots are reused by multiple model builders. It does not power ChatGPT search.
Should I allow CCBot in robots.txt?
Only if you want to opt out of training-style collection. Blocking CCBot does not, by itself, remove you from live AI search citations — those use the search and user-fetch tokens from Common Crawl.
How do I know a request is really CCBot?
Common Crawl has not published a range file for this token. Treat the user-agent as a claimed identity, and do not build a hard IP allowlist from blog posts or screenshots.
Letting CCBot in is the start. Getting cited is the job.
Rankealo checks whether AI crawlers can reach you, then publishes pages built to be retrieved and quoted in ChatGPT, Claude, Perplexity, and Gemini.
