Get discovered by the bots that feed AI answers

The fetch ChatGPT makes when a person asks it to open, browse, or cite a specific page. This is a live citation request, not GPTBot training.

Claude’s on-demand fetcher when a person needs a live page for an answer. Blocking it hides you from in-product browsing and citations.

Perplexity’s live fetch for a user question and the citations shown next to the answer. This is how a page becomes a source in the moment.

A Google user-triggered fetcher used by AI and product experiences. If Google-Agent cannot read you, those surfaces have nothing to quote.

The fetcher behind NotebookLM-style answer workflows. Discovery here means your page can be loaded into a notebook and cited as a source.

Google’s fetcher for read-aloud and assistant playback. If it cannot parse the article text, the assistant has nothing to speak.

A second Google agent token seen in the wild for user-triggered AI fetches. Treat it like Google-Agent: allow it if you want Google products to read the page.

User-triggered Microsoft Copilot fetches for AI answers. Blocking Copilot (or Bing preview UAs) keeps your pages out of Copilot citations.

Mistral’s user-triggered fetcher. If Le Chat needs your page to answer, this is the UA that has to succeed.

Amazon’s user-triggered fetch for fresh answers in Alexa and related products. Discovery here is being readable when Amazon needs the live page.

DuckDuckGo’s real-time crawler for AI-assisted answers with citations. If DuckAssistBot is blocked, you will not be a Duck.ai source.

xAI’s user-triggered search fetcher for Grok answers. Allow it if you want Grok to be able to retrieve and cite the page.

Grok’s deeper user-triggered research fetch. This is how a page gets pulled into a Grok DeepSearch-style answer.

Meta’s fetcher when a person requests or shares a specific URL. Blocking it breaks Meta AI reads and some share previews.

Moonshot’s user-triggered Kimi fetcher. If Kimi cannot open the page, it cannot cite you in that answer.

Alibaba’s user-triggered Qwen fetcher. Discovery means Qwen can actually read the page when someone asks.

OpenAI’s search-index crawler. This is the bot that makes a site eligible to appear in ChatGPT search results and citations.

Anthropic’s crawler for Claude search discovery. If Claude-SearchBot cannot index you, Claude search has no page to retrieve.

Perplexity’s index crawler. This is how pages enter the corpus Perplexity answers from — a search bot, not a training bot.

The UA Search Console’s URL Inspection uses. If it cannot fetch you, you cannot debug Google’s view of the page — including AI-related features that ride on Search.

Google’s Search crawler. Cloudflare notes Googlebot traffic can also support AI features such as AI Overviews — blocking it is not an AI-only decision.

Microsoft’s Search crawler. Copilot and Bing AI features depend on Bing’s index — if Bingbot cannot crawl you, Copilot has a thinner source set.

Legacy Microsoft search crawler still seen in logs. Allow it unless you have a specific reason to drop old Microsoft UAs.

Mistral’s search/index crawler. This is how pages become eligible for Mistral search-style answers, not the live user fetch.

Amazon’s crawler for making content eligible in Amazon search experiences. Blocking it hides you from those surfaces even if Amazonbot still trains.

Meta’s crawler for Meta AI search quality. If it cannot index you, Meta AI search has a weaker view of the site.

Moonshot’s Kimi search-index crawler. This is the discovery path into Kimi search, not the live user fetch.

ByteDance’s search/index spider. If you want TikTok/ByteDance search surfaces to know the site exists, this bot has to succeed.

Baidu’s search crawler. Baidu AI features sit on top of Baidu’s index — no crawl, no discovery in that ecosystem.

You.com’s search/index crawler. Allow it if you want You.com AI search to be able to retrieve your pages.

OpenAI’s training crawler. Blocking GPTBot opts you out of future model training. It does not remove you from ChatGPT search if OAI-SearchBot stays allowed.

Anthropic’s training crawler for Claude. Blocking ClaudeBot is a training opt-out. Keep Claude-SearchBot allowed if you still want to be cited.

Google’s generic crawler for product teams, research, and development fetches. It is not Googlebot and does not replace Search indexing.

A robots.txt-only switch for Gemini training and grounding. It is not an HTTP crawler. Disallowing it does not change Google Search crawling.

Google crawler for targeted Vertex AI agent crawls that a site owner requested. It is not the public Search crawler.

Apple’s robots.txt-only opt-out for AI training. Disallowing it does not remove you from Apple search, Spotlight, or Siri results that use Applebot.

Apple’s crawler for search and, depending on Applebot-Extended, AI training. Cloudflare classifies it across search and training use cases.

Amazon’s broad crawler for improving Amazon products and services, including search and AI. Blocking it is a product-opt-out, not a Google ranking move.

Meta’s crawler for indexing or improving Meta products and AI systems. Publishers have disputed how consistently it honors robots.txt.

Moonshot’s public-content crawler. Blocking KimiBot is a training-style opt-out; keep Kimi-SearchBot allowed for discovery.

ByteDance’s public-content crawler. It is widely reported to ignore robots.txt — a Disallow alone may not stop it.

Baidu’s ERNIE public-content crawler. Blocking it is a training opt-out; Baiduspider remains the Search discovery path.

Alibaba’s Qwen training-style crawler. Keep Qwen-User allowed if you still want live Qwen answers to be able to fetch you.

DeepSeek’s public-content crawler. DeepSeek’s chat product does not ship a first-party web search bot comparable to OAI-SearchBot.

Zhipu’s ChatGLM public-content spider. Treat it as a training-style crawl unless Zhipu documents a separate search UA.

Cohere’s public-content crawler token. Blocking it opts you out of Cohere collection; it is not a ChatGPT citation switch.

Cohere’s named training-data crawler. A Disallow here is a clear training opt-out.

Allen Institute crawler for AI research datasets. Blocking it is a research-crawl opt-out, not a ChatGPT ranking move.

Common Crawl’s CCBot builds open web datasets that many labs reuse for pretraining. Blocking it is the widest training opt-out you can make in one token.

Anthropic’s legacy training token. Rankealo still tracks it because older robots.txt files and crawlers still emit anthropic-ai.

OpenAI’s ads and landing-page fetcher. Allow it if you run ChatGPT ads or need OpenAI to preview a destination URL.

An xAI Grok crawler token seen in the wild. Allow Grok UAs if you want Grok products to be able to fetch the site.

A generic xAI bot token. If it cannot fetch you, some Grok product fetches fail even when SearchBot is allowed.

Another xAI/Grok identity in logs. Rankealo lists it so you can allow the real token instead of guessing.

xAI’s web-crawler token. Allow it if you want xAI systems to discover pages beyond a single user click.

A short Grok user-agent token. Short tokens are easy to miss in robots.txt because they do not look like GPTBot.

Meta’s ads-related fetcher. Allow it on destinations you run as Facebook/Instagram ads or the preview will not match the live page.

Meta’s link-preview crawler for Facebook, Instagram, and Messenger. Blocking it breaks unfurls, not ChatGPT citations.

Meta crawler traffic associated with Facebook bot fetches. Allow it unless you are deliberately dropping Facebook product crawls.

ByteDance’s Doubao product crawler. Allow it if Doubao should be able to fetch your pages; block at the edge if you want a hard no.

Baidu’s Ernie/Yiyan assistant crawler token. Distinct from Baiduspider (Search) and ERNIEBot (training-style collection).

Alibaba’s Tongyi assistant crawler. Allow it for Tongyi fetches; Qwen-User is the closer “live answer” twin.

Alibaba Cloud’s crawler token. Usually an infrastructure/product fetch, not ChatGPT search — still needs a 200 if you want those products to read you.