Meta · Training crawlers
meta-externalagent
Meta’s crawler for indexing or improving Meta products and AI systems. Publishers have disputed how consistently it honors robots.txt.
- Operator
- Meta
- Traffic type
- Training crawlers
- Verification
- Not independently verifiable
- robots.txt
- Robots.txt compliance disputed
Usually means: Bots that collect public pages for future models. Blocking them opts you out of training. It does not, by itself, remove you from live AI search citations.
User-agent
robots.txt and most WAF rules match the token, not the full Mozilla string. Operators often wrap meta-externalagent in extra product or version text.
meta-externalagent
What meta-externalagent does
meta-externalagent crawls public content for Meta products and AI. It is not facebookexternalhit (previews) and not meta-webindexer (search quality).
How to get discovered
For Meta AI search, allow meta-webindexer and meta-externalfetcher. Treat meta-externalagent as the training/improvement crawl you may choose to block — and enforce at the edge if you need a hard stop.
Why meta-externalagent might skip you
Intentional Disallow, or the reverse: you Disallowed it and still see hits, which is why edge enforcement exists.
robots.txt rule
This snippet opts out of training-style collection. Keep the search and user-fetch tokens from the same operator allowed if you still want citations.
# Optional training opt-out — does not remove you from AI search by itself
User-agent: meta-externalagent
Disallow: /Meta has not published a machine-readable range file for this token. Do not treat the user-agent alone as proof of identity.
Sources
JavaScript: render behavior is not documented.
Other Meta bots
Training, search, and user-fetch tokens from the same operator are not interchangeable. Allow the discovery path even when you opt out of training.
meta-externalagent FAQ
What is meta-externalagent?
meta-externalagent crawls public content for Meta products and AI. It is not facebookexternalhit (previews) and not meta-webindexer (search quality).
Should I allow meta-externalagent in robots.txt?
Only if you want to opt out of training-style collection. Blocking meta-externalagent does not, by itself, remove you from live AI search citations — those use the search and user-fetch tokens from Meta.
How do I know a request is really meta-externalagent?
Meta has not published a range file for this token. Treat the user-agent as a claimed identity, and do not build a hard IP allowlist from blog posts or screenshots.
Letting meta-externalagent in is the start. Getting cited is the job.
Rankealo checks whether AI crawlers can reach you, then publishes pages built to be retrieved and quoted in ChatGPT, Claude, Perplexity, and Gemini.
