ByteDance · Training crawlers
Bytespider
ByteDance’s public-content crawler. It is widely reported to ignore robots.txt — a Disallow alone may not stop it.
- Operator
- ByteDance
- Traffic type
- Training crawlers
- Verification
- Not independently verifiable
- robots.txt
- Robots.txt compliance disputed
Usually means: Bots that collect public pages for future models. Blocking them opts you out of training. It does not, by itself, remove you from live AI search citations.
User-agent
robots.txt and most WAF rules match the token, not the full Mozilla string. Operators often wrap Bytespider in extra product or version text.
Bytespider
What Bytespider does
Bytespider collects pages for ByteDance/TikTok models and products. TikTokSpider is the search-index counterpart.
How to get discovered
For ByteDance search, allow TikTokSpider. For Bytespider, treat robots.txt as a request; use firewall/CDN rules if you need a real block.
Why Bytespider might skip you
Edge blocks, or it may not skip at all despite Disallow — verify in logs, do not trust robots.txt as enforcement.
robots.txt rule
This snippet opts out of training-style collection. Keep the search and user-fetch tokens from the same operator allowed if you still want citations.
# Optional training opt-out — does not remove you from AI search by itself
User-agent: Bytespider
Disallow: /ByteDance has not published a machine-readable range file for this token. Do not treat the user-agent alone as proof of identity.
JavaScript: reads raw HTML only.
Other ByteDance bots
Training, search, and user-fetch tokens from the same operator are not interchangeable. Allow the discovery path even when you opt out of training.
Bytespider FAQ
What is Bytespider?
Bytespider collects pages for ByteDance/TikTok models and products. TikTokSpider is the search-index counterpart.
Should I allow Bytespider in robots.txt?
Only if you want to opt out of training-style collection. Blocking Bytespider does not, by itself, remove you from live AI search citations — those use the search and user-fetch tokens from ByteDance.
How do I know a request is really Bytespider?
ByteDance has not published a range file for this token. Treat the user-agent as a claimed identity, and do not build a hard IP allowlist from blog posts or screenshots.
Letting Bytespider in is the start. Getting cited is the job.
Rankealo checks whether AI crawlers can reach you, then publishes pages built to be retrieved and quoted in ChatGPT, Claude, Perplexity, and Gemini.
