ByteDance · Search indexes
TikTokSpider
ByteDance’s search/index spider. If you want TikTok/ByteDance search surfaces to know the site exists, this bot has to succeed.
- Operator
- ByteDance
- Traffic type
- Search indexes
- Verification
- Not independently verifiable
- robots.txt
- Compliance not documented
Usually means: Crawlers that build the indexes ChatGPT search, Claude search, Perplexity, Google, and Bing draw from. Blocking them is how brands disappear from AI answers without noticing.
User-agent
robots.txt and most WAF rules match the token, not the full Mozilla string. Operators often wrap TikTokSpider in extra product or version text.
TikTokSpider
What TikTokSpider does
TikTokSpider is tracked as a search or answer-index crawler. Bytespider is the broader public-content crawler — different job, same company.
How to get discovered
Allow TikTokSpider if ByteDance search visibility matters. Keep public pages 200 and crawlable; do not rely on an in-app webview copy of the site.
Why TikTokSpider might skip you
Company-wide ByteDance blocks that also catch TikTokSpider, or challenge pages that only humans pass.
robots.txt rule
Allow this token on public pages you want retrieved or cited. A CDN “Block AI bots” toggle can still 403 it after robots.txt says Allow.
# Keep TikTokSpider eligible to fetch this site
User-agent: TikTokSpider
Allow: /ByteDance has not published a machine-readable range file for this token. Do not treat the user-agent alone as proof of identity.
JavaScript: render behavior is not documented.
Other ByteDance bots
Training, search, and user-fetch tokens from the same operator are not interchangeable. Allow the discovery path even when you opt out of training.
TikTokSpider FAQ
What is TikTokSpider?
TikTokSpider is tracked as a search or answer-index crawler. Bytespider is the broader public-content crawler — different job, same company.
Should I allow TikTokSpider in robots.txt?
Yes if you want ByteDance to be able to fetch and cite this site. Blocking TikTokSpider is how pages stay invisible to that product even when Googlebot still crawls them.
How do I know a request is really TikTokSpider?
ByteDance has not published a range file for this token. Treat the user-agent as a claimed identity, and do not build a hard IP allowlist from blog posts or screenshots.
Letting TikTokSpider in is the start. Getting cited is the job.
Rankealo checks whether AI crawlers can reach you, then publishes pages built to be retrieved and quoted in ChatGPT, Claude, Perplexity, and Gemini.
