ByteDance

Common mistakes with ByteDance's crawlers

The errors we see most often with ByteDance's crawlers in the files and firewalls we test:

1. Assuming that blocking ByteDance's crawlers removes you from AI answers. It is a training crawler; AI answers are fed by retrieval and user-triggered crawlers with different tokens.
2. Blocking ByteDance's crawlers with a broad User-agent: * rule that also catches search engines. Name Bytespider in its own group instead.
3. Expecting a ByteDance's crawlers block to remove content already collected. robots.txt only affects future visits.

All three hide inside a file that looks correct, so check ByteDance's crawlers from outside after every change.

Spotting ByteDance in your logs

robots.txt says what ByteDance / Toutiao Search and Doubao (ByteDance) and TikTok Spider and Trae Agent (ByteDance) may do; your access logs show what they actually did.

Count requests

On a server with standard Nginx or Apache logs, this counts matching requests per day (adjust the log path for your host):

grep -iE "Bytespider|DoubaoBot|TikTokSpider|Trae" /var/log/nginx/access.log \
  | awk '{print $4}' | cut -d: -f1 | sort | uniq -c

And this shows which paths it requested most, which is the quickest way to see whether your Disallow lines are being respected:

grep -iE "Bytespider|DoubaoBot|TikTokSpider|Trae" /var/log/nginx/access.log \
  | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

Check the requests are genuine

Anyone can send these user-agents, so confirm the source before acting on a match:

• Bytespider: we hold no published IP list, so use reverse DNS where ByteDance documents a hostname and treat an unverifiable ByteDance / Toutiao Search request with caution. Check it separately from Doubao (ByteDance) and TikTok Spider and Trae Agent (ByteDance).
• DoubaoBot: we hold no published IP list, so use reverse DNS where ByteDance documents a hostname and treat an unverifiable Doubao (ByteDance) request with caution. Check it separately from ByteDance / Toutiao Search and TikTok Spider and Trae Agent (ByteDance).
• TikTokSpider: we hold no published IP list, so use reverse DNS where ByteDance documents a hostname and treat an unverifiable TikTok Spider request with caution. Check it separately from ByteDance / Toutiao Search and Doubao (ByteDance) and Trae Agent (ByteDance).
• Trae: we hold no published IP list, so use reverse DNS where ByteDance documents a hostname and treat an unverifiable Trae Agent (ByteDance) request with caution. Check it separately from ByteDance / Toutiao Search and Doubao (ByteDance) and TikTok Spider.

What our own data shows about ByteDance

Everything below comes from the AI Crawler Check bot database and our two most recent robots.txt scans of the Tranco top 10,000, not from ByteDance's marketing. It covers 4 ByteDance tokens in turn.

Bytespider in our records

We first recorded Bytespider on March 17, 2026. ByteDance documents it on toutiao.com, and we recheck that page before each release. Between our scans of October 17, 2026 and October 20, 2026, files naming Bytespider went from 706 to 758 (+52) and files blocking it from 828 to 887 (+59), so more sites wrote a rule for Bytespider and more of those rules block it.

Among the 39 data scrapers in our database, Bytespider sits first by priority (Tier 2), level with TikTokSpider. 4 of the 39 share its aggressive rating. Other operators with a crawler that collects pages both for training and for search or answers: PerplexityBot (Perplexity AI, Tier 1), Andibot (Andi, Tier 2), cohere-ai (Cohere, Tier 2), DeepSeekBot (DeepSeek, Tier 2), GrokBot (xAI, Tier 2). A policy for Bytespider is usually a policy for these too.

A decision record for Bytespider

Teams that review crawlers once a quarter keep one short entry per token, so anyone can see why a rule exists and when it was last checked. Filled in from our data, the Bytespider entry reads:

• Operator and job: ByteDance; it collects pages both for training and for search or answers.
• Priority and behaviour: Tier 2, rated aggressive, blocking impact low.
• Suggested starting point: Disallow, and back it with a server rule.
• Enforcement: Bytespider's operator does not clearly document how it treats robots.txt. Write the Bytespider rule anyway, then confirm in your logs that its requests stop.

User-agent: Bytespider
Disallow: /

Paste the line into the validator with one of your own URLs before it goes live. Revisit the Bytespider entry sooner than most: a aggressive rating means ByteDance traffic deserves a monthly look at your logs.

DoubaoBot in our records

We first recorded DoubaoBot on September 22, 2026. We could not find a page where ByteDance names it, so treat the token as reported rather than documented. DoubaoBot is outside our tracked token set, but 15 of 5,583 top-site files already name it, which is enough to show owners treat it as a real crawler.

Among the 104 AI and LLM crawlers in our database, DoubaoBot sits 37th by priority (Tier 2), level with AddSearchBot, AI2Bot, AI2Bot-DeepResearchEval, aiHitBot. 22 of the 104 share its caution rating. Other operators with a crawler that collects pages both for training and for search or answers: PerplexityBot (Perplexity AI, Tier 1), Andibot (Andi, Tier 2), cohere-ai (Cohere, Tier 2), DeepSeekBot (DeepSeek, Tier 2), GrokBot (xAI, Tier 2). A policy for DoubaoBot is usually a policy for these too.

TikTokSpider in our records

We first recorded TikTokSpider on September 22, 2026. We could not find a page where ByteDance names it, so treat the token as reported rather than documented. TikTokSpider is outside our tracked token set, but 97 of 5,583 top-site files already name it, which is enough to show owners treat it as a real crawler.

Among the 39 data scrapers in our database, TikTokSpider sits second by priority (Tier 2), level with Bytespider. 4 of the 39 share its aggressive rating. Other operators with a crawler that collects pages to train AI models: ClaudeBot (Anthropic, Tier 1), Google-Extended (Google, Tier 1), GPTBot (OpenAI, Tier 1), Meta-ExternalAgent (Meta, Tier 1), AI2Bot (Allen Institute for AI, Tier 2). A policy for TikTokSpider is usually a policy for these too.

Trae in our records

We first recorded Trae on September 22, 2026. We could not find a page where ByteDance names it, so treat the token as reported rather than documented. Trae is outside our tracked token set, but 15 of 5,583 top-site files already name it, which is enough to show owners treat it as a real crawler.

Among the 104 AI and LLM crawlers in our database, Trae sits 102th by priority (Tier 3), level with AIWebIndex-Agent, alphalens-bot, amazon-QBusiness, Anomura. 81 of the 104 share its safe rating. Other operators with a crawler that fetches a page when a person asks an assistant or app to open it: ChatGPT Agent (OpenAI, Tier 1), ChatGPT Operator (OpenAI, Tier 1), ChatGPT-User (OpenAI, Tier 1), ChatGPT-User/2.0 (OpenAI, Tier 1), Claude-SearchBot (Anthropic, Tier 1). A policy for Trae is usually a policy for these too.

How top sites treat ByteDance's crawlers

From our robots.txt study of the Tranco top 10,000 (5,583 valid files, fetched October 20, 2026):

• Bytespider, the only one tracked among the ByteDance's crawlers tokens: blocked in 15.9% of files (664 with its own group, 223 through User-agent: *), more often by the top 1,000 (23.8%) than the long tail (14.9%).

For scale, 20.1% of all files block at least one AI training crawler. Bytespider is one of the tokens driving that figure.

More questions about ByteDance

Will blocking ByteDance affect my Google rankings?

Only if one of its tokens is a Google search crawler. None of ByteDance's tokens are Googlebot, so classic Google rankings are unaffected.

How quickly does a robots.txt change take effect for ByteDance?

Crawlers cache robots.txt, commonly for up to a day. Expect a change to apply within about 24 hours, and confirm it in your logs.

Can I allow ByteDance on some sections and block it on others?

Yes. Inside the group for Bytespider, add Disallow lines for the paths to keep private and Allow lines for exceptions. The longest matching rule wins.

Does blocking ByteDance delete content it already collected?

No. robots.txt only controls future visits. For content already collected, check whether ByteDance offers a removal or opt-out process.

Is a firewall block better than robots.txt for ByteDance?

A firewall enforces the decision, while robots.txt asks politely. Many operators recommend robots.txt because blocking their IPs can stop them reading your rules at all. Use robots.txt first, and add a firewall rule only if requests continue.