Doubao (ByteDance)

Doubao (ByteDance) in context

Doubao (ByteDance) is a training crawler operated by ByteDance. In plain terms, it collects pages both for training and for search or answers. Our database files it under AI and LLM crawlers and ranks it Tier 2, important crawlers from established AI and search operators. 74 of the 104 AI and LLM crawlers in our database share this tier.

Its safety rating is caution and its blocking impact is low. So the decision about Doubao (ByteDance) rests on what you want from ByteDance, not on its behaviour. How we rate safety and impact.

How site owners treat Doubao (ByteDance)

In our own scan of the Tranco top 10,000 (5,583 valid robots.txt files), 15 files (0.3%) name DoubaoBot in a User-agent line. That shows site owners treat the token as a real crawler worth a rule. It does not, on its own, prove who operates it.

Does it follow robots.txt?

Doubao (ByteDance)'s operator does not clearly document how it treats robots.txt. Write the Doubao (ByteDance) rule anyway, then confirm in your logs that its requests stop.

ByteDance crawlers and close alternatives

Doubao (ByteDance) is one of 4 ByteDance crawlers in our database. A User-agent: DoubaoBot group controls Doubao (ByteDance) only; the other ByteDance tokens below need their own lines.

• DoubaoBot: this crawler (Tier 2)
• Bytespider: ByteDance / Toutiao Search, collects pages both for training and for search or answers (Tier 2)
• TikTokSpider: TikTok Spider, collects pages to train AI models (Tier 2)
• Trae: Trae Agent (ByteDance), fetches a page when a person asks an assistant or app to open it (Tier 3)

To give Doubao (ByteDance) the same answer as its siblings, stack User-agent: DoubaoBot with their lines above one set of rules; to treat Doubao (ByteDance) differently, give it its own group.

Other AI and LLM crawlers that do the same job (collects pages both for training and for search or answers) include PerplexityBot from Perplexity AI, Andibot from Andi, cohere-ai from Cohere, DeepSeekBot from DeepSeek, GrokBot from xAI, iAskBot from iAsk. Deciding about Doubao (ByteDance) is usually a decision about the whole group, so check how your file treats these too.

We first recorded DoubaoBot on September 22, 2026. We could not find a page where ByteDance names it, so treat the token as reported rather than documented.

Which rule Doubao (ByteDance) actually follows

Doubao (ByteDance) reads exactly one group in robots.txt, the most specific one that names DoubaoBot, and never combines rules across groups. This short file shows the trap:

User-agent: *
Disallow: /private/

User-agent: DoubaoBot
Disallow: /drafts/

1. Find the group. Doubao (ByteDance) looks for a group naming DoubaoBot. One exists, so it ignores User-agent: * completely.
2. Apply only that group. It is refused /drafts/ and nothing else.
3. The surprise. /private/ is open to Doubao (ByteDance), because that rule lives in the * group it no longer reads. To keep it out of both, repeat Disallow: /private/ inside its own group.

Should you allow Doubao (ByteDance)? Decide by situation

Check Doubao (ByteDance) on your own site

1. Open /robots.txt on your domain and search for DoubaoBot. If it is not named, it follows User-agent: *.
2. List the Disallow lines in the group Doubao (ByteDance) follows and test your key URLs against them; the longest match wins, and Allow wins a tie.
3. Look beyond the file. A firewall or bot manager can refuse Doubao (ByteDance) even when robots.txt allows it, and that never shows in the file.
4. Run our GPTBot and ClaudeBot access check to see how Doubao (ByteDance) and every other crawler we track are treated in one pass, including firewall refusals.