ByteDance / Toutiao Search

ByteDance / Toutiao Search in context

ByteDance / Toutiao Search is a training crawler operated by ByteDance. In plain terms, it collects pages both for training and for search or answers. Our database files it under data scrapers and ranks it Tier 2, important crawlers from established AI and search operators. 2 of the 39 data scrapers in our database share this tier.

Its safety rating is aggressive and its blocking impact is low. So ByteDance / Toutiao Search is heavy on servers and cheap to lose: the clearest case for blocking it. How we rate safety and impact.

How site owners treat ByteDance / Toutiao Search

In our own study of robots.txt files across the Tranco top 10,000 sites (fetched October 20, 2026, 5,583 valid files), 758 files (13.6%) name Bytespider in a User-agent line, and 887 (15.9%) block it from the homepage. 664 of those blocks come from a group written for this crawler by name, and 223 come from a broad User-agent: * rule that catches it by accident. Larger sites are stricter: 23.8% of the top 1,000 block it, against 15.4% of the middle band and 14.9% of the tail.

Does it follow robots.txt?

ByteDance / Toutiao Search's operator does not clearly document how it treats robots.txt. Write the ByteDance / Toutiao Search rule anyway, then confirm in your logs that its requests stop.

ByteDance crawlers and close alternatives

ByteDance / Toutiao Search is one of 4 ByteDance crawlers in our database. A User-agent: Bytespider group controls ByteDance / Toutiao Search only; the other ByteDance tokens below need their own lines.

• Bytespider: this crawler (Tier 2)
• DoubaoBot: Doubao (ByteDance), collects pages both for training and for search or answers (Tier 2)
• TikTokSpider: TikTok Spider, collects pages to train AI models (Tier 2)
• Trae: Trae Agent (ByteDance), fetches a page when a person asks an assistant or app to open it (Tier 3)

To give ByteDance / Toutiao Search the same answer as its siblings, stack User-agent: Bytespider with their lines above one set of rules; to treat ByteDance / Toutiao Search differently, give it its own group.

No other crawler in our data scrapers category has exactly the same purpose, which makes ByteDance / Toutiao Search an unusual case: decide about it on its own merits.

We first recorded Bytespider on March 17, 2026. ByteDance documents it on toutiao.com, and we recheck that page before each release.

Which rule ByteDance / Toutiao Search actually follows

ByteDance / Toutiao Search reads exactly one group in robots.txt, the most specific one that names Bytespider, and never combines rules across groups. This short file shows the trap:

User-agent: *
Disallow: /private/

User-agent: Bytespider
Disallow: /drafts/

1. Find the group. ByteDance / Toutiao Search looks for a group naming Bytespider. One exists, so it ignores User-agent: * completely.
2. Apply only that group. It is refused /drafts/ and nothing else.
3. The surprise. /private/ is open to ByteDance / Toutiao Search, because that rule lives in the * group it no longer reads. To keep it out of both, repeat Disallow: /private/ inside its own group.

Should you allow ByteDance / Toutiao Search? Decide by situation

Check ByteDance / Toutiao Search on your own site

1. Open /robots.txt on your domain and search for Bytespider. If it is not named, it follows User-agent: *.
2. List the Disallow lines in the group ByteDance / Toutiao Search follows and test your key URLs against them; the longest match wins, and Allow wins a tie.
3. Look beyond the file. A firewall or bot manager can refuse ByteDance / Toutiao Search even when robots.txt allows it, and that never shows in the file.
4. Run our live AI crawler test to see how ByteDance / Toutiao Search and every other crawler we track are treated in one pass, including firewall refusals.