Bytespider: What It Is, How Much It Crawls in 2026, and How to Block It

Bytespider is the web crawler run by ByteDance, the company that owns TikTok. In 2024 it was the busiest AI crawler on Cloudflare's network and the one site owners complained about most. In 2026 the picture is very different: its share of AI crawling has collapsed, newer training crawlers send far more requests, and the real question for most sites is no longer "how do I stop Bytespider?" but "which aggressive crawler is costing me the most?"

This guide answers both. It explains what Bytespider is, what the 2024 to 2026 data shows, how to block it properly, how to tell the real crawler from impostors, and how to deal with the wider group of heavy AI scrapers. We use public network data from Cloudflare and Hostinger, plus our own study of the top 10,000 sites. To see whether Bytespider can reach your site right now, run the free AI crawler checker.

Bytespider: who runs it and what it collects

Bytespider identifies itself with the token Bytespider in its user agent string. It belongs to ByteDance, the Chinese technology company behind TikTok, Douyin and the Toutiao news app. Cloudflare's 2024 analysis described it as "reportedly used to gather training data for its large language models (LLMs), including those that support its ChatGPT rival, Doubao" (Cloudflare, July 2024). Public crawler registries also link it to Toutiao search.

What ByteDance has not done, as far as we could confirm, is publish a crawler documentation page in the style of OpenAI, Anthropic or Google: no stated purpose list, no robots.txt policy and no IP ranges. That matters for two reasons. You cannot verify a request the way you can with Googlebot, and you cannot quote an official promise that it follows your rules. Our Bytespider profile lists what is known and links the sources, and the scraper crawlers directory covers similar bots.

Like most AI crawlers, Bytespider does not run JavaScript. Vercel and MERJ tested the major AI crawlers across Vercel's network and listed ByteDance's Bytespider among those that "currently" do not render JavaScript (Vercel and MERJ). It reads the HTML your server sends and nothing more.

Bytespider in 2026: what the traffic data shows

In mid-2024 Bytespider was the most active AI crawler Cloudflare saw. Its July 2024 post said Bytespider "not only leads in terms of number of requests but also in both the extent of its Internet property crawling and the frequency with which it is blocked", reaching 40.40% of websites behind Cloudflare, ahead of GPTBot at 35.46%.

One year later the order had flipped. Cloudflare's August 2025 report tracked a fixed set of customers and found:

Looking only at AI crawlers, the change is even larger: Bytespider dropped from 37.3% of AI-only crawler traffic to 5.8%, a fall of 31.5 percentage points, while GPTBot grew from 11.9% to 28.1% (Cloudflare, August 2025).

Hostinger's January 2026 study of 66.7 billion bot requests across about 5 million hosted sites tells a similar story from a different network. The heaviest training crawler on those sites was Meta-ExternalAgent, at 57.33% average site coverage and about 4 billion requests, followed by OpenAI's GPTBot at 55.67% coverage (Hostinger). Hostinger grouped TikTok's assistant-facing bot separately, at 25.67% coverage and 1.4 billion requests.

Crawl-to-refer: why blocking it costs almost nothing

Cloudflare measures a "crawl-to-refer ratio": how many pages an operator crawls for every visitor it sends back. For ByteDance, the ratio fell from 18 in January 2025 to 0.9 in July 2025. That sounds generous, but it reflects much less crawling, not many more visitors. For comparison, Cloudflare measured OpenAI at 1,091 and Anthropic at 38,065 crawls per referral in July 2025.

For most sites outside China, ByteDance sends no measurable search or assistant traffic. That is why blocking Bytespider is one of the lowest-risk changes you can make. You give up nothing you can see in analytics.

How many sites block Bytespider?

In our study, 746 of 4,962 valid robots.txt files from the Tranco top 10,000 block Bytespider from the whole site (15%). That places it second only to Common Crawl's CCBot, and above GPTBot.

The last row is telling. 180 sites block Bytespider while leaving GPTBot open. For those sites, Bytespider is not about "AI training" in general; it is a specific crawler they decided they did not want.

TikTok blocks other companies' AI crawlers too

We read tiktok.com's own robots.txt on 5 October 2026. Its first group disallows the whole site for a long list of AI agents: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, anthropic-ai, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, meta-externalagent, CCBot, DuckAssistBot, AI2Bot, MistralAI-User and several Google agent tokens. Bytespider is on that list too. ByteDance's own consumer site, in other words, opts out of almost every AI crawler, its own included.

How to block Bytespider

Use two layers. Robots.txt states your wish, and a server rule enforces it for any client that ignores the file. Blocking Bytespider has no effect on Google or Bing rankings, and it does not stop TikTok link previews, which use a different agent.

# ByteDance crawler: no referral value for this site
User-agent: Bytespider
Disallow: /

# Everyone else keeps the default
User-agent: *
Allow: /

Give Bytespider its own group, or add it to a training-only group. Do not add it to a group with search crawlers you want to keep. RFC 9309 merges every User-agent line in a group into one set of rules.

# In the server block
if ($http_user_agent ~* "Bytespider") {
    return 403;
}

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} Bytespider [NC]
RewriteRule .* - [F,L]

(http.user_agent contains "Bytespider")
# Action: Block

Cloudflare users have a simpler option too. Its AI traffic settings include a Training category, and Cloudflare lists Bytespider among the crawlers it manages. Our guide to blocking bad bots without hurting AI SEO explains how to build a wider block list safely.

Does Bytespider respect robots.txt?

There is no ByteDance statement we could confirm either way, and the public record is mixed. In 2024 many site owners reported Bytespider crawling despite robots.txt blocks, and some hosts blocked it at the network level. Cloudflare's 2024 post noted that robots.txt blocks "are reliant on the bot operator respecting robots.txt", and that "user agents are trivial for bot operators to change".

If you have evidence that Bytespider ignored your robots.txt, keep it: the log lines with timestamps, IPs and the full user agent, plus a copy of your robots.txt at the time. That record helps your host or CDN tune a block, and it is the same evidence Semrush, Anthropic and other operators ask for when you report a misbehaving crawler to them.

Our honest answer: treat robots.txt as a request and the server rule as the control. That is the right approach for any crawler whose operator does not publish a policy and an IP list. It also protects you if someone else uses the Bytespider name.

Telling real Bytespider from impostors

With Googlebot, OpenAI or Anthropic, you can check an IP against a published list or a reverse DNS pattern. For Bytespider we could not find an official list, so verification is limited. Here is what you can do:

1. Look at the full user agent string. Note any version number and the URL it links to.
2. Look up the IP owner (ASN) with a tool such as whois. Requests from residential or consumer networks are unlikely to be a large company's crawler.
3. Check the pattern. A real crawler usually fetches robots.txt first and then works through many pages. An impostor often hits a few expensive URLs, such as search pages, very fast.
4. Block by user agent anyway. If you do not want Bytespider, it does not matter whether a request is real or fake.

grep -i "Bytespider" /var/log/nginx/access.log | awk '{print $1}' \
  | sort | uniq -c | sort -rn | head -10 \
  | while read n ip; do echo "$n $ip $(whois $ip | grep -im1 -E 'orgname|org-name|netname')"; done

Other aggressive AI crawlers to watch

"Aggressive" can mean two things: high volume, or poor manners such as ignoring rules or hiding identity. Here is how the main candidates compare in 2026, using public data.

The worst behavior is usually not from named crawlers at all. Hostinger found that unlabeled scripts, empty user agents and generic bots made up 34.6% of the bot requests it analyzed, reaching more than 92% of sites. Those clients do not read robots.txt, so only rate limits and bot scoring stop them. Our AI crawler rate limiting guide shows how.

A defense plan that does not hurt AI search

Blocking aggressive crawlers is easy. Blocking them without also blocking the bots that send you visitors takes a little care. Work through these steps in order.

1. Measure. Count requests and bandwidth per user agent for one week. Our monitoring guide has the commands.
2. Sort bots into three groups. Search and assistant bots you want (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot), training bots you have an opinion on, and bots you do not want at all.
3. Write robots.txt groups for each. Never mix the first group with the others.
4. Add server rules for the "do not want" group. Bytespider usually belongs here.
5. Add rate limits for everything else. A limit protects you from unknown scrapers without naming them.
6. Test. Run the AI crawler diagnostic to confirm the bots you want still get a 200.

One warning for Cloudflare users: from 15 September 2026, new Cloudflare domains block Training and Agent traffic by default on pages that show ads, and Cloudflare says a "block Training" choice can also block multi-purpose crawlers such as Googlebot and Bingbot (Cloudflare, 1 July 2026). Check that your Bytespider block did not take Googlebot with it.

Bytespider vs TikTok's other agents

ByteDance runs more than one agent, and they are easy to confuse. Bytespider is the general crawler. TikTokSpider is a separate token that public registries link to TikTok search and link handling. Cloudflare's 2025 data listed TikTokSpider at 0.7% of AI-only crawler traffic, a new entry that year. In our study, 86 files name TikTokSpider in their own group.

Blocking Bytespider does not block TikTokSpider, and the reverse is also true, because each token matches only its own group. If you want neither, list both. If you want TikTok users to see rich previews of your links, test any rule against a shared link before you deploy it.

What an aggressive crawler actually costs you

It helps to put a number on the problem before you decide how hard to fight it. You can estimate the cost of any crawler from your own logs with three figures: requests per day, average response size, and how expensive each request is for your server.

Here is a worked example with round numbers. Suppose a crawler makes 50,000 requests a day and your average page is 120 KB. That is about 6 GB of transfer a day, or roughly 180 GB a month. On a host that charges for bandwidth, that is a visible line on the bill. On a site that renders pages from a database for every request, the CPU cost can matter more than the bandwidth, because each fetch runs the same queries as a human visit.

grep -i "Bytespider" /var/log/nginx/access.log \
  | awk '{bytes+=$10; n++} END {printf "%d requests, %.1f MB\n", n, bytes/1048576}'

Run the same line for GPTBot, ClaudeBot and Meta-ExternalAgent, then compare. Multiply by the number of days your log covers to get a monthly figure. If one crawler is a small share of the total, a block is a policy choice, not a cost saving. If it dominates, a block or a rate limit will show up on your bill.

Remember that crawlers waste requests too. Vercel and MERJ found that OpenAI's and Anthropic's crawlers spent about 34% of their fetches on 404 pages in their sample (Vercel and MERJ). Fixing broken internal links and old asset URLs cuts cost for every crawler at once, the good ones included.

Should you block Bytespider? It depends on your site

Whichever you choose, write the reason as a comment in robots.txt. A year from now, someone on your team will want to know why the line is there.

Bytespider checklist

• □You measured Bytespider's share of your bot traffic before acting.
• □Bytespider has its own robots.txt group or sits in a training-only group.
• □A server or CDN rule returns 403 for the Bytespider user agent.
• □Your search and assistant bots are still allowed and return 200.
• □Your CDN's training block did not also block Googlebot or Bingbot.
• □You set rate limits for unnamed scrapers.
• □You re-ran the AI crawler checker after the change.

Sources and further reading

Every claim above links to the page it came from. These are the primary sources, all read or re-read on 5 October 2026. Where we give a figure from our own data, it comes from our robots.txt study of the top 10,000 sites, and the raw counts are in the downloadable CSV.

1. Cloudflare: Declaring your AIndependence (3 July 2024)
2. Cloudflare: The crawl-to-click gap (29 August 2025)
3. Cloudflare: Content Independence Day, new AI options (1 July 2026)
4. Hostinger: 66 billion bot requests analysis (20 January 2026)
5. Vercel and MERJ: The rise of the AI crawler (December 2024)
6. RFC 9309, Robots Exclusion Protocol (IETF, September 2022)
7. Tranco: a research-oriented top sites ranking