Data Scrapers Directory

Common mistakes with data scrapers

The errors we see most often with data scrapers in the files and firewalls we test:

1. Treating data scrapers as an AI or search crawler. Blocking it has little effect on visibility; decide based on load and what it does with your data.
2. Relying on robots.txt alone when data scrapers is rated caution or aggressive. Back the rule with a server-side block.
3. Blocking data scrapers by IP alone. Ranges change; match the token and verify with logs.

All three hide inside a file that looks correct, so check data scrapers from outside after every change.

Spotting data scrapers in your logs

robots.txt says what Bytespider and TikTokSpider and Amazon Kendra and ApifyBot and Barkrowler and BLEXBot may do; your access logs show what they actually did.

Count requests

On a server with standard Nginx or Apache logs, this counts matching requests per day (adjust the log path for your host):

grep -iE "Bytespider|TikTokSpider|Amazon Kendra|ApifyBot|Barkrowler|BLEXBot" /var/log/nginx/access.log \
  | awk '{print $4}' | cut -d: -f1 | sort | uniq -c

And this shows which paths it requested most, which is the quickest way to see whether your Disallow lines are being respected:

grep -iE "Bytespider|TikTokSpider|Amazon Kendra|ApifyBot|Barkrowler|BLEXBot" /var/log/nginx/access.log \
  | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

Check the requests are genuine

Anyone can send these user-agents, so confirm the source before acting on a match:

• Bytespider: we hold no published IP list, so use reverse DNS where data scrapers documents a hostname and treat an unverifiable Bytespider request with caution. Check it separately from TikTokSpider and Amazon Kendra and ApifyBot and Barkrowler and BLEXBot.
• TikTokSpider: we hold no published IP list, so use reverse DNS where data scrapers documents a hostname and treat an unverifiable TikTokSpider request with caution. Check it separately from Bytespider and Amazon Kendra and ApifyBot and Barkrowler and BLEXBot.
• Amazon Kendra: we hold no published IP list, so use reverse DNS where data scrapers documents a hostname and treat an unverifiable Amazon Kendra request with caution. Check it separately from Bytespider and TikTokSpider and ApifyBot and Barkrowler and BLEXBot.
• ApifyBot: we hold no published IP list, so use reverse DNS where data scrapers documents a hostname and treat an unverifiable ApifyBot request with caution. Check it separately from Bytespider and TikTokSpider and Amazon Kendra and Barkrowler and BLEXBot.
• Barkrowler: we hold no published IP list, so use reverse DNS where data scrapers documents a hostname and treat an unverifiable Barkrowler request with caution. Check it separately from Bytespider and TikTokSpider and Amazon Kendra and ApifyBot and BLEXBot.
• BLEXBot: we hold no published IP list, so use reverse DNS where data scrapers documents a hostname and treat an unverifiable BLEXBot request with caution. Check it separately from Bytespider and TikTokSpider and Amazon Kendra and ApifyBot and Barkrowler.

Related: how AI engines choose their sources

Recent changes among data scrapers

The newest data scrapers in our database: TikTokSpider (September 22, 2026), Panscient (September 22, 2026), Brightbot (September 22, 2026), ApifyBot (September 22, 2026), yacybot (March 17, 2026), yacy (March 17, 2026). A User-agent: * default decides what each of these gets until you name it.

Movement among the tracked data scrapers: 2026 and October 20, 2026, files naming Bytespider went from 706 to 758 (+52) and files blocking it from 828 to 887 (+59), so more sites wrote a rule for Bytespider and more of those rules block it. 2026 and October 20, 2026, files naming CCBot went from 783 to 845 (+62) and files blocking it from 834 to 899 (+65), so more sites wrote a rule for CCBot and more of those rules block it.

How top sites treat data scrapers

From our robots.txt study of the Tranco top 10,000 (5,583 valid files, fetched October 20, 2026):

• CCBot, the most blocked of the data scrapers tokens: blocked in 16.1% of files (680 with its own group, 219 through User-agent: *), more often by the top 1,000 (22.4%) than the long tail (15%).
• Bytespider, second of the data scrapers tokens: blocked in 15.9% of files (664 with its own group, 223 through User-agent: *), more often by the top 1,000 (23.8%) than the long tail (14.9%).

For scale, 20.1% of all files block at least one AI training crawler. CCBot is one of the tokens driving that figure.

More questions about data scrapers

How many data scrapers should I name in robots.txt?

Only the ones that need a different answer from your default. Every crawler you do not name follows User-agent: *, so a short file with a sensible default is easier to maintain than a long list.

Do new data scrapers appear often?

Yes. New crawlers launch every month, which is why a deliberate User-agent: * default matters: it decides what happens to crawlers you have never heard of.

Can I copy another site's rules for data scrapers?

Use them as a pattern, not a template. Each file reflects that site's licensing and infrastructure. Copy the structure, then name the crawlers that matter to you.

Where do I check how my site treats data scrapers?

Run the GPTBot and ClaudeBot access check. It reports every crawler in this category in one pass, including firewall refusals that robots.txt never shows.