Data Scrapers Directory

Data aggregation and research crawlers. No SEO impact from blocking.

39 tokens
29 safe
6 caution
4 aggressive

Common providers

Pick a provider to see only its crawlers. Each provider also has a full profile under Bot Operators.

All 39 Data Scrapers

FAQ

How many Data Scrapers are tracked?
We currently track 39 Data Scrapers with user-agent strings, safety ratings, and blocking rules.
Which Data Scrapers should I never block?
Never block bots with 'critical' impact rating - there are 0 in this category. Blocking them removes your site from their search results.
Are all Data Scrapers safe?
Out of 39 bots in this category: 29 safe, 6 caution, 4 aggressive.
How do I block all Data Scrapers at once?
Generate a combined robots.txt block rule for all Data Scrapers using our bulk robots.txt generator, or manually list each User-agent + Disallow: / directive.