Diffbot
SafeUser-agent: Diffbot
Category: Data Scrapers
SEO impact if blocked: 2/10
Diffbot
2 crawlers operated by Diffbot
Who Diffbot is, for site owners
Diffbot operates 2 crawlers in our directory, filed under Data Scrapers and AI & LLM Bots. Of those 2, 2 rated safe for crawl behaviour.
Blocking Diffbot carries an SEO impact score of 2/10 across its crawlers. Every crawler here carries the same blocking impact: Low - No SEO ranking impact.
Our recommendation for Diffbot, set by its least reassuring crawler (Diffbot, rated safe): Generally safe to allow; provides legitimate crawling value.
Diffbot at a glance
| Operator | Diffbot |
|---|---|
| Crawlers in directory | 2 |
| Categories | Data Scrapers, AI & LLM Bots |
| SEO impact if blocked | 2/10 |
| Crawl behaviour | 2 rated safe |
| Official crawler docs | Diffbot crawler documentation |
Crawlers operated by Diffbot
How to block or allow Diffbot in robots.txt
These rules cover all 2 crawlers Diffbot operates. Add them to the robots.txt file at the root of your domain.
Block Diffbot completely
User-agent: Diffbot
User-agent: Diffbot-User
Disallow: /
Allow Diffbot explicitly
User-agent: Diffbot
User-agent: Diffbot-User
Allow: /
robots.txt is a request, not an enforcement mechanism. A crawler that ignores it needs a block at the server or CDN layer. Rules are case-sensitive on the path and matched against the user-agent token, not the full user-agent string.
Frequently asked questions about Diffbot
What crawlers does Diffbot run?
Diffbot operates 2 crawlers in our directory: Diffbot (user-agent Diffbot), Diffbot Fetcher (user-agent Diffbot-User).
Should I block Diffbot?
Generally safe to allow; provides legitimate crawling value. That guidance follows this operator's least reassuring crawler, Diffbot, rated safe, because allowing an operator allows every crawler it runs. The SEO impact score for blocking is 2/10, where 10 means blocking costs you the most visibility.
How do I block Diffbot in robots.txt?
Add the following to robots.txt at your domain root: User-agent: Diffbot then User-agent: Diffbot-User then Disallow: /. One Disallow line covers all 2 user-agent tokens.
Does blocking Diffbot hurt my search rankings?
Every crawler here carries the same blocking impact: Low - No SEO ranking impact. Google's own ranking crawler is Googlebot, which this operator does not run, so blocking Diffbot does not by itself remove you from Google's index. What it can change is whether AI assistants and analytics tools can read your pages.