Diffbot

Common mistakes with Diffbot's crawlers

The errors we see most often with Diffbot's crawlers in the files and firewalls we test:

1. Assuming that blocking Diffbot's crawlers removes you from AI answers. It is a training crawler; AI answers are fed by retrieval and user-triggered crawlers with different tokens.
2. Blocking Diffbot's crawlers with a broad User-agent: * rule that also catches search engines. Name Diffbot in its own group instead.
3. Expecting a Diffbot's crawlers block to remove content already collected. robots.txt only affects future visits.

All three hide inside a file that looks correct, so check Diffbot's crawlers from outside after every change.

Spotting Diffbot in your logs

robots.txt says what Diffbot and Diffbot Fetcher may do; your access logs show what they actually did.

Count requests

On a server with standard Nginx or Apache logs, this counts matching requests per day (adjust the log path for your host):

grep -iE "Diffbot|Diffbot-User" /var/log/nginx/access.log \
  | awk '{print $4}' | cut -d: -f1 | sort | uniq -c

And this shows which paths it requested most, which is the quickest way to see whether your Disallow lines are being respected:

grep -iE "Diffbot|Diffbot-User" /var/log/nginx/access.log \
  | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

Check the requests are genuine

Anyone can send these user-agents, so confirm the source before acting on a match:

• Diffbot: we hold no published IP list, so use reverse DNS where the operator documents a hostname and treat an unverifiable Diffbot request with caution. Check it separately from Diffbot Fetcher.
• Diffbot-User: we hold no published IP list, so use reverse DNS where Diffbot documents a hostname and treat an unverifiable Diffbot Fetcher request with caution. Check it separately from Diffbot.

What our own data shows about Diffbot

Everything below comes from the AI Crawler Check bot database and our two most recent robots.txt scans of the Tranco top 10,000, not from Diffbot's marketing. It covers 2 Diffbot tokens in turn.

Diffbot in our records

We first recorded Diffbot on March 17, 2026. Diffbot documents it on docs.diffbot.com, and we recheck that page before each release. No file in our top-10,000 scans names Diffbot, so any rule you write for it puts you ahead of almost every large site.

Among the 39 data scrapers in our database, Diffbot sits 14th by priority (Tier 6), level with Amazon Kendra, ApifyBot, Barkrowler, BLEXBot. 29 of the 39 share its safe rating. Other operators with a crawler that harvests pages for a dataset or data product: FirecrawlAgent (Firecrawl, Tier 2), iaskspider (iAsk, Tier 2), NovaAct (Amazon, Tier 2), CloudflareBrowserRenderingCrawler (Cloudflare, Tier 3), Amazon Kendra (Amazon, Tier 6). A policy for Diffbot is usually a policy for these too.

A decision record for Diffbot

Teams that review crawlers once a quarter keep one short entry per token, so anyone can see why a rule exists and when it was last checked. Filled in from our data, the Diffbot entry reads:

• Operator and job: Diffbot; it harvests pages for a dataset or data product.
• Priority and behaviour: Tier 6, rated safe, blocking impact low.
• Suggested starting point: Your licensing call: Disallow to refuse training, Allow to permit it.
• Enforcement: Diffbot's robots.txt behaviour is documented with exceptions or has been publicly questioned, so pair a Diffbot rule in robots.txt with a server-side rule if blocking matters to you.

User-agent: Diffbot
Allow: /

Paste the line into the validator with one of your own URLs before it goes live. Revisit the Diffbot entry once a quarter, or when Diffbot publishes a new token next to it.

Diffbot-User in our records

We first recorded Diffbot-User on September 22, 2026. Diffbot documents it on docs.diffbot.com, and we recheck that page before each release. No file in our top-10,000 scans names Diffbot-User, so any rule you write for it puts you ahead of almost every large site.

Among the 104 AI and LLM crawlers in our database, Diffbot-User sits 97th by priority (Tier 3), level with AIWebIndex-Agent, alphalens-bot, amazon-QBusiness, Anomura. 81 of the 104 share its safe rating. Other operators with a crawler that fetches a page when a person asks an assistant or app to open it: ChatGPT Agent (OpenAI, Tier 1), ChatGPT Operator (OpenAI, Tier 1), ChatGPT-User (OpenAI, Tier 1), ChatGPT-User/2.0 (OpenAI, Tier 1), Claude-SearchBot (Anthropic, Tier 1). A policy for Diffbot-User is usually a policy for these too.

A decision record for Diffbot-User

Teams that review crawlers once a quarter keep one short entry per token, so anyone can see why a rule exists and when it was last checked. Filled in from our data, the Diffbot-User entry reads:

• Operator and job: Diffbot; it fetches a page when a person asks an assistant or app to open it.
• Priority and behaviour: Tier 3, rated safe, blocking impact low.
• Suggested starting point: Allow by default; revisit if it costs you bandwidth.
• Enforcement: Diffbot-User's operator does not clearly document how it treats robots.txt. Write the Diffbot-User rule anyway, then confirm in your logs that its requests stop.

User-agent: Diffbot-User
Allow: /

Paste the line into the validator with one of your own URLs before it goes live. Revisit the Diffbot-User entry once a quarter, or when Diffbot publishes a new token next to it.

How top sites treat Diffbot's crawlers

From our robots.txt study of the Tranco top 10,000 (5,583 valid files, fetched October 20, 2026):

None of the tokens for Diffbot's crawlers is tracked by name in our study. For scale, 20.1% of top-site files block at least one AI training crawler and 4.3% refuse every unnamed crawler with User-agent: * and Disallow: /, which would catch Diffbot's crawlers too.

More questions about Diffbot

Will blocking Diffbot affect my Google rankings?

Only if one of its tokens is a Google search crawler. None of Diffbot's tokens are Googlebot, so classic Google rankings are unaffected.

How quickly does a robots.txt change take effect for Diffbot?

Crawlers cache robots.txt, commonly for up to a day. Expect a change to apply within about 24 hours, and confirm it in your logs.

Can I allow Diffbot on some sections and block it on others?

Yes. Inside the group for Diffbot, add Disallow lines for the paths to keep private and Allow lines for exceptions. The longest matching rule wins.

Does blocking Diffbot delete content it already collected?

No. robots.txt only controls future visits. For content already collected, check whether Diffbot offers a removal or opt-out process.

Is a firewall block better than robots.txt for Diffbot?

A firewall enforces the decision, while robots.txt asks politely. Many operators recommend robots.txt because blocking their IPs can stop them reading your rules at all. Use robots.txt first, and add a firewall rule only if requests continue.