DuckDuckGo

Common mistakes with DuckDuckGo's crawlers

The errors we see most often with DuckDuckGo's crawlers in the files and firewalls we test:

1. Blocking DuckDuckGo's crawlers while trying to stop AI training. Retrieval crawlers fetch pages so answers can cite them; blocking it can remove you from those answers without affecting training.
2. Letting a firewall or bot-fight mode refuse DuckDuckGo's crawlers while robots.txt allows it. The file looks right, but the crawler still gets a 403 or a challenge page.
3. Forgetting that once a User-agent: DuckAssistBot group exists, DuckDuckGo's crawlers ignores every rule under User-agent: *.

All three hide inside a file that looks correct, so check DuckDuckGo's crawlers from outside after every change.

Spotting DuckDuckGo in your logs

robots.txt says what DuckAssistBot and DuckDuckBot may do; your access logs show what they actually did.

Count requests

On a server with standard Nginx or Apache logs, this counts matching requests per day (adjust the log path for your host):

grep -iE "DuckAssistBot|DuckDuckBot" /var/log/nginx/access.log \
  | awk '{print $4}' | cut -d: -f1 | sort | uniq -c

And this shows which paths it requested most, which is the quickest way to see whether your Disallow lines are being respected:

grep -iE "DuckAssistBot|DuckDuckBot" /var/log/nginx/access.log \
  | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

Check the requests are genuine

Anyone can send these user-agents, so confirm the source before acting on a match:

• DuckAssistBot: we hold no published IP list, so use reverse DNS where DuckDuckGo documents a hostname and treat an unverifiable DuckAssistBot request with caution. Check it separately from DuckDuckBot.
• DuckDuckBot: we hold no published IP list, so use reverse DNS where DuckDuckGo documents a hostname and treat an unverifiable DuckDuckBot request with caution. Check it separately from DuckAssistBot.

What our own data shows about DuckDuckGo

Everything below comes from the AI Crawler Check bot database and our two most recent robots.txt scans of the Tranco top 10,000, not from DuckDuckGo's marketing. It covers 2 DuckDuckGo tokens in turn.

DuckAssistBot in our records

We first recorded DuckAssistBot on March 17, 2026. DuckDuckGo documents it on help.duckduckgo.com, and we recheck that page before each release. Between our scans of October 17, 2026 and October 20, 2026, files naming DuckAssistBot went from 292 to 304 (+12) and files blocking it from 442 to 461 (+19), so more sites wrote a rule for DuckAssistBot and more of those rules block it.

Among the 104 AI and LLM crawlers in our database, DuckAssistBot sits 38th by priority (Tier 2), level with AddSearchBot, AI2Bot, AI2Bot-DeepResearchEval, aiHitBot. 81 of the 104 share its safe rating. Other operators with a crawler that fetches a page when a person asks an assistant or app to open it: ChatGPT Agent (OpenAI, Tier 1), ChatGPT Operator (OpenAI, Tier 1), ChatGPT-User (OpenAI, Tier 1), ChatGPT-User/2.0 (OpenAI, Tier 1), Claude-SearchBot (Anthropic, Tier 1). A policy for DuckAssistBot is usually a policy for these too.

A decision record for DuckAssistBot

Teams that review crawlers once a quarter keep one short entry per token, so anyone can see why a rule exists and when it was last checked. Filled in from our data, the DuckAssistBot entry reads:

• Operator and job: DuckDuckGo; it fetches a page when a person asks an assistant or app to open it.
• Priority and behaviour: Tier 2, rated safe, blocking impact low.
• Suggested starting point: Allow, and watch it in your logs.
• Enforcement: DuckAssistBot's operator says it follows robots.txt, so a correctly written DuckAssistBot group is enough to control it.

User-agent: DuckAssistBot
Allow: /

Paste the line into the validator with one of your own URLs before it goes live. Revisit the DuckAssistBot entry whenever DuckDuckGo announces a change, because a Tier 2 crawler affects visibility directly.

DuckDuckBot in our records

We first recorded DuckDuckBot on March 17, 2026. DuckDuckGo documents it on duckduckgo.com, and we recheck that page before each release. No file in our top-10,000 scans names DuckDuckBot, so any rule you write for it puts you ahead of almost every large site.

Among the 63 search engines in our database, DuckDuckBot sits 16th by priority (Tier 3), level with AdIdxBot, Amzn-SearchBot, Applebot, baidu. 62 of the 63 share its safe rating. Other operators with a crawler that builds a search or answer index: AddSearchBot (AddSearch, Tier 2), AIWebIndex (Lyrenth, Tier 2), atlassian-bot (Atlassian, Tier 2), AzureAI-SearchBot (Microsoft, Tier 2), Cloudflare-AI-Search (Cloudflare, Tier 2). A policy for DuckDuckBot is usually a policy for these too.

A decision record for DuckDuckBot

Teams that review crawlers once a quarter keep one short entry per token, so anyone can see why a rule exists and when it was last checked. Filled in from our data, the DuckDuckBot entry reads:

• Operator and job: DuckDuckGo; it builds a search or answer index.
• Priority and behaviour: Tier 3, rated safe, blocking impact critical.
• Suggested starting point: Allow by default; revisit if it costs you bandwidth.
• Enforcement: DuckDuckBot's operator does not clearly document how it treats robots.txt. Write the DuckDuckBot rule anyway, then confirm in your logs that its requests stop.

User-agent: DuckDuckBot
Allow: /

Paste the line into the validator with one of your own URLs before it goes live. Revisit the DuckDuckBot entry once a quarter, or when DuckDuckGo publishes a new token next to it.

How top sites treat DuckDuckGo's crawlers

From our robots.txt study of the Tranco top 10,000 (5,583 valid files, fetched October 20, 2026):

• DuckAssistBot, the only one tracked among the DuckDuckGo's crawlers tokens: blocked in 8.3% of files (236 with its own group, 225 through User-agent: *), more often by the top 1,000 (14.2%) than the long tail (7.1%).

For scale, 20.1% of all files block at least one AI training crawler. None of the tokens for DuckDuckGo's crawlers comes close to that rate, so blocking them is still a minority choice.

More questions about DuckDuckGo

Will blocking DuckDuckGo affect my Google rankings?

Only if one of its tokens is a Google search crawler. None of DuckDuckGo's tokens are Googlebot, so classic Google rankings are unaffected.

How quickly does a robots.txt change take effect for DuckDuckGo?

Crawlers cache robots.txt, commonly for up to a day. Expect a change to apply within about 24 hours, and confirm it in your logs.

Can I allow DuckDuckGo on some sections and block it on others?

Yes. Inside the group for DuckAssistBot, add Disallow lines for the paths to keep private and Allow lines for exceptions. The longest matching rule wins.

Does blocking DuckDuckGo delete content it already collected?

No. robots.txt only controls future visits. For content already collected, check whether DuckDuckGo offers a removal or opt-out process.

Is a firewall block better than robots.txt for DuckDuckGo?

A firewall enforces the decision, while robots.txt asks politely. Many operators recommend robots.txt because blocking their IPs can stop them reading your rules at all. Use robots.txt first, and add a firewall rule only if requests continue.