Anthropic

Common mistakes with Anthropic's crawlers

The errors we see most often with Anthropic's crawlers in the files and firewalls we test:

1. Assuming that blocking Anthropic's crawlers removes you from AI answers. It is a training crawler; AI answers are fed by retrieval and user-triggered crawlers with different tokens.
2. Blocking Anthropic's crawlers with a broad User-agent: * rule that also catches search engines. Name anthropic-ai in its own group instead.
3. Expecting a Anthropic's crawlers block to remove content already collected. robots.txt only affects future visits.

All three hide inside a file that looks correct, so check Anthropic's crawlers from outside after every change.

Spotting Anthropic in your logs

robots.txt says what Anthropic AI (Deprecated) and Claude Code Agent and Claude Web (Deprecated) and Claude-SearchBot and Claude-User and ClaudeBot may do; your access logs show what they actually did.

Count requests

On a server with standard Nginx or Apache logs, this counts matching requests per day (adjust the log path for your host):

grep -iE "anthropic-ai|Claude-Code|Claude-Web|Claude-SearchBot|Claude-User|ClaudeBot" /var/log/nginx/access.log \
  | awk '{print $4}' | cut -d: -f1 | sort | uniq -c

And this shows which paths it requested most, which is the quickest way to see whether your Disallow lines are being respected:

grep -iE "anthropic-ai|Claude-Code|Claude-Web|Claude-SearchBot|Claude-User|ClaudeBot" /var/log/nginx/access.log \
  | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

Check the requests are genuine

Anyone can send these user-agents, so confirm the source before acting on a match:

• anthropic-ai: we hold no published IP list, so use reverse DNS where Anthropic documents a hostname and treat an unverifiable Anthropic AI (Deprecated) request with caution. Check it separately from Claude Code Agent and Claude Web (Deprecated) and Claude-SearchBot and Claude-User and ClaudeBot.
• Claude-Code: we hold no published IP list, so use reverse DNS where Anthropic documents a hostname and treat an unverifiable Claude Code Agent request with caution. Check it separately from Anthropic AI (Deprecated) and Claude Web (Deprecated) and Claude-SearchBot and Claude-User and ClaudeBot.
• Claude-Web: we hold no published IP list, so use reverse DNS where Anthropic documents a hostname and treat an unverifiable Claude Web (Deprecated) request with caution. Check it separately from Anthropic AI (Deprecated) and Claude Code Agent and Claude-SearchBot and Claude-User and ClaudeBot.
• Claude-SearchBot: compare each address with the ranges at claude.com/crawling/bots.json. A request claiming to be Claude-SearchBot from outside them is not genuine. Check it separately from Anthropic AI (Deprecated) and Claude Code Agent and Claude Web (Deprecated) and Claude-User and ClaudeBot.
• Claude-User: compare each address with the ranges at claude.com/crawling/bots.json. A request claiming to be Claude-User from outside them is not genuine. Check it separately from Anthropic AI (Deprecated) and Claude Code Agent and Claude Web (Deprecated) and Claude-SearchBot and ClaudeBot.
• ClaudeBot: compare each address with the ranges at claude.com/crawling/bots.json. A request claiming to be ClaudeBot from outside them is not genuine. Check it separately from Anthropic AI (Deprecated) and Claude Code Agent and Claude Web (Deprecated) and Claude-SearchBot and Claude-User.

What our own data shows about Anthropic

Everything below comes from the AI Crawler Check bot database and our two most recent robots.txt scans of the Tranco top 10,000, not from Anthropic's marketing. It covers 6 Anthropic tokens in turn.

anthropic-ai in our records

We first recorded anthropic-ai on March 17, 2026. We could not find a page where Anthropic names it, so treat the token as reported rather than documented. Between our scans of October 17, 2026 and October 20, 2026, files naming anthropic-ai went from 543 to 593 (+50) and files blocking it from 639 to 696 (+57), so more sites wrote a rule for anthropic-ai and more of those rules block it.

Among the 104 AI and LLM crawlers in our database, anthropic-ai sits 23th by priority (Tier 2), level with AddSearchBot, AI2Bot, AI2Bot-DeepResearchEval, aiHitBot. 81 of the 104 share its safe rating. Other operators with a crawler that collects pages to train AI models: Google-Extended (Google, Tier 1), GPTBot (OpenAI, Tier 1), Meta-ExternalAgent (Meta, Tier 1), AI2Bot (Allen Institute for AI, Tier 2), aiHitBot (aiHit, Tier 2). A policy for anthropic-ai is usually a policy for these too.

A decision record for anthropic-ai

Teams that review crawlers once a quarter keep one short entry per token, so anyone can see why a rule exists and when it was last checked. Filled in from our data, the anthropic-ai entry reads:

• Operator and job: Anthropic; it collects pages to train AI models.
• Priority and behaviour: Tier 2, rated safe, blocking impact low.
• Suggested starting point: Your licensing call: Disallow to refuse training, Allow to permit it.
• Enforcement: anthropic-ai's operator does not clearly document how it treats robots.txt. Write the anthropic-ai rule anyway, then confirm in your logs that its requests stop.

User-agent: anthropic-ai
Allow: /

Paste the line into the validator with one of your own URLs before it goes live. Revisit the anthropic-ai entry whenever Anthropic announces a change, because a Tier 2 crawler affects visibility directly.

Claude-Code in our records

We first recorded Claude-Code on March 17, 2026. We could not find a page where Anthropic names it, so treat the token as reported rather than documented. Claude-Code is outside our tracked token set, but 22 of 5,583 top-site files already name it, which is enough to show owners treat it as a real crawler.

Among the 104 AI and LLM crawlers in our database, Claude-Code sits 29th by priority (Tier 2), level with AddSearchBot, AI2Bot, AI2Bot-DeepResearchEval, aiHitBot. 81 of the 104 share its safe rating. Other operators with a crawler that fetches a page when a person asks an assistant or app to open it: ChatGPT Agent (OpenAI, Tier 1), ChatGPT Operator (OpenAI, Tier 1), ChatGPT-User (OpenAI, Tier 1), ChatGPT-User/2.0 (OpenAI, Tier 1), GrokAgent (xAI, Tier 1). A policy for Claude-Code is usually a policy for these too.

Claude-Web in our records

We first recorded Claude-Web on March 17, 2026. We could not find a page where Anthropic names it, so treat the token as reported rather than documented. Claude-Web is outside our tracked token set, but 477 of 5,583 top-site files already name it, which is enough to show owners treat it as a real crawler.

Among the 104 AI and LLM crawlers in our database, Claude-Web sits 30th by priority (Tier 2), level with AddSearchBot, AI2Bot, AI2Bot-DeepResearchEval, aiHitBot. 81 of the 104 share its safe rating. Other operators with a crawler that fetches a page when a person asks an assistant or app to open it: ChatGPT Agent (OpenAI, Tier 1), ChatGPT Operator (OpenAI, Tier 1), ChatGPT-User (OpenAI, Tier 1), ChatGPT-User/2.0 (OpenAI, Tier 1), GrokAgent (xAI, Tier 1). A policy for Claude-Web is usually a policy for these too.

Claude-SearchBot in our records

We first recorded Claude-SearchBot on March 17, 2026. Anthropic documents it on support.claude.com, and we recheck that page before each release. Between our scans of October 17, 2026 and October 20, 2026, files naming Claude-SearchBot went from 341 to 359 (+18) and files blocking it from 404 to 424 (+20), so more sites wrote a rule for Claude-SearchBot and more of those rules block it.

Among the 104 AI and LLM crawlers in our database, Claude-SearchBot sits fifth by priority (Tier 1), level with ChatGPT Agent, ChatGPT Operator, ChatGPT-User, ChatGPT-User/2.0. 81 of the 104 share its safe rating. Other operators with a crawler that fetches a page when a person asks an assistant or app to open it: ChatGPT Agent (OpenAI, Tier 1), ChatGPT Operator (OpenAI, Tier 1), ChatGPT-User (OpenAI, Tier 1), ChatGPT-User/2.0 (OpenAI, Tier 1), GrokAgent (xAI, Tier 1). A policy for Claude-SearchBot is usually a policy for these too.

Claude-User in our records

We first recorded Claude-User on March 17, 2026. Anthropic documents it on support.claude.com, and we recheck that page before each release. Between our scans of October 17, 2026 and October 20, 2026, files naming Claude-User went from 339 to 357 (+18) and files blocking it from 408 to 430 (+22), so more sites wrote a rule for Claude-User and more of those rules block it.

Among the 104 AI and LLM crawlers in our database, Claude-User sits sixth by priority (Tier 1), level with ChatGPT Agent, ChatGPT Operator, ChatGPT-User, ChatGPT-User/2.0. 81 of the 104 share its safe rating. Other operators with a crawler that fetches a page when a person asks an assistant or app to open it: ChatGPT Agent (OpenAI, Tier 1), ChatGPT Operator (OpenAI, Tier 1), ChatGPT-User (OpenAI, Tier 1), ChatGPT-User/2.0 (OpenAI, Tier 1), GrokAgent (xAI, Tier 1). A policy for Claude-User is usually a policy for these too.

ClaudeBot in our records

We first recorded ClaudeBot on March 17, 2026. Anthropic documents it on support.claude.com, and we recheck that page before each release. Between our scans of October 17, 2026 and October 20, 2026, files naming ClaudeBot went from 800 to 858 (+58) and files blocking it from 745 to 805 (+60), so more sites wrote a rule for ClaudeBot and more of those rules block it.

Among the 104 AI and LLM crawlers in our database, ClaudeBot sits seventh by priority (Tier 1), level with ChatGPT Agent, ChatGPT Operator, ChatGPT-User, ChatGPT-User/2.0. 81 of the 104 share its safe rating. Other operators with a crawler that collects pages to train AI models: Google-Extended (Google, Tier 1), GPTBot (OpenAI, Tier 1), Meta-ExternalAgent (Meta, Tier 1), AI2Bot (Allen Institute for AI, Tier 2), aiHitBot (aiHit, Tier 2). A policy for ClaudeBot is usually a policy for these too.

How top sites treat Anthropic's crawlers

From our robots.txt study of the Tranco top 10,000 (5,583 valid files, fetched October 20, 2026):

• ClaudeBot, the most blocked of the Anthropic's crawlers tokens: blocked in 14.4% of files (594 with its own group, 211 through User-agent: *), more often by the top 1,000 (20.4%) than the long tail (13.3%).
• anthropic-ai, second of the Anthropic's crawlers tokens: blocked in 12.5% of files (469 with its own group, 227 through User-agent: *), more often by the top 1,000 (15.6%) than the long tail (11.6%).
• Claude-User, third of the Anthropic's crawlers tokens: blocked in 7.7% of files (214 with its own group, 216 through User-agent: *), more often by the top 1,000 (14%) than the long tail (6.3%).
• Claude-SearchBot, fourth of the Anthropic's crawlers tokens: blocked in 7.6% of files (207 with its own group, 217 through User-agent: *), more often by the top 1,000 (13.6%) than the long tail (6.2%).

For scale, 20.1% of all files block at least one AI training crawler. ClaudeBot is one of the tokens driving that figure.

More questions about Anthropic

Will blocking Anthropic affect my Google rankings?

Only if one of its tokens is a Google search crawler. None of Anthropic's tokens are Googlebot, so classic Google rankings are unaffected.

How quickly does a robots.txt change take effect for Anthropic?

Crawlers cache robots.txt, commonly for up to a day. Expect a change to apply within about 24 hours, and confirm it in your logs.

Can I allow Anthropic on some sections and block it on others?

Yes. Inside the group for anthropic-ai, add Disallow lines for the paths to keep private and Allow lines for exceptions. The longest matching rule wins.

Does blocking Anthropic delete content it already collected?

No. robots.txt only controls future visits. For content already collected, check whether Anthropic offers a removal or opt-out process.

Is a firewall block better than robots.txt for Anthropic?

A firewall enforces the decision, while robots.txt asks politely. Many operators recommend robots.txt because blocking their IPs can stop them reading your rules at all. Use robots.txt first, and add a firewall rule only if requests continue.