Safe Official docs AI & LLM Bots

ClaudeBot

Q: What is the official user-agent string for ClaudeBot?

The official user-agent string for ClaudeBot is: ClaudeBot . This is the exact string you must use in robots.txt, Nginx, Apache, or Cloudflare firewall rules to target this bot. User-agent matching in robots.txt is case-insensitive, but the string must be spelled correctly. You can verify that a request genuinely comes from ClaudeBot by performing a reverse-DNS lookup on the source IP - legitimate bots resolve back to their operator's domain.

Q: Is ClaudeBot safe?

Yes. ClaudeBot is a documented bot: Anthropic publishes a page that names it and explains what it does.

Q: How do I block ClaudeBot in robots.txt?

Add the following lines to your /robots.txt file: User-agent: ClaudeBot Disallow: / This instructs ClaudeBot not to crawl any path on your site. The Disallow: / directive covers the entire domain including subfolders. To only block specific sections, replace / with the path (e.g., Disallow: /blog/ ). Note: robots.txt is publicly readable - any bot or human can inspect it at yourdomain.com/robots.txt .

Q: How do I verify if ClaudeBot is crawling my site?

Search your web server access logs for the string ClaudeBot (case-insensitive grep: grep -i "ClaudeBot" /var/log/nginx/access.log ). You can also check Google Search Console → Coverage → Crawl Stats for Googlebot variants. For ClaudeBot specifically, filter by user-agent in your log analysis tool (GoAccess, AWStats, etc.).

Operated by Anthropic

Scout's quick take

ClaudeBot collects pages for Anthropic's AI training. Blocking it keeps your content out of future training runs and does not change your Google or Bing rankings.

User-agent: ClaudeBot
Disallow: /

Test my robots.txt against AI bots Build a full robots.txt

Quick Facts

User-Agent:: ClaudeBot
Category:: AI & LLM Bots
Operator:: Anthropic
Safety:: Safe
Blocking Impact:: Low - No SEO ranking impact
SEO Impact Score:: 2/10
Official docs:: support.claude.com

Confirmed by the operator

Anthropic names ClaudeBot on its own documentation page, linked above. The token, purpose and robots.txt behaviour on this profile come from that page, re-checked on October 19, 2026.

What is ClaudeBot?

ClaudeBot is Anthropic's training crawler. Anthropic says it collects "web content that could potentially contribute to" model training.

Anthropic: restricting ClaudeBot "signals that the site's future materials should be excluded from our AI model training datasets". It identifies itself with the token ClaudeBot. Anthropic documents it at support.claude.com.

What happens if you block ClaudeBot?

No effect on Google, Bing or Claude search (Claude-SearchBot is separate).

Generally safe to allow; provides legitimate crawling value.

Want to see whether ClaudeBot actually visits you? Read how do I monitor AI bot traffic to my website?

How to block ClaudeBot with robots.txt

<code>User-agent: ClaudeBot</code> - Matching is case-insensitive. Robots.txt is fetched from the root of each subdomain separately.

Block completely (robots.txt)

User-agent: ClaudeBot
Disallow: /

Allow all (robots.txt)

User-agent: ClaudeBot
Allow: /

Block private only (robots.txt)

User-agent: ClaudeBot
Disallow: /private/
Disallow: /api/
Disallow: /admin/
Allow: /

Nginx server block

# Nginx: Hard-block ClaudeBot
if ($http_user_agent ~* "ClaudeBot") {
    return 403 "Bot blocked";
}

Apache .htaccess

# Apache: Hard-block ClaudeBot
SetEnvIfNoCase User-Agent "ClaudeBot" bad_bot
Order Allow,Deny
Allow from all
Deny from env=bad_bot

Meta robots tag

X-Robots-Tag header

X-Robots-Tag: noindex, nofollow

Is ClaudeBot safe to allow?

Yes. ClaudeBot is a documented bot: Anthropic publishes a page that names it and explains what it does.

Verify by reverse-DNS lookup: legitimate ClaudeBot requests resolve to anthropic's domain.

What does ClaudeBot do?

From Anthropic's own documentation.

ClaudeBot is Anthropic's training crawler.
Content may be used to train AI models
No effect on Google, Bing or Claude search (Claude-SearchBot is separate).

ClaudeBot in context

ClaudeBot is a training crawler operated by Anthropic. In plain terms, it collects pages to train AI models. Our database files it under AI and LLM crawlers and ranks it Tier 1, the small set whose absence you would notice in AI answers or search results. 14 of the 104 AI and LLM crawlers in our database share this tier.

Its safety rating is safe and its blocking impact is low. So the decision about ClaudeBot rests on what you want from Anthropic, not on its behaviour. How we rate safety and impact.

How site owners treat ClaudeBot

In our own study of robots.txt files across the Tranco top 10,000 sites (fetched October 20, 2026, 5,583 valid files), 858 files (15.4%) name ClaudeBot in a User-agent line, and 805 (14.4%) block it from the homepage. 594 of those blocks come from a group written for this crawler by name, and 211 come from a broad User-agent: * rule that catches it by accident. Larger sites are stricter: 20.4% of the top 1,000 block it, against 14.6% of the middle band and 13.3% of the tail.

Does it follow robots.txt?

ClaudeBot's operator says it follows robots.txt, so a correctly written ClaudeBot group is enough to control it.

Anthropic crawlers and close alternatives

ClaudeBot is one of 6 Anthropic crawlers in our database. A User-agent: ClaudeBot group controls ClaudeBot only; the other Anthropic tokens below need their own lines.

ClaudeBot: this crawler (Tier 1)
anthropic-ai: Anthropic AI (Deprecated), collects pages to train AI models (Tier 2)
Claude-Code: Claude Code Agent, fetches a page when a person asks an assistant or app to open it (Tier 2)
Claude-SearchBot: Claude Search, fetches a page when a person asks an assistant or app to open it (Tier 1)
Claude-User: Claude User Agent, fetches a page when a person asks an assistant or app to open it (Tier 1)
Claude-Web: Claude Web (Deprecated), fetches a page when a person asks an assistant or app to open it (Tier 2)

To give ClaudeBot the same answer as its siblings, stack User-agent: ClaudeBot with their lines above one set of rules; to treat ClaudeBot differently, give it its own group.

Other AI and LLM crawlers that do the same job (collects pages to train AI models) include Google-Extended from Google, GPTBot from OpenAI, Meta-ExternalAgent from Meta, AI2Bot from Allen Institute for AI, aiHitBot from aiHit, bedrockbot from Amazon. Deciding about ClaudeBot is usually a decision about the whole group, so check how your file treats these too.

We first recorded ClaudeBot on March 17, 2026. Anthropic documents it on support.claude.com, and we recheck that page before each release.

Which rule ClaudeBot actually follows

ClaudeBot reads exactly one group in robots.txt, the most specific one that names ClaudeBot, and never combines rules across groups. This short file shows the trap:

User-agent: *
Disallow: /private/

User-agent: ClaudeBot
Disallow: /drafts/

Find the group. ClaudeBot looks for a group naming ClaudeBot. One exists, so it ignores User-agent: * completely.
Apply only that group. It is refused /drafts/ and nothing else.
The surprise. /private/ is open to ClaudeBot, because that rule lives in the * group it no longer reads. To keep it out of both, repeat Disallow: /private/ inside its own group.

Test yourself: does a rule under User-agent: * apply to ClaudeBot if the file also has a User-agent: ClaudeBot group?

No. Once a group names it, ClaudeBot reads only that group. Put every rule you want it to follow inside its own group.

Test yourself: will blocking ClaudeBot remove your pages from AI answers?

No. ClaudeBot collects training data. Blocking it changes what future models learn, not which pages AI answers cite today.

Test yourself: is User-agent: claudebot the same as User-agent: ClaudeBot?

Yes. RFC 9309 makes user-agent matching case-insensitive. Paths are a different story: Disallow: /Blog/ does not block /blog/.

Should you allow ClaudeBot? Decide by situation

Recommended setting for ClaudeBot by type of site
If you are...	Recommended setting
You publish content and want AI answers to cite it	Your call. ClaudeBot is a training crawler, so blocking it does not remove you from AI answers; retrieval crawlers do that job.
You license content and want to stop AI training	Block it by name with `User-agent: ClaudeBot` and `Disallow: /`.
You run an online shop	Allow it unless your logs show it is costly.
Your server is under load from bots	Rate-limit before blocking. It is rated safe in our database.

Check ClaudeBot on your own site

Open /robots.txt on your domain and search for ClaudeBot. If it is not named, it follows User-agent: *.
List the Disallow lines in the group ClaudeBot follows and test your key URLs against them; the longest match wins, and Allow wins a tie.
Look beyond the file. A firewall or bot manager can refuse ClaudeBot even when robots.txt allows it, and that never shows in the file.
Run our AI search crawlability check to see how ClaudeBot and every other crawler we track are treated in one pass, including firewall refusals.

Common mistakes with ClaudeBot

The errors we see most often with ClaudeBot in the files and firewalls we test:

Assuming that blocking ClaudeBot removes you from AI answers. It is a training crawler; AI answers are fed by retrieval and user-triggered crawlers with different tokens.
Blocking ClaudeBot with a broad User-agent: * rule that also catches search engines. Name ClaudeBot in its own group instead.
Expecting a ClaudeBot block to remove content already collected. robots.txt only affects future visits.

All three hide inside a file that looks correct, so check ClaudeBot from outside after every change.

Spotting ClaudeBot in your logs

robots.txt says what ClaudeBot may do; your access logs show what it actually did.

Count requests

On a server with standard Nginx or Apache logs, this counts matching requests per day (adjust the log path for your host):

grep -iE "ClaudeBot" /var/log/nginx/access.log \
  | awk '{print $4}' | cut -d: -f1 | sort | uniq -c

And this shows which paths it requested most, which is the quickest way to see whether your Disallow lines are being respected:

grep -iE "ClaudeBot" /var/log/nginx/access.log \
  | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

Check the requests are genuine

Anyone can send the ClaudeBot user-agent, so confirm the source before acting on a match:

ClaudeBot: compare each address with the ranges at claude.com/crawling/bots.json. A request claiming to be ClaudeBot from outside them is not genuine.

Frequently Asked Questions

What is the official user-agent string for ClaudeBot?

The official user-agent string for ClaudeBot is: ClaudeBot. This is the exact string you must use in robots.txt, Nginx, Apache, or Cloudflare firewall rules to target this bot. User-agent matching in robots.txt is case-insensitive, but the string must be spelled correctly. You can verify that a request genuinely comes from ClaudeBot by performing a reverse-DNS lookup on the source IP - legitimate bots resolve back to their operator's domain.

Is ClaudeBot safe?

Yes. ClaudeBot is a documented bot: Anthropic publishes a page that names it and explains what it does.

Will blocking ClaudeBot hurt my SEO?

No effect on Google, Bing or Claude search (Claude-SearchBot is separate).

How do I block ClaudeBot in robots.txt?

Add the following lines to your /robots.txt file:

User-agent: ClaudeBot
Disallow: /

This instructs ClaudeBot not to crawl any path on your site. The Disallow: / directive covers the entire domain including subfolders. To only block specific sections, replace / with the path (e.g., Disallow: /blog/). Note: robots.txt is publicly readable - any bot or human can inspect it at yourdomain.com/robots.txt.

Does ClaudeBot respect robots.txt?

Yes, and it also honours the non-standard Crawl-delay directive, according to Anthropic.

How do I verify if ClaudeBot is crawling my site?

Search your web server access logs for the string ClaudeBot (case-insensitive grep: grep -i "ClaudeBot" /var/log/nginx/access.log). You can also check Google Search Console → Coverage → Crawl Stats for Googlebot variants. For ClaudeBot specifically, filter by user-agent in your log analysis tool (GoAccess, AWStats, etc.).

What is the crawl frequency of ClaudeBot?

ClaudeBot crawls at a moderate rate. If you notice excessive traffic in your logs, you can add a Crawl-delay directive:

User-agent: ClaudeBot
Crawl-delay: 10

(10 second delay between requests).

Can I block ClaudeBot from specific pages only?

Yes. Instead of a global Disallow: / you can restrict ClaudeBot to specific paths:

User-agent: ClaudeBot
Disallow: /private/
Disallow: /staging/
Allow: /

This allows ClaudeBot everywhere except the listed paths. Path matching in robots.txt uses prefix matching - Disallow: /private/ blocks /private/page.html but NOT /public/private/.

Does blocking ClaudeBot prevent AI training on my content?

A robots.txt Disallow for ClaudeBot is the opt-out Anthropic documents. Robots.txt is voluntary, so for enforcement add a firewall or CDN rule. Anthropic: restricting ClaudeBot "signals that the site's future materials should be excluded from our AI model training datasets".

Is there an alternative to robots.txt to opt out of ClaudeBot?

The opt-out Anthropic documents is robots.txt. A firewall, CDN or server rule that returns 403 for ClaudeBot enforces it. Per-bot meta tags and llms.txt are not opt-out mechanisms any major operator documents.