Claude-SearchBot (Anthropic): User Agent & robots.txt Rules

Q: What is the official user-agent string for Claude-SearchBot?

The official user-agent string for Claude-SearchBot is: Claude-SearchBot. This is the exact string you must use in robots.txt, Nginx, Apache, or Cloudflare firewall rules to target this bot. User-agent matching in robots.txt is case-insensitive, but the string must be spelled correctly. You can verify that a request genuinely comes from Claude-SearchBot by performing a reverse-DNS lookup on the source IP - legitimate bots resolve back to their operator's domain.

Q: Is Claude-SearchBot safe?

Yes, Claude-SearchBot is a safe and legitimate crawler. It is operated by Anthropic, which publicly documents its crawler at an official URL and follows the Robots Exclusion Protocol (RFC 9309). The user-agent string Claude-SearchBot is verifiable via reverse-DNS lookup on the crawling IP addresses. You can safely allow it unless you have a specific reason to block (e.g., AI training opt-out or SEO tool visibility).

Q: Will blocking Claude-SearchBot hurt my SEO?

✅ No SEO Impact - Blocking Claude-SearchBot does not affect your rankings in Google, Bing, or any other search engine. Claude-SearchBot is an AI training crawler, not a search indexer. You can freely block it via User-agent: Claude-SearchBot / Disallow: / without any SEO penalty. This is the recommended approach if you want to opt out of Anthropic's LLM training datasets.

Q: How do I block Claude-SearchBot in robots.txt?

Add the following lines to your /robots.txt file: User-agent: Claude-SearchBot Disallow: / This instructs Claude-SearchBot not to crawl any path on your site. The Disallow: / directive covers the entire domain including subfolders. To only block specific sections, replace / with the path (e.g., Disallow: /blog/). Note: robots.txt is publicly readable - any bot or human can inspect it at yourdomain.com/robots.txt.

Q: How do I verify if Claude-SearchBot is crawling my site?

Search your web server access logs for the string Claude-SearchBot (case-insensitive grep: grep -i "Claude-SearchBot" /var/log/nginx/access.log). You can also check Google Search Console → Coverage → Crawl Stats for Googlebot variants. For Claude-SearchBot specifically, filter by user-agent in your log analysis tool (GoAccess, AWStats, etc.).

Quick Facts

User-Agent:: Claude-SearchBot
Category:: AI & LLM Bots
Operator:: Anthropic
Safety:: Safe
Blocking Impact:: Low - No SEO ranking impact
SEO Impact Score:: 2/10
Official docs:: support.claude.com

Confirmed by the operator

Anthropic names Claude-SearchBot on its own documentation page, linked above. The token, purpose and robots.txt behaviour on this profile come from that page, re-checked on September 22, 2026.

What is Claude-SearchBot?

Claude-SearchBot is a crawler used to index content specifically for search-related features within the Claude ecosystem.

Claude-SearchBot is a crawler used to index content specifically for search-related features within the Claude ecosystem.
Claude-SearchBot is an AI data-collection crawler operated by Anthropic. It harvests web content to build or expand training datasets for large language models (LLMs). Unlike search crawlers, Claude-SearchBot does NOT influence your page ranking in any search engine. The user-agent string Claude-SearchBot can be safely blocked via robots.txt, meta tags (noai), or the emerging llms.txt standard without any SEO penalty. Robots.txt is voluntary; for hard enforcement, combine it with server-level IP blocking.

What happens if you block Claude-SearchBot?

✅ No SEO Impact - Blocking Claude-SearchBot does not affect your rankings in Google, Bing, or any other search engine. Claude-SearchBot is an AI training crawler, not a search indexer. You can freely block it via User-agent: Claude-SearchBot / Disallow: / without any SEO penalty. This is the recommended approach if you want to opt out of Anthropic's LLM training datasets.

Generally safe to allow; provides legitimate crawling value.

Claude-SearchBot in one paragraph

Claude-SearchBot is the Anthropic crawler that indexes pages for Claude's search features, and blocking it removes you from Claude search results. It is controlled independently of ClaudeBot, which collects training data, so you can allow one and block the other. For monitoring, see how do I monitor AI bot traffic to my website?

User-agent: Claude-SearchBot
Allow: /

User-agent: ClaudeBot
Disallow: /

Claude-SearchBot vs ClaudeBot vs Claude-User

Anthropic runs three bots and documents what blocking each one does (Anthropic crawler documentation, updated 7 April 2026). Claude-SearchBot is the one that decides whether Claude's search features can find and cite you.

Anthropic's three crawlers compared
Question	ClaudeBot	Claude-SearchBot	Claude-User
Purpose	Collects training data	Indexes pages for Claude search	Fetches a page a user asked about
Block it and...	Future training excludes your site	Claude search may not show or cite you	Claude cannot retrieve your page on request
Triggered by	Anthropic crawl schedule	Anthropic index schedule	A person in a chat
Crawl-delay	Supported	Supported	Not applicable (single fetch)
Evidence tier here	Official	Official	Official

Anthropic publishes its IP ranges at claude.com/crawling/bots.json but warns that blocking by IP may not work reliably; use robots.txt tokens.

Why Claude-SearchBot is the Anthropic bot to keep

Anthropic crawls far more than it refers. Cloudflare, 29 August 2025 measured 38,065 pages crawled for each visitor referred in July 2025, down from 286,930 in January. Most of that crawling is ClaudeBot training; the search and user bots are the part that can send people back.

Anthropic crawl-to-refer ratio, January vs July 2025.

ClaudeBot's share of all AI and search crawling also grew, from 6% to 9.9% over the year to July 2025.

Share of AI and search crawling by crawler.

Claude-SearchBot use cases

Use cases: what to do with this bot
If you are...	Recommended setting and why
You want Claude citations but not training	Allow Claude-SearchBot and Claude-User, block ClaudeBot.
You block every AI bot	Block all three by name; a wildcard block also works, but named groups are explicit.
Server load from Anthropic	Add Crawl-delay to ClaudeBot first; it is the high-volume one.

User-agent: Claude-SearchBot
User-agent: Claude-User
Allow: /

User-agent: ClaudeBot
Disallow: /

Verify with a best free AI crawler check and compare Anthropic with OpenAI's matching three tiers in ClaudeBot vs Claude-SearchBot and the Anthropic operator page.

Test Anthropic crawler access

AI crawler checker: tests 248 bots against your robots.txt in one scan.
AI crawler diagnostic: fetches a page as each AI bot and names the layer that blocks it.
robots.txt validator: shows allowed, blocked and partial status per bot.
robots.txt generator: builds a file with presets for search, agents and training.
batch URL checker: compares up to 20 sites side by side, with CSV export.
llms.txt generator: writes the file AI assistants read for a site summary.
schema markup generator: produces JSON-LD for eight common types.
GEO audit: scores a page for generative engine optimization.

How to block Claude-SearchBot with robots.txt

<code>User-agent: Claude-SearchBot</code> - Matching is case-insensitive. Robots.txt is fetched from the root of each subdomain separately.

Block completely (robots.txt)

User-agent: Claude-SearchBot
Disallow: /

Allow all (robots.txt)

User-agent: Claude-SearchBot
Allow: /

Block private only (robots.txt)

User-agent: Claude-SearchBot
Disallow: /private/
Disallow: /api/
Disallow: /admin/
Allow: /

Nginx server block

# Nginx: Hard-block Claude-SearchBot
if ($http_user_agent ~* "Claude\-SearchBot") {
    return 403 "Bot blocked";
}

Apache .htaccess

# Apache: Hard-block Claude-SearchBot
SetEnvIfNoCase User-Agent "Claude\-SearchBot" bad_bot
Order Allow,Deny
Allow from all
Deny from env=bad_bot

Meta robots tag

X-Robots-Tag header

X-Robots-Tag: noindex, nofollow

Is Claude-SearchBot safe to allow?

Yes, Claude-SearchBot is a safe and legitimate crawler. It is operated by Anthropic, which publicly documents its crawler at an official URL and follows the Robots Exclusion Protocol (RFC 9309). The user-agent string Claude-SearchBot is verifiable via reverse-DNS lookup on the crawling IP addresses. You can safely allow it unless you have a specific reason to block (e.g., AI training opt-out or SEO tool visibility).

Verify by reverse-DNS lookup: legitimate Claude-SearchBot requests resolve to anthropic's domain.

What does Claude-SearchBot do?

Understanding Claude-SearchBot's purpose helps you decide whether to allow or block it.

Collects web content for AI model training
Gathers data to improve language understanding
Scrapes text, code, and other content for datasets
May be used for chatbot and AI assistant development
Typically does NOT affect search engine rankings

Frequently Asked Questions

What is the official user-agent string for Claude-SearchBot?

The official user-agent string for Claude-SearchBot is: Claude-SearchBot. This is the exact string you must use in robots.txt, Nginx, Apache, or Cloudflare firewall rules to target this bot. User-agent matching in robots.txt is case-insensitive, but the string must be spelled correctly. You can verify that a request genuinely comes from Claude-SearchBot by performing a reverse-DNS lookup on the source IP - legitimate bots resolve back to their operator's domain.

Is Claude-SearchBot safe?

Yes, Claude-SearchBot is a safe and legitimate crawler. It is operated by Anthropic, which publicly documents its crawler at an official URL and follows the Robots Exclusion Protocol (RFC 9309). The user-agent string Claude-SearchBot is verifiable via reverse-DNS lookup on the crawling IP addresses. You can safely allow it unless you have a specific reason to block (e.g., AI training opt-out or SEO tool visibility).

Will blocking Claude-SearchBot hurt my SEO?

✅ No SEO Impact - Blocking Claude-SearchBot does not affect your rankings in Google, Bing, or any other search engine. Claude-SearchBot is an AI training crawler, not a search indexer. You can freely block it via User-agent: Claude-SearchBot / Disallow: / without any SEO penalty. This is the recommended approach if you want to opt out of Anthropic's LLM training datasets.

How do I block Claude-SearchBot in robots.txt?

Add the following lines to your /robots.txt file:

User-agent: Claude-SearchBot
Disallow: /

This instructs Claude-SearchBot not to crawl any path on your site. The Disallow: / directive covers the entire domain including subfolders. To only block specific sections, replace / with the path (e.g., Disallow: /blog/). Note: robots.txt is publicly readable - any bot or human can inspect it at yourdomain.com/robots.txt.

Does Claude-SearchBot respect robots.txt?

Yes - Claude-SearchBot is a well-behaved bot operated by Anthropic. It fetches and parses /robots.txt before crawling any page, following RFC 9309.

How do I verify if Claude-SearchBot is crawling my site?

Search your web server access logs for the string Claude-SearchBot (case-insensitive grep: grep -i "Claude-SearchBot" /var/log/nginx/access.log). You can also check Google Search Console → Coverage → Crawl Stats for Googlebot variants. For Claude-SearchBot specifically, filter by user-agent in your log analysis tool (GoAccess, AWStats, etc.).

What is the crawl frequency of Claude-SearchBot?

Claude-SearchBot crawls at a moderate rate. If you notice excessive traffic in your logs, you can add a Crawl-delay directive:

User-agent: Claude-SearchBot
Crawl-delay: 10

(10 second delay between requests).

Can I block Claude-SearchBot from specific pages only?

Yes. Instead of a global Disallow: / you can restrict Claude-SearchBot to specific paths:

User-agent: Claude-SearchBot
Disallow: /private/
Disallow: /staging/
Allow: /

This allows Claude-SearchBot everywhere except the listed paths. Path matching in robots.txt uses prefix matching - Disallow: /private/ blocks /private/page.html but NOT /public/private/.

Does blocking Claude-SearchBot prevent AI training on my content?

Blocking Claude-SearchBot via robots.txt signals to Anthropic that your content should not be used for AI training. However, robots.txt is a voluntary protocol - there is no technical enforcement. For stronger protection:
1. Add <meta name="Claude-SearchBot" content="noai, noimageai, noindex"> to your pages.
2. Add a llms.txt file at your domain root (emerging standard).
3. Use Cloudflare WAF or Nginx to return 403 for this user-agent.
4. Consider IP blocklists for Anthropic's known crawler IP ranges.

Is there an alternative to robots.txt to opt out of Claude-SearchBot?

Yes. Several additional opt-out mechanisms exist for AI crawlers:
• Meta tag: <meta name="Claude-SearchBot" content="noindex">
• X-Robots-Tag HTTP header: X-Robots-Tag: noai, noimageai
• llms.txt: Add a /llms.txt file (similar to robots.txt but for LLMs)
• Server block: Return 403 or 429 for this user-agent via WAF or Nginx
Using multiple layers provides the strongest protection.

Can I block ClaudeBot but allow Claude-SearchBot?

Yes. They are separate robots.txt tokens. Anthropic documents that blocking ClaudeBot excludes your content from future training, while blocking Claude-SearchBot may stop Claude search from showing or citing your pages.