Meta-ExternalAgent vs ClaudeBot: Which AI Crawler Should You Allow or Block?

Mistakes when writing rules for Meta-ExternalAgent and ClaudeBot together

Meta-ExternalAgent is named in 10.7% of the top-site robots.txt files in our study and blocked in 12.6%; ClaudeBot is named in 15.4% of the top-site robots.txt files in our study and blocked in 14.4%. Large sites block ClaudeBot more often, which usually reflects a decision about training rather than about Meta-ExternalAgent.

1. Both are a training crawlers, so the usual error is treating a rule for one as a rule for both. A group for Meta-ExternalAgent says nothing about ClaudeBot; if you want the same treatment, list both lines above one set of rules.
2. They come from different operators (Meta and Anthropic), with different documentation and different update cycles. Do not assume a change one of them announces applies to the other.
3. Putting User-agent: Meta-ExternalAgent and User-agent: ClaudeBot in separate groups and then editing only one. Two groups drift apart over time; keep a comment above each that says why it differs from the other.

Each crawler's own profile lists the mistakes that apply to it alone; this list is only about combining the two.

Spotting Meta-ExternalAgent and ClaudeBot in your logs

robots.txt says what Meta-ExternalAgent and ClaudeBot may do; your access logs show what they actually did.

Count requests

On a server with standard Nginx or Apache logs, this counts matching requests per day (adjust the log path for your host):

grep -iE "Meta-ExternalAgent|ClaudeBot" /var/log/nginx/access.log \
  | awk '{print $4}' | cut -d: -f1 | sort | uniq -c

And this shows which paths it requested most, which is the quickest way to see whether your Disallow lines are being respected:

grep -iE "Meta-ExternalAgent|ClaudeBot" /var/log/nginx/access.log \
  | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

Check the requests are genuine

Anyone can send these user-agents, so confirm the source before acting on a match:

• Meta-ExternalAgent: we hold no published IP list, so use reverse DNS where Meta-ExternalAgent and ClaudeBot documents a hostname and treat an unverifiable Meta-ExternalAgent request with caution. Check it separately from ClaudeBot.
• ClaudeBot: compare each address with the ranges at claude.com/crawling/bots.json. A request claiming to be ClaudeBot from outside them is not genuine. Check it separately from Meta-ExternalAgent.

How Meta-ExternalAgent and ClaudeBot moved in our latest scans

Between our scans of October 17, 2026 and October 20, 2026, files naming Meta-ExternalAgent went from 553 to 598 (+45) and files blocking it from 658 to 706 (+48), so more sites wrote a rule for Meta-ExternalAgent and more of those rules block it. For comparison, 2026 and October 20, 2026, files naming ClaudeBot went from 800 to 858 (+58) and files blocking it from 745 to 805 (+60), so more sites wrote a rule for ClaudeBot and more of those rules block it.

Over the 3 days between those scans, blocking of Meta-ExternalAgent moved +48 files and blocking of ClaudeBot moved +60. Owners are acting faster on ClaudeBot, which is the one to recheck first if you have not touched your file this month.

We first recorded Meta-ExternalAgent on March 17, 2026. Meta documents it on developers.facebook.com, and we recheck that page before each release. We first recorded ClaudeBot on March 17, 2026. Anthropic documents it on support.claude.com, and we recheck that page before each release.

Among the 104 AI and LLM crawlers in our database, Meta-ExternalAgent sits 11th by priority (Tier 1), level with ChatGPT Agent, ChatGPT Operator, ChatGPT-User, ChatGPT-User/2.0. 22 of the 104 share its caution rating. On the ClaudeBot side: Among the 104 AI and LLM crawlers in our database, ClaudeBot sits seventh by priority (Tier 1), level with ChatGPT Agent, ChatGPT Operator, ChatGPT-User, ChatGPT-User/2.0. 81 of the 104 share its safe rating.

How top sites treat Meta-ExternalAgent and ClaudeBot

From our robots.txt study of the Tranco top 10,000 (5,583 valid files, fetched October 20, 2026):

• ClaudeBot, the most blocked of the Meta-ExternalAgent and ClaudeBot tokens: blocked in 14.4% of files (594 with its own group, 211 through User-agent: *), more often by the top 1,000 (20.4%) than the long tail (13.3%).
• Meta-ExternalAgent, second of the Meta-ExternalAgent and ClaudeBot tokens: blocked in 12.6% of files (486 with its own group, 220 through User-agent: *), more often by the top 1,000 (19.2%) than the long tail (11%).

For scale, 20.1% of all files block at least one AI training crawler. ClaudeBot is one of the tokens driving that figure.

More questions about Meta-ExternalAgent and ClaudeBot

Can I allow one and block the other?

Yes. Give each its own User-agent group with its own rules. Each crawler reads only the group that names it.

Which of the two visits more often?

It depends on your site, your audience and how often your content changes. Your access logs are the reliable answer; the commands above count both.

How long until a change applies?

Crawlers commonly cache robots.txt for up to a day, so allow about 24 hours and confirm in your logs.

Do I need to update my rules when either operator launches a new crawler?

Often, yes. A new token is not covered by rules written for the old ones. Check this comparison and both profiles periodically, or rerun the AI bot access checker after any announcement.