Meta

Common mistakes with Meta's crawlers

The errors we see most often with Meta's crawlers in the files and firewalls we test:

1. Treating Meta's crawlers as an AI or search crawler. Blocking it has little effect on visibility; decide based on load and what it does with your data.
2. Relying on robots.txt alone when Meta's crawlers is rated caution or aggressive. Back the rule with a server-side block.
3. Blocking Meta's crawlers by IP alone. Ranges change; match the token and verify with logs.

All three hide inside a file that looks correct, so check Meta's crawlers from outside after every change.

Spotting Meta in your logs

robots.txt says what Facebook Crawler and facebookexternalhit and Meta Ads Crawler and Meta Muse (muse-) and Meta Web Indexer and Meta-ExternalAgent and Meta-ExternalFetcher and Meta-ExternalTest may do; your access logs show what they actually did.

Count requests

On a server with standard Nginx or Apache logs, this counts matching requests per day (adjust the log path for your host):

grep -iE "FacebookBot|facebookexternalhit|Meta-ExternalAds|muse-|meta-webindexer|Meta-ExternalAgent|Meta-ExternalFetcher|meta-externaltest" /var/log/nginx/access.log \
  | awk '{print $4}' | cut -d: -f1 | sort | uniq -c

And this shows which paths it requested most, which is the quickest way to see whether your Disallow lines are being respected:

grep -iE "FacebookBot|facebookexternalhit|Meta-ExternalAds|muse-|meta-webindexer|Meta-ExternalAgent|Meta-ExternalFetcher|meta-externaltest" /var/log/nginx/access.log \
  | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

Check the requests are genuine

Anyone can send these user-agents, so confirm the source before acting on a match:

• FacebookBot: we hold no published IP list, so use reverse DNS where Meta documents a hostname and treat an unverifiable Facebook Crawler request with caution. Check it separately from facebookexternalhit and Meta Ads Crawler and Meta Muse (muse-) and Meta Web Indexer and Meta-ExternalAgent and Meta-ExternalFetcher and Meta-ExternalTest.
• facebookexternalhit: we hold no published IP list, so use reverse DNS where Meta documents a hostname and treat an unverifiable facebookexternalhit request with caution. Check it separately from Facebook Crawler and Meta Ads Crawler and Meta Muse (muse-) and Meta Web Indexer and Meta-ExternalAgent and Meta-ExternalFetcher and Meta-ExternalTest.
• Meta-ExternalAds: we hold no published IP list, so use reverse DNS where Meta documents a hostname and treat an unverifiable Meta Ads Crawler request with caution. Check it separately from Facebook Crawler and facebookexternalhit and Meta Muse (muse-) and Meta Web Indexer and Meta-ExternalAgent and Meta-ExternalFetcher and Meta-ExternalTest.
• muse-: we hold no published IP list, so use reverse DNS where Meta documents a hostname and treat an unverifiable Meta Muse (muse-) request with caution. Check it separately from Facebook Crawler and facebookexternalhit and Meta Ads Crawler and Meta Web Indexer and Meta-ExternalAgent and Meta-ExternalFetcher and Meta-ExternalTest.
• meta-webindexer: we hold no published IP list, so use reverse DNS where Meta documents a hostname and treat an unverifiable Meta Web Indexer request with caution. Check it separately from Facebook Crawler and facebookexternalhit and Meta Ads Crawler and Meta Muse (muse-) and Meta-ExternalAgent and Meta-ExternalFetcher and Meta-ExternalTest.
• Meta-ExternalAgent: we hold no published IP list, so use reverse DNS where Meta documents a hostname and treat an unverifiable Meta-ExternalAgent request with caution. Check it separately from Facebook Crawler and facebookexternalhit and Meta Ads Crawler and Meta Muse (muse-) and Meta Web Indexer and Meta-ExternalFetcher and Meta-ExternalTest.
• Meta-ExternalFetcher: we hold no published IP list, so use reverse DNS where Meta documents a hostname and treat an unverifiable Meta-ExternalFetcher request with caution. Check it separately from Facebook Crawler and facebookexternalhit and Meta Ads Crawler and Meta Muse (muse-) and Meta Web Indexer and Meta-ExternalAgent and Meta-ExternalTest.
• meta-externaltest: we hold no published IP list, so use reverse DNS where Meta documents a hostname and treat an unverifiable Meta-ExternalTest request with caution. Check it separately from Facebook Crawler and facebookexternalhit and Meta Ads Crawler and Meta Muse (muse-) and Meta Web Indexer and Meta-ExternalAgent and Meta-ExternalFetcher.

Related: how AI engines choose their sources

What our own data shows about Meta

Everything below comes from the AI Crawler Check bot database and our two most recent robots.txt scans of the Tranco top 10,000, not from Meta's marketing. It covers 6 Meta tokens in turn.

FacebookBot in our records

We first recorded FacebookBot on March 17, 2026. We could not find a page where Meta names it, so treat the token as reported rather than documented. FacebookBot is outside our tracked token set, but 395 of 5,583 top-site files already name it, which is enough to show owners treat it as a real crawler.

Among the 9 social preview bots in our database, FacebookBot sits second by priority (Tier 5), level with facebookexternalhit, LinkedInBot, Pinterestbot, Quora-Bot. 9 of the 9 share its safe rating. Other operators with a crawler that fetches a page to build a link preview when someone shares it: MicrosoftPreview (Microsoft, Tier 3), GoogleMessages (Google, Tier 5), LinkedInBot (LinkedIn, Tier 5), Pinterestbot (Pinterest, Tier 5), Quora-Bot (Quora, Tier 5). A policy for FacebookBot is usually a policy for these too.

A decision record for FacebookBot

Teams that review crawlers once a quarter keep one short entry per token, so anyone can see why a rule exists and when it was last checked. Filled in from our data, the FacebookBot entry reads:

• Operator and job: Meta; it fetches a page to build a link preview when someone shares it.
• Priority and behaviour: Tier 5, rated safe, blocking impact not documented.
• Suggested starting point: Allow by default; revisit if it costs you bandwidth.
• Enforcement: FacebookBot's operator does not clearly document how it treats robots.txt. Write the FacebookBot rule anyway, then confirm in your logs that its requests stop.

User-agent: FacebookBot
Allow: /

Paste the line into the validator with one of your own URLs before it goes live. Revisit the FacebookBot entry once a quarter, or when Meta publishes a new token next to it.

facebookexternalhit in our records

We first recorded facebookexternalhit on March 17, 2026. Meta documents it on developers.facebook.com, and we recheck that page before each release. No file in our top-10,000 scans names facebookexternalhit, so any rule you write for it puts you ahead of almost every large site.

Among the 9 social preview bots in our database, facebookexternalhit sits third by priority (Tier 5), level with FacebookBot, LinkedInBot, Pinterestbot, Quora-Bot. 9 of the 9 share its safe rating. Other operators with a crawler that fetches a page to build a link preview when someone shares it: MicrosoftPreview (Microsoft, Tier 3), GoogleMessages (Google, Tier 5), LinkedInBot (LinkedIn, Tier 5), Pinterestbot (Pinterest, Tier 5), Quora-Bot (Quora, Tier 5). A policy for facebookexternalhit is usually a policy for these too.

Meta-ExternalAds in our records

We first recorded Meta-ExternalAds on September 22, 2026. Meta documents it on developers.facebook.com, and we recheck that page before each release. No file in our top-10,000 scans names Meta-ExternalAds, so any rule you write for it puts you ahead of almost every large site.

Among the 9 social preview bots in our database, Meta-ExternalAds sits first by priority (Tier 4), alone in its tier. 9 of the 9 share its safe rating. Other operators with a crawler that checks pages for an SEO, uptime or analytics service: OAI-AdsBot (OpenAI, Tier 2), AdIdxBot (Microsoft, Tier 3), AdsBot-Google (Google, Tier 3), AdsBot-Google-Mobile (Google, Tier 3), Baiduspider-ads (Baidu, Tier 3). A policy for Meta-ExternalAds is usually a policy for these too.

muse- in our records

We first recorded muse- on October 11, 2026. We could not find a page where Meta names it, so treat the token as reported rather than documented. muse- is outside our tracked token set, but 0 of 5,583 top-site files already name it, which is enough to show owners treat it as a real crawler.

Among the 104 AI and LLM crawlers in our database, muse- sits 12th by priority (Tier 1), level with ChatGPT Agent, ChatGPT Operator, ChatGPT-User, ChatGPT-User/2.0. 22 of the 104 share its caution rating. Other operators with a crawler that fetches a page when a person asks an assistant or app to open it: ChatGPT Agent (OpenAI, Tier 1), ChatGPT Operator (OpenAI, Tier 1), ChatGPT-User (OpenAI, Tier 1), ChatGPT-User/2.0 (OpenAI, Tier 1), Claude-SearchBot (Anthropic, Tier 1). A policy for muse- is usually a policy for these too.

meta-webindexer in our records

We first recorded meta-webindexer on June 18, 2026. Meta documents it on developers.facebook.com, and we recheck that page before each release. No file in our top-10,000 scans names meta-webindexer, so any rule you write for it puts you ahead of almost every large site.

Among the 104 AI and LLM crawlers in our database, meta-webindexer sits 64th by priority (Tier 2), level with AddSearchBot, AI2Bot, AI2Bot-DeepResearchEval, aiHitBot. 22 of the 104 share its caution rating. Other operators with a crawler that builds a search or answer index: AddSearchBot (AddSearch, Tier 2), AIWebIndex (Lyrenth, Tier 2), atlassian-bot (Atlassian, Tier 2), AzureAI-SearchBot (Microsoft, Tier 2), Cloudflare-AI-Search (Cloudflare, Tier 2). A policy for meta-webindexer is usually a policy for these too.

Meta-ExternalAgent in our records

We first recorded Meta-ExternalAgent on March 17, 2026. Meta documents it on developers.facebook.com, and we recheck that page before each release. Between our scans of October 17, 2026 and October 20, 2026, files naming Meta-ExternalAgent went from 553 to 598 (+45) and files blocking it from 658 to 706 (+48), so more sites wrote a rule for Meta-ExternalAgent and more of those rules block it.

Among the 104 AI and LLM crawlers in our database, Meta-ExternalAgent sits 11th by priority (Tier 1), level with ChatGPT Agent, ChatGPT Operator, ChatGPT-User, ChatGPT-User/2.0. 22 of the 104 share its caution rating. Other operators with a crawler that collects pages to train AI models: ClaudeBot (Anthropic, Tier 1), Google-Extended (Google, Tier 1), GPTBot (OpenAI, Tier 1), AI2Bot (Allen Institute for AI, Tier 2), aiHitBot (aiHit, Tier 2). A policy for Meta-ExternalAgent is usually a policy for these too.

How top sites treat Meta's crawlers

From our robots.txt study of the Tranco top 10,000 (5,583 valid files, fetched October 20, 2026):

• Meta-ExternalAgent, the only one tracked among the Meta's crawlers tokens: blocked in 12.6% of files (486 with its own group, 220 through User-agent: *), more often by the top 1,000 (19.2%) than the long tail (11%).

For scale, 20.1% of all files block at least one AI training crawler. Meta-ExternalAgent is one of the tokens driving that figure.

More questions about Meta

Will blocking Meta affect my Google rankings?

Only if one of its tokens is a Google search crawler. None of Meta's tokens are Googlebot, so classic Google rankings are unaffected.

How quickly does a robots.txt change take effect for Meta?

Crawlers cache robots.txt, commonly for up to a day. Expect a change to apply within about 24 hours, and confirm it in your logs.

Can I allow Meta on some sections and block it on others?

Yes. Inside the group for FacebookBot, add Disallow lines for the paths to keep private and Allow lines for exceptions. The longest matching rule wins.

Does blocking Meta delete content it already collected?

No. robots.txt only controls future visits. For content already collected, check whether Meta offers a removal or opt-out process.

Is a firewall block better than robots.txt for Meta?

A firewall enforces the decision, while robots.txt asks politely. Many operators recommend robots.txt because blocking their IPs can stop them reading your rules at all. Use robots.txt first, and add a firewall rule only if requests continue.