Caution Official docs AI & LLM Bots

Meta Web Indexer

Q: What is the official user-agent string for Meta Web Indexer?

The official user-agent string for Meta Web Indexer is: meta-webindexer . Use this exact string in robots.txt, Nginx, Apache, or Cloudflare firewall rules to target this bot. Matching in robots.txt is case-insensitive. Verify a request genuinely comes from Meta Web Indexer by performing a reverse-DNS lookup on the source IP.

Q: Is Meta Web Indexer safe?

Meta Web Indexer is generally legitimate but warrants caution. It is operated by Meta. Review its crawl behaviour in your logs and apply robots.txt or rate-limiting if its activity is heavier than you expect.

Q: Will blocking Meta Web Indexer hurt my SEO?

✅ No SEO Impact - Blocking Meta Web Indexer does not affect your rankings in Google, Bing, or any other search engine. Meta Web Indexer is an AI crawler, not a traditional search indexer. You can freely block it via User-agent / Disallow: / without any SEO penalty.

Q: How do I block Meta Web Indexer in robots.txt?

Add the following lines to your /robots.txt file: User-agent: meta-webindexer Disallow: / This instructs Meta Web Indexer not to crawl any path on your site. To block only specific sections, replace / with the path (e.g., Disallow: /blog/ ).

Q: How do I verify if Meta Web Indexer is crawling my site?

Search your web server access logs for the string meta-webindexer (case-insensitive: grep -i "meta-webindexer" /var/log/nginx/access.log ). Filter by user-agent in your log analytics tool (GoAccess, AWStats, etc.).

Operated by Meta

Scout's quick take

meta-webindexer (Meta Web Indexer) fetches pages so Meta's AI search can answer and cite them. Block it and you can drop out of those answers, so most sites keep it allowed.

# Block only if you really mean to
User-agent: meta-webindexer
Disallow: /

Audit my AI crawler access Build a full robots.txt

Quick Facts

User-Agent:: meta-webindexer
Category:: AI & LLM Bots
Operator:: Meta
Safety:: Use Caution
Blocking Impact:: Low - No SEO ranking impact
SEO Impact Score:: 2/10
Official docs:: developers.facebook.com

Confirmed by the operator

Meta names meta-webindexer on its own documentation page, linked above. The token, purpose and robots.txt behaviour on this profile come from that page, re-checked on October 19, 2026.

What is Meta Web Indexer?

Meta Web Indexer is an AI search-index crawler operated by Meta. It collects and indexes web content so it can be surfaced as citations or answers inside Meta's AI search experience. Allowing Meta Web Indexer can increase your visibility in AI-powered search results, which is why most sites choose to permit it.
Meta Web Indexer uses the user-agent token meta-webindexer. You can control it via robots.txt, meta tags (noai), or the emerging llms.txt standard. Robots.txt is voluntary; for hard enforcement, combine it with server-level IP blocking.

What happens if you block Meta Web Indexer?

✅ No SEO Impact - Blocking Meta Web Indexer does not affect your rankings in Google, Bing, or any other search engine. Meta Web Indexer is an AI crawler, not a traditional search indexer. You can freely block it via User-agent / Disallow: / without any SEO penalty.

Generally safe to allow; provides legitimate crawling value.

Meta-WebIndexer vs Meta-ExternalAgent vs Meta-ExternalFetcher

Meta runs four crawlers with four different jobs, and its Meta web crawler documentation (updated 21 May 2026) says allowing Meta-WebIndexer, user agent meta-webindexer/1.1, helps Meta AI cite and link your content.

Meta's crawlers compared
Crawler	Job	Follows robots.txt?	Block it and...
meta-webindexer	Indexes pages to improve Meta AI search	Yes	Meta AI is less likely to cite and link you
Meta-ExternalAgent	Crawls to train AI models	Yes	Opt out of Meta training
Meta-ExternalFetcher	Fetches a link a user shared with Meta AI	May bypass robots.txt (Meta)	Needs a server rule to stop
facebookexternalhit	Builds link previews on Facebook and Instagram	May bypass robots.txt (Meta)	Broken link previews

Meta caches robots.txt for up to 24 hours, so changes take up to a day.

Meta crawl volume: the data behind the caution rating

Meta-ExternalAgent is now one of the largest AI trainers. It grew from 0.9% to 7.5% of all AI and search crawling between July 2024 and July 2025 (Cloudflare, 29 August 2025) and reached 57.33% of sites with about 4 billion requests in Hostinger, 20 January 2026. Publishers on Reddit reported 7.9 million Meta requests in 30 days using more than 900 GB, and 11 million in 15 days on another site.

Meta-ExternalAgent is among the fastest-growing AI crawlers.

Those numbers are the training crawler, not the indexer. That is the reason to separate them rather than block "Meta" wholesale.

Meta-WebIndexer use cases

Use cases: what to do with this bot
If you are...	Recommended setting and why
You want Meta AI citations but not training	Allow meta-webindexer, block Meta-ExternalAgent.
Meta traffic is overloading the server	Rate limit Meta-ExternalAgent at the edge; robots.txt changes take up to 24 hours.
You rely on Facebook sharing	Never block facebookexternalhit, or link previews break.

User-agent: meta-webindexer
Allow: /

User-agent: Meta-ExternalAgent
Disallow: /

Check each Meta token with an AI crawler check, and see the Meta operator page and the Meta-ExternalAgent guide for the full family.

Test Meta crawler access

AI crawler checker: tests 265 bots against your robots.txt in one scan.
AI crawler diagnostic: fetches a page as each AI bot and names the layer that blocks it.
robots.txt validator: shows allowed, blocked and partial status per bot.
robots.txt generator: builds a file with presets for search, agents and training.
batch URL checker: compares up to 20 sites side by side, with CSV export.
llms.txt generator: writes the file AI assistants read for a site summary.
schema markup generator: produces JSON-LD for eight common types.
GEO audit: scores a page for generative engine optimization.

Want to see whether Meta Web Indexer actually visits you? Read how do I monitor AI bot traffic to my website?

How to block Meta Web Indexer with robots.txt

<code>User-agent: meta-webindexer</code> - Matching is case-insensitive. Robots.txt is fetched from the root of each subdomain separately.

Block completely (robots.txt)

User-agent: meta-webindexer
Disallow: /

Allow all (robots.txt)

User-agent: meta-webindexer
Allow: /

Block private only (robots.txt)

User-agent: meta-webindexer
Disallow: /private/
Disallow: /api/
Disallow: /admin/
Allow: /

Nginx server block

# Nginx: Hard-block Meta Web Indexer
if ($http_user_agent ~* "meta-webindexer") {
    return 403 "Bot blocked";
}

Apache .htaccess

# Apache: Hard-block Meta Web Indexer
SetEnvIfNoCase User-Agent "meta-webindexer" bad_bot
Order Allow,Deny
Allow from all
Deny from env=bad_bot

Meta robots tag

X-Robots-Tag header

X-Robots-Tag: noindex, nofollow

Is Meta Web Indexer safe to allow?

Meta Web Indexer is generally legitimate but warrants caution. It is operated by Meta. Review its crawl behaviour in your logs and apply robots.txt or rate-limiting if its activity is heavier than you expect.

Verify by reverse-DNS lookup: legitimate Meta Web Indexer requests resolve to Meta's domain.

What does Meta Web Indexer do?

Understanding Meta Web Indexer's purpose helps you decide whether to allow or block it.

Indexes web pages for AI-powered search answers
Surfaces your content as citations inside AI search results
Builds a knowledge index used to answer user questions
Allowing it can increase visibility in AI search
Respects robots.txt directives

Meta Web Indexer in context

Meta Web Indexer is a retrieval crawler that feeds AI search answers operated by Meta. In plain terms, it builds a search or answer index. Our database files it under AI and LLM crawlers and ranks it Tier 2, important crawlers from established AI and search operators. 74 of the 104 AI and LLM crawlers in our database share this tier.

Its safety rating is caution and its blocking impact is low. So the decision about Meta Web Indexer rests on what you want from Meta, not on its behaviour. How we rate safety and impact.

How site owners treat Meta Web Indexer

Our robots.txt study of the Tranco top 10,000 did not track meta-webindexer as a named token, so we do not publish an adoption figure for it. If you run a large site, your own logs are the best evidence of how often it visits.

Does it follow robots.txt?

Meta Web Indexer's operator says it follows robots.txt, so a correctly written Meta Web Indexer group is enough to control it.

Meta crawlers and close alternatives

Meta Web Indexer is one of 8 Meta crawlers in our database. A User-agent: meta-webindexer group controls Meta Web Indexer only; the other Meta tokens below need their own lines.

meta-webindexer: this crawler (Tier 2)
FacebookBot: Facebook Crawler, fetches a page to build a link preview when someone shares it (Tier 5)
facebookexternalhit: Facebook Preview, fetches a page to build a link preview when someone shares it (Tier 5)
Meta-ExternalAds: Meta Ads Crawler, checks pages for an SEO, uptime or analytics service (Tier 4)
Meta-ExternalAgent: Meta AI Training, collects pages to train AI models (Tier 1)
Meta-ExternalFetcher: Meta AI Fetcher, fetches a page when a person asks an assistant or app to open it (Tier 2)
meta-externaltest: Meta-ExternalTest, checks pages for an SEO, uptime or analytics service (Tier 4)
muse-: Meta Muse (personal agent), fetches a page when a person asks an assistant or app to open it (Tier 1)

To give Meta Web Indexer the same answer as its siblings, stack User-agent: meta-webindexer with their lines above one set of rules; to treat Meta Web Indexer differently, give it its own group.

Other AI and LLM crawlers that do the same job (builds a search or answer index) include AddSearchBot from AddSearch, AIWebIndex from Lyrenth, atlassian-bot from Atlassian, AzureAI-SearchBot from Microsoft, Cloudflare-AI-Search from Cloudflare, Cloudflare-AutoRAG from Cloudflare. Deciding about Meta Web Indexer is usually a decision about the whole group, so check how your file treats these too.

We first recorded meta-webindexer on June 18, 2026. Meta documents it on developers.facebook.com, and we recheck that page before each release.

Which rule Meta Web Indexer actually follows

Meta Web Indexer reads exactly one group in robots.txt, the most specific one that names meta-webindexer, and never combines rules across groups. This short file shows the trap:

User-agent: *
Disallow: /private/

User-agent: meta-webindexer
Disallow: /drafts/

Find the group. Meta Web Indexer looks for a group naming meta-webindexer. One exists, so it ignores User-agent: * completely.
Apply only that group. It is refused /drafts/ and nothing else.
The surprise. /private/ is open to Meta Web Indexer, because that rule lives in the * group it no longer reads. To keep it out of both, repeat Disallow: /private/ inside its own group.

Test yourself: does a rule under User-agent: * apply to Meta Web Indexer if the file also has a User-agent: meta-webindexer group?

No. Once a group names it, Meta Web Indexer reads only that group. Put every rule you want it to follow inside its own group.

Test yourself: will blocking Meta Web Indexer remove your pages from AI answers?

Often, yes. Meta Web Indexer is a retrieval crawler that feeds AI search answers, so pages it cannot fetch can drop out of the answers or results it supports.

Test yourself: is User-agent: meta-webindexer the same as User-agent: meta-webindexer?

Yes. RFC 9309 makes user-agent matching case-insensitive. Paths are a different story: Disallow: /Blog/ does not block /blog/.

Should you allow Meta Web Indexer? Decide by situation

Recommended setting for Meta Web Indexer by type of site
If you are...	Recommended setting
You publish content and want AI answers to cite it	Allow it. Meta Web Indexer fetches pages for answers, so blocking it can remove you from them.
You license content and want to stop AI training	Blocking it does not stop training, because Meta Web Indexer is a retrieval crawler that feeds AI search answers. Block the operator's training crawler instead.
You run an online shop	Allow it on product and category pages, which are the pages shoppers ask assistants about.
Your server is under load from bots	Rate-limit before blocking. It is rated caution in our database.

Check Meta Web Indexer on your own site

Open /robots.txt on your domain and search for meta-webindexer. If it is not named, it follows User-agent: *.
List the Disallow lines in the group Meta Web Indexer follows and test your key URLs against them; the longest match wins, and Allow wins a tie.
Look beyond the file. A firewall or bot manager can refuse Meta Web Indexer even when robots.txt allows it, and that never shows in the file.
Run our robots.txt AI crawler test to see how Meta Web Indexer and every other crawler we track are treated in one pass, including firewall refusals.

Common mistakes with Meta Web Indexer

The errors we see most often with Meta Web Indexer in the files and firewalls we test:

Blocking Meta Web Indexer while trying to stop AI training. Retrieval crawlers fetch pages so answers can cite them; blocking it can remove you from those answers without affecting training.
Letting a firewall or bot-fight mode refuse Meta Web Indexer while robots.txt allows it. The file looks right, but the crawler still gets a 403 or a challenge page.
Forgetting that once a User-agent: meta-webindexer group exists, Meta Web Indexer ignores every rule under User-agent: *.

All three hide inside a file that looks correct, so check Meta Web Indexer from outside after every change.

Spotting Meta Web Indexer in your logs

robots.txt says what Meta Web Indexer may do; your access logs show what it actually did.

Count requests

On a server with standard Nginx or Apache logs, this counts matching requests per day (adjust the log path for your host):

grep -iE "meta-webindexer" /var/log/nginx/access.log \
  | awk '{print $4}' | cut -d: -f1 | sort | uniq -c

And this shows which paths it requested most, which is the quickest way to see whether your Disallow lines are being respected:

grep -iE "meta-webindexer" /var/log/nginx/access.log \
  | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

Check the requests are genuine

Anyone can send the Meta Web Indexer user-agent, so confirm the source before acting on a match:

meta-webindexer: we hold no published IP list, so use reverse DNS where the operator documents a hostname and treat an unverifiable Meta Web Indexer request with caution.

Frequently Asked Questions

What is the official user-agent string for Meta Web Indexer?

The official user-agent string for Meta Web Indexer is: meta-webindexer. Use this exact string in robots.txt, Nginx, Apache, or Cloudflare firewall rules to target this bot. Matching in robots.txt is case-insensitive. Verify a request genuinely comes from Meta Web Indexer by performing a reverse-DNS lookup on the source IP.

Is Meta Web Indexer safe?

Will blocking Meta Web Indexer hurt my SEO?

How do I block Meta Web Indexer in robots.txt?

Add the following lines to your /robots.txt file:

User-agent: meta-webindexer
Disallow: /

This instructs Meta Web Indexer not to crawl any path on your site. To block only specific sections, replace / with the path (e.g., Disallow: /blog/).

Does Meta Web Indexer respect robots.txt?

Meta Web Indexer is operated by Meta and is expected to fetch and parse /robots.txt before crawling, following RFC 9309. For hard enforcement, combine robots.txt with server-level IP or user-agent blocking.

How do I verify if Meta Web Indexer is crawling my site?

Search your web server access logs for the string meta-webindexer (case-insensitive: grep -i "meta-webindexer" /var/log/nginx/access.log). Filter by user-agent in your log analytics tool (GoAccess, AWStats, etc.).

Does blocking Meta-ExternalAgent also block meta-webindexer?

No. They are separate tokens. Meta says Meta-WebIndexer improves Meta AI search and helps Meta cite and link your content, while Meta-ExternalAgent crawls for AI training.

AI2BotSafesame_category