Meta Web Indexer
Operated by Meta
Quick Facts
- User-Agent:
- meta-webindexer
- Category:
- AI & LLM Bots
- Operator:
- Meta
- Safety:
- Use Caution
- Blocking Impact:
- Low - No SEO ranking impact
- SEO Impact Score:
- 2/10
- Official docs:
- developers.facebook.com
Confirmed by the operator
Meta names meta-webindexer on its own documentation page, linked above. The token, purpose and robots.txt behaviour on this profile come from that page, re-checked on September 22, 2026.
What is Meta Web Indexer?
Meta Web Indexer is an AI search-index crawler operated by Meta. It collects and indexes web content so it can be surfaced as citations or answers inside Meta's AI search experience. Allowing Meta Web Indexer can increase your visibility in AI-powered search results, which is why most sites choose to permit it.
Meta Web Indexer is an AI search-index crawler operated by Meta. It collects and indexes web content so it can be surfaced as citations or answers inside Meta's AI search experience. Allowing Meta Web Indexer can increase your visibility in AI-powered search results, which is why most sites choose to permit it.
Meta Web Indexer uses the user-agent token meta-webindexer. You can control it via robots.txt, meta tags (noai), or the emerging llms.txt standard. Robots.txt is voluntary; for hard enforcement, combine it with server-level IP blocking.
What happens if you block Meta Web Indexer?
User-agent / Disallow: / without any SEO penalty.Meta-WebIndexer vs Meta-ExternalAgent vs Meta-ExternalFetcher
Meta runs four crawlers with four different jobs, and its Meta web crawler documentation (updated 21 May 2026) says allowing Meta-WebIndexer, user agent meta-webindexer/1.1, helps Meta AI cite and link your content.
| Crawler | Job | Follows robots.txt? | Block it and... |
|---|---|---|---|
| meta-webindexer | Indexes pages to improve Meta AI search | Yes | Meta AI is less likely to cite and link you |
| Meta-ExternalAgent | Crawls to train AI models | Yes | Opt out of Meta training |
| Meta-ExternalFetcher | Fetches a link a user shared with Meta AI | May bypass robots.txt (Meta) | Needs a server rule to stop |
| facebookexternalhit | Builds link previews on Facebook and Instagram | May bypass robots.txt (Meta) | Broken link previews |
Meta caches robots.txt for up to 24 hours, so changes take up to a day.
Meta crawl volume: the data behind the caution rating
Meta-ExternalAgent is now one of the largest AI trainers. It grew from 0.9% to 7.5% of all AI and search crawling between July 2024 and July 2025 (Cloudflare, 29 August 2025) and reached 57.33% of sites with about 4 billion requests in Hostinger, 20 January 2026. Publishers on Reddit reported 7.9 million Meta requests in 30 days using more than 900 GB, and 11 million in 15 days on another site.
Those numbers are the training crawler, not the indexer. That is the reason to separate them rather than block "Meta" wholesale.
Meta-WebIndexer use cases
| If you are... | Recommended setting and why |
|---|---|
| You want Meta AI citations but not training | Allow meta-webindexer, block Meta-ExternalAgent. |
| Meta traffic is overloading the server | Rate limit Meta-ExternalAgent at the edge; robots.txt changes take up to 24 hours. |
| You rely on Facebook sharing | Never block facebookexternalhit, or link previews break. |
User-agent: meta-webindexer
Allow: /
User-agent: Meta-ExternalAgent
Disallow: /Check each Meta token with a AI crawler check, and see the Meta operator page and the Meta-ExternalAgent guide for the full family.
Test Meta crawler access
- AI crawler checker: tests 248 bots against your robots.txt in one scan.
- AI crawler diagnostic: fetches a page as each AI bot and names the layer that blocks it.
- robots.txt validator: shows allowed, blocked and partial status per bot.
- robots.txt generator: builds a file with presets for search, agents and training.
- batch URL checker: compares up to 20 sites side by side, with CSV export.
- llms.txt generator: writes the file AI assistants read for a site summary.
- schema markup generator: produces JSON-LD for eight common types.
- GEO audit: scores a page for generative engine optimization.
Want to see whether Meta Web Indexer actually visits you? Read how do I monitor AI bot traffic to my website?
How to block Meta Web Indexer with robots.txt
<code>User-agent: meta-webindexer</code> - Matching is case-insensitive. Robots.txt is fetched from the root of each subdomain separately.
Is Meta Web Indexer safe to allow?
What does Meta Web Indexer do?
Understanding Meta Web Indexer's purpose helps you decide whether to allow or block it.
- Indexes web pages for AI-powered search answers
- Surfaces your content as citations inside AI search results
- Builds a knowledge index used to answer user questions
- Allowing it can increase visibility in AI search
- Respects robots.txt directives
Frequently Asked Questions
What is the official user-agent string for Meta Web Indexer?
meta-webindexer. Use this exact string in robots.txt, Nginx, Apache, or Cloudflare firewall rules to target this bot. Matching in robots.txt is case-insensitive. Verify a request genuinely comes from Meta Web Indexer by performing a reverse-DNS lookup on the source IP.Is Meta Web Indexer safe?
Will blocking Meta Web Indexer hurt my SEO?
User-agent / Disallow: / without any SEO penalty.How do I block Meta Web Indexer in robots.txt?
/robots.txt file:User-agent: meta-webindexer Disallow: /
This instructs Meta Web Indexer not to crawl any path on your site. To block only specific sections, replace / with the path (e.g.,
Disallow: /blog/).Does Meta Web Indexer respect robots.txt?
/robots.txt before crawling, following RFC 9309. For hard enforcement, combine robots.txt with server-level IP or user-agent blocking.How do I verify if Meta Web Indexer is crawling my site?
meta-webindexer (case-insensitive: grep -i "meta-webindexer" /var/log/nginx/access.log). Filter by user-agent in your log analytics tool (GoAccess, AWStats, etc.).