How We Calculate Your AI Visibility Score

Score Composition (100 points)

The 2026 Bot Taxonomy We Use

Not all AI bots are equal. We classify every user-agent into five behavioral types, because each type needs a different decision from you:

How the Check Works

1. We fetch your robots.txt and parse it per RFC 9309, including longest-match precedence and group merging.
2. If a WAF or CDN firewall blocks our primary fetch, we retry with fallback strategies and clearly flag results when we cannot confirm the real file.
3. We evaluate all 265 known user-agent tokens against your rules, including wildcard inheritance.
4. We check for llms.txt, llms-full.txt, your XML sitemap, homepage structured data, HTTPS, and the IETF Web Bot Auth directory at /.well-known/http-message-signatures-directory.
5. We scan your robots.txt for deprecated or undocumented tokens (anthropic-ai, claude-web, google-notebooklm, notebooklm, operator, chatgpt agent, teoma) that vendors have retired or never published, a common source of false security.

What We Cannot Detect (And Nobody Can)

• Whether a bot actually honors your robots.txt. We report documented vendor behavior and known violations (for example, Cloudflare's August 2025 findings on Perplexity stealth crawling).
• Agentic browser traffic that presents a standard Chrome user-agent.
• Server-side blocks (WAF rules, IP blocks) that do not appear in robots.txt. Your robots.txt may allow a bot your firewall silently rejects.
• Whether AI engines actually cite you. Access is necessary but not sufficient for citations.

How We Verify Every Bot (Evidence Tiers)

A bot list is only as good as its sources. Every one of the 265 tokens in our database sits in exactly one of two tiers, and the build fails if a token has no evidence at all. We do not copy entries from other bot directories or community lists.

Verified by AI Crawler Check: the robots.txt study

On October 20, 2026 we fetched /robots.txt from every domain in the Tranco top 10,000 list (Tranco ranks sites by real traffic). 5,583 returned a valid robots.txt file. For each Reported token we counted the files with a User-agent line naming it exactly (case-insensitive, as RFC 9309 specifies). A high count means site owners treat the name as a real crawler worth a rule. It does not prove who operates the bot, which is why these tokens keep the Reported badge.

Our study fetcher identifies itself as AICrawlerCheck-Research/1.0 and links to this page. It requests only robots.txt, once per domain. Bot intelligence version: 2026-10-19.

Data Sources

Our bot database is built from primary vendor documentation, independent primary registries and our own robots.txt study, refreshed continuously. Intelligence version: 2026-10-19.

• OpenAI, Anthropic, Google, Microsoft, Apple, Meta, Perplexity official crawler documentation
• Published IP ranges and verification endpoints where vendors provide them
• Cloudflare Radar verified bots data and security research
• Our own robots.txt study of the Tranco top 10,000 (see evidence tiers above)
• IETF Web Bot Auth draft specifications