AI Crawler Diagnostic Tool

Find out exactly why an AI crawler cannot read a page. Enter one URL and we request it as 18 crawlers (GPTBot, ClaudeBot, PerplexityBot, Googlebot and more), check the two AI opt-out tokens, then run six gates for each: robots.txt, page directives, HTTP status, redirects, firewall challenges and JavaScript.

One page per run, usually 3 to 8 seconds. For up to 20 sites at once use the batch checker.

The six gates, in the order a crawler meets them

  1. robots.txt. Only the group that names the crawler applies (RFC 9309), so a User-agent: GPTBot group replaces the * group entirely for GPTBot. Within the group the longest matching path wins, query string included. If the page redirects to another host, that host's robots.txt is checked too. A robots.txt that returns 5xx or 429 blocks the whole site; a 404 means no restrictions. Guide: robots.txt disallow and allow.
  2. Page directives. noindex or none in <meta name="robots">, in a tag named for the bot, or in the X-Robots-Tag header. These are indexing rules, so they block search engines and AI search crawlers; a user-triggered fetcher such as ChatGPT-User still reads the page, and training crawlers follow robots.txt instead, so for those two the directive is shown as a note.
  3. HTTP status. The page is requested with the crawler's own user agent, and once more as a normal browser for comparison. A 403 or 429 that the browser never sees means the site treats that crawler differently. Because our request does not come from the crawler's own network, that result is labelled Firewall block: verify: it is either a real block or fake-bot protection that lets the real crawler through, and your CDN or WAF log tells you which. If the browser is refused too, the site is filtering our server and the result is Unverifiable, never a block.
  4. Redirect chain. Every hop is followed and counted. A loop is a hard block; a long chain wastes crawl budget. Read redirects and AI crawler access, and the case study of a loop only proxies trigger.
  5. Challenge pages. Cloudflare, Akamai, Imperva, Sucuri, DataDome, PerimeterX, AWS WAF and DDoS-Guard challenge or block pages served instead of content. See is your WAF blocking AI crawlers?
  6. Content without JavaScript. Most AI fetchers do not run JavaScript, so a client-rendered page reads as empty. See JavaScript rendering and AI crawlers.

Frequently asked questions

What does the AI crawler diagnostic tool check?

For each of 16 major AI crawlers plus Googlebot and Bingbot it runs six gates against one URL: robots.txt (with RFC 9309 group selection and longest-match rules), page directives such as noindex, the HTTP status returned to that user agent, the redirect chain, CDN or WAF challenge pages, and whether the content is present without JavaScript.

How is this different from the AI Crawler Check homepage?

The homepage scores your whole site against every bot in the database using robots.txt and site-level files. This diagnostic goes deeper on one page: it actually requests the page as each crawler and tells you which gate fails and how to fix it.

Can a page pass here and still be blocked?

Yes, in one case. We send each crawler's user-agent string, but from our servers, not from the crawler's own IP range. A site that blocks or allows by IP address can behave differently for the real crawler. Everything else, including robots.txt, directives, redirects and JavaScript dependence, is the same for the real crawler.

What does "Firewall block: verify" mean?

Your site refused the request when it claimed to be that crawler, but served a normal browser. There are two possible causes and they look identical from outside the crawler's network: your firewall blocks that crawler (a real block), or it blocks requests that claim to be that crawler from an IP address the operator does not own (fake-bot protection, which lets the real crawler through). Check your CDN or WAF bot log for the crawler to see which one applies. We never count it as reachable, and we never call it a confirmed block.

Is the AI crawler diagnostic free?

Yes. It needs no account and no API key, because every check is a plain HTTP request plus parsing.

How do I audit my website for AI crawler accessibility?

Run this diagnostic on your homepage and your most important pages, fix the issues in the order listed (the top issue blocks the most crawlers), then re-run it after each release.