AI crawler diagnostic six gates, one URL

The six gates, in the order a crawler meets them

1. robots.txt. Only the group that names the crawler applies (RFC 9309), so a User-agent: GPTBot group replaces the * group entirely for GPTBot. Within the group the longest matching path wins, query string included. If the page redirects to another host, that host's robots.txt is checked too. A robots.txt that returns 5xx or 429 blocks the whole site; a 404 means no restrictions. Guide: robots.txt disallow and allow.
2. Page directives. noindex or none in <meta name="robots">, in a tag named for the bot, or in the X-Robots-Tag header. These are indexing rules, so they block search engines and AI search crawlers. A user-triggered fetcher such as ChatGPT-User still reads the page, and training crawlers follow robots.txt instead, so for those two the directive is shown as a note.
3. HTTP status. The page is requested with the crawler's own user agent, and once more as a normal browser for comparison. A 403 or 429 that the browser never sees means the site treats that crawler differently.
4. Redirect chain. Every hop is followed and counted. A loop is a hard block; a long chain wastes crawl budget. Read redirects and AI crawler access, and the case study of a loop that only proxies trigger.
5. Challenge pages. Cloudflare, Akamai, Imperva, Sucuri, DataDome, PerimeterX, AWS WAF and DDoS-Guard challenge or block pages served instead of content. See is your WAF blocking AI crawlers?
6. Content without JavaScript. Most AI fetchers do not run JavaScript, so a client-rendered page reads as empty. See JavaScript rendering and AI crawlers.

What each verdict means

Reachable

The crawler passed all six gates and received the page content.

Blocked by robots.txt

A rule in the crawler's group refuses the URL. The fix list names the line.

Blocked by page directive

A noindex or none directive applies to this crawler.

Blocked by status

The crawler got a refusal or an error that a normal browser did not.

Firewall block: verify

The site refused the request when it claimed to be the crawler but served a normal browser. Because our request does not come from the crawler's own network, this is either a real block or fake-bot protection that lets the real crawler through. Your CDN or WAF log tells you which.

Interstitial

A challenge or block page was served instead of content.

Requires JS

The HTML has too little text without scripts, so the crawler sees an almost empty page.

Redirect loop

The redirects never reach a final page.

Unverifiable

The site refused our normal browser too, so it is filtering our server. The result says nothing about the real crawler and is never counted as a block.

Opt-out token: allowed

Google-Extended and Applebot-Extended never fetch pages; robots.txt is the only gate that applies to them.

Reading the per-crawler table

Under the fix list, every crawler gets one row. Hover or focus a Pass or Fail cell to see the evidence behind it.

Crawler

The name, linked to its profile in our directory, with the operator underneath.

Verdict

The first gate that failed, or Reachable when none did.

robots.txt

Whether the crawler's group allows this exact URL.

Directive

Whether a noindex or none directive applies to this crawler.

Status

The HTTP status the crawler received, compared with a normal browser.

Redirects

Whether the redirect chain ends in a page, and how many hops it took.

Challenge

Whether a bot-protection page replaced the content.

No-JS

Whether the raw HTML holds enough text to read without scripts. "n/a" means an earlier gate already stopped the crawler.

Which crawlers are tested, and why these

Each run tests 20 names: 5 training crawlers, 5 AI search crawlers, 6 user-triggered fetchers, the 2 opt-out tokens and the 2 search engines that AI answer engines also depend on. They are the crawlers whose behaviour changes whether you appear in AI answers. For every one of the 265 bots in our database, use the whole-site AI crawler check.

Working through the fix list

Fixes are ranked by how many crawlers each issue blocks, so the top item usually unlocks the most. Change one thing at a time and rerun after each change, because one blocked gate can hide the next: a page that fails at the firewall may also have a robots.txt problem that only shows once the firewall lets the crawler through.

Why our result can differ from the real crawler

Requests come from our servers using each crawler's user-agent string, not from the crawler's own IP range. A site that trusts verified IPs, or blocks unknown data-centre ranges, can treat the real crawler better or worse than our test. That is why a firewall refusal is labelled "verify" rather than "blocked", and why a site that refuses our browser too is "Unverifiable". Everything else, including robots.txt, directives, redirects and JavaScript dependence, is the same for the real crawler. How each gate is evaluated is on the methodology page.

Which pages to diagnose

• A page that is never cited although competitors are. Start here; the answer is often a gate you cannot see in a browser.
• One URL per template: homepage, a category or listing page, a product or service page and an article. Templates share their gates, so one URL tells you about thousands.
• Any URL behind a new CDN rule, security preset or redirect, the day it ships.
• Pages with tracking parameters, because a wildcard Disallow written for faceted URLs can catch them too.