Fixing a crawler that is blocked by mistake

The rules that most often block a crawler unintentionally, and how to undo them.

6 min read

If the report shows something blocked that you did not intend, the cause is almost always one of a small number of robots.txt patterns. These are worth knowing because the file does not behave the way most people assume.

A crawler obeys exactly one group

This is the single most misunderstood rule in robots.txt, and it is written into the standard (RFC 9309). A crawler finds the group that names it most specifically and ignores every other group, including User-agent: *.

User-agent: * Disallow: /admin/ Sitemap: https://example.com/sitemap.xml User-agent: GPTBot Disallow: /drafts/

GPTBot here is not blocked from /admin/. It reads only its own group. If you want a rule to apply to a named crawler, you have to repeat it inside that crawler's group.

The practical consequence

People add a specific group for one AI bot, assume the shared rules still apply to it, and end up with a policy that does not do what they think. If you maintain per-bot groups, every shared rule must be repeated in every group.

Longest match wins, not first match

Within a group, conflicts are resolved by the longest matching pattern, not document order. If two rules tie in length, Allow wins. That is what makes this work:

User-agent: * Disallow: /*?* Allow: /static/

A stylesheet at /static/app.css?v=123 stays crawlable, because Allow: /static/ is the longer match. Without that Allow line, a broad query-string rule can block your own rendering assets and stop search engines rendering your pages properly.

A question mark is a literal character

Only * and a trailing $ are special in robots.txt. A ? means an actual question mark, not "optional" as it would in a regular expression. Reading it as a regex is what makes /*?* look like it matches everything, when it only matches URLs that contain a query string.

When robots.txt is not the cause

If robots.txt looks correct and access still fails, work through these in order:

  1. Check for a firewall rule

    A WAF or bot-management product can block a crawler before robots.txt is ever consulted. The report flags when it sees this. Firewall rules are invisible in robots.txt, so this is a common blind spot.

  2. Check meta robots and X-Robots-Tag

    robots.txt controls crawling. A noindex in a meta tag or an X-Robots-Tag header controls indexing. A page can be perfectly crawlable and still be kept out of results by a header you forgot about.

  3. Confirm the file is actually served

    It must be reachable at the domain root, over HTTP, as plain text. A robots.txt that returns a 404, redirects oddly, or is served as HTML will not be honoured.

  4. Verify after deploying

    Re-run the check once the change is live. Reading your own repository is not evidence about what your server is serving.

Generate rather than hand-write

Most accidental blocks come from hand-edited files. The robots.txt generator composes the file from per-bot toggles and repeats shared rules into every group for you, and the validator warns when a rule would block something SEO-critical.

Common questions

Why is GPTBot still blocked after I removed the Disallow line?
Three usual causes: the change is not deployed yet, a broader group still applies, or a firewall is blocking it before robots.txt matters. Open your live robots.txt in a browser first, then re-run the check.
Does User-agent: * apply to a bot that has its own group?
No, and this surprises almost everyone. A crawler obeys exactly one group, the most specific one that names it, and ignores the wildcard group entirely. Shared rules must be repeated inside each named group.
Can I block AI crawlers without hurting search?
Yes. Target the training crawlers specifically and leave search engines and retrieval crawlers allowed. Use the generator so the groups are written correctly rather than copying a broad snippet.
Is Disallow enough to remove a page from AI answers?
Not reliably. Disallow asks a crawler not to fetch a URL, and well-behaved crawlers comply, but a page can still be described from other sources that link to or quote it. For content you need kept out of indexes, use noindex or remove the page.
How do I stop a crawler that ignores robots.txt?
robots.txt is voluntary, so a crawler that ignores it needs enforcement at the network layer: a firewall or WAF rule, or rate limiting. The bot directory notes which crawlers have a documented history of inconsistent robots.txt behaviour.

Ready to check your site?

196 crawlers, 58 GEO checks, no sign-up.

Run a free check