How to Fix Blocked AI Crawlers in robots.txt

If the report shows something blocked that you did not intend, the cause is almost always one of a small number of robots.txt patterns. These are worth knowing because the file does not behave the way most people assume.

A crawler obeys exactly one group

This is the single most misunderstood rule in robots.txt, and it is written into the standard (RFC 9309). A crawler finds the group that names it most specifically and ignores every other group, including User-agent: *.

User-agent: *
Disallow: /admin/
Sitemap: https://example.com/sitemap.xml

User-agent: GPTBot
Disallow: /drafts/

GPTBot here is not blocked from /admin/. It reads only its own group. If you want a rule to apply to a named crawler, you have to repeat it inside that crawler's group.

The practical consequence

People add a specific group for one AI bot, assume the shared rules still apply to it, and end up with a policy that does not do what they think. If you maintain per-bot groups, every shared rule must be repeated in every group.

Longest match wins, not first match

Within a group, conflicts are resolved by the longest matching pattern, not document order. If two rules tie in length, Allow wins. That is what makes this work:

User-agent: *
Disallow: /*?*
Allow: /static/

A stylesheet at /static/app.css?v=123 stays crawlable, because Allow: /static/ is the longer match. Without that Allow line, a broad query-string rule can block your own rendering assets and stop search engines rendering your pages properly.

A question mark is a literal character

Only * and a trailing $ are special in robots.txt. A ? means an actual question mark, not "optional" as it would in a regular expression. Reading it as a regex is what makes /*?* look like it matches everything, when it only matches URLs that contain a query string.

When robots.txt is not the cause

If robots.txt looks correct and access still fails, work through these in order:

Check for a firewall rule

A WAF or bot-management product can block a crawler before robots.txt is ever consulted. The report flags when it sees this. Firewall rules are invisible in robots.txt, so this is a common blind spot.
Check meta robots and X-Robots-Tag

robots.txt controls crawling. A noindex in a meta tag or an X-Robots-Tag header controls indexing. A page can be perfectly crawlable and still be kept out of results by a header you forgot about.
Confirm the file is actually served

It must be reachable at the domain root, over HTTP, as plain text. A robots.txt that returns a 404, redirects oddly, or is served as HTML will not be honoured.
Verify after deploying

Re-run the check once the change is live. Reading your own repository is not evidence about what your server is serving.

Generate rather than hand-write

Most accidental blocks come from hand-edited files. The robots.txt generator composes the file from per-bot toggles and repeats shared rules into every group for you, and the validator warns when a rule would block something SEO-critical.

A troubleshooting flow

Work through these in order

Does /robots.txt load as plain text with HTTP 200? If not, fix the server first.
Which group does the crawler follow? Search for its token; if absent, it follows User-agent: *.
Which rule decides the URL? Apply the longest match; Allow wins ties.
Does the page return 200 to the crawler, or 403, 429 or a challenge?
Is there a noindex directive in the page or headers?
Is the content present in the raw HTML?

The AI Crawler Diagnostic runs this whole flow for 18 crawlers at once.

Before and after: a typical fix

# Before: SearchBot never sees /blog/
User-agent: *
Disallow: /tmp/

User-agent: OAI-SearchBot
Disallow: /

# After: SearchBot allowed, tmp still private
User-agent: *
Disallow: /tmp/

User-agent: OAI-SearchBot
Disallow: /tmp/

In this example the site meant to block a training crawler but named the search crawler instead. Swapping the token and repeating the shared rule fixed both problems.

When the file is not the problem

robots.txt is a request, not a lock. Many of the blocks we find are not in the file at all. In our robots.txt study of the Tranco top 10,000 (5,583 valid files), 1,117 domains answered our robots.txt request with HTTP 403 and 501 returned an HTML page instead of rules. Both usually mean a firewall, CDN or bot manager stepped in before the file could be read.

HTTP 403 or 429 on robots.txt or the page: a firewall or rate limit is refusing the crawler.
A challenge page (a CAPTCHA or JavaScript check): the crawler cannot pass it, so it sees no content.
Redirect loops: a misconfigured HTTPS or trailing-slash rule sends the crawler in circles.
Content built by JavaScript: many AI crawlers read raw HTML only, so a page that renders in the browser can be empty to them.

When to stop and ask for help

If you have checked the file, the CDN, the firewall and the server and a crawler is still refused, stop changing things. Each extra change makes the cause harder to find. Write down what you changed and when, save the latest result and send both to whoever owns the layer you could not see. Most hosting providers and CDNs can show the exact rule that refused a request if you give them the time and the user-agent.

The three causes we see most

A named group that silently drops the rules under User-agent: *.
CDN bot protection that challenges every crawler it does not recognise.
A plugin or platform that rewrites robots.txt after you edit it.

Who to ask when the cause is not yours

Owners of each layer
Layer	Usually owned by
robots.txt	The site owner or the CMS plugin that writes it
CDN bot protection	Whoever manages the CDN account
Web application firewall	The hosting provider or the security team
Server errors	The developer or the hosting provider

Proving the fix worked

After any change, request the page again with the crawler's user-agent, run the checker and save both results with the date. If the crawler still fails, the cause is in a layer you have not changed yet; go back to the troubleshooting flow above and move one layer outward.

Crawl-delay is rarely the fix

651 top-site files in our study use Crawl-delay, most often with a value of 10 seconds. Googlebot ignores the line, and 28 files set it for Googlebot anyway. If a crawler is overloading you, a rate limit at the server or CDN works for every crawler; the rate limiting guide covers both.

Structural errors we see in real files

483 files block CSS or JavaScript folders, which can stop crawlers that render pages from seeing them properly.
25 files put rules before any User-agent line, where parsers ignore them.
41 files start with a byte-order mark, which some parsers read as part of the first line.
501 sites return an HTML page at /robots.txt instead of a rules file.

Paste your file into the validator to find any of these in seconds.

Test yourself

robots.txt returns HTML with HTTP 200. What do crawlers see?

Not valid rules. Many treat it as no restrictions; some treat it as an error. Serve plain text.

Common questions

Why is GPTBot still blocked after I removed the Disallow line?

Three usual causes: the change is not deployed yet, a broader group still applies, or a firewall is blocking it before robots.txt matters. Open your live robots.txt in a browser first, then re-run the check.

Does User-agent: * apply to a bot that has its own group?

No, and this surprises almost everyone. A crawler obeys exactly one group, the most specific one that names it, and ignores the wildcard group entirely. Shared rules must be repeated inside each named group.

Can I block AI crawlers without hurting search?

Yes. Target the training crawlers specifically and leave search engines and retrieval crawlers allowed. Use the generator so the groups are written correctly rather than copying a broad snippet.

Is Disallow enough to remove a page from AI answers?

Not reliably. Disallow asks a crawler not to fetch a URL, and well-behaved crawlers comply, but a page can still be described from other sources that link to or quote it. For content you need kept out of indexes, use noindex or remove the page.

How do I stop a crawler that ignores robots.txt?

robots.txt is voluntary, so a crawler that ignores it needs enforcement at the network layer: a firewall or WAF rule, or rate limiting. The bot directory notes which crawlers have a documented history of inconsistent robots.txt behaviour.

Ready to check your site?

265 crawlers, 42 GEO checks, no sign-up.

Run a free check