Doing it yourself

AI Crawler Check vs reading your robots.txt by hand

Manual checking is free, requires no tool, and for a five-line robots.txt on a site you know well it is genuinely sufficient. It stops being sufficient the moment the file has multiple groups, the moment a firewall is involved, or the moment you need to be right about 196 user-agents rather than the four you can name from memory.

Competitor facts verified 2026-08-08 Source: RFC 9309, the Robots Exclusion Protocol

Short answer

Read it by hand when the file is short, you wrote it, and you only care about a few well-known crawlers. Use an audit when groups overlap, when precedence decides the outcome, when a WAF might be intercepting requests, or when being wrong has a cost.

Choose checking robots.txt by hand when

The file is short, you wrote it, and you remember why each line is there.
You only care about two or three crawlers you can name.
You want to understand the protocol properly. Reading RFC 9309 once is worth more than any tool.
You are offline, or working on a site that is not publicly reachable.
You need to inspect something no audit models, such as an unusual header or a conditional response.

Choose AI Crawler Check when

The file has several groups and you are unsure which one a given bot obeys.
You need path-level precedence resolved correctly rather than by intuition.
You suspect a firewall rather than robots.txt is doing the blocking.
You want to be correct about 196 user-agents, not the handful anyone can recall.
You need something to show a client or a colleague that is not your own reading of a text file.

Side by side

AI Crawler Check compared with checking robots.txt by hand
What you are comparing	AI Crawler Check	checking robots.txt by hand
Comparable. Cost and setup	Free, no sign-up	Free, no tool at all
Advantage AI Crawler Check. Coverage of user-agents	196 checked every time, in 8 categories	However many you remember to look for
Advantage AI Crawler Check. Group resolution	Applies the one-group rule and reports which group won for each bot	Easy to assume rules combine. They do not
Advantage AI Crawler Check. Path precedence	Longest match wins, Allow wins exact-length ties, resolved per bot	Commonly misread as first match or last match
Advantage AI Crawler Check. Wildcard handling	Treats `*` and `$` as special and `?` as literal, as the standard requires	`?` is very often assumed to be a wildcard
Advantage AI Crawler Check. Detects WAF or firewall blocking	Yes, and distinguishes it from a missing file	A browser request may succeed where a crawler request is blocked, so this is invisible
Advantage AI Crawler Check. Checks llms.txt as well	Yes	Only if you remember to look
Comparable. Explains what to do next	Prioritised recommendations, tied to 6 bot tiers	Your own judgement, which may be excellent
Advantage checking robots.txt by hand. Builds your understanding	Shows the reasoning, but does the work for you	Reading the standard yourself teaches you more than any report
Advantage checking robots.txt by hand. Handles edge cases no tool models	No. It models the standard and common failure modes	Yes. Nothing beats looking directly at the response
Advantage checking robots.txt by hand. Judgement about your own site	None. It does not know your licensing position, your commercial priorities or why a rule exists	You know why the rule was added and what it is protecting, which is context no report can supply
Advantage checking robots.txt by hand. Trust in the result	You are trusting our parser. The methodology is published so you can check it, but it is still a dependency	You verified it yourself, which is the strongest form of confidence available
Advantage AI Crawler Check. Repeatable across many sites	Yes, up to 20 URLs per batch	Linear in your time and attention

Dot colour shows which tool has the edge on that row: AI Crawler Check, checking robots.txt by hand, comparable.

Four things manual reading gets wrong, in order of frequency

Manual checking is not a bad habit, and this page is not an argument against understanding your own configuration. It is an argument about where human reading reliably diverges from how crawlers actually behave.

1. Assuming rules combine across groups

A compliant crawler obeys exactly one group: the most specific one that names it. If GPTBot has its own User-agent: GPTBot block, every rule under User-agent: * is invisible to GPTBot. All of them.

User-agent: *
Disallow: /private/

User-agent: GPTBot
Disallow: /drafts/

Read casually, this looks like GPTBot is kept out of both /private/ and /drafts/. It is not. GPTBot obeys only its own group, so /private/ is fully open to it. Everything the wildcard group protects is unprotected against every bot that has its own group.

2. Reading precedence as file order

Within a group, order does not decide. The longest matching path wins, and on an exact-length tie Allow beats Disallow.

User-agent: GPTBot
Disallow: /blog/
Allow: /blog/public/

Here /blog/public/post is allowed, because /blog/public/ is a longer match than /blog/. A reader working top to bottom concludes the whole blog is blocked. Move the lines around and nothing changes, which surprises people who expect order to matter.

3. Expecting `?` to be a wildcard

Only * and $ are special in robots.txt. A ? is a literal question mark. A rule written to block query strings by using ? as a wildcard blocks only paths containing a literal ?, which is almost never what was intended.

4. Missing the firewall entirely

This is the one that costs the most, because manual checking is structurally blind to it. You open /robots.txt in your browser and it loads. Conclusion: the file is fine and reachable.

But your browser is not GPTBot. It sends a browser user-agent, from a residential address, with normal headers. A WAF or bot-protection rule can allow that request and block one with an AI crawler user-agent from a datacentre range. Your view is not the crawler's view, and no amount of careful reading closes that gap.

Why this matters more than a syntax error A robots.txt mistake is visible in a text file, so somebody eventually spots it. A WAF block leaves no trace in any file you own. The rule lives in a dashboard, was often added by somebody else for an unrelated reason, and the only symptom is an absence: crawlers stop arriving. When we detect this we report it as a firewall block rather than a missing file, because those have entirely different fixes.

What manual checking is genuinely better at

Two things, and they are not trivial:

Learning. Reading RFC 9309 once will make you better at this than any tool can. Understanding the one-group rule and longest-match precedence means you write correct files the first time rather than fixing them afterwards.
Unmodelled edge cases. If something strange is happening with headers, redirects, content negotiation or conditional responses, looking directly at the raw response with curl is the right move. No audit models everything.

The practical division: read the standard to build judgement, use an audit to apply it consistently at 196-bot scale without relying on memory.

If you prefer to do it by hand

Then do it properly. A useful manual pass with curl, which at least sends a crawler user-agent:

curl -sA "GPTBot" https://example.com/robots.txt
curl -sI -A "GPTBot" https://example.com/
curl -s https://example.com/llms.txt -o /dev/null -w "%{http_code}\n"

Compare the first result against a normal browser user-agent. If they differ, something is treating crawlers differently, and that something is not robots.txt.

When you want the same reasoning applied to all 196 bots at once, with group resolution and precedence handled for you, the free AI crawler check does it in about a minute. The fixing blocked crawlers guide explains each rule it applies, so you can verify the tool rather than trust it.

Frequently asked questions

Can I just read my robots.txt myself?

For a short file you wrote, on a site you know, checking two or three crawlers, yes. The failure points are group resolution, longest-match precedence, the literal ?, and firewall blocks that never appear in the file at all. Those are where careful readers still reach wrong conclusions.

My robots.txt loads fine in my browser. Does that mean crawlers can read it?

Not necessarily. Your browser sends a browser user-agent from a residential address. A WAF can allow that and block the same request from an AI crawler user-agent in a datacentre range. If a firewall blocks our fetch, we report it as a firewall block rather than a missing file, because the fix is completely different.

Why is my Disallow rule being ignored?

Most often because the bot has its own User-agent group, so your User-agent: * rules do not apply to it at all. A crawler obeys exactly one group. Second most often, a longer Allow path overrides your Disallow regardless of line order.

Does line order matter in robots.txt?

No, not within a group. The longest matching path wins, and on an exact-length tie Allow beats Disallow. Reordering lines changes nothing, which catches out anyone reading the file top to bottom like code.

Is a tool more reliable than reading the file myself?

On consistency, yes: it applies the same rules to 196 user-agents every time and does not get tired. On understanding, no: reading RFC 9309 yourself is more valuable than any report. The best position is both, which is why the methodology is published rather than hidden.

See where your site actually stands

Free check against 196 AI crawlers. No sign-up, nothing to install.

Run a free check