Most robots.txt problems are not strategic. They are small technical faults that change what crawlers see without anyone noticing: a file that returns HTML, a server error that tells crawlers to stay away, a rule that sits outside any group. We fetched robots.txt from all 10,000 domains in the Tranco top list on 17 October 2026 and measured how often these faults appear. The results are a useful checklist for any site, and a reminder that AI visibility starts with a file most teams never look at.
For each fault, we explain what the standard (RFC 9309) and Google's robots.txt specification say happens, how many top sites have it, and how to fix it.
Key Takeaways
- Only 5,254 of 10,000 domains returned a usable robots.txt to our research fetcher. The rest returned errors, HTML or nothing at all.
- 483 returned an HTML page at /robots.txt, usually the homepage or a 404 page served with status 200.
- 77 returned a server error. RFC 9309 says crawlers must then assume complete disallow; Google stops crawling for the first 12 hours.
- 226 valid files block the whole site for every crawler not named elsewhere through
User-agent: *withDisallow: /. - 612 files use crawl-delay, which Google does not support; 27 set it for Googlebot specifically.
How we measured
We requested https://domain/robots.txt for each of the 10,000 domains, falling back to the www. host, following up to five redirects, with a clearly labelled research user agent that links to our methodology page. We parsed every 200 response with an RFC 9309 parser and recorded the faults below. A file counted as valid when it returned status 200 and was not HTML.
Read the error counts with care
Our fetcher is not a search engine crawler. Some sites block unknown bots at the CDN, so a 403 to us does not always mean Googlebot sees a 403. The counts for valid files are reliable; the counts for errors show how often a non-major crawler is turned away, which is closer to what a new AI search crawler experiences.
The status code faults
What a crawler does depends first on the HTTP status of robots.txt, before it reads a single rule.
| Response | Domains | What the rules say happens |
|---|---|---|
| 200 with valid rules | 5,254 | The crawler follows the rules |
| 200 but HTML | 483 | Google says it tries to extract rules from the content and ignores everything else; in practice usually no rules |
| 404 not found | 650 | RFC 9309: the crawler may access any resource. Google: no crawl restrictions |
| 403 forbidden | 1,062 | Treated like other 4xx errors: no restrictions for Google. Often a CDN block of unknown bots |
| 429 too many requests | 32 | Google excludes 429 from its "4xx means no restrictions" rule; it is a signal to slow down, not a missing file |
| 5xx server error | 77 | RFC 9309: the crawler must assume complete disallow. Google: stops crawling for 12 hours, then uses its last good copy |
| No response (DNS, timeout, TLS) | 2,329 | Unreachable; many of these domains serve no website at all |
Related: how AI engines choose their sources
The 5xx trap
A server error on robots.txt is the most dangerous fault on the list. RFC 9309 says that if robots.txt is unreachable because of server or network errors, the crawler "MUST assume complete disallow". So a site whose robots.txt fails with a 500 or 503 can stop being crawled, even though every page works for humans. Google's specification gives the detail for Googlebot: for the first 12 hours it stops crawling the site and keeps retrying; then for up to 30 days it uses the last good copy it has. Newer crawlers, which may have no cached copy at all, are more likely to simply stay away. This often happens when robots.txt is generated by the application rather than served as a static file, and the application throws an error for bot requests.
The 403 misunderstanding
Some teams return 403 on robots.txt to "keep bots out". It does the opposite. Google's specification says it treats 4xx errors, except 429, as if no valid robots.txt existed, which means no crawl restrictions. It also says not to use 401 and 403 to limit crawl rate. If a firewall blocks robots.txt for a crawler, that crawler loses your rules but may still be allowed to fetch pages, or may be blocked from those too. Either way your policy is not what you wrote. See fixing 403 errors for AI crawlers.
The HTML problem
483 domains answered /robots.txt with an HTML page and status 200. Usually a single-page app or a catch-all route returns the homepage for any unknown path. Google says it will try to extract rules from HTML and ignore the rest, but an HTML page rarely contains valid lines, so the effective result is no rules. If you rely on robots.txt to block training crawlers, a missing file means you are blocking nothing.
Google and the standard do not always agree
RFC 9309 sets the minimum behaviour; each crawler adds its own details. Google documents more than most, and it differs from the bare standard in useful ways.
| Situation | RFC 9309 | Google (from its specification) |
|---|---|---|
| Server error on robots.txt | Assume complete disallow; after a long period (for example 30 days) may treat as unavailable or use a cached copy | Stop crawling for 12 hours, then use the last good copy for 30 days; after that, behaviour depends on whether the site is generally available |
| DNS errors, timeouts, broken connections | Unreachable, so complete disallow | Treated as a server error |
| 4xx such as 403 or 404 | Crawler may access any resource | No crawl restrictions, except 429 |
| Caching | Up to 24 hours by default | Generally up to 24 hours, longer if refreshing fails |
The practical lesson is the same for both: make robots.txt boring. A static file, a 200 status and plain text avoid every edge case in the table.
The parsing faults
| Fault | Valid files with it | Share | What happens |
|---|---|---|---|
No User-agent: * group | 304 | 5.8% | Crawlers not named anywhere have no rules and may fetch everything |
| No groups at all | 198 | 3.8% | Often only a Sitemap line or comments; no rules apply |
| Rules before the first user-agent line | 26 | 0.5% | Rules belong to no group and are ignored |
| Byte order mark at the start | 38 | 0.7% | Google ignores the BOM; some other parsers may misread the first line |
noindex: line | 19 | 0.4% | Not a robots.txt rule; Google does not support it |
| Relative sitemap URL | 23 | 0.4% | Google asks for the complete URL of a sitemap |
Disallow: * with nothing else | 15 | 0.3% | Paths should start with /; behaviour varies between parsers |
The first two rows matter most for AI crawlers. A file with no star group and no named group for a new AI search crawler gives that crawler no instructions at all. That may be fine, but it should be a decision rather than an accident.
The policy faults
Blocking everything by accident
226 valid files (4.3%) block the root for every crawler that is not named in its own group. Some sites mean this: they allow a short list of named crawlers and close the door to the rest. But when we see it together with no named group for any AI search crawler, it usually means the site has removed itself from ChatGPT, Claude and Perplexity search without discussing it. Check whether OAI-SearchBot, Claude-SearchBot and PerplexityBot have their own allow group.
User-agent: Googlebot
User-agent: Bingbot
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /
User-agent: *
Disallow: /Blocking CSS and JavaScript
457 valid files disallow CSS or JavaScript files or common asset folders. Crawlers that render pages need those files to see the content the way a visitor does. Google has said for years that it needs CSS and JavaScript to render. For AI crawlers, many of which do not render JavaScript at all, the bigger issue is the reverse: content that only appears after JavaScript runs may be invisible to them. See JavaScript rendering and AI crawlers.
Crawl-delay
612 files (11.6%) contain a Crawl-delay line. Google's specification lists it among fields Google does not support, so it does nothing for Googlebot; 27 files set it for Googlebot anyway. Some other crawlers do honour it; Anthropic says it supports the non-standard extension. Values of 10 seconds or more can slow a crawler to a few thousand pages a day, which may be what you want for a training crawler and is rarely what you want for a search crawler.
The size limit
RFC 9309 says crawlers must parse at least 500 kibibytes, and Google enforces exactly that limit: content after 500 KiB is ignored. None of the valid files in our scan exceeded it, so this is a rare problem, but it does happen on sites that generate one rule per product or per user. If your file is approaching the limit, group the excluded content under a common folder and disallow the folder.
A five-minute health check
- Fetch the file and look at the status code. It should be 200. A 404 is acceptable if you want no rules. A 5xx is an emergency.
- Look at the content type and first line. It should be plain text and start with a comment, a user-agent line or a sitemap line, not
<!DOCTYPE html>. - Fetch it with a crawler user agent. Some CDNs answer browsers and bots differently. Our AI crawler diagnostic requests a page with a crawler's user agent and again as a normal browser, and shows the difference. It sends the user agent from our servers, not the crawler's own IP range, so IP-based rules can still behave differently.
- Check the groups. Every rule should sit under a user-agent line, and there should be a deliberate star group.
- Check AI crawlers by name. Run AI Crawler Check to see whether each tracked crawler is allowed, partly allowed or blocked, and which rule decides it.
curl -sI https://example.com/robots.txt | head -5
curl -s https://example.com/robots.txt | head -3
curl -s -A "Mozilla/5.0 (compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot)" \
-o /dev/null -w "%{http_code}\n" https://example.com/robots.txtWhy this matters more for AI crawlers
Googlebot has decades of handling for broken robots.txt files and is often allowed through CDNs by default. New AI search crawlers have neither advantage. They are more likely to be challenged by a bot manager, more likely to receive an error page, and less likely to be on a firewall's allow list. A file that works for Google can fail for them. That is why we test with several crawler identities rather than one, and why the error counts above are worth taking seriously even with the caveat about our fetcher.
The fix is rarely complicated: serve robots.txt as a static file, return 200, keep every rule inside a group, and make sure your CDN does not challenge requests for that one path. Most of the faults above take minutes to correct once someone looks.
Common causes and their fixes
| Symptom | Usual cause | Fix |
|---|---|---|
| HTML at /robots.txt | Single-page app or CMS route catches every path | Serve a static file before the app router, or add an explicit route |
| 5xx only for bots | Robots.txt generated by app code that fails on unusual headers | Serve it statically; cache it at the edge |
| 403 for crawlers, 200 for browsers | Bot manager challenges every non-browser request | Exclude /robots.txt from challenges |
| Rules ignored after a CMS update | Plugin rewrote the file or added a BOM | Check the file after every plugin update |
| Different file on www and bare domain | Two hosts, two files, one forgotten | Redirect one host to the other, or keep both files identical |
The last row is easy to miss. Robots.txt applies per host and protocol, so example.com and www.example.com each need their own file, or one must redirect to the other. Check both.
What to do now
- □Confirm robots.txt returns 200 and plain text, for a browser and for a crawler user agent.
- □Exclude /robots.txt from bot challenges and rate limits in your CDN or firewall.
- □Move any rule above the first user-agent line into a group.
- □Remove
noindex:lines; use meta robots or X-Robots-Tag instead. - □Use full URLs in Sitemap lines.
- □Decide on crawl-delay per crawler, knowing Google ignores it.
- □Review AI crawler tokens with our stale AI token audit, and wildcard rules with robots.txt wildcards.
