Robots.txt validator with a verdict per bot

A free robots.txt validator that answers per crawler

Use this robots.txt validator free and without sign-up. Robots.txt optimaliseren (optimising) starts with knowing what each rule does, so it reports which of 265 bots every rule blocks, across 8 categories. Most robots.txt validators stop at syntax or test Googlebot alone; this one gives a verdict for every crawler, so you see at a glance whether GPTBot, ClaudeBot or Googlebot is affected.

1. Fetch a live file by URL, or switch to Paste Content. The sample buttons load five real patterns, from the accidental block-all to a faceted e-commerce file.
2. Read the verdicts by category, then the SEO Safety Score and any warning.
3. Fix the file, or build a new one in the robots.txt generator, then validate the draft again before you publish it.

Reading the report: statuses and the SEO Safety Score

Allowed

No rule in the crawler's group refuses the URL.

Blocked

The crawler's group refuses the whole site with Disallow: /.

Partial

The homepage is allowed but some paths are refused, for example /admin/. Usually intended; check that nothing you want cited sits inside a blocked path.

Not named

The crawler has no group of its own and follows User-agent: *. That is fine if your default is what you want for it.

The SEO Safety Score looks only at search engine crawlers, because those decide whether you stay in search results. It starts from the share of search engines allowed, counts a partial block as half, and runs from 0 to 100. A score of 100 means no search engine is refused; under 50 the report raises a critical warning naming the engines you block. Blocking AI training crawlers or scrapers never lowers it.

Syntax the parser accepts, and what it ignores

• Field names are case-insensitive: user-agent, User-Agent and USER-AGENT all work.
• Several User-agent lines in a row form one group, which is the tidy way to give many crawlers the same rules.
• Blank lines and # comments are ignored. A comment can sit at the end of a rule line.
• An empty Disallow: allows everything; it is not an error.
• Lines the standard does not define, such as Host or Clean-param, are skipped. Some crawlers read them; the verdicts here do not depend on them.
• Rules before the first User-agent line belong to no group and are ignored, exactly as most crawlers ignore them.

robots.txt in four rules

Every verdict in the report follows from four rules in RFC 9309, the robots.txt standard.

1. One group per crawler. A crawler obeys only the most specific group that names it. If none does, it follows User-agent: *. Rules never combine across groups.
2. Longest match wins. Inside the group, the rule with the longest matching path decides. If an Allow and a Disallow tie on length, Allow wins.
3. Only two wildcards. * matches any characters and $ anchors the end of the URL. A question mark is a literal character.
4. Names ignore case, paths do not. gptbot and GPTBot are the same token, but /Blog/ and /blog/ are different paths.

Seven checks to run on any file

1. Search for Disallow: / under User-agent: *. If present, every crawler you have not named is refused everything.
2. Find every named group and confirm it repeats the shared rules you still want that crawler to follow. A bot with its own group ignores the * group entirely.
3. Look for the same crawler named in two groups. Parsers handle this differently, so merge them.
4. Check that CSS, JavaScript and image folders are not disallowed.
5. Confirm the file loads as plain text with HTTP 200. Under RFC 9309 a 4xx means "no rules", while a 5xx tells Google and Semrush to crawl nothing (Semrush bot documentation).
6. Keep it under 500 KiB. Crawlers only have to parse that much; rules after it are ignored.
7. Remember that Google-Extended and Applebot-Extended never fetch pages. They only take effect as lines in this file. Crawlers may also cache the file for up to 24 hours, and Meta says it does (Meta web crawler documentation), so validate before you deploy, not after.

This validator compared with other ways to test

Sample files to test, and when to paste or fetch

Fetch from URL tests the live file exactly as it is served, which catches server problems such as an HTML error page or a redirect. Paste content tests a draft before you publish it, the safe way to try a change. A good habit: paste first, publish, then fetch to confirm the live result matches. Copy either sample below into Paste Content.

# 1. AI search allowed, AI training blocked
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /

User-agent: *
Allow: /

# 2. The accidental block-all (fails the SEO Safety Score)
User-agent: *
Disallow: /

More patterns, with the reasoning behind each, are in Allow vs Disallow in robots.txt and robots.txt best practices for AI SEO.

What each sample button teaches

Edit any sample and analyse again to see how one line changes the verdicts. It is the quickest way to learn how group selection and longest-match work.

When the file is not the problem

robots.txt is a request, not a lock, and many blocks are not in the file at all. In our robots.txt study of the Tranco top 10,000 (5,583 valid files), 1,117 domains answered our robots.txt request with HTTP 403 and 501 returned an HTML page instead of rules. Both usually mean a firewall, CDN or bot manager stepped in first. If the validator says a crawler is allowed but it never visits, look for:

• HTTP 403 or 429 on the page: a firewall or rate limit is refusing the crawler.
• A challenge page (a CAPTCHA or JavaScript check): the crawler cannot pass it.
• Redirect loops from a misconfigured HTTPS or trailing-slash rule.
• Content built by JavaScript: many AI crawlers read raw HTML only.

The AI crawler diagnostic tests all four for each crawler on one page.