Robots.txt Validator

Paste your robots.txt or enter a URL to analyze it against our database of 248+ bots. See which crawlers are blocked, allowed, or partially restricted - with SEO impact analysis.

Robots txt validator free of sign-ups

This checker is free and needs no account: paste a file or fetch one from any URL, and it reports syntax errors and which of 248 bots each rule blocks. Unlike most robots.txt validators it shows the verdict per crawler, so you can see at a glance whether GPTBot, ClaudeBot or Googlebot is affected. Robots.txt optimaliseren (optimising) starts here: fix the errors, then check the robots.txt best practices.

How to validate a robots.txt file in three steps

Fetch a live file by URL, or switch to Paste Content. The new sample buttons load five real-world patterns (AI search allowed, training blocked, the accidental block-all, and more) so you can see how each is scored.
Read the per-bot verdict. Each of 248 bots is marked Allowed, Blocked or Partial using RFC 9309 longest-match rules.
Fix and retest. Build a corrected file in the robots.txt generator, deploy it, then run a crawler check on the live site.

What this validator checks versus other robots.txt testers

robots.txt testing methods compared
Check	This validator	Search Console robots.txt report	Manual reading
Googlebot verdict	Yes	Yes	Error-prone
AI bots (88 in the database)	Yes, each one	No	Error-prone
Training vs search tier impact	Yes, by category	No	Only if you know every token
Works on sites you do not own	Yes	No	Yes
Paste a draft before deploying	Yes	No	Yes
CDN or firewall 403	No, use the diagnostic	No	No

Robots.txt validation rules that trip AI crawlers

File size. RFC 9309 only requires crawlers to parse 500 KiB; rules after that are ignored.
Status code. A 4xx means "no rules"; a 5xx means "crawl nothing" for Semrush and Google (Semrush bot documentation).
Caching. Crawlers may cache the file for up to 24 hours, and Meta says it does (Meta web crawler documentation). Validate before you deploy, not after.
Group isolation. A bot with its own group ignores User-agent: * entirely.
Control tokens. Google-Extended and Applebot-Extended are never fetched; they only take effect as lines in this file.

Training blocks are usually deliberate; retrieval blocks usually are not.

Sample robots.txt files to test

Copy any of these into Paste Content, or use the sample buttons above the text box.

# 1. AI search allowed, AI training blocked (the 2026 default for most sites)
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /

User-agent: *
Allow: /

# 2. The accidental block-all (fails the SEO Safety Score)
User-agent: *
Disallow: /

More patterns, with the reasoning behind each, are in Allow vs Disallow in robots.txt and robots.txt best practices for AI SEO.

Next steps after validation

AI crawler checker: tests 248 bots against your robots.txt in one scan.
AI crawler diagnostic: fetches a page as each AI bot and names the layer that blocks it.
robots.txt generator: builds a file with presets for search, agents and training.
batch URL checker: compares up to 20 sites side by side, with CSV export.
llms.txt generator: writes the file AI assistants read for a site summary.
schema markup generator: produces JSON-LD for eight common types.
GEO audit: scores a page for generative engine optimization.

Frequently Asked Questions

Can a robots.txt validator detect a Cloudflare block?

No. A robots.txt validator only reads the file. A CDN or firewall can still return 403 to AI bots. Use a live-fetch diagnostic that requests the page as each bot to see that layer.

What does the Robots.txt Validator check?

Our validator parses your robots.txt file and checks each rule against 248 known bot user-agents across 8 categories: AI bots, search engines, Google bots, SEO tools, social media bots, data scrapers, cloud services, and other agents. It shows you exactly which bots are blocked, allowed, or partially restricted.

What is the SEO Safety Score?

The SEO Safety Score (0-100) measures how your robots.txt impacts search engine visibility. Blocking critical search engines (Googlebot, Bingbot) drastically reduces your score. Blocking AI bots and scrapers has zero negative impact. A score of 100 means all search engine bots are allowed.

How do I fix a robots.txt that blocks search engines?

If your robots.txt blocks Googlebot, Bingbot, or other search engine crawlers, use our Robots.txt Generator to create a new one. Choose the "SEO Optimized" preset which blocks AI bots and scrapers while keeping all search engines allowed.

Can I validate robots.txt for any website?

Yes! Enter any public URL and we will fetch its robots.txt file automatically. Alternatively, you can paste robots.txt content directly if you have it locally. The analysis is instant and covers all 248+ bots in our database.

What's the difference between blocked and partially blocked?

Blocked means the bot is completely denied access to all paths (Disallow: /). Partially blocked means some paths are allowed while others are restricted - for example, allowing the homepage but blocking /admin/ or /api/ paths. Both are reflected in the analysis.