Everything here is free and needs no account. This page is about picking the right one for the job in front of you.
Crawler check
The main scan. Resolves your robots.txt against all 265 crawlers across 8 categories, detects your AI files, runs the 42 GEO checks and returns a scored report with fixes. Start here unless you already know exactly what you need.
robots.txt generator
Builds a correct file from per-bot toggles and presets. Use it when you want to change policy. It repeats shared rules into every group, which is the step people miss when writing the file by hand.
robots.txt validator
Checks a file you already have and warns when a rule would block something SEO-critical. Use it before deploying a hand-edited robots.txt, and after inheriting a site whose file you did not write.
llms.txt generator
Produces an llms.txt describing what your site is about. robots.txt grants permission; llms.txt supplies context. They solve different problems and you can publish both.
Schema generator
Generates JSON-LD structured data. Structured data is one of the more direct ways to make a page machine-readable, which is why it carries real weight in the GEO checks.
Batch checker
Checks up to 20 URLs in one run and exports CSV. Built for agencies and anyone auditing a portfolio, where running scans one at a time is the bottleneck.
GEO audit
A deeper pass focused on whether your content can be understood and cited, scored across 5 areas. Use it once access is already correct and you want to improve how you are represented.
Bot directory
384 reference pages: every crawler in the database, 8 category overviews and 112 operator profiles. Use it when you have a user agent from your logs and need to know who it belongs to and whether to allow it.
Which tool for which job
| Your situation | Use this |
|---|---|
| "Can AI read my site at all?" | Crawler check |
| "I want to change what I allow" | Generator |
| "Is my existing file safe?" | Validator |
| "I have many sites to audit" | Batch checker |
| "What is this user agent in my logs?" | Bot directory |
| "Access is fine, citations are not" | GEO audit |
Combining the tools
Three common workflows
| Goal | Steps |
|---|---|
| Launch a new site | Generator, then validator, then crawler check, then GEO audit |
| Investigate a drop in AI referrals | Crawler check, then diagnostic on the affected pages, then agent log checker |
| Monthly client report | Batch checker, then diagnostic on any site that dropped |
A glossary of AI crawler terms
- robots.txt
- A plain-text file at the root of each host that tells crawlers which paths they may fetch. Defined by RFC 9309. Voluntary: well-behaved crawlers follow it, others may not.
- User-agent
- The line in robots.txt that names which crawler a group applies to, and the header a crawler sends to identify itself.
- Group
- One or more
User-agentlines followed by rules. A crawler obeys only the most specific group that names it. - Disallow / Allow
- Rules that refuse or permit a path. The longest matching rule wins; Allow wins a tie.
- Training crawler
- Collects pages to train AI models, for example GPTBot and ClaudeBot.
- Retrieval crawler
- Fetches pages so AI answers can cite them, for example OAI-SearchBot and Claude-SearchBot.
- User-triggered agent
- Fetches a page because a person asked an assistant to open it, for example ChatGPT-User. Often not bound by robots.txt.
- Opt-out token
- A name you set in robots.txt that controls how content is used, but never fetches pages itself, for example Google-Extended.
- AI agent
- A program that uses a real browser to act for a person: booking, buying, filling forms. Stopped by challenges and login walls rather than robots.txt.
- WAF / bot manager
- A firewall layer, often at a CDN, that can refuse crawlers regardless of robots.txt.
- llms.txt
- A proposed Markdown file listing a site's most useful pages for language models. Not a standard and not an access control.
- Structured data
- Machine-readable JSON-LD describing a page, such as an Organization or Article.
- GEO
- Generative Engine Optimisation: making pages readable and citable by AI answer engines. Also called AI search optimisation.
- AI Visibility Score
- Our 0 to 100 measure: 70 points for crawler access, 15 for AI files and 15 for technical readiness.
- Tier
- One of 6 priority levels in our database. Tier 1 crawlers are the ones whose absence you would notice.
Tools and your workflow
Most people keep the checker and the validator open side by side while working on a file. Edit in the validator until every crawler gives the result you want, publish, then confirm with the checker on the live site. The generators are for starting from scratch or rebuilding a file that has grown messy. Once access is right, the GEO audit is a monthly habit rather than a one-off.
What the tools have in common
All the tools run in your browser against our server, need no account and keep no list of the sites you check. They share one bot database of 265 crawlers, so a crawler named in the checker means the same thing in the generator, the validator and the directory. When we add a crawler, every tool picks it up in the same release, and the changelog says so.
Choosing between the checker and the validator
The checker tests your live site, including firewalls, redirects and server errors, which is what crawlers actually experience. The validator tests a robots.txt file on its own, which is what you want before you publish a change. Use the validator while editing, then the checker once the file is live.
Which tool answers which question
- Can GPTBot read my home page?
- The crawler check.
- Which line in my file blocks ClaudeBot?
- The validator, with the ClaudeBot result expanded.
- What should my file say if I want search but not training?
- The generator, with training groups set to block.
- Is my site ready to be cited, beyond access?
- The GEO audit.
- How do twenty client home pages compare?
- The batch checker.
Limits worth knowing
- The batch checker accepts up to 20 URLs per run.
- Checks read public files only. Pages behind a login or a paywall are reported as the crawler would see them: refused.
- Results are a snapshot. A cache or a CDN rule can change what a crawler sees an hour later.
- The generators write files; they never upload anything to your server. You publish the file yourself.
Getting help with a tool
Each tool page has its own short guide under the form. If a result still looks wrong, note the URL, the time and what you expected, and send it through the contact page.
Inputs and outputs at a glance
| Tool | You give it | You get back |
|---|---|---|
| Crawler check | A URL | Access for 104 AI crawlers, the score and fixes |
| Generator | Your choices per bot group | A ready robots.txt |
| Validator | A file or a URL | Syntax errors and per-bot results |
| llms.txt generator | Your key pages | An llms.txt file |
| GEO audit | A URL | 42 checks on AI readiness |
Tools in the order most people need them
- Run the crawler check to see the problem.
- Use the validator to find the exact line that causes it.
- Use the generator to write the corrected file.
- Run the GEO audit once access is fixed, to see what else limits citations.
- Add an llms.txt with the generator if the audit flags it.
Test yourself
Which tool runs entirely in your browser?
The log checkers. Pasted log lines never leave your machine.
Common questions
How many URLs can the batch checker handle at once?
What is the difference between the crawler check and the GEO audit?
Do I need both robots.txt and llms.txt?
Can I export a report to share with a client?
Is there an API?
Ready to check your site?
265 crawlers, 42 GEO checks, no sign-up.
Run a free check