Documentation5 min read

Tools reference

What each tool does, and when to reach for it.

All tools

Every free tool, what it is for and when to use it. Not sure? The crawler check is always the right first step.

Everything here is free and needs no account. This page is about picking the right one for the job in front of you.

Crawler check

The main scan. Resolves your robots.txt against all 265 crawlers across 8 categories, detects your AI files, runs the 42 GEO checks and returns a scored report with fixes. Start here unless you already know exactly what you need.

robots.txt generator

Builds a correct file from per-bot toggles and presets. Use it when you want to change policy. It repeats shared rules into every group, which is the step people miss when writing the file by hand.

robots.txt validator

Checks a file you already have and warns when a rule would block something SEO-critical. Use it before deploying a hand-edited robots.txt, and after inheriting a site whose file you did not write.

llms.txt generator

Produces an llms.txt describing what your site is about. robots.txt grants permission; llms.txt supplies context. They solve different problems and you can publish both.

Schema generator

Generates JSON-LD structured data. Structured data is one of the more direct ways to make a page machine-readable, which is why it carries real weight in the GEO checks.

Batch checker

Checks up to 20 URLs in one run and exports CSV. Built for agencies and anyone auditing a portfolio, where running scans one at a time is the bottleneck.

GEO audit

A deeper pass focused on whether your content can be understood and cited, scored across 5 areas. Use it once access is already correct and you want to improve how you are represented.

Bot directory

384 reference pages: every crawler in the database, 8 category overviews and 112 operator profiles. Use it when you have a user agent from your logs and need to know who it belongs to and whether to allow it.

Which tool for which job

Choosing a tool by task
Your situation Use this
"Can AI read my site at all?"Crawler check
"I want to change what I allow"Generator
"Is my existing file safe?"Validator
"I have many sites to audit"Batch checker
"What is this user agent in my logs?"Bot directory
"Access is fine, citations are not"GEO audit

Combining the tools

Three common workflows

Tool workflows
GoalSteps
Launch a new siteGenerator, then validator, then crawler check, then GEO audit
Investigate a drop in AI referralsCrawler check, then diagnostic on the affected pages, then agent log checker
Monthly client reportBatch checker, then diagnostic on any site that dropped

A glossary of AI crawler terms

robots.txt
A plain-text file at the root of each host that tells crawlers which paths they may fetch. Defined by RFC 9309. Voluntary: well-behaved crawlers follow it, others may not.
User-agent
The line in robots.txt that names which crawler a group applies to, and the header a crawler sends to identify itself.
Group
One or more User-agent lines followed by rules. A crawler obeys only the most specific group that names it.
Disallow / Allow
Rules that refuse or permit a path. The longest matching rule wins; Allow wins a tie.
Training crawler
Collects pages to train AI models, for example GPTBot and ClaudeBot.
Retrieval crawler
Fetches pages so AI answers can cite them, for example OAI-SearchBot and Claude-SearchBot.
User-triggered agent
Fetches a page because a person asked an assistant to open it, for example ChatGPT-User. Often not bound by robots.txt.
Opt-out token
A name you set in robots.txt that controls how content is used, but never fetches pages itself, for example Google-Extended.
AI agent
A program that uses a real browser to act for a person: booking, buying, filling forms. Stopped by challenges and login walls rather than robots.txt.
WAF / bot manager
A firewall layer, often at a CDN, that can refuse crawlers regardless of robots.txt.
llms.txt
A proposed Markdown file listing a site's most useful pages for language models. Not a standard and not an access control.
Structured data
Machine-readable JSON-LD describing a page, such as an Organization or Article.
GEO
Generative Engine Optimisation: making pages readable and citable by AI answer engines. Also called AI search optimisation.
AI Visibility Score
Our 0 to 100 measure: 70 points for crawler access, 15 for AI files and 15 for technical readiness.
Tier
One of 6 priority levels in our database. Tier 1 crawlers are the ones whose absence you would notice.

Tools and your workflow

Most people keep the checker and the validator open side by side while working on a file. Edit in the validator until every crawler gives the result you want, publish, then confirm with the checker on the live site. The generators are for starting from scratch or rebuilding a file that has grown messy. Once access is right, the GEO audit is a monthly habit rather than a one-off.

What the tools have in common

All the tools run in your browser against our server, need no account and keep no list of the sites you check. They share one bot database of 265 crawlers, so a crawler named in the checker means the same thing in the generator, the validator and the directory. When we add a crawler, every tool picks it up in the same release, and the changelog says so.

Choosing between the checker and the validator

The checker tests your live site, including firewalls, redirects and server errors, which is what crawlers actually experience. The validator tests a robots.txt file on its own, which is what you want before you publish a change. Use the validator while editing, then the checker once the file is live.

Which tool answers which question

Can GPTBot read my home page?
The crawler check.
Which line in my file blocks ClaudeBot?
The validator, with the ClaudeBot result expanded.
What should my file say if I want search but not training?
The generator, with training groups set to block.
Is my site ready to be cited, beyond access?
The GEO audit.
How do twenty client home pages compare?
The batch checker.

Limits worth knowing

  • The batch checker accepts up to 20 URLs per run.
  • Checks read public files only. Pages behind a login or a paywall are reported as the crawler would see them: refused.
  • Results are a snapshot. A cache or a CDN rule can change what a crawler sees an hour later.
  • The generators write files; they never upload anything to your server. You publish the file yourself.

Getting help with a tool

Each tool page has its own short guide under the form. If a result still looks wrong, note the URL, the time and what you expected, and send it through the contact page.

Inputs and outputs at a glance

Tool inputs and outputs
ToolYou give itYou get back
Crawler checkA URLAccess for 104 AI crawlers, the score and fixes
GeneratorYour choices per bot groupA ready robots.txt
ValidatorA file or a URLSyntax errors and per-bot results
llms.txt generatorYour key pagesAn llms.txt file
GEO auditA URL42 checks on AI readiness

Tools in the order most people need them

  1. Run the crawler check to see the problem.
  2. Use the validator to find the exact line that causes it.
  3. Use the generator to write the corrected file.
  4. Run the GEO audit once access is fixed, to see what else limits citations.
  5. Add an llms.txt with the generator if the audit flags it.

Test yourself

Which tool runs entirely in your browser?

The log checkers. Pasted log lines never leave your machine.

Common questions

How many URLs can the batch checker handle at once?
Up to 20 per run, with CSV export. For a larger portfolio, run it in batches.
What is the difference between the crawler check and the GEO audit?
The crawler check answers whether AI systems can reach your content. The GEO audit answers whether they can understand and cite it once they arrive. Access first, comprehension second.
Do I need both robots.txt and llms.txt?
They are complementary, not alternatives. robots.txt controls access. llms.txt supplies context about what your site is and which pages matter. Publishing both is reasonable; neither replaces the other.
Can I export a report to share with a client?
Yes. Reports export to PDF and Excel, and you can put your own company name on them. The batch checker exports CSV.
Is there an API?
The site exposes JSON endpoints that the interface itself uses, including the bot database at /api/bots. These are rate limited and are not a documented public contract yet, so treat them as subject to change. If you need bulk access to the data for a project, email hello@aicrawlercheck.com and ask; we would rather give you a sensible answer than have you scrape it.

Ready to check your site?

265 crawlers, 42 GEO checks, no sign-up.

Run a free check
WhatsApp us