How to use AI Crawler Check

Start here, in this order

1. Quick start: run your first check and read the result in about two minutes.
2. Understanding your score: what the number means, how tiers are weighted, and why training and retrieval are different.
3. Fixing blocked crawlers: the robots.txt rules that cause most accidental blocks, and what to do when the file is not the cause.
4. Tools reference: every tool, what it is for and when to use it.
5. For agencies: a repeatable audit workflow and how to explain it to clients.

Key terms

Crawler

A program that fetches pages automatically, for search, AI training or AI answers.

Token

The name a crawler uses in robots.txt, such as GPTBot.

Training crawler

Collects pages to train future AI models. Blocking it does not remove you from AI answers today.

Retrieval crawler

Fetches pages so AI answers can cite them. Blocking it can remove you from those answers.

User-triggered agent

Fetches a page because a person asked an assistant to open it.

Tier

One of 6 priority levels. Tier 1 crawlers are the ones whose absence you would notice.

AI Visibility Score

A 0 to 100 measure of how readable your site is to AI systems: 70 points for crawler access, 15 for AI files, 15 for technical readiness.

GEO

Generative Engine Optimisation: preparing pages to be cited by AI answers. Our audit has 42 checks.

What the database covers

We track 265 bots from 112 operators in 8 categories, including 104 AI and LLM crawlers, with 262 profile pages in the bot directory. 20 bot-vs-bot pages explain common pairs, 8 tool comparisons explain how we differ from other products, and 159 articles cover strategy and news.

What changes when the tool changes

When we change how a check works, for example a new crawler, a new score component or a new kind of refusal we can detect, the docs page that explains it is updated in the same release. Old results are not recalculated, so a score from last month may differ from today's for reasons that have nothing to do with your site. If you compare results across a release, read the release notes in the changelog first. They list every change that can move a score, so you can tell a real improvement on your site from a change in how we measure it.

Docs for different roles

Why the docs are short on purpose

Each docs page is meant to be finished in one sitting and acted on the same day. Background, history and opinion live on the blog, so the docs stay focused on what to click, what the result means and what to change. When a page grows past that, we split it.

A typical first week with the tools

The vocabulary the docs assume

Crawler

A program that requests pages automatically and identifies itself with a user-agent token.

Token

The short name a robots.txt group uses to address a crawler, such as GPTBot.

Group

A block of robots.txt that starts with one or more User-agent lines and lists rules for them.

Tier

Our priority ranking, from Tier 1 (the crawlers you would notice losing) downwards, used across the 262 profiles.

GEO

Generative engine optimisation: making a page easy for AI engines to read, trust and cite.

When the docs and the tool disagree

The tool is the source of truth for your site, because it reads your live files; the docs describe how the tool works in general. If a result looks wrong after you have read the matching docs page, run the check again in a private window, then send us the URL and the time. Real discrepancies are fixed and listed in the changelog.

Which page of the docs answers which question

What the docs do not cover, and where that lives

The docs explain how to use AI Crawler Check. They deliberately leave three things to other parts of the site, so each fact has one home:

• Facts about a single crawler (user-agent, operator, purpose, safety rating) live on its profile in the bot directory, which covers 265 crawlers across 8 categories.
• Head-to-head decisions between two crawlers live in the 20 comparison pages.
• Research and news, including our robots.txt study of 5,583 top-site files, live on the blog and in the 6 case studies.

If a docs page and a profile ever disagree about a crawler, the profile wins: it is the page we recheck against the operator's own documentation before every release, as described in our methodology.

How long each docs page takes

Most people read the quick start, run a check, then come back for the page that matches what the check found. That order works better than reading the docs from top to bottom.

Keeping your own notes as you go

Every check is a snapshot of one moment. Before you change anything, save the result page or export it, and write down the date. After the change, run the same check again. The two results side by side are the only reliable proof that a fix worked, and they are what a client, a manager or your future self will ask for. The agency guide shows a simple log format that also works for a single site.

Test yourself