How to Use AI Crawler Check - Quick Start Guide (265 Bots)

AI Crawler Check answers one question: can AI systems and search engines actually read your website? You do not need an account, a credit card, or anything installed. If you only have a minute, this is the whole workflow.

Enter your domain

Type your site into the box on the home page and press Check. You can paste a bare domain like example.com or a full URL. There is nothing to install and no account to create.
Wait for the scan

The scan fetches your robots.txt, resolves it against all 265 crawlers in the database, then looks for llms.txt and runs the GEO checks. It usually takes a few seconds.
Read your score first

The score out of 100 and its letter grade are a summary, not the answer. Treat them as a way to see whether anything urgent is wrong, then move to the detail below.
Check the Tier 1 row

Tier 1 is the group that decides whether AI assistants can quote you at all. If anything there is blocked and you did not intend it, fix that before anything else on the page.
Work the recommendations

Each recommendation comes with the exact lines to paste. They are ordered by impact, so working top-down is the fastest route to a better result.
Re-run after deploying

The check reads your live site, so it only reflects a change once that change is deployed. Re-run it to confirm the fix landed rather than assuming.

What the scan actually reads

It fetches your live robots.txt, your homepage HTML, and (if present) llms.txt and llms-full.txt. It does not crawl your whole site, so it is fast and it does not add meaningful load to your server.

Reading the results

The report is organised so the most consequential information is nearest the top.

What each section of the report tells you
Section	What it answers
Score and grade	Roughly how discoverable you are right now.
Bot access table	Which of the 265 crawlers you allow, block, or partly block.
AI files	Whether you publish `llms.txt`, and whether it is valid.
GEO report	42 checks on whether your content can be understood and cited.
Recommendations	What to change, with copy-paste code, ranked by impact.

Blocked is not always bad

Blocking a training crawler is a legitimate editorial choice and costs you nothing in search rankings. What you almost never want is to block a retrieval crawler, because those are the ones that fetch your page in order to cite it in an answer. The report labels which is which so you are not guessing.

Where to go next

Understanding your score if the number surprised you.
Fixing blocked crawlers if something is blocked that should not be.
Tools reference for the generator, validator and batch checker.

Going further after your first check

A worked example of reading a report

Score: 58. Lower than expected, so look at what is missing before anything else.
Tier 1: two crawlers blocked. GPTBot and ClaudeBot are refused. Both are training crawlers, so this may be intended.
Retrieval crawlers allowed. OAI-SearchBot and Claude-SearchBot can read the site, so AI answers can still cite it.
No llms.txt. A small, optional gain.
Decision. If the training block is deliberate, nothing urgent needs fixing. If it is not, edit the two groups and rerun.

Your first week

Day 1: run the check on your homepage and one key template.
Day 2: fix any accidental blocks and validate the new file.
Day 3: run the GEO audit on your most important page.
Day 5: rerun the check to confirm crawlers have picked up the change.
Day 7: add structured data or llms.txt if the audit suggests it.

robots.txt in four rules

Every tool on this site rests on the same four rules from RFC 9309, the robots.txt standard. Knowing them makes every result easier to read.

One group per crawler. A crawler obeys only the most specific group that names it. If none does, it follows User-agent: *. Rules never combine across groups.
Longest match wins. Inside the group, the rule with the longest matching path decides. If an Allow and a Disallow tie on length, Allow wins.
Only two wildcards. * matches any characters and $ anchors the end of the URL. A question mark is a literal character, not a wildcard.
Names ignore case, paths do not. gptbot and GPTBot are the same token, but /Blog/ and /blog/ are different paths.

If something looks wrong

Sometimes a first result looks impossible: a crawler you never blocked shows as refused, or every crawler shows an error. Run the check again after a minute, because a slow server or a CDN warming up can cause a one-off failure. If the result repeats, open the reason shown next to the crawler. A status code such as 403 points to a firewall or CDN, a 5xx points to the server, and a robots.txt line points to the file. Each of these has its own section in fixing blocked crawlers, which is the natural next page.

Saving your first result

Before you change anything, save the result: export it or take a screenshot that includes the date. That single file is your baseline. Every later check is compared with it, and it is the clearest way to show a manager or a client what changed.

Questions after a first check

Do I need a robots.txt at all?: No file means everything is allowed. A file is only needed when you want to refuse something or point to your sitemap.
Why do some crawlers show as tokens?: Tokens such as Google-Extended never visit; they control how content is used. The result lists them so your file can address them.
How often should I check?: After any change to robots.txt, your CDN or your hosting, and otherwise once a month.

What a good first result looks like

A healthy site usually shows every Tier 1 AI crawler allowed, a valid robots.txt, a sitemap and HTTPS. Our database lists 14 Tier 1 crawlers, including ChatGPT Agent, ChatGPT Operator, ChatGPT-User, ChatGPT-User/2.0. If all of them can reach your home page, your next steps are about content and the GEO items, not access; the GEO audit is the place to start.

Checking more than the home page

robots.txt applies to the whole host, but firewall rules and page errors can differ by section. After the home page, check one URL from each template that matters: a product or service page, an article and a category page. If those agree, you have covered the site.

Your first check, minute by minute

First check timeline
Minute	What you do
0 to 1	Paste your home page URL into the checker and run it
1 to 2	Read the score and the list of crawlers that cannot reach you
2 to 4	Open the reason for each refusal: robots.txt, firewall or an error
4 to 5	Copy the suggested fix or open the generator

Three results that need a second look

Every crawler blocked: Usually a User-agent: * with Disallow: /, or a firewall that refuses unknown agents. Check which.
Only one operator blocked: Often deliberate. Confirm with whoever owns the file before you change it.
Errors instead of blocks: A 5xx or timeout is a server problem, not a policy. See fixing blocked crawlers.

Test yourself

Your score is low because GPTBot is blocked on purpose. Should you unblock it?

Only if you want your content used for training. Blocking it does not stop AI answers citing you.

Common questions

Do I need an account to use AI Crawler Check?

No. Every tool is free and requires no sign-up, no email and no credit card. You can run a check immediately.

How long does a check take?

Usually a few seconds. It fetches your robots.txt, your homepage and your AI files rather than crawling your entire site, which is why it stays fast.

Does running a check affect my site or my SEO?

No. It makes a small number of ordinary HTTP requests, the same as any visitor. It changes nothing on your site and has no effect on your rankings.

Can I check a site I do not own?

Yes. It only reads publicly available files, so you can check competitors, clients or prospects. Nothing private is accessed.

Why did my score not change after I edited robots.txt?

The scan reads your live site, so the edit has to be deployed first. If you changed a local file or a staging environment, the production robots.txt is still the old one. Confirm by opening yourdomain.com/robots.txt in a browser.

Ready to check your site?

265 crawlers, 42 GEO checks, no sign-up.

Run a free check