Why AI Crawler Check Is Different From Other AI Bot Checkers

Questions to ask of any checker, including ours

1. How many crawlers does it test, and are they current? New AI crawlers launch every month.
2. Does it explain why a crawler is blocked? A verdict without the group and line is hard to act on.
3. Does it apply RFC 9309 correctly? One group per crawler, longest match, Allow wins ties, ? is literal.
4. Does it detect firewalls? A 403 or a challenge page is a different problem from a robots.txt rule.
5. Does it separate training from retrieval? The trade-offs are different.
6. Does it say what it cannot see? Every outside-in tool has limits; honest tools name them.

Where we are honest about our limits

• We test from our servers, not from each crawler's own IP range.
• We test the URLs you give us; we do not crawl your whole site.
• We never change your site; enforcement happens in your robots.txt, server or CDN.
• We report access, not whether an AI system will choose to cite you.

Our tool comparisons name the cases where another product is the better choice.

What the database covers

265 bots from 112 operators across 8 categories, 104 of them AI and LLM crawlers, each with a tier, a purpose and an evidence label. Facts come from operators' own documentation where it exists, and our own robots.txt study of the Tranco top 10,000 adds adoption data no vendor publishes.

What we ask of you in return

Nothing is required. If the tools help you, the most useful things you can do are to tell us when a fact is wrong, to link to the page you used rather than copying it, and to send the questions you could not answer. Those three things make the directory more accurate and decide what we write next. Agencies who want help with a project can reach Horatos.ai through the contact page, but the tools stay free and complete whether you do or not, and nothing on the site is held back for paying clients.

Questions people ask before trusting a free tool

Do you store the sites I check?

No list of checked URLs is stored against you. The privacy policy has the detail.

How current is the bot list?

It is rechecked before every release; the date on each profile shows the last change.

Can I use the results for clients?

Yes, with the attribution described in the terms.

A worked example: one site, three checkers

Picture a site whose robots.txt allows every AI crawler, behind a CDN whose bot protection challenges unfamiliar agents. A checker that only reads the file reports a perfect result. A checker that fetches the page once with a browser user-agent also reports success. AI Crawler Check requests the page as each crawler would, so it sees the challenge page that GPTBot or PerplexityBot receives and reports the refusal with its status code. The invisible block case study walks through a real site where exactly this happened.

What you can verify yourself

1. Open any bot profile and follow the link to the operator's own documentation.
2. Read the method behind our robots.txt study on the methodology page.
3. Run your own site through the checker, then compare the robots.txt result with the validator line by line.
4. Check the changelog for what changed in each release and when.

How the free check stays free

AI Crawler Check is built and run by Horatos.ai, an AI search agency. The tools are free because they show, in public, the kind of work the agency does for clients. There is no paid tier hiding the useful parts, and no account is needed. The site is supported by that agency work and by clearly labelled sponsorships, described on the contact page; sponsors never change a profile, a score or a study figure.

What we measure that most checkers do not

• Firewall and CDN refusals. A file can allow a crawler that a firewall then blocks with a 403 or a challenge. We request the page as each crawler would, so that refusal shows up.
• Tokens, not just crawlers. Control tokens such as Google-Extended never visit, but they change how your content may be used. We report them separately.
• Group matching. We apply the RFC 9309 rule that a crawler obeys only its most specific group, which is where most hand-written files go wrong.
• The GEO side. Access is 70 of the points; the rest covers llms.txt, sitemap, structured data and HTTPS, explained in how the score works.

Where our numbers come from

The directory is our own: 265 crawlers from 112 operators, each checked against the operator's own documentation where it exists. The market figures come from our own robots.txt scan of the Tranco top 10,000, in which 5,583 sites returned a valid file. We publish the method on the methodology page and the raw findings in the study write-up, so you can check any figure we quote.

What we will not claim

We do not say we are the largest or the best tool, and we do not show ratings we have not earned. A free check cannot see your server logs, so it cannot tell you how often a crawler actually visits; it can tell you whether it is allowed to and whether it would get through. For visit counts, the log guide shows how to read your own data.

Who uses it and for what

Test yourself