AI Crawler Check vs the ai.robots.txt community blocklist

Side by side

We measured the overlap instead of claiming one

Comparison pages usually put two numbers side by side and let the bigger one imply superiority. We ran the actual comparison instead, because the interesting information is not in the totals.

On the verification date, the published robots.txt in the ai.robots.txt repository contained 163 user-agent entries. Our database contained 196 profiled bots. Matching case-insensitively:

• 103 agents appear in both.
• 93 bots we track are not in that list, largely search engine crawlers, SEO tools and cloud fetchers that a blocklist has no reason to include.
• 57 agents on their list were not in our database.

The two sets differ because the purposes differ. A blocklist wants breadth: a string it has never heard of is a string it cannot block. A profiled database wants confidence: every entry carries a category, a tier, an operator and a recommendation, and adding an entry we cannot describe would weaken the report rather than strengthen it.

The problem with pasting a blanket list

Suppose you paste all 163 entries into your robots.txt with Disallow: /. You have now blocked, in one action:

• Crawlers that collect training data, which may well be what you wanted.
• Crawlers that fetch your page because a user asked a question it answers, which removes you from AI answers.
• Crawlers belonging to assistants your own customers use to find you.

For a publisher protecting a licensed archive, that trade may be correct and deliberate. For a business that wants to be recommended when someone asks an assistant for a supplier, it is a self-inflicted wound that will not show up in any dashboard, because the traffic it prevents never existed to be measured.

This is the case for auditing before acting. See training bots vs search bots for the strategic version and understanding your score for how we weight it.

Correct strings, incorrect rules

The second gap a list cannot close: robots.txt is easy to get subtly wrong, and a wrong file fails silently. There is no error message and no validation step. The most common mistakes we see:

• Assuming rules combine. A crawler obeys exactly one group. If a bot has its own User-agent group, your User-agent: * rules are ignored for that bot entirely. Many sites believe a global rule is protecting paths that a specific group has already overridden.
• Expecting ? to be a wildcard. In robots.txt it is a literal character. Only * and $ are special.
• Assuming order decides. It does not. The longest matching path wins, and Allow beats Disallow on an exact-length tie. A rule further down the file can override one above it.
• Blocking a bot that never reads robots.txt. Some scrapers ignore it. For those, robots.txt is documentation of intent, not enforcement, and the block has to happen at your edge.

An audit resolves each rule the way a compliant crawler resolves it and shows you the winner. The fixing blocked crawlers guide works through each of these, and the robots.txt generator emits rules that follow the standard.

Use both, in this order

1. Audit. Run the free AI crawler check to see what your site permits today. Do this before changing anything, so you know what you are changing from.
2. Decide by category, not in bulk. Training, retrieval, search and scraping are separate decisions with separate consequences.
3. Borrow the strings. The ai.robots.txt list is a good source of user-agent tokens for the ones you have decided to block, and it is MIT licensed.
4. Re-audit. Confirm the file parses as intended and that nothing you wanted is now blocked. A rule you believed you wrote is not the same as a rule you actually wrote.