AI Crawler Comparisons

Why bots come in pairs

Most AI companies run more than one crawler, each with its own job and its own robots.txt token. The most important split is between training and retrieval. Blocking one does not block the other, which is exactly the point: you can refuse training while staying visible in AI answers. Each comparison explains one pair that site owners often confuse.

The split is common. In our robots.txt study of the Tranco top 10,000 (5,583 valid files), 427 sites block GPTBot but keep OAI-SearchBot open, and 575 block Google-Extended while keeping Googlebot allowed.

What every comparison contains

• A side-by-side table of tokens, operators, safety, impact and official documentation.
• Numbers from our robots.txt study: how many top sites name and block each crawler.
• A recommendation by goal: citations, training opt-out, server load and search traffic.
• Copy-paste robots.txt rules for each crawler and for both together.
• Common mistakes, log commands and short self-tests.

The most-blocked AI crawlers among top sites

Deciding your AI crawler policy in five questions

1. Do you want AI assistants to cite and link to you? If yes, allow retrieval crawlers and user-triggered agents.
2. Do you license your content, or object to it being used for training? If yes, block training crawlers by name.
3. Are bots costing you real money in bandwidth? If yes, rate-limit first, then block the lowest-tier scrapers.
4. Does your firewall agree with your robots.txt? Check that bot protection is not refusing crawlers you allow.
5. Who checks it next month? Policies drift as new crawlers launch. Put a recurring check in the calendar.

Most commercial sites land on the same answer: allow search and retrieval, decide training on its merits, and verify from outside after every change with the free AI crawler check.

Comparisons and your own data

A comparison tells you how two crawlers differ in general and how large sites treat them. It cannot tell you how often either one visits your site, or which pages they request. For that, read your access logs: count requests for each token per day and note the paths. If one of the pair barely visits, the decision matters less than the comparison suggests. If one visits heavily, a rate limit may solve the problem without a block. The server log guide shows the commands, and the comparison pages give the exact patterns for each pair.

After you have decided

1. Write the groups for both crawlers in your file, or tick both in the generator.
2. Test a real URL for each crawler in the validator.
3. Publish the file and run the checker on your live site.
4. Note the decision and the date, so the next review knows why each rule exists.

What every comparison has in common

Whatever the pair, three rules apply. Each crawler reads only the group that names it, so a rule for one never covers the other. A named group ignores everything under User-agent: *, so shared rules must be repeated. And a robots.txt line is a request: if either crawler ignores it, the fix is a server or CDN rule, not a stronger line. The troubleshooting guide covers all three in detail.

Reading a comparison page from top to bottom

1. Quick verdict. One paragraph on which crawler has the better safety profile and what that means for you.
2. Side-by-side table. Operator, user-agent string, category, safety rating, impact if blocked and the operator's own documentation.
3. Rules for the pair. A short robots.txt that gives each crawler the answer that matches its job.
4. Mistakes when combining them. The errors specific to these two, not general robots.txt advice.
5. Latest movement. How top sites changed their rules for each crawler between our two most recent scans.
6. Copy-paste blocks. A ready group for each crawler, and a link to the generator for larger files.

If you only have a minute, read the verdict and the rules for the pair; everything below them is the evidence.

Training crawlers in the comparison set

Most comparisons involve at least one training crawler, because that is where site owners hesitate. Our database lists 29 crawlers whose only job is training and 11 that both train and build an index. In our study, 1,123 top-site files (20.1%) block at least one training crawler, against 689 (12.3%) that block an AI search crawler. That gap is the reason many comparisons end with "block one, allow the other".

Asking for a new comparison

We add a comparison when people regularly weigh two crawlers against each other and the answer is not obvious from their profiles. If you keep comparing two names that are not listed here, send us the pair and the decision you are trying to make. Until then, open both profiles from the directory: the "in context" and "family" sections on each profile answer most pair questions.

The pairs people compare most, and why

Each comparison puts both crawlers side by side, then gives the rules for the pair, the mistakes specific to writing them together and how top sites treated each one in our latest scans.

Same job or different job

Every comparison starts from one fact: whether the two crawlers do the same kind of job. If both are training crawlers, most sites give them the same answer and the only question is who receives your content. If one trains and the other fetches pages for answers, they usually deserve different answers, and treating them alike is the most common mistake we see. Our database lists 40 crawlers that train or both train and index, against 75 that index and 58 that fetch for a person's request.

When a comparison is not the right page

If you are deciding about one crawler, its profile in the directory is shorter and more complete. If you are deciding about a whole company, the operator page lists all its tokens. Comparisons are for the moment you have two names in front of you and need to know whether one rule can cover both.

Test yourself