What the world's top sites decide about AI crawlers

Why a second radar

Traffic tells you who visits. Rules tell you who is welcome.

Network-scale radars, such as Cloudflare Radar, measure crawler traffic from inside the network. That is the server's view, and nobody outside a large network can match it. Scout Radar measures the opposite: the decisions publishers write down. The two answer different questions, so use them together.

What Scout Radar does not show: traffic, request volume or whether a crawler obeys the rules. robots.txt is a request, not a lock. For traffic, use a network radar.

The radar

Which crawlers top sites keep out

Each dot is a crawler. The further it sits from the centre, the more top sites block it. Dots are grouped by job, because sites treat a training crawler very differently from a search crawler or an assistant fetching a page for one user.

Source: Scout Radar, Tranco top 10,000, fetched 20 October 2026, 5,583 valid robots.txt files, RFC 9309 parser. "Top 1,000" is the share among the highest-ranked sites in the list; a higher number means the biggest sites block it more often.

The big split

Sites say no to training and yes to answers

The most common decision is not "block AI". It is "block training, stay visible in AI answers". 508 files (9.1%) block at least one AI training crawler while leaving every AI search crawler open. Only 11.1% block both.

Source: Scout Radar, Tranco top 10,000, fetched 20 October 2026, 5,583 valid robots.txt files, RFC 9309 parser.

What goes wrong

The robots.txt mistakes Scout finds most

A traffic radar cannot see these, because the mistake is in the file, not the request. Each one means a site is blocking something it wanted, or allowing something it meant to stop.

1. Name a retired token

   683 files 12.2%

   Rules for tokens the operator no longer uses, such as anthropic-ai or Claude-Web, block nothing.

2. Answer robots.txt with HTTP 403

   1,117 domains 11.2%

   Usually a firewall. Under RFC 9309 some crawlers treat an unreachable file as "allow all", others as "stay away".

3. Block CSS or JavaScript paths

   483 files 8.7%

   Crawlers that render pages see a broken layout, which can hurt how the page is understood.

4. Serve an HTML page instead of rules

   501 domains 5.0%

   A soft 404 or a login page at /robots.txt. Crawlers read it as no rules, or give up.

5. Block everyone with Disallow: /

   240 files 4.3%

   Often a staging file that went live. Search engines and AI engines alike are told to leave.

6. Have no User-agent group at all

   217 files 3.9%

   Rules without a group apply to no one. The file looks strict and does nothing.

7. Block Googlebot itself

   154 files 2.8%

   Sometimes deliberate. Often a site meant to block Google-Extended (training) and blocked search by mistake.

8. Name only the retired token, not its replacement

   151 files 2.7%

   The site thinks it opted out. The live crawler is not named at all.

Source: Scout Radar, Tranco top 10,000, fetched 20 October 2026, 5,583 valid robots.txt files, RFC 9309 parser. HTTP 403 and HTML responses are counted over all 10,000 domains requested; the rest are counted over the 5,583 valid files.

New signals

What publishers are starting to write

robots.txt is changing. These newer lines are still rare, which is exactly why they are worth watching: a radar shows a trend before it becomes the default.

Source: Scout Radar, Tranco top 10,000, fetched 20 October 2026, 5,583 valid robots.txt files, RFC 9309 parser.

Movement

What changed between scans

Scout Radar keeps every scan, so a change is a measurement, not a guess. These rows compare the same metric in the last two scans, built by the same script, so every row is defined the same way. A few days move little; the value is in the method, which repeats with every scan.

Both scans fetched every domain in the Tranco top 10,000. The number of files we can read differs a little from day to day (5,254 and 5,583) because some servers time out or refuse one request and answer the next, so compare the percentages, not the raw counts.

Daily pulse

What Scout re-read in the last few days

A full scan of 10,000 sites is a big job, so between full scans Scout keeps watch on the 1,000 highest-ranked sites that publish a valid robots.txt. Every day it re-reads a slice of that list and records every crawler that a site newly blocks or newly allows. A change here is a real edit to a real file, compared with Scout's own last reading of the same site.

• 300sites read by the pulse30% of the watch list
• 252files re-read on 24 October 2026
• 0crawler decisions changedlast 4 days
• 24 Octlast reading2026, UTC

No crawler decision has changed in the sites Scout re-read so far. That is normal: most sites edit robots.txt a few times a year.

Blocked today among the 300 sites the pulse has read: Bytespider 25.3%, CCBot 23.3%, ClaudeBot 21.3%, GPTBot 21.0%, Meta-ExternalAgent 20.7%, Google-Extended 19.3%.

How it runs: up to 6 files at a time, at most once a minute and 250 files a day, with the user agent AICrawlerCheck-Research/1.0. Watch list: Tranco list GQG5K, generated 2026-10-19. The live numbers are also in the pulse feed.

Early warning

Tokens sites name before operators confirm them

Some crawlers have no official documentation, yet site owners already write rules for them. Scout Radar counts those rules. It is how a token earns a place in the database as "Reported", and how a dead token is caught when sites keep naming it after the operator moved on.

Source: Scout Radar, Tranco top 10,000, fetched 20 October 2026, 5,583 valid robots.txt files, RFC 9309 parser. Naming a token shows that site owners write rules for it. It does not prove who runs it.

The database behind the radar

Verified, not oversized

We do not claim the biggest list of crawlers; large networks see more bots than any public site can. We claim a list you can check. Every entry has a source, and every removal has a reason.

• 193confirmed by the operator's own documentation
• 72reported by a named registry or by our robots.txt study, labelled as such
• 39strings we deliberately removed, each with a written reason

Browse every entry in the crawler directory, or read how we verify a token.

Method

How Scout Radar is measured

1. Take a public list of top sitesThe Tranco top 10,000 is a research ranking built to resist manipulation, so anyone can use the same list.
2. Fetch every robots.txtOne request per domain, with an honest user agent. Timeouts, HTTP errors and HTML pages are recorded, not dropped.
3. Parse by the standardFiles are read with an RFC 9309 parser, so a rule counts the way a compliant crawler would read it, including wildcard groups.
4. Count decisions, not mentionsA crawler is "blocked" only if its group disallows the whole site. A file that names a crawler but allows it is counted as named, not blocked.
5. Publish the numbers and the scriptsThe aggregate numbers are free to download as JSON. They come from five scripts: robots-study-fetch.py (fetch), trend-stats.py (decisions and signals), hygiene-stats.py (mistakes), stale-token-stats.py (retired tokens) and count-reported-tokens.py (sightings).

Cite Scout Radar as: AI Crawler Check, Scout Radar, 20 October 2026, aicrawlercheck.com/radar. The data is free to reuse under CC BY 4.0.