Case Studies

New series, October 2026

How five of the web's biggest sites treat AI crawlers

On 17 October 2026 Scout Agent ran the same AI bot checker on five sites that sit at the centre of the argument about AI and content: The New York Times, Reddit, The Guardian, the BBC and Wikipedia. Each one gave a different answer to the same question, may an AI company read our pages? Together they map every strategy a site owner can choose, from a total block to a fully open door.

Five strategies, one spectrum

1. Block everything, license by contract (Reddit). A two-rule robots.txt refuses all 257 bots we tested, Googlebot included, and the network edge enforces it. Partners get in through paid agreements. It only works if buyers are queuing for your data.
2. Block almost all AI, keep search, license selectively (The New York Times). 25 AI crawlers are refused by name, Googlebot stays in, and Amazon, the one company with a published licence, is allowed. A bot challenge backs up the file at the edge.
3. Block AI except one commercial section (BBC). A detailed written refusal of every AI use, then Allow: /storyworks for OpenAI and Google, so the branded content advertisers pay for stays visible. Our checker also found newer AI search agents the file never names.
4. Allow the partner, block the rest (The Guardian). OpenAI, its announced partner, is left unnamed and therefore allowed. Anthropic, Perplexity and Meta are listed in one Disallow group. 9 AI crawlers blocked in total.
5. Stay open, protect expensive pages, sell a faster door (Wikipedia). 0 AI crawlers blocked. Only bots that misbehave are refused, costly dynamic pages are off-limits to everyone, and companies that need scale pay for an API.

What the series teaches site owners

Three lessons repeat across all five. First, robots.txt is a statement, not a lock. Three of the five sites (the Times, Reddit and Wikipedia) refused our Googlebot test at the network edge, because the request did not come from Google's verified IP ranges. Serious policies are enforced by a firewall or bot manager, and the file records intent. Second, naming matters more than blocking. A crawler with its own group follows only that group, and a crawler with none follows User-agent: *. The Guardian lets OpenAI in simply by not naming it, and the BBC lets Anthropic's search agents in by accident for the same reason. Third, these strategies only make sense for content AI companies will pay for. For most businesses, being cited by ChatGPT, Gemini, Claude and Perplexity is how new customers arrive, and the right policy is the opposite of the Times': keep AI search open and decide about training separately.

How every study is built

Each study follows the same method so the results can be compared. Scout Agent runs a single AI Crawler Check against the live site and saves the raw result, including the full robots.txt, a SHA-256 fingerprint of the file, the verdict for each of the 257 bots, the six-step Googlebot crawlability test and the 42-point GEO audit. Every number on the study pages is read from that saved result when the page is built, so the article cannot drift from its evidence. Background facts, such as licensing deals and published traffic data, are cited once each to the organisation that published them. A human editor at Horatos.ai reviews every study before it goes live, and you can rerun any test yourself with the free checker.

Each study also ends with a checklist you can tick off as you work through your own robots.txt, and a short test yourself section that checks you understood the key rule. Both are plain HTML, so they work with JavaScript switched off and on any device.

How to read the numbers in each study

AI crawlers blocked

How many of the 97 AI and LLM crawlers in our database the site's robots.txt refuses on the homepage path. It covers training crawlers, AI search crawlers and user-triggered assistants such as ChatGPT-User.

Tier 1 bots blocked

How many of the 14 most influential crawlers, the ones that feed the largest AI assistants and search engines, the file refuses. A single Tier 1 block can remove a site from a whole assistant.

AI Visibility Score

A 0 to 100 score for how readable the site is to AI assistants. It weighs which crawlers are allowed, how important each one is and whether the server actually answers them. Lower means fewer AI answers can quote the site.

GEO score

The result of the 42-point Generative Engine Optimisation audit: crawler access, technical SEO, content structure, structured data and trust signals. It shows how well a page is prepared to be cited once a crawler gets in.

The two scores answer different questions. The AI Visibility Score asks can AI read you, and the GEO score asks will AI quote you. A site can be fully open and still score poorly on GEO if its pages lack structure, and a well-built site can score badly on visibility because one line of robots.txt shuts the door.

Run your own case study in ten minutes

1. Open the free checker on the homepage and enter your domain. Keep the result tab open, because it is your evidence.
2. Note the AI crawlers blocked count and look for any Tier 1 crawler marked Blocked that you did not intend to refuse.
3. Read the Googlebot crawlability steps. A failure at the edge, such as HTTP 403 or a challenge page, means your firewall is overriding your robots.txt.
4. Compare your file with the strategy above that matches your goal, then edit one group at a time and rerun the check after each change.
5. Record the date and score before and after. That pair of numbers is your own case study, and it is the clearest way to show a client or manager what changed.

Frequently asked questions about the series

Why did you choose these five sites?

Each one represents a distinct, public strategy, and each is large enough that its robots.txt is maintained deliberately. Together they cover the full range from total refusal to near total openness, so any reader can find a policy close to their own.

Will the numbers change?

Yes. Large sites edit robots.txt often, especially as new AI crawlers launch. Every study shows the date of the test and the file fingerprint, so you can tell whether the live file still matches. Run the checker on any of these domains to see today's result.

Does blocking AI crawlers hurt Google rankings?

Blocking AI-specific crawlers such as GPTBot or ClaudeBot does not affect classic Google Search, as long as Googlebot itself stays allowed. Blocking Googlebot itself is different: pages it cannot crawl drop out of Google Search over time, which is why four of the five sites in this series keep it allowed.

Can I copy one of these robots.txt files?

Use them as patterns, not templates. Each file reflects that publisher's licensing deals and infrastructure. Copy the structure that fits your goal, then name the crawlers you actually want to allow or refuse.

Start with the study closest to your situation

• You run a publication and want to stop AI training: read the New York Times study, then the Guardian study for a middle path.
• You want AI assistants to see only part of your site: read the BBC study on path-level rules.
• Your robots.txt says Disallow: / and you are not sure why: read the Reddit study, then fix it.
• Bots are costing you bandwidth but you want to stay visible: read the Wikipedia study.
• Your site passes every SEO test but AI crawlers still fail: read The Invisible AI Crawler Block, our original investigation of a hidden redirect loop.