The Guardian Lets OpenAI In and Keeps Others Out

The New York Times blocks almost every AI crawler. The Guardian takes a different route. Its robots.txt carries the same kind of legal notice against AI use, yet OpenAI's crawlers can read the site freely while Anthropic's, Perplexity's and Meta's cannot. We ran our AI search visibility check on theguardian.com to show how a publisher turns robots.txt into a partner list, and what that means for anyone who wants to be cited by one AI assistant but not another.

• 54/100AI Visibility Score (Moderate)
• 9/97AI crawlers blocked
• 5/14Tier 1 bots blocked
• 29/257All bots blocked
• 77/100GEO audit score

The short version

The Guardian blocks 9 of 97 AI crawlers, far fewer than the Times (25), and its AI Visibility Score is 54/100 (Moderate). The blocks are not random. Every OpenAI crawler is allowed: GPTBot, OAI-SearchBot and ChatGPT-User all fall through to the open wildcard group. Every Anthropic crawler (ClaudeBot, Claude-SearchBot, Claude-User) is named and blocked, and so are PerplexityBot, Meta-ExternalAgent and DuckAssistBot.

The pattern lines up with the business side. In February 2025 Guardian Media Group announced a strategic partnership with OpenAI that brings Guardian journalism into ChatGPT, with attribution and links. The crawler that belongs to the partner is in. Crawlers from companies without a public agreement are out.

How the file is built

Like the Times, the Guardian opens with terms rather than rules: content is available under its terms and conditions, and any use "for large language models (LLMs), machine learning and/or artificial intelligence-related purposes" is not permitted. The file then lists a licensing email for anyone who wants permission. So the default stance in the comments is "no". The rules underneath make exceptions by leaving some crawlers unnamed.

That is the clever part. In robots.txt, a crawler that has its own named group follows only that group. A crawler with no named group follows User-agent: *. The Guardian's wildcard group only blocks a few housekeeping paths, so any AI crawler the file does not mention is allowed. The block list is one long group of 37 or so user agents sharing a single Disallow: /. Being left off that list is the permission.

Crawler by crawler

Google-Extended is also allowed, so Guardian content can be used for Gemini. Googlebot is allowed, so the paper appears in Google Search and AI Overviews. Applebot-Extended and CCBot (Common Crawl) are blocked, as are ByteDance's Bytespider and Amazon's Amazonbot.

Our checker also flagged that the file blocks a few search engines outright: 7 of 62, including Mojeek, PetalBot (Huawei), SeznamBot and Amazon's new search crawler. For a UK and US publisher those engines matter little. For a business selling in the Czech Republic, blocking SeznamBot would be a real mistake, which is why we never recommend copying a publisher's list without reading it.

The trade-off in one sentence

A question asked in ChatGPT can be answered with a Guardian link. The same question asked in Claude or Perplexity cannot, because those assistants are told not to fetch Guardian pages, not even when a user asks for one. That is the real meaning of a selective AI policy: you choose which AI audiences can find you.

For a news publisher that is a defensible choice. The partner pays, attributes and links. Others crawl without an agreement, and Cloudflare's 2025 data shows how lopsided that exchange can be: Anthropic's crawlers were measured fetching tens of thousands of pages for every visitor sent back. For most other businesses the arithmetic runs the other way. Every assistant you block is a place a prospect asks a question and hears about a competitor instead.

The technical side is in good shape

• Passrobots.txt allows GooglebotNo rule in the User-agent: * group matches /
• PassPage answers GooglebotHTTP 200 to the Googlebot smartphone user agent, after a redirect to https://www.theguardian.com/us
• PassNo challenge or block pageThe response is the real page, not a challenge
• PassPage is indexable (no noindex)Directives present but indexable: max-image-preview:large
• WarningCanonical points to this URLPoints to another URL: https://www.theguardian.com. Google will usually index that one instead
• PassContent is in the HTML before JavaScript21,829 characters of text are present without JavaScript.

Tool verdict: Crawlable and indexable: Googlebot can fetch this page and may index it.

Googlebot gets a clean HTTP 200 and the page carries 21,829 characters of text are present without JavaScript. The only warning is the canonical: the homepage redirects US visitors to /us, which declares the main homepage as canonical. That is normal for an edition-based site and not a problem.

The Guardian scores 77 on our GEO audit, far above the Times, because our auditor could actually read the page. The weak spot is structured data: our tool found no Schema.org markup on the homepage. Article pages on large news sites usually carry NewsArticle markup, but the homepage is where an AI system learns who the publisher is, and an Organization block there is cheap to add. Our schema generator builds one in a minute.

How to build a selective policy for your own site

1. Write down who you want to be cited by. Most brands want ChatGPT, Google, Perplexity, Claude and Bing Copilot, because that is where buyers ask questions.
2. Separate training from search. If you object to training, block the training tokens (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot) and keep the search and user tokens open. Our guide to training bots versus search bots lists which is which.
3. Use one named group for the block list. Several User-agent lines can share one Disallow: /, as the Guardian does. It keeps the file short and easy to audit.
4. Keep the wildcard group permissive. Then any new crawler you have not decided about is allowed by default, and you review it when it appears.
5. Re-check when a partner changes. A deal that ends, or a new one that starts, should change the file the same week.

Sources: robots.txt and homepage fetched by AI Crawler Check on 17 October 2026. Partnership: Guardian Media Group press office, 14 February 2025. Crawl-to-referral data: Cloudflare, August 2025. We describe a public file. We do not know the terms of any agreement beyond what the companies have published.