How The New York Times Blocks AI Crawlers

The New York Times is the most visible publisher in the fight over AI and news. It sued OpenAI and Microsoft in December 2023, it licensed its journalism to Amazon in May 2025, and its robots.txt now names 69 user agents. We ran our AI bot checker on nytimes.com to see what that policy looks like to a crawler, and what it costs the paper in AI search.

• 29/100AI Visibility Score (Critical)
• 25/97AI crawlers blocked
• 10/14Tier 1 bots blocked
• 49/257All bots blocked
• 40/100GEO audit score

The short version

The Times blocks 25 of the 97 AI and LLM crawlers we track, including every crawler from OpenAI, Anthropic and Perplexity. It blocks Google-Extended too, so its content cannot be used to train or ground Gemini. It does not block Googlebot, which keeps the paper in classic Google results and in Google's AI Overviews. And it leaves one AI company's crawler mostly open: Amazon, the only one on the list with a published licensing deal.

That gives nytimes.com an AI Visibility Score of 29/100 (Critical). That is a low number, but it is not an error. The score measures how visible a site is to AI assistants. The Times has decided it does not want to be visible to most of them unless they pay. The useful question is what that choice costs, and who should copy it.

What the robots.txt file actually says

The file opens with a legal notice rather than a crawl rule. It states that the content is for personal, non-commercial use, and that using it to develop "software, machine learning, artificial intelligence (AI), and/or large language models (LLMs)" is prohibited without written permission. It also cites Article 4 of the EU Directive on Copyright in the Digital Single Market, the text and data mining exception that rights holders in Europe can opt out of in a machine-readable way. Robots.txt is one of the places European publishers make that opt-out.

After the notice come three blocks of rules:

• A shared group for User-agent: * and Googlebot, with 93 path rules that keep crawlers out of ad servers, login and checkout pages and search results. These are housekeeping rules, not an AI policy.
• A long list of named groups that end in Disallow: /. This is the AI policy. Every major AI crawler is named one by one rather than caught by a wildcard.
• Three narrow exceptions: an ad bot allowed everywhere, and the Facebook, X (Twitter) and Reddit preview bots allowed only on URLs that carry the paper's share parameter, so shared links still show a title and image.

Naming each bot is the more careful way to write this. A wildcard block would also hit Googlebot, Bingbot and every SEO tool. Named groups block only what you list, and because a named group replaces the wildcard group for that crawler, each bot gets exactly the rules written for it.

Crawler by crawler

This is the result for the 18 crawlers most site owners ask about. "Named group" means the Times wrote a rule for that bot. "User-agent: *" means the bot falls back to the wildcard group.

Three findings stand out.

1. Search bots are blocked along with training bots

Many publishers block the training crawlers (GPTBot, ClaudeBot, CCBot) and keep the search crawlers that send readers back, so their articles are still cited in ChatGPT search and Claude. The Times does not make that distinction. OAI-SearchBot, Claude-SearchBot and PerplexityBot are blocked, and so are the user-triggered fetchers ChatGPT-User, Claude-User and Perplexity-User. The message is consistent: no access without a contract, for any purpose.

One caveat applies to that last group. Perplexity's own documentation says Perplexity-User "generally ignores robots.txt rules" because a person asked for the page. A Disallow line for it is a statement of intent, not a lock. If a publisher really needs it stopped, the block has to happen at the firewall or CDN, which is exactly where the Times also acts (see the Googlebot test below).

2. Amazon is the exception, and it lines up with the deal

Amazonbot has its own group with a single rule, Full content access; 1 path rule (/wirecutter/). That keeps Amazon out of Wirecutter, the paper's affiliate shopping section, and lets it read everything else. The Times announced a multi-year licensing agreement with Amazon in May 2025 covering its newsroom, NYT Cooking and The Athletic for products such as Alexa, reported at roughly $20 to $25 million a year. The robots.txt is the technical side of that commercial agreement: the one AI company with a contract is the one crawler left in.

Google gets a similar, narrower exception. Google-CloudVertexBot, the crawler for Vertex AI agents that site owners or customers configure, is blocked except on /wirecutter/ and /athletic/. Our tool reports it as partial.

3. The file still carries retired tokens

Our checker flagged 2 outdated tokens: anthropic-ai and claude-web. Anthropic stopped documenting them after it consolidated its crawlers into ClaudeBot, Claude-User and Claude-SearchBot. The Times blocks those current names too, so nothing leaks. But a file this long tends to collect dead entries, and dead entries make it harder to see whether the live ones are right. If you copy a publisher's list, validate it first.

The chart shows that the policy is aimed precisely. 25 AI crawlers and 15 of 39 scrapers are blocked, while 24 of 24 Google crawlers and 61 of 62 search engines can still get in. The single search-engine block is Applebot-Extended, Apple's training opt-out token. Applebot itself, which powers Siri and Spotlight, is allowed.

The second wall: Googlebot meets a bot challenge

Robots.txt is only half the picture. Our crawlability test also fetches the homepage as Googlebot and checks what actually comes back.

• Passrobots.txt allows GooglebotNo rule in its own Googlebot group matches /
• FailPage answers GooglebotHTTP 403 to the Googlebot smartphone user agent. A firewall, CDN or rate limit is refusing it.
• FailNo challenge or block pageDataDome challenge was served instead of the page
• Not readPage is indexable (no noindex)Page not read
• Not readCanonical points to this URLPage not read
• Not readContent is in the HTML before JavaScriptPage not read

Tool verdict: Not crawlable: the page returns HTTP 403 to Googlebot.

Our request carried Googlebot's user agent, but it came from our server, not from Google's published IP ranges. The site's bot protection (our tool identified a DataDome challenge) treated it as a fake Googlebot and refused it. That is correct behaviour. Real Googlebot is verified by reverse DNS, and impostors using its name are one of the most common scraping tricks. The result shows that the Times does not rely on robots.txt alone. Bots that ignore the file, or pretend to be Google, meet a second wall at the edge.

For your own site, the lesson cuts both ways. A bot-management layer this strict is how you enforce a policy, and it is also how sites block real crawlers by accident. If you turn one on, confirm with Search Console's URL Inspection that the real Googlebot still gets a 200. Our guide to firewalls that block AI crawlers by accident covers the checks.

How the homepage scores for AI search readiness

The GEO audit is capped at 40 because, in the audit's own words, "robots.txt blocks the crawlers that feed AI answers." That cap is the intended result of the policy. The low content and structured-data scores have a different cause: our auditor received the bot challenge, not the page, so it could not read headings, schema or body text. A reader in a browser sees a rich homepage. A crawler without a contract sees a wall. That is the whole strategy in one measurement.

What the policy costs, and what it buys

Blocking AI search bots has a price. When someone asks ChatGPT, Claude or Perplexity about a story the Times broke, those assistants cannot fetch the original, so they cite whoever summarised it. Cloudflare's 2025 measurements show why publishers accept that trade: AI platforms crawl far more pages than they send visitors back for. Cloudflare put Anthropic's crawl-to-referral ratio in the tens of thousands to one and OpenAI's in the hundreds to one, while traditional search sends readers back at a much higher rate. For a subscription business, a crawler that reads everything and returns almost no readers is a cost, not a channel.

What the policy buys is leverage. The Times' lawsuit argues that its archive was used for training without permission. A robots.txt that names and refuses each crawler, plus a written notice reserving text and data mining rights, documents that permission was withheld. A licence with Amazon shows the content has a market price. Together they turn "AI uses our work" into "AI pays for our work, or does not get it."

Should you copy the New York Times?

Almost certainly not wholesale. The Times can afford to be invisible in AI answers because readers seek it out by name, it has a legal team, and it has buyers for a licence. Most businesses are in the opposite position: being cited by an AI assistant is how new customers find them. Use the Times as a pattern, not a template:

• Decide per purpose, not per company. Training, search and user-triggered fetching are different jobs with different trade-offs. Our guide to blocking AI training while allowing AI search shows the middle path most brands should take.
• Name crawlers explicitly. Never use a wildcard Disallow to block AI, or you will block Google and Bing as well.
• Put your terms in the file. A short comment that reserves text and data mining rights costs nothing and records your intent.
• Enforce at the edge if it matters. Robots.txt is a request. A firewall rule is a lock.
• Retest after every change. A AI crawlability test takes 30 seconds.

Sources: robots.txt and homepage fetched by AI Crawler Check on 17 October 2026. Licensing deal: The New York Times, 29 May 2025. Crawl-to-referral data: Cloudflare, August 2025. Perplexity-User behaviour: Perplexity crawler documentation. Google-Extended scope: Google Search Central. This study describes a public file and a public page. It is not legal advice and makes no claim about the merits of any lawsuit.