The BBC Blocks AI, Except on One Commercial Path

The BBC's robots.txt opens with the plainest anti-AI statement of any major publisher: "Please use our site like a human, not a robot." It then rules out training, retrieval-augmented generation, AI-powered search, agentic AI and even AI summaries for personal use. And yet our website crawlability test found OpenAI's and Google's AI crawlers allowed on one part of bbc.com. This case study explains the exception and what it teaches about writing precise rules.

• 64/100AI Visibility Score (Moderate)
• 15/97AI crawlers blocked
• 3/14Tier 1 bots blocked
• 33/257All bots blocked
• 82/100GEO audit score

The short version

bbc.com blocks 15 of 97 AI crawlers outright and gives 5 crawlers partial access. Its AI Visibility Score is 64/100 (Moderate), higher than you might expect from the notice, because the file is written path by path. GPTBot, OAI-SearchBot, ChatGPT-User and Google-Extended are each told Disallow: / and then Allow: /storyworks. Everything is closed to them except one section.

That section is BBC StoryWorks, the commercial content studio of BBC Studios, which makes branded content for advertisers. In plain terms: the BBC keeps AI away from its journalism, but lets the biggest AI assistants read the content brands have paid to place. For an advertiser, being visible in ChatGPT and Gemini is part of what they buy.

The notice, line by line

Most publishers write one sentence about AI. The BBC lists every use it refuses. The file says no scraping or systematic extraction; no use for training or fine-tuning AI models; no retrieval-augmented generation, AI-powered search, agentic AI or grounding; no datasets; no text and data mining under Article 4 of the EU copyright directive; and no summaries "for your own use". It ends: "TL;DR: Browse, read, watch, enjoy, like a human."

The list matters because AI use has split into many separate jobs, and each AI company maps those jobs to different crawlers. A notice that only says "no training" leaves room to argue about search and grounding. The BBC closes every one of those gaps in writing, then backs the words with named rules for each crawler it knows.

Crawler by crawler

Three groups emerge:

• Blocked everywhere: ClaudeBot, PerplexityBot, Perplexity-User, Meta-ExternalAgent, CCBot, Bytespider, Amazonbot and a long tail of scraping and AI search APIs (Firecrawl, Diffbot, Exa, Tavily, You.com).
• Allowed only on /storyworks: GPTBot, OAI-SearchBot, ChatGPT-User and Google-Extended. Our tool reports these as partial.
• Not named, so the wildcard applies: Claude-SearchBot, Claude-User, DuckAssistBot and newer agents such as Mistral's. The BBC's wildcard group sets housekeeping rules only, so these crawlers can read the site.

The third group is the interesting finding. The BBC blocks ClaudeBot (Anthropic's training crawler) by name, but not Anthropic's newer search and user agents, which Anthropic introduced when it split its crawling by purpose. A notice that forbids AI search, sitting above a file that leaves an AI search crawler unnamed, is the typical result of a policy written by one team and a file maintained by another. Our checker also flagged 2 retired tokens (anthropic-ai, claude-web) still in the file, which points the same way: the list is updated by hand as names appear, and some never get added.

How the StoryWorks exception works

Under the robots.txt standard (RFC 9309), when an Allow and a Disallow both match a URL, the more specific rule, the one with the longer path, wins. /storyworks/ is longer than /, so every URL under StoryWorks is allowed and every other URL is blocked. Listing both /storyworks and /storyworks/ covers the section's landing page with and without a trailing slash. It is a clean, correct example of carving one section out of a full block. Our guide to robots.txt pattern matching covers the precedence rules in detail.

The same pattern is useful for any business. A software company might block AI training on its documentation but allow its pricing and product pages. A retailer might block its account and checkout paths for every bot while leaving product pages open. Path-level rules let you say exactly that.

Technical health: strong

• Passrobots.txt allows GooglebotNo rule in the User-agent: * group matches /
• PassPage answers GooglebotHTTP 200 to the Googlebot smartphone user agent
• PassNo challenge or block pageThe response is the real page, not a challenge
• PassPage is indexable (no noindex)Directives present but indexable: noodp, noydir
• PassCanonical points to this URLSelf-referencing: https://www.bbc.com
• PassContent is in the HTML before JavaScript15,821 characters of text are present without JavaScript.

Tool verdict: Crawlable and indexable: Googlebot can fetch this page and may index it.

bbc.com passes every Googlebot gate and scores 82 on our GEO audit, the best result among the five publishers we tested. The homepage is server-rendered, has a self-referencing canonical, carries structured data and loads real text without JavaScript. Whatever the BBC thinks of AI, its site is built in exactly the way AI and search crawlers read best. That is worth remembering: a strict AI policy and good technical SEO are separate decisions.

The legal context

In June 2025 the BBC threatened legal action against Perplexity, asking it to stop scraping BBC content, delete what it held and propose compensation. It was the broadcaster's first such move against an AI company. Perplexity disputed the claims. The robots.txt we tested names PerplexityBot and Perplexity-User and blocks both. As with the New York Times, a detailed file plus a written notice is evidence of what the publisher refused, whatever a court eventually decides.

What to take from the BBC file

• Say what you refuse, use by use. Training, search, grounding, agents and summaries are different. Name the ones you mean.
• Use Allow to carve out sections. Disallow: / plus Allow: /section/ is precise and standard.
• Audit for new crawler names every quarter. AI companies split and rename crawlers often. The BBC's file shows how easy it is to fall behind. Our AI crawler directory lists every current name.
• Keep your technical SEO excellent regardless. Good structure helps every crawler you do allow.

Sources: robots.txt and homepage fetched by AI Crawler Check on 17 October 2026. StoryWorks: BBC StoryWorks. Legal letter: BBC News, 20 June 2025. Rule precedence: RFC 9309, Robots Exclusion Protocol.