Content-Signal vs Content-Usage: Which AI Preference Line to Use

There are now two ways to write an AI usage preference into robots.txt. Content-Signal comes from Cloudflare and is written into every site that switches on Cloudflare's managed robots.txt. Content-Usage comes from the IETF AIPREF working group and is on its way to becoming an open standard. They look alike, they overlap, and they disagree on one important point: whether feeding your page into a live AI answer is its own category.

This guide puts the two side by side, shows how the top 10,000 sites use them today from our own scan, and gives you a robots.txt that speaks both without contradicting itself.

The two formats at a glance

How the categories line up

The definitions are close, but not identical. Here they are in each document's own terms, summarised.

search

Both formats describe the same thing: building a search index and showing results with links and short excerpts. contentsignals.org says search "does not include providing AI-generated search summaries". The AIPREF draft says the search category "does not include the use of assets to generate summaries". So in both, search=yes means classic results, not an AI answer written from your page.

ai-input

Only Content Signals has it. It covers "inputting content into one or more AI models", with retrieval augmented generation and grounding as examples, which is how ChatGPT search, Perplexity and Google's AI features turn your page into an answer in real time. This is the category that matters most for AI visibility, because it decides whether an AI answer may quote and cite you today.

AIPREF draft 07 does not have an equivalent. Under AIPREF, an AI answer engine that grounds on your page is neither "search" (because it summarises) nor "train-ai" (because it does not change model weights). The preference for that use is therefore unknown. Expect this gap to be one of the main debates as the vocabulary matures.

ai-train and train-ai

Same idea, different spelling. Content Signals defines it as "training or fine-tuning AI models". AIPREF defines it as using content to change the learned parameters of a model that generates content. For a site owner the two are interchangeable.

What the top 10,000 sites actually say

We fetched every robots.txt in the Tranco top 10,000 on 17 October 2026 and parsed 5,254 valid files. 164 files (3.1%) contain at least one Content-Signal line. These are the most common combinations, counted once per file.

Two patterns stand out. First, the split position (yes to AI answers, no to training) is almost as common as fully open. That matches what we see with crawler tokens: 457 files block at least one training crawler while leaving every AI search crawler we track open. Second, 40 files say nothing about ai-input. That is the line Cloudflare's managed robots.txt writes by default, and it leaves the most important category for AI visibility unstated.

For Content-Usage the picture is simpler: 5 files, and every value we could read says train-ai=n. 4 files use both formats.

Inside Cloudflare's managed robots.txt

36 files in the top 10,000 contain the marker Cloudflare adds to its managed robots.txt. Every one of them starts with the same long comment block, and it is worth reading, because it explains how Cloudflare wants the signals understood.

• Yes means you may collect for that use; no means you may not. The comment frames the signals as a condition of access, not a technical block.
• Silence is neutral. If a signal is missing, the comment says the operator "neither grants nor restricts permission" for that use. That is the same idea as AIPREF's "unknown".
• The comment cites Article 4 of the EU copyright directive, calling any restriction an express reservation of rights. We cover the legal side below.
• The signal line is short. Most of these files write search=yes,ai-train=no,use=reference, with no ai-input value at all.

The same managed file also adds a list of training crawlers with Disallow: /, such as GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot and Bytespider. So on these sites the signal is backed by token rules that crawlers honour today. If you rely on Cloudflare's version, open your live robots.txt once and read what it actually says; the list changes as Cloudflare updates it.

The optional use= field

Cloudflare added a fourth field, use=, with three values: immediate (interact, keep nothing), reference (index, excerpt and link back) and full (summarise and reproduce). It has no AIPREF equivalent. In our scan 39 files used use=reference, almost all through the managed file. Cloudflare's Search, Agent and Training defaults explains how Cloudflare enforces it.

How a reader resolves the two together

No AI company has published how it would combine both formats, but the AIPREF vocabulary gives the obvious rule: if any statement says no, the answer is no. Here is what that means for three files.

The middle row is the trap. A team adds an AIPREF line for "safety" without noticing the older Content-Signal line said yes. The combined answer flips to no, which may not be what marketing wanted. Keep one owner for robots.txt and write both lines at the same time.

A quick decision guide by site type

These are starting points, not rules. Our AI SEO for publishers and how to opt out of AI training guides go deeper on the trade-offs, and zero-click survival guide covers what to measure once AI answers start quoting you.

Which one should you use?

You do not have to choose. Both lines are ignored by parsers that do not know them, so they cannot break Googlebot or any other crawler. The question is what each one buys you.

• Content-Signal has a live enforcer. If your site is behind Cloudflare, Cloudflare says it watches how verified bots use content against these signals. It is also the only format that lets you speak about AI answers separately.
• Content-Usage is where the open standard is heading. It supports paths and an HTTP header, and its strictest-wins rule is clear. Adding it now puts your position on record for any company that adopts AIPREF.
• Crawler tokens remain the only thing every major AI company documents and honours today. Use them for the decisions that matter, as described in block AI training, allow AI search.

A robots.txt that speaks both

This is the setup we recommend for most businesses that want to be cited in AI answers but not used for training. Both preference lines say the same thing, and the token rules do the actual work.

# Training crawlers: no
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /

# Everyone else, including AI search and AI assistants
User-agent: *
Allow: /
Disallow: /account/
Content-Signal: search=yes, ai-input=yes, ai-train=no
Content-Usage: search=y, train-ai=n

Sitemap: https://example.com/sitemap.xml

Note what stays open: OAI-SearchBot, Claude-SearchBot, PerplexityBot and the user agents such as ChatGPT-User all read the star group. If you put preference lines only in the star group, crawlers you name in their own group will not see them, because a crawler reads only the most specific group that names it. Copy the lines into any named group that allows crawling.

If you want to go further: per-folder rules

Content-Signal applies to the whole group, but Content-Usage can take a path. If you publish open research you are happy for models to learn from, you can say so for that folder alone.

User-agent: *
Allow: /
Content-Signal: search=yes, ai-input=yes, ai-train=no
Content-Usage: search=y, train-ai=n
Content-Usage: /research/ train-ai=y

Be aware that the two formats now disagree for /research/: Content-Signal says no to training for the whole group. A company reading both and applying "strictest wins" would treat that folder as no. If the folder matters, give it its own explanation in an llms.txt file or on the page, and keep both lines consistent elsewhere.

Legal context: the EU text and data mining reservation

Part of the push for machine-readable preferences comes from Article 4 of EU Directive 2019/790, which lets rightholders reserve text and data mining rights "in an appropriate manner, such as machine-readable means". Cloudflare's managed robots.txt adds a comment that ties its signals to that reservation, and both formats are designed so that a "no" can be read by machines. We are not lawyers, and this is not legal advice: whether a given line counts as a valid reservation is for courts and regulators to decide. If the legal side matters to your business, read content licensing vs blocking and take advice.

Common mistakes

• Lines outside a group. A preference line above the first User-agent belongs to no group, so crawlers that follow RFC 9309 never associate it with any rules. Keep every preference line inside a group.
• Only in the star group. Named groups do not inherit from *. Repeat the line in each named group that allows crawling.
• Mixed spelling. ai-train=n and train-ai=no are both wrong.
• Uppercase in Content-Usage. AIPREF is case sensitive; Train-AI makes the preference unknown.
• Expecting a block. A "no" is a request. Abusive crawlers ignore it. Use edge rules for those, carefully, as explained in WAFs that block AI crawlers by accident.

What to do now

• □Decide three answers: classic search, AI answers, training. Write them down before you touch robots.txt.
• □Put the decisions that must work today into crawler-token groups, using our robots.txt generator.
• □Add matching Content-Signal and Content-Usage lines to every group that allows crawling.
• □If you are on Cloudflare's managed robots.txt, add ai-input=yes if you want to be cited in AI answers; the default leaves it unstated.
• □Validate the file, then run AI Crawler Check on a key page to confirm search crawlers are allowed.
• □Read our AIPREF explainer for the details of the IETF format, and check back as the drafts change.