Robots.txt Best Practices for AI SEO in 2026 (With Data From 10,000 Sites)

A good robots.txt for AI SEO does one job well: it lets the bots that can send you visitors and citations in, and it tells the rest what you want. In 2026 that means treating AI search crawlers and AI training crawlers as two different decisions. Get that split wrong and you can disappear from ChatGPT, Claude or Perplexity answers without noticing, while doing nothing to stop training.

This guide gives you the rules that matter, copy-paste templates for common sites, the mistakes we found in real files, and a test plan. The advice is based on the RFC 9309 standard, each operator's own documentation, and our study of 4,962 robots.txt files from the top 10,000 sites. Before you change anything, run the free AI crawler checker to see what your current file does.

Robots txt SEO: what the file can and cannot do

Robots.txt is a plain text file at the root of your host that asks crawlers which URLs they may fetch. RFC 9309, the standard published by the IETF in 2022, is blunt about its limits: "These rules are not a form of access authorization." Well-behaved crawlers follow it. Others may not, and anyone can read it.

For SEO, the most important fact is that robots.txt controls crawling, not indexing. Google's own documentation says it "can't index the content of pages which are disallowed for crawling, but it may still index the URL and show it in search results without a snippet." So if you want a page out of Google, use noindex and let Google crawl the page so it can see that tag.

Our study shows how often sites still confuse these jobs. 21 of the top 10,000 sites have a noindex: line in robots.txt, which Google stopped supporting on 1 September 2019 (Google, July 2019). It does nothing today.

Robots.txt for AI: allow search bots, decide on training bots

Every major AI company now runs separate bots for separate jobs. That is good news, because it lets you make two decisions instead of one. OpenAI puts it plainly: each setting "is independent of the others", so a webmaster "can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot" for training (OpenAI crawler docs).

The search row is the one that sends traffic. OpenAI says sites "opted out of OAI-SearchBot will not be shown in ChatGPT search answers". Anthropic says disabling Claude-SearchBot "may reduce your site's visibility and accuracy in user search results". Perplexity says it does not use PerplexityBot "to crawl content for AI foundation models". There is no reason to block these bots unless you want to stay out of AI search entirely.

# AI search crawlers: allowed, so assistants can cite you
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /

# AI training: opted out
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Bytespider
User-agent: Meta-ExternalAgent
Disallow: /

# Everyone else, including Googlebot and Bingbot
User-agent: *
Disallow: /admin/
Disallow: /cart/

Sitemap: https://www.example.com/sitemap.xml

Notice that Googlebot is not named. It falls into the * group, which allows everything except the admin and cart paths. That is the right default for most sites.

News publishers are the extreme case. BuzzStream reviewed 100 top US and UK news sites and found 79% block at least one training bot, but 71% also block at least one retrieval or live search bot, and 49% block OAI-SearchBot (BuzzStream via Search Engine Journal). Some of that is a deliberate licensing choice. Some of it is a copied list.

How crawlers read your file: the four rules people miss

Most broken robots.txt files are broken because the writer assumed crawlers read them top to bottom like a script. They do not. Here are the four rules from RFC 9309 and Google's specification that cause the most surprises.

1. A crawler obeys exactly one group

A crawler looks for the group whose User-agent line matches its token, case-insensitively. If it finds one, it uses that group and ignores the * group completely. RFC 9309 says crawlers must obey the * group only "if no matching group exists". Google says the same: "User agent specific groups and global groups (*) are not combined."

User-agent: *
Disallow: /private/

User-agent: GPTBot
Allow: /blog/

# GPTBot reads ONLY its own group. /private/ is NOT blocked for it,
# and nothing outside /blog/ is blocked either.

2. The longest matching path wins

Inside a group, the most specific rule wins, measured by the length of the path. If an Allow and a Disallow are the same length, Allow wins. That lets you open one folder inside a blocked area.

User-agent: *
Disallow: /docs/
Allow: /docs/public/

# /docs/public/guide.html  -> allowed (longer rule)
# /docs/internal.html      -> blocked

3. Paths are case-sensitive, tokens are not

User-agent: gptbot and User-agent: GPTBot are the same. But Disallow: /Blog/ does not block /blog/. Google lists this explicitly: /fish does not match /Fish.asp.

4. Each host and protocol has its own file

A robots.txt file applies only to "the host, protocol, and port number where the robots.txt file is hosted", in Google's words. example.com/robots.txt does not cover blog.example.com or shop.example.com. Every subdomain that serves pages needs its own file.

Templates for five common sites

Pick the template closest to your business, then adjust the paths. Test it in the robots.txt validator before you publish, or build one with presets in the robots.txt generator.

A local business or SaaS that wants every AI to recommend it

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /checkout/

Sitemap: https://www.example.com/sitemap.xml

If you sell a product, being in the training data can help models describe you correctly. Blocking nothing but private paths is a valid choice, and it is what most of the web still does: in our study, 80.8% of top sites block no major training crawler.

A publisher that licenses content

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Bytespider
User-agent: Meta-ExternalAgent
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
Disallow: /

User-agent: *
Allow: /

The user-triggered agents at the end are a preference, not a wall. OpenAI and Perplexity both say those fetchers may not follow robots.txt, because a person asked for the page. If you have a licensing reason to stop them, add a firewall rule too.

An online store with faceted navigation

User-agent: *
Disallow: /cart
Disallow: /checkout
Disallow: /*?*sort=
Disallow: /*?*filter=
Allow: /

Sitemap: https://www.example.com/sitemap-products.xml

Google's crawl-rate guide names faceted navigation as one of the most common causes of a sudden jump in crawling. The same URL explosion wastes AI crawler fetches too.

A documentation site

User-agent: *
Disallow: /v1/
Disallow: /search
Allow: /

Sitemap: https://docs.example.com/sitemap.xml

A staging or private site

User-agent: *
Disallow: /

Remember RFC 9309's warning: listing paths "exposes them publicly". A staging site needs authentication, not just this file.

Mistakes we found in the top 10,000 sites

We read 4,962 real files for our robots.txt study. These are the errors that change crawler behavior, with how often each appeared.

Two status-code mistakes deserve their own warning. If your robots.txt returns a 4xx error, Google treats it "as if a valid robots.txt file didn't exist" and crawls everything. If it returns a 5xx error, Google stops crawling your site for the first 12 hours. A firewall rule that blocks bots from robots.txt itself can trigger either one. More on that in robots.txt mistakes that block AI.

Allow, Disallow and wildcards in plain English

Google, Bing and other major engines support two wildcard characters. * means "any characters" and $ means "end of the URL". Everything else is matched literally from the start of the path. If you need the full side-by-side rules, read our robots.txt disallow vs allow guide.

Google enforces a size limit of 500 kibibytes and ignores anything after it. In our study, no top-10,000 file came near that limit, but very long generated block lists are getting closer every year.

Where Google AI Overviews and AI Mode fit

This is the area that changed most in 2026. Google's AI features use Googlebot. Google says "robots.txt directives for Googlebot is the control for site owners to manage access to how their sites are crawled for Search", and AI Overviews and AI Mode are part of Search (Google, AI features). Blocking Google-Extended does not remove you from them.

On 31 August 2026 Google rolled out a new option to every site: the Search generative AI control in Search Console (Search Console Help, AI control). It lets you exclude your links and content from AI Overviews, AI Mode and generative features in Discover. Google says this control "doesn't affect AI training", and points to Google-Extended for that.

Google's new Generative AI performance report shows impressions from AI Overviews and AI Mode per page, so you can see what you would give up before you exclude anything. Our Google-Extended vs Googlebot guide covers the full picture.

New signals: Content-Signal and AI preferences

Cloudflare adds a Content-Signal line to sites that use its managed robots.txt. It expresses preferences like this:

User-agent: *
Content-Signal: search=yes,ai-train=no,use=reference
Allow: /

The use value was added on 1 July 2026 and has three levels: immediate, reference and full (Cloudflare, 1 July 2026). Cloudflare also changed its defaults: from 15 September 2026, new domains on its network block Training and Agent traffic by default on pages that show ads, while Search stays allowed. If your robots.txt says "allow" but bots still get blocked, check that setting.

These signals are preferences. RFC 9309 says crawlers may interpret other records, and the IETF AI Preferences working group is still standardizing a shared vocabulary (IETF aipref). In our study, 125 top sites (2.5%) already use Content-Signal. Adding it does no harm, but write your real decisions as User-agent groups, because that is what every crawler understands today.

How to test your robots.txt before and after you publish

1. Open the live file in a browser. It must be plain text. If you see a login page, a CAPTCHA or your homepage, bots see it too.
2. Check the status code. Run curl -I https://www.example.com/robots.txt. You want 200, not 403, 404 or 5xx.
3. Test the rules per bot. Paste the file into the robots.txt validator and check GPTBot, OAI-SearchBot, Googlebot and the others you care about.
4. Test the whole site. Run the AI crawler checker, which tests 265 bots against your live file at once.
5. Test past robots.txt. Run the AI crawler diagnostic to fetch a real page as each bot. It shows when a CDN or firewall blocks what robots.txt allows.
6. Wait, then watch your logs. OpenAI says changes can take about 24 hours to show in search. Perplexity says up to 24 hours. Semrush says up to one hour.

for ua in "GPTBot/1.4" "OAI-SearchBot/1.4" "ClaudeBot/1.0" "Googlebot/2.1"; do
  printf "%-20s" "$ua"
  curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 (compatible; $ua)" https://www.example.com/robots.txt
done

Quick reference: which token controls what

Use this table when you write or audit a file. It lists the token you put on the User-agent line, who runs it, and whether that company says the bot follows robots.txt. Profiles for each one are in our bot directory.

Anthropic is the one operator that names its user-triggered agent as a robots.txt control: it says disabling Claude-User "prevents our system from retrieving your content in response to a user query". OpenAI and Perplexity say the opposite for their own agents. That is why the same line can behave differently for three companies.

A short maintenance checklist

• □Review the file every quarter, and whenever a major AI company announces a new bot.
• □Keep search and training tokens in separate groups.
• □Remove tokens operators no longer document, once the current ones are in place.
• □Keep a comment at the top with the date and the reason for each block.
• □List every sitemap with a full URL.
• □Give every subdomain that serves pages its own file.
• □Make sure your firewall lets bots read robots.txt itself.
• □Compare your choices with the top-10,000 study once a year.

If you manage many sites, the batch URL checker compares up to 20 at once and exports a CSV.

Sources and further reading

Every claim above links to the page it came from. These are the primary sources, all read or re-read on 5 October 2026. Where we give a figure from our own data, it comes from our robots.txt study of the top 10,000 sites, and the raw counts are in the downloadable CSV.

1. RFC 9309, Robots Exclusion Protocol (IETF, September 2022)
2. Google Search Central: How Google interprets the robots.txt specification
3. Google Search Central Blog: A note on unsupported rules in robots.txt (2 July 2019)
4. Google Search Central: AI features and your website
5. Search Console Help: Search generative AI control (rolled out 31 August 2026)
6. Search Console Help: Generative AI performance report (31 August 2026)
7. Google Search Central: Google's common crawlers (Google-Extended entry)
8. Google Search Central: Reduce the Google crawl rate
9. OpenAI: Overview of OpenAI crawlers
10. Anthropic: Does Anthropic crawl data from the web? (7 April 2026)
11. Perplexity: Perplexity crawlers
12. Cloudflare: Content Independence Day, new AI options (1 July 2026)
13. IETF AI Preferences (aipref) working group charter
14. BuzzStream via Search Engine Journal: Most major news publishers block AI training and retrieval bots (January 2026)
15. Tranco: a research-oriented top sites ranking