Bot directoryCategory

AI & LLM Bots Directory

AI training crawlers collecting content for LLM datasets. Blocking has zero SEO impact.

Scan my site for AI crawlersBuild a robots.txt

These crawlers feed AI products, but not all in the same way. Some train models, some fetch pages for live AI answers, and some only visit when a person asks. Use the purpose filter: that tag, not the company name, is what decides whether to block.
  • 104tokens
  • 101profiles
  • 79safe
  • 21caution

Common providers

Pick a provider to see only its crawlers. Each provider also has a full profile under Bot Operators.

All 104 AI & LLM Bots

104 user-agent tokens on 101 profiles. 3 unversioned tokens are covered on the profile of their versioned twin, so searching either spelling finds the same card.

The AI and LLM crawlers category in numbers

Our database lists 104 AI and LLM crawlers from 58 operators. 81 are rated safe, 22 caution and 1 aggressive. What they are for:

AI and LLM crawlers by purpose
PurposeCrawlers
fetches a page when a person asks an assistant or app to open it45
collects pages to train AI models22
builds a search or answer index21
collects pages both for training and for search or answers10
harvests pages for a dataset or data product4
checks pages for an SEO, uptime or analytics service2

Priority tiers

How the 104 AI and LLM crawlers spread across our six priority tiers, where Tier 1 is what you would notice losing:

  • Tier 1: 14 crawlers
  • Tier 2: 74 crawlers
  • Tier 3: 15 crawlers
  • Tier 4: 1 crawler

The highest-priority AI and LLM crawlers here are AddSearchBot, AI2Bot, AI2Bot-DeepResearchEval, aiHitBot, AIWebIndex, AmazonBuyForMe, Amzn-User, Andibot, anthropic-ai, atlassian-bot. Start your review with these.

How top sites treat this category

From our robots.txt study of the Tranco top 10,000 (5,583 valid files, fetched October 20, 2026):

AI and LLM crawlers named and blocked in top-site robots.txt files
TokenFiles naming itFiles blocking it
anthropic-ai10.6%12.5%
ChatGPT-User11.8%9.5%
Claude-SearchBot6.4%7.6%
Claude-User6.4%7.7%
ClaudeBot15.4%14.4%
cohere-ai9.7%12.4%
DuckAssistBot5.4%8.3%
Google-Extended14.3%13%
GPTBot18.1%15.4%
GrokBot1.1%5.2%
Meta-ExternalAgent10.7%12.6%
OAI-SearchBot10.3%7.8%
Perplexity-User6.4%8%
PerplexityBot12.9%10.9%
xAI-SearchBot0.1%4.4%

Tokens in AI and LLM crawlers can be blocked without being named, through User-agent: * with Disallow: /, which is why some blocking shares in this table exceed the naming shares.

How to review AI and LLM crawlers on your site

  1. List the ones that matter. Start with Tier 1 and 2 in the table above; the rest can usually follow a default.
  2. Choose a default. Decide what User-agent: * should say, because every crawler you do not name follows it.
  3. Name the exceptions. Give each crawler that needs a different answer its own group, and repeat any shared Disallow lines inside it.
  4. Check the edge. Firewalls and bot managers often treat these crawlers differently from robots.txt. Run the AI crawlability checker to see both layers together.
Test yourself: a crawler in this category is not named anywhere in your robots.txt. Which rules does it follow?

The User-agent: * group. If there is no such group, it may crawl everything.

Test yourself: does a safety rating of "safe" mean you should allow a crawler?

Not necessarily. Safety describes behaviour. Whether to allow it depends on what it does with your content and what you lose by blocking it.

The AI and LLM crawlers worth knowing first

Ranked by priority tier, these are the AI and LLM crawlers with the biggest effect on a typical site:

  • ChatGPT Agent (OpenAI, Tier 1): a user-triggered agent that fetches a page when a person asks an assistant or app to open it.
  • ChatGPT Operator (OpenAI, Tier 1): a user-triggered agent that fetches a page when a person asks an assistant or app to open it.
  • ChatGPT-User (OpenAI, Tier 1): a user-triggered agent that fetches a page when a person asks an assistant or app to open it.
  • ChatGPT-User/2.0 (OpenAI, Tier 1): a user-triggered agent that fetches a page when a person asks an assistant or app to open it.
  • Claude-SearchBot (Anthropic, Tier 1): a retrieval crawler that feeds AI search answers that fetches a page when a person asks an assistant or app to open it.
  • Claude-User (Anthropic, Tier 1): a user-triggered agent that fetches a page when a person asks an assistant or app to open it.
  • ClaudeBot (Anthropic, Tier 1): a training crawler that collects pages to train AI models.
  • Google-Extended (Google, Tier 1): an opt-out token rather than a crawler that collects pages to train AI models.

Of the 104 AI and LLM crawlers we track, 25 are documented as following robots.txt, 1 as not following it for some requests, 4 with exceptions or disputed behaviour, and 73 with no clear statement from the operator. Where behaviour is unclear, back a robots.txt rule with a server-side one.

What to do with AI and LLM crawlers, by type of site

Recommended approach to AI and LLM crawlers by type of site
If you run...Recommended approach
Publisher that wants AI citationsAllow retrieval and user-triggered crawlers; decide separately about training crawlers.
Publisher that licenses contentBlock training crawlers by name; keep retrieval crawlers if you still want to be cited.
Online shopAllow retrieval and assistant fetches on product pages; training is a separate decision.
Site under heavy bot loadRate-limit first, then block the lowest-tier crawlers.

Common mistakes with AI and LLM crawlers

The errors we see most often with AI and LLM crawlers in the files and firewalls we test:

  1. Assuming a robots.txt rule will stop AI and LLM crawlers. User-triggered fetches act for a person, and some operators say robots.txt may not apply to them. Check the operator's statement above.
  2. Blocking AI and LLM crawlers as if it were a bulk crawler. It fetches one page because a person asked; blocking it means that person's assistant cannot read your page.
  3. Counting AI and LLM crawlers visits as crawl volume. Each request maps to a real user question, which makes the paths it requests a useful signal of what people ask about you.

All three hide inside a file that looks correct, so check AI and LLM crawlers from outside after every change.

Spotting AI and LLM crawlers in your logs

robots.txt says what ChatGPT Agent and ChatGPT Operator and ChatGPT-User and ChatGPT-User/2.0 and Claude-SearchBot and Claude-User may do; your access logs show what they actually did.

Count requests

On a server with standard Nginx or Apache logs, this counts matching requests per day (adjust the log path for your host):

grep -iE "ChatGPT Agent|ChatGPT Operator|ChatGPT-User|ChatGPT-User/2\.0|Claude-SearchBot|Claude-User" /var/log/nginx/access.log \
  | awk '{print $4}' | cut -d: -f1 | sort | uniq -c

And this shows which paths it requested most, which is the quickest way to see whether your Disallow lines are being respected:

grep -iE "ChatGPT Agent|ChatGPT Operator|ChatGPT-User|ChatGPT-User/2\.0|Claude-SearchBot|Claude-User" /var/log/nginx/access.log \
  | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

Check the requests are genuine

Anyone can send these user-agents, so confirm the source before acting on a match:

  • ChatGPT Agent: we hold no published IP list, so use reverse DNS where AI and LLM crawlers documents a hostname and treat an unverifiable ChatGPT Agent request with caution. Check it separately from ChatGPT Operator and ChatGPT-User and ChatGPT-User/2.0 and Claude-SearchBot and Claude-User.
  • ChatGPT Operator: we hold no published IP list, so use reverse DNS where AI and LLM crawlers documents a hostname and treat an unverifiable ChatGPT Operator request with caution. Check it separately from ChatGPT Agent and ChatGPT-User and ChatGPT-User/2.0 and Claude-SearchBot and Claude-User.
  • ChatGPT-User: compare each address with the ranges at openai.com/chatgpt-user.json. A request claiming to be ChatGPT-User from outside them is not genuine. Check it separately from ChatGPT Agent and ChatGPT Operator and ChatGPT-User/2.0 and Claude-SearchBot and Claude-User.
  • ChatGPT-User/2.0: we hold no published IP list, so use reverse DNS where AI and LLM crawlers documents a hostname and treat an unverifiable ChatGPT-User/2.0 request with caution. Check it separately from ChatGPT Agent and ChatGPT Operator and ChatGPT-User and Claude-SearchBot and Claude-User.
  • Claude-SearchBot: compare each address with the ranges at claude.com/crawling/bots.json. A request claiming to be Claude-SearchBot from outside them is not genuine. Check it separately from ChatGPT Agent and ChatGPT Operator and ChatGPT-User and ChatGPT-User/2.0 and Claude-User.
  • Claude-User: compare each address with the ranges at claude.com/crawling/bots.json. A request claiming to be Claude-User from outside them is not genuine. Check it separately from ChatGPT Agent and ChatGPT Operator and ChatGPT-User and ChatGPT-User/2.0 and Claude-SearchBot.

Related: how AI engines choose their sources

Recent changes among AI and LLM crawlers

The newest AI and LLM crawlers in our database: KeenableBot (October 19, 2026), Keenable-User (October 19, 2026), CloudflareBrowserRenderingCrawler (October 19, 2026), Cloudflare-AI-Search-External (October 19, 2026), Cloudflare-AI-Search (October 19, 2026), Channel3Bot (October 19, 2026). A User-agent: * default decides what each of these gets until you name it.

Movement among the tracked AI and LLM crawlers: 2026 and October 20, 2026, files naming anthropic-ai went from 543 to 593 (+50) and files blocking it from 639 to 696 (+57), so more sites wrote a rule for anthropic-ai and more of those rules block it. 2026 and October 20, 2026, files naming ChatGPT-User went from 629 to 661 (+32) and files blocking it from 502 to 530 (+28), so more sites wrote a rule for ChatGPT-User and more of those rules block it. 2026 and October 20, 2026, files naming Claude-SearchBot went from 341 to 359 (+18) and files blocking it from 404 to 424 (+20), so more sites wrote a rule for Claude-SearchBot and more of those rules block it. 2026 and October 20, 2026, files naming Claude-User went from 339 to 357 (+18) and files blocking it from 408 to 430 (+22), so more sites wrote a rule for Claude-User and more of those rules block it.

How top sites treat AI and LLM crawlers

From our robots.txt study of the Tranco top 10,000 (5,583 valid files, fetched October 20, 2026):

  • GPTBot, the most blocked of the AI and LLM crawlers tokens: blocked in 15.4% of files (663 with its own group, 196 through User-agent: *), more often by the top 1,000 (20.4%) than the long tail (14.5%).
  • ClaudeBot, second of the AI and LLM crawlers tokens: blocked in 14.4% of files (594 with its own group, 211 through User-agent: *), more often by the top 1,000 (20.4%) than the long tail (13.3%).
  • Google-Extended, third of the AI and LLM crawlers tokens: blocked in 13% of files (519 with its own group, 204 through User-agent: *), more often by the top 1,000 (18.6%) than the long tail (11.9%).
  • Meta-ExternalAgent, fourth of the AI and LLM crawlers tokens: blocked in 12.6% of files (486 with its own group, 220 through User-agent: *), more often by the top 1,000 (19.2%) than the long tail (11%).
  • anthropic-ai, fifth of the AI and LLM crawlers tokens: blocked in 12.5% of files (469 with its own group, 227 through User-agent: *), more often by the top 1,000 (15.6%) than the long tail (11.6%).
  • cohere-ai, sixth of the AI and LLM crawlers tokens: blocked in 12.4% of files (466 with its own group, 227 through User-agent: *), more often by the top 1,000 (18.2%) than the long tail (11.3%).

For scale, 20.1% of all files block at least one AI training crawler. GPTBot is one of the tokens driving that figure.

More questions about AI and LLM crawlers

How many AI and LLM crawlers should I name in robots.txt?
Only the ones that need a different answer from your default. Every crawler you do not name follows User-agent: *, so a short file with a sensible default is easier to maintain than a long list.
Do new AI and LLM crawlers appear often?
Yes. New crawlers launch every month, which is why a deliberate User-agent: * default matters: it decides what happens to crawlers you have never heard of.
Can I copy another site's rules for AI and LLM crawlers?
Use them as a pattern, not a template. Each file reflects that site's licensing and infrastructure. Copy the structure, then name the crawlers that matter to you.
Where do I check how my site treats AI and LLM crawlers?
Run the AI search crawlability check. It reports every crawler in this category in one pass, including firewall refusals that robots.txt never shows.

FAQ

How many AI & LLM Bots are tracked?
We currently track 104 user-agent tokens in AI & LLM Bots, documented on 101 profile pages with user-agent strings, safety ratings and blocking rules. 3 unversioned tokens share a profile with their versioned twin.
Which AI & LLM Bots should I never block?
Never block bots with 'critical' impact rating - there are 0 in this category. Blocking them removes your site from their search results.
Are all AI & LLM Bots safe?
Out of the 101 profiles in this category: 79 safe, 21 caution, 1 aggressive.
How do I block all AI & LLM Bots at once?
Generate a combined robots.txt block rule for all AI & LLM Bots using our bulk robots.txt generator, or manually list each User-agent + Disallow: / directive.
WhatsApp us