Web Crawler & Bot Directory

Find a bot fast

1. Know the name from your logs? Search the category page that fits it, or open its operator.
2. Only know the company? Start at operators and open the company profile; it lists every token it runs.
3. Not sure what a bot does? Each profile explains its purpose, tier, safety rating and how it treats robots.txt, with the operator's own documentation where it exists.

What each profile tells you

Token

The exact name to use in robots.txt.

Evidence

Official when the operator documents the token; Reported when only an independent source does.

Tier

One of 6 priority levels, by how much the crawler's access matters to a typical site.

Safety

How the crawler behaves: identification, rule-following and request rate.

Blocking impact

What you lose if you refuse it.

Adoption

How many top sites name and block it, from our own robots.txt study.

How top sites treat the biggest names

From our robots.txt study of the Tranco top 10,000 (5,583 valid files).

Related: how AI engines choose their sources

What top sites actually do: our latest scan

On October 20, 2026 we fetched /robots.txt from every domain in the Tranco top 10,000. 8,122 answered, 5,583 returned a valid file, 1,117 refused us with HTTP 403 and 689 had no file at all. Here is how those valid files treat the crawlers that matter most:

• 20.1% of sites block at least one AI training crawler, and 12.3% block at least one AI search or assistant crawler.
• 427 sites block GPTBot but leave OAI-SearchBot open: they opt out of training while staying in ChatGPT search.
• 575 block Google-Extended while keeping Googlebot, opting out of Gemini training without leaving Google Search.
• 382 treat Anthropic's crawlers differently from each other, a sign that site owners are learning the training and retrieval split.
• 3,764 files list a sitemap, 651 set a Crawl-delay and only 118 mention llms.txt.

The bigger the site, the stricter it tends to be: the top 1,000 block GPTBot at 20.4%, against 14.5% in the long tail. The full write-up is in our robots.txt study.

Suggesting a crawler

If you see a crawler in your logs that the directory does not list, send us the user-agent string, a sample log line and any page where the operator describes it. We add it once we can confirm it exists, with a note on where that evidence came from.

What each profile contains

• The user-agent token and the full string the crawler sends.
• The operator, with a link to its own documentation where one exists.
• Purpose, priority tier, safety rating and the impact of blocking it.
• How top sites treat it in our robots.txt study, where we track it.
• Copy-paste rules to allow or block it, and the mistakes we see most often.

Safety ratings in one sentence each

Safe

Identifies itself, follows its documented rules and keeps a reasonable request rate.

Caution

Some behaviour is unclear, disputed or heavier than most; watch it in your logs.

Aggressive

Heavy request rates or a record of ignoring rules; back any block with a server rule.

Using the directory to write a file

Start from the categories that affect visibility: AI and LLM crawlers and search engines. Decide a default with User-agent: *, then add a named group only for crawlers that need a different answer. Most sites end up naming a dozen tokens or fewer. Paste the result into the generator to check the structure before publishing.

Finding a bot you saw in your logs

1. Copy the distinctive part of the user-agent string, usually the word ending in "bot" or "agent".
2. Search for it with the box at the top of this page, or open the category it most likely belongs to.
3. On the profile, compare the full user-agent string and the operator's IP list with your log line.
4. If nothing matches, the request may be a scraper borrowing a name; the spoofed user-agent guide explains how to tell.

How profiles are kept current

Before every release we recheck the operator documentation behind each profile and rerun our robots.txt scan, then update the date on any profile that changed. The changelog lists new crawlers and corrections.

The directory by the numbers

226 of the 265 crawlers are rated safe, 34 caution and 5 aggressive (Brightbot, Bytespider, DeepSeekBot, news-please, TikTokSpider). Safety describes behaviour, not value: a safe crawler can still be one you choose to block.

How a bot gets into the directory

A crawler is added when we can show it exists: an operator page that names the token, a reputable registry, or our own scans finding the token in top-site robots.txt files. Each profile says which of these applies. Crawlers that are retired stay listed with a note, because old tokens still sit in many files; the stale token audit explains why that matters.

Test yourself