Bot Operators

Why operators matter

robots.txt works token by token, but decisions are usually made company by company: "do we want our content in this company's products?" An operator profile bridges the two. It lists every token a company runs, what each one does and how each treats robots.txt, so you can turn one business decision into the right set of rules.

A three-step operator policy

1. Decide per job. For each operator that matters, decide separately about training, search and user-triggered fetching.
2. Map jobs to tokens. Use the operator profile to find which token does which job.
3. Write and verify. One group per decision, then confirm the result with the website crawlability test.

The operators most top sites have an opinion about

From our robots.txt study of the Tranco top 10,000 (5,583 valid files).

Reporting a wrong operator

If an operator page names the wrong company, links to the wrong documentation or misses a token, tell us through the contact page with a link to the operator's own source. We check it and correct the page in the next release.

How we name operators

We use the name the operator uses in its own crawler documentation, so a search for that name leads straight to the right source. Where a crawler has changed hands or brands, its profile notes the old token when the operator still honours it.

Single-crawler operators and what they tend to be

74 operators in our database run exactly one crawler. They are mostly specialists: SEO and marketing platforms that audit sites, research archives, plagiarism and brand-monitoring services, and smaller search engines. For these, the operator page and the crawler profile cover the same token, so the decision is simple: one User-agent line and one rule. Start with what the operator's service does for you. If you use it, allow it; if you have never heard of it and it is rated caution or aggressive, a block costs you little.

Large operators and the cost of getting them wrong

The operators at the top of this table carry most of the risk in a robots.txt file, because a single broad rule can catch several of their tokens at once. Write their groups first and test each token on its own in the validator.

Questions people ask about operators

Is an operator the same as a company?

Usually. Where one company runs crawlers under separate brands, we list the brand the crawler identifies with, so the documentation link matches the token.

Can I block a whole operator with one line?

No. robots.txt matches tokens, not companies. Stack every token you mean above one set of rules.

Why does an operator page show crawlers I have never seen?

Many tokens only visit when a person asks an assistant to read your page, or only crawl certain kinds of site. Absence from your logs does not mean the crawler is retired.

Operators by the job their crawlers do

An operator can appear in several rows. Google, for example, runs crawlers for search indexing, user requests and monitoring, which is why its operator page groups tokens by job rather than listing them alphabetically.

The operators with the most crawlers

The top of the list is dominated by search engines and the large AI labs, which split their crawling into separate tokens for training, search and user requests. 74 of the 112 operators run a single crawler, so for most of the list one robots.txt group is the whole decision.

Why operators split their crawlers

An operator with several tokens is giving you a choice. OpenAI, Anthropic and Google each run a training crawler or token next to the agents that fetch pages for search answers or for one person's request. Blocking one does not block the others. Our study found 427 top-site files that block GPTBot while leaving OAI-SearchBot open, and 575 that block Google-Extended but keep Googlebot: owners using exactly that choice. Each operator page lists every token and what it is for, and the hybrid setup guide shows the file.

Operators we cannot fully identify

6 crawlers in our database have no operator we can confirm. They appear under "Unknown" rather than under a guessed name, because a wrong operator would lead you to the wrong documentation and the wrong IP list. If you can point to an operator's own page that names one of these tokens, tell us and we will move it.

How to use an operator page

1. Open the operator and read which of its tokens does which job.
2. Decide per job: training, search, user requests, monitoring.
3. Write one group per decision, naming every token that shares it.
4. Paste the file into the validator and test a real URL for each token.
5. Check the operator page again when it announces a new crawler; a new token is not covered by old rules.

Test yourself