Cohere

Common mistakes with Cohere's crawlers

The errors we see most often with Cohere's crawlers in the files and firewalls we test:

1. Assuming that blocking Cohere's crawlers removes you from AI answers. It is a training crawler; AI answers are fed by retrieval and user-triggered crawlers with different tokens.
2. Blocking Cohere's crawlers with a broad User-agent: * rule that also catches search engines. Name cohere-ai in its own group instead.
3. Expecting a Cohere's crawlers block to remove content already collected. robots.txt only affects future visits.

All three hide inside a file that looks correct, so check Cohere's crawlers from outside after every change.

Spotting Cohere in your logs

robots.txt says what Cohere AI and Cohere Training may do; your access logs show what they actually did.

Count requests

On a server with standard Nginx or Apache logs, this counts matching requests per day (adjust the log path for your host):

grep -iE "cohere-ai|cohere-training-data-crawler" /var/log/nginx/access.log \
  | awk '{print $4}' | cut -d: -f1 | sort | uniq -c

And this shows which paths it requested most, which is the quickest way to see whether your Disallow lines are being respected:

grep -iE "cohere-ai|cohere-training-data-crawler" /var/log/nginx/access.log \
  | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

Check the requests are genuine

Anyone can send these user-agents, so confirm the source before acting on a match:

• cohere-ai: we hold no published IP list, so use reverse DNS where Cohere documents a hostname and treat an unverifiable Cohere AI request with caution. Check it separately from Cohere Training.
• cohere-training-data-crawler: we hold no published IP list, so use reverse DNS where Cohere documents a hostname and treat an unverifiable Cohere Training request with caution. Check it separately from Cohere AI.

What our own data shows about Cohere

Everything below comes from the AI Crawler Check bot database and our two most recent robots.txt scans of the Tranco top 10,000, not from Cohere's marketing. It covers 2 Cohere tokens in turn.

cohere-ai in our records

We first recorded cohere-ai on March 17, 2026. We could not find a page where Cohere names it, so treat the token as reported rather than documented. Between our scans of October 17, 2026 and October 20, 2026, files naming cohere-ai went from 500 to 542 (+42) and files blocking it from 640 to 693 (+53), so more sites wrote a rule for cohere-ai and more of those rules block it.

Among the 104 AI and LLM crawlers in our database, cohere-ai sits 33th by priority (Tier 2), level with AddSearchBot, AI2Bot, AI2Bot-DeepResearchEval, aiHitBot. 81 of the 104 share its safe rating. Other operators with a crawler that collects pages both for training and for search or answers: PerplexityBot (Perplexity AI, Tier 1), Andibot (Andi, Tier 2), Bytespider (ByteDance, Tier 2), DeepSeekBot (DeepSeek, Tier 2), DoubaoBot (ByteDance, Tier 2). A policy for cohere-ai is usually a policy for these too.

A decision record for cohere-ai

Teams that review crawlers once a quarter keep one short entry per token, so anyone can see why a rule exists and when it was last checked. Filled in from our data, the cohere-ai entry reads:

• Operator and job: Cohere; it collects pages both for training and for search or answers.
• Priority and behaviour: Tier 2, rated safe, blocking impact low.
• Suggested starting point: Your licensing call: Disallow to refuse training, Allow to permit it.
• Enforcement: cohere-ai's operator does not clearly document how it treats robots.txt. Write the cohere-ai rule anyway, then confirm in your logs that its requests stop.

User-agent: cohere-ai
Allow: /

Paste the line into the validator with one of your own URLs before it goes live. Revisit the cohere-ai entry whenever Cohere announces a change, because a Tier 2 crawler affects visibility directly.

cohere-training-data-crawler in our records

We first recorded cohere-training-data-crawler on March 17, 2026. We could not find a page where Cohere names it, so treat the token as reported rather than documented. cohere-training-data-crawler is outside our tracked token set, but 237 of 5,583 top-site files already name it, which is enough to show owners treat it as a real crawler.

Among the 104 AI and LLM crawlers in our database, cohere-training-data-crawler sits 34th by priority (Tier 2), level with AddSearchBot, AI2Bot, AI2Bot-DeepResearchEval, aiHitBot. 22 of the 104 share its caution rating. Other operators with a crawler that collects pages to train AI models: ClaudeBot (Anthropic, Tier 1), Google-Extended (Google, Tier 1), GPTBot (OpenAI, Tier 1), Meta-ExternalAgent (Meta, Tier 1), AI2Bot (Allen Institute for AI, Tier 2). A policy for cohere-training-data-crawler is usually a policy for these too.

A decision record for cohere-training-data-crawler

Teams that review crawlers once a quarter keep one short entry per token, so anyone can see why a rule exists and when it was last checked. Filled in from our data, the cohere-training-data-crawler entry reads:

• Operator and job: Cohere; it collects pages to train AI models.
• Priority and behaviour: Tier 2, rated caution, blocking impact low.
• Suggested starting point: Your licensing call: Disallow to refuse training, Allow to permit it.
• Enforcement: cohere-training-data-crawler's operator does not clearly document how it treats robots.txt. Write the cohere-training-data-crawler rule anyway, then confirm in your logs that its requests stop.

User-agent: cohere-training-data-crawler
Allow: /

Paste the line into the validator with one of your own URLs before it goes live. Revisit the cohere-training-data-crawler entry sooner than most: a caution rating means Cohere traffic deserves a monthly look at your logs.

How top sites treat Cohere's crawlers

From our robots.txt study of the Tranco top 10,000 (5,583 valid files, fetched October 20, 2026):

• cohere-ai, the only one tracked among the Cohere's crawlers tokens: blocked in 12.4% of files (466 with its own group, 227 through User-agent: *), more often by the top 1,000 (18.2%) than the long tail (11.3%).

For scale, 20.1% of all files block at least one AI training crawler. cohere-ai is one of the tokens driving that figure.

More questions about Cohere

Will blocking Cohere affect my Google rankings?

Only if one of its tokens is a Google search crawler. None of Cohere's tokens are Googlebot, so classic Google rankings are unaffected.

How quickly does a robots.txt change take effect for Cohere?

Crawlers cache robots.txt, commonly for up to a day. Expect a change to apply within about 24 hours, and confirm it in your logs.

Can I allow Cohere on some sections and block it on others?

Yes. Inside the group for cohere-ai, add Disallow lines for the paths to keep private and Allow lines for exceptions. The longest matching rule wins.

Does blocking Cohere delete content it already collected?

No. robots.txt only controls future visits. For content already collected, check whether Cohere offers a removal or opt-out process.

Is a firewall block better than robots.txt for Cohere?

A firewall enforces the decision, while robots.txt asks politely. Many operators recommend robots.txt because blocking their IPs can stop them reading your rules at all. Use robots.txt first, and add a firewall rule only if requests continue.