Cohere Training
Operated by Cohere
Quick Facts
- User-Agent:
- cohere-training-data-crawler
- Category:
- AI & LLM Bots
- Operator:
- Cohere
- Safety:
- Use Caution
- Blocking Impact:
- Low - No SEO ranking impact
- SEO Impact Score:
- 2/10
- Official docs:
- None published by Cohere
Reported, not operator documented
Cohere has not published a crawler page that names cohere-training-data-crawler. It is listed here on our own evidence. Verified by AI Crawler Check: 237 of the 5,583 valid robots.txt files in the Tranco top 10,000 carry a rule for cohere-training-data-crawler, so site owners treat it as a real crawler name (how we measure this). That proves the spelling is in use, not who sends it. Reported Cohere training crawler. Cohere publishes no crawler page.
What that means for you: the cohere-training-data-crawler rule on this page is the best available spelling, but Cohere has made no statement of purpose, crawl rate or robots.txt compliance. Treat a cohere-training-data-crawler request in your logs as a clue, not proof of who sent it, and add a firewall rule if that traffic matters to you. If the operator publishes documentation, this profile moves to Official docs.
What is Cohere Training?
Cohere Training is an AI data-collection crawler operated by Cohere. It gathers public web content to build or refine training datasets for large language models. Like other AI training crawlers, Cohere Training does not influence your search-engine rankings, so you can block it via robots.txt without any SEO penalty if you wish to opt out of AI training.
Cohere Training is an AI data-collection crawler operated by Cohere. It gathers public web content to build or refine training datasets for large language models. Like other AI training crawlers, Cohere Training does not influence your search-engine rankings, so you can block it via robots.txt without any SEO penalty if you wish to opt out of AI training.
Cohere Training uses the user-agent token cohere-training-data-crawler. You can control it via robots.txt, meta tags (noai), or the emerging llms.txt standard. Robots.txt is voluntary; for hard enforcement, combine it with server-level IP blocking.
What happens if you block Cohere Training?
User-agent / Disallow: / without any SEO penalty.Want to see whether Cohere Training actually visits you? Read how do I monitor AI bot traffic to my website?
How to block Cohere Training with robots.txt
<code>User-agent: cohere-training-data-crawler</code> - Matching is case-insensitive. Robots.txt is fetched from the root of each subdomain separately.
Is Cohere Training safe to allow?
What does Cohere Training do?
Understanding Cohere Training's purpose helps you decide whether to allow or block it.
- Collects public web content to train large language models
- Builds and refreshes AI training datasets
- Harvests text, code, and structured data for model improvement
- Does NOT affect search-engine rankings
- Can be blocked to opt out of AI training without SEO loss
Cohere Training in context
Cohere Training is a training crawler operated by Cohere. In plain terms, it collects pages to train AI models. Our database files it under AI and LLM crawlers and ranks it Tier 2, important crawlers from established AI and search operators. 74 of the 104 AI and LLM crawlers in our database share this tier.
Its safety rating is caution and its blocking impact is low. So the decision about Cohere Training rests on what you want from Cohere, not on its behaviour. How we rate safety and impact.
How site owners treat Cohere Training
In our own scan of the Tranco top 10,000 (5,583 valid robots.txt files), 237 files (4.2%) name cohere-training-data-crawler in a User-agent line. That shows site owners treat the token as a real crawler worth a rule. It does not, on its own, prove who operates it.
Does it follow robots.txt?
Cohere Training's operator does not clearly document how it treats robots.txt. Write the Cohere Training rule anyway, then confirm in your logs that its requests stop.
Cohere crawlers and close alternatives
Cohere Training is one of 2 Cohere crawlers in our database. A User-agent: cohere-training-data-crawler group controls Cohere Training only; the other Cohere tokens below need their own lines.
cohere-training-data-crawler: this crawler (Tier 2)cohere-ai: Cohere AI, collects pages both for training and for search or answers (Tier 2)
To give Cohere Training the same answer as its siblings, stack User-agent: cohere-training-data-crawler with their lines above one set of rules; to treat Cohere Training differently, give it its own group.
Other AI and LLM crawlers that do the same job (collects pages to train AI models) include ClaudeBot from Anthropic, Google-Extended from Google, GPTBot from OpenAI, Meta-ExternalAgent from Meta, AI2Bot from Allen Institute for AI, aiHitBot from aiHit. Deciding about Cohere Training is usually a decision about the whole group, so check how your file treats these too.
We first recorded cohere-training-data-crawler on March 17, 2026. We could not find a page where Cohere names it, so treat the token as reported rather than documented.
Which rule Cohere Training actually follows
Cohere Training reads exactly one group in robots.txt, the most specific one that names cohere-training-data-crawler, and never combines rules across groups. This short file shows the trap:
User-agent: *
Disallow: /private/
User-agent: cohere-training-data-crawler
Disallow: /drafts/- Find the group. Cohere Training looks for a group naming
cohere-training-data-crawler. One exists, so it ignoresUser-agent: *completely. - Apply only that group. It is refused
/drafts/and nothing else. - The surprise.
/private/is open to Cohere Training, because that rule lives in the*group it no longer reads. To keep it out of both, repeatDisallow: /private/inside its own group.
Test yourself: does a rule under User-agent: * apply to Cohere Training if the file also has a User-agent: cohere-training-data-crawler group?
No. Once a group names it, Cohere Training reads only that group. Put every rule you want it to follow inside its own group.
Test yourself: will blocking Cohere Training remove your pages from AI answers?
No. Cohere Training collects training data. Blocking it changes what future models learn, not which pages AI answers cite today.
Test yourself: is User-agent: cohere-training-data-crawler the same as User-agent: cohere-training-data-crawler?
Yes. RFC 9309 makes user-agent matching case-insensitive. Paths are a different story: Disallow: /Blog/ does not block /blog/.
Should you allow Cohere Training? Decide by situation
| If you are... | Recommended setting |
|---|---|
| You publish content and want AI answers to cite it | Your call. Cohere Training is a training crawler, so blocking it does not remove you from AI answers; retrieval crawlers do that job. |
| You license content and want to stop AI training | Block it by name with User-agent: cohere-training-data-crawler and Disallow: /. |
| You run an online shop | Allow it unless your logs show it is costly. |
| Your server is under load from bots | Rate-limit before blocking. It is rated caution in our database. |
Check Cohere Training on your own site
- Open
/robots.txton your domain and search forcohere-training-data-crawler. If it is not named, it followsUser-agent: *. - List the
Disallowlines in the group Cohere Training follows and test your key URLs against them; the longest match wins, andAllowwins a tie. - Look beyond the file. A firewall or bot manager can refuse Cohere Training even when robots.txt allows it, and that never shows in the file.
- Run our blocked AI crawler check to see how Cohere Training and every other crawler we track are treated in one pass, including firewall refusals.
Common mistakes with Cohere Training
The errors we see most often with Cohere Training in the files and firewalls we test:
- Assuming that blocking Cohere Training removes you from AI answers. It is a training crawler; AI answers are fed by retrieval and user-triggered crawlers with different tokens.
- Blocking Cohere Training with a broad
User-agent: *rule that also catches search engines. Namecohere-training-data-crawlerin its own group instead. - Expecting a Cohere Training block to remove content already collected. robots.txt only affects future visits.
All three hide inside a file that looks correct, so check Cohere Training from outside after every change.
Spotting Cohere Training in your logs
robots.txt says what Cohere Training may do; your access logs show what it actually did.
Count requests
On a server with standard Nginx or Apache logs, this counts matching requests per day (adjust the log path for your host):
grep -iE "cohere-training-data-crawler" /var/log/nginx/access.log \
| awk '{print $4}' | cut -d: -f1 | sort | uniq -cAnd this shows which paths it requested most, which is the quickest way to see whether your Disallow lines are being respected:
grep -iE "cohere-training-data-crawler" /var/log/nginx/access.log \
| awk '{print $7}' | sort | uniq -c | sort -rn | head -20Check the requests are genuine
Anyone can send the Cohere Training user-agent, so confirm the source before acting on a match:
cohere-training-data-crawler: we hold no published IP list, so use reverse DNS where the operator documents a hostname and treat an unverifiable Cohere Training request with caution.
Frequently Asked Questions
What user-agent string does Cohere Training use?
cohere-training-data-crawler. Evidence that the spelling is in real use comes from our own robots.txt study: 237 of the 5,583 valid robots.txt files in the Tranco top 10,000 name it (fetched October 20, 2026), not from Cohere, which has not published crawler documentation naming it. Use it in robots.txt as the best available match, and treat a request carrying it as a clue rather than proof of who sent it.Is Cohere Training safe?
Will blocking Cohere Training hurt my SEO?
User-agent / Disallow: / without any SEO penalty.How do I block Cohere Training in robots.txt?
/robots.txt file:User-agent: cohere-training-data-crawler Disallow: /
This instructs Cohere Training not to crawl any path on your site. To block only specific sections, replace / with the path (e.g.,
Disallow: /blog/).Does Cohere Training respect robots.txt?
cohere-training-data-crawler will stop a client that chooses to honour it; for enforcement, add a firewall or CDN rule as well.How do I verify if Cohere Training is crawling my site?
cohere-training-data-crawler (for example grep -i "cohere-training-data-crawler" /var/log/nginx/access.log). Because the operator publishes no IP ranges or reverse-DNS pattern, a match shows that something used the name, not that Cohere sent it.