Cohere AI
Operated by Cohere
Quick Facts
- User-Agent:
- cohere-ai
- Category:
- AI & LLM Bots
- Operator:
- Cohere
- Safety:
- Safe
- Blocking Impact:
- Low - No SEO ranking impact
- SEO Impact Score:
- 2/10
- Official docs:
- None published by Cohere
Reported, not operator documented
Cohere has not published a crawler page that names cohere-ai. It is listed here on our own evidence. Verified by AI Crawler Check: 542 of the 5,583 valid robots.txt files in the Tranco top 10,000 carry a rule for cohere-ai, so site owners treat it as a real crawler name (how we measure this). That proves the spelling is in use, not who sends it. Reported Cohere retrieval token. Cohere publishes no crawler page.
What that means for you: the cohere-ai rule on this page is the best available spelling, but Cohere has made no statement of purpose, crawl rate or robots.txt compliance. Treat a cohere-ai request in your logs as a clue, not proof of who sent it, and add a firewall rule if that traffic matters to you. If the operator publishes documentation, this profile moves to Official docs.
What is Cohere AI?
Cohere AI is a dual-purpose AI crawler operated by Cohere. It both indexes content for AI search answers and may retrieve pages on demand for user queries. Allowing it can improve your visibility inside Cohere's AI answers.
Cohere AI is a dual-purpose AI crawler operated by Cohere. It both indexes content for AI search answers and may retrieve pages on demand for user queries. Allowing it can improve your visibility inside Cohere's AI answers.
Cohere AI uses the user-agent token cohere-ai. You can control it via robots.txt, meta tags (noai), or the emerging llms.txt standard. Robots.txt is voluntary; for hard enforcement, combine it with server-level IP blocking.
What happens if you block Cohere AI?
User-agent / Disallow: / without any SEO penalty.Want to see whether Cohere AI actually visits you? Read how do I monitor AI bot traffic to my website?
How to block Cohere AI with robots.txt
<code>User-agent: cohere-ai</code> - Matching is case-insensitive. Robots.txt is fetched from the root of each subdomain separately.
Is Cohere AI safe to allow?
What does Cohere AI do?
Understanding Cohere AI's purpose helps you decide whether to allow or block it.
- Indexes content for AI search answers
- Retrieves pages on demand for user queries
- Can surface your content as AI citations
- Allowing it improves AI-answer visibility
- robots.txt behaviour not documented by the operator
Cohere AI in context
Cohere AI is a training crawler operated by Cohere. In plain terms, it collects pages both for training and for search or answers. Our database files it under AI and LLM crawlers and ranks it Tier 2, important crawlers from established AI and search operators. 74 of the 104 AI and LLM crawlers in our database share this tier.
Its safety rating is safe and its blocking impact is low. So the decision about Cohere AI rests on what you want from Cohere, not on its behaviour. How we rate safety and impact.
How site owners treat Cohere AI
In our own study of robots.txt files across the Tranco top 10,000 sites (fetched October 20, 2026, 5,583 valid files), 542 files (9.7%) name cohere-ai in a User-agent line, and 693 (12.4%) block it from the homepage. 466 of those blocks come from a group written for this crawler by name, and 227 come from a broad User-agent: * rule that catches it by accident. Larger sites are stricter: 18.2% of the top 1,000 block it, against 12.6% of the middle band and 11.3% of the tail.
Does it follow robots.txt?
Cohere AI's operator does not clearly document how it treats robots.txt. Write the Cohere AI rule anyway, then confirm in your logs that its requests stop.
Cohere crawlers and close alternatives
Cohere AI is one of 2 Cohere crawlers in our database. A User-agent: cohere-ai group controls Cohere AI only; the other Cohere tokens below need their own lines.
cohere-ai: this crawler (Tier 2)cohere-training-data-crawler: Cohere Training, collects pages to train AI models (Tier 2)
To give Cohere AI the same answer as its siblings, stack User-agent: cohere-ai with their lines above one set of rules; to treat Cohere AI differently, give it its own group.
Other AI and LLM crawlers that do the same job (collects pages both for training and for search or answers) include PerplexityBot from Perplexity AI, Andibot from Andi, DeepSeekBot from DeepSeek, DoubaoBot from ByteDance, GrokBot from xAI, iAskBot from iAsk. Deciding about Cohere AI is usually a decision about the whole group, so check how your file treats these too.
We first recorded cohere-ai on March 17, 2026. We could not find a page where Cohere names it, so treat the token as reported rather than documented.
Which rule Cohere AI actually follows
Cohere AI reads exactly one group in robots.txt, the most specific one that names cohere-ai, and never combines rules across groups. This short file shows the trap:
User-agent: *
Disallow: /private/
User-agent: cohere-ai
Disallow: /drafts/- Find the group. Cohere AI looks for a group naming
cohere-ai. One exists, so it ignoresUser-agent: *completely. - Apply only that group. It is refused
/drafts/and nothing else. - The surprise.
/private/is open to Cohere AI, because that rule lives in the*group it no longer reads. To keep it out of both, repeatDisallow: /private/inside its own group.
Test yourself: does a rule under User-agent: * apply to Cohere AI if the file also has a User-agent: cohere-ai group?
No. Once a group names it, Cohere AI reads only that group. Put every rule you want it to follow inside its own group.
Test yourself: will blocking Cohere AI remove your pages from AI answers?
No. Cohere AI collects training data. Blocking it changes what future models learn, not which pages AI answers cite today.
Test yourself: is User-agent: cohere-ai the same as User-agent: cohere-ai?
Yes. RFC 9309 makes user-agent matching case-insensitive. Paths are a different story: Disallow: /Blog/ does not block /blog/.
Should you allow Cohere AI? Decide by situation
| If you are... | Recommended setting |
|---|---|
| You publish content and want AI answers to cite it | Your call. Cohere AI is a training crawler, so blocking it does not remove you from AI answers; retrieval crawlers do that job. |
| You license content and want to stop AI training | Block it by name with User-agent: cohere-ai and Disallow: /. |
| You run an online shop | Allow it unless your logs show it is costly. |
| Your server is under load from bots | Rate-limit before blocking. It is rated safe in our database. |
Check Cohere AI on your own site
- Open
/robots.txton your domain and search forcohere-ai. If it is not named, it followsUser-agent: *. - List the
Disallowlines in the group Cohere AI follows and test your key URLs against them; the longest match wins, andAllowwins a tie. - Look beyond the file. A firewall or bot manager can refuse Cohere AI even when robots.txt allows it, and that never shows in the file.
- Run our robots.txt AI crawler test to see how Cohere AI and every other crawler we track are treated in one pass, including firewall refusals.
Common mistakes with Cohere AI
The errors we see most often with Cohere AI in the files and firewalls we test:
- Assuming that blocking Cohere AI removes you from AI answers. It is a training crawler; AI answers are fed by retrieval and user-triggered crawlers with different tokens.
- Blocking Cohere AI with a broad
User-agent: *rule that also catches search engines. Namecohere-aiin its own group instead. - Expecting a Cohere AI block to remove content already collected. robots.txt only affects future visits.
All three hide inside a file that looks correct, so check Cohere AI from outside after every change.
Spotting Cohere AI in your logs
robots.txt says what Cohere AI may do; your access logs show what it actually did.
Count requests
On a server with standard Nginx or Apache logs, this counts matching requests per day (adjust the log path for your host):
grep -iE "cohere-ai" /var/log/nginx/access.log \
| awk '{print $4}' | cut -d: -f1 | sort | uniq -cAnd this shows which paths it requested most, which is the quickest way to see whether your Disallow lines are being respected:
grep -iE "cohere-ai" /var/log/nginx/access.log \
| awk '{print $7}' | sort | uniq -c | sort -rn | head -20Check the requests are genuine
Anyone can send the Cohere AI user-agent, so confirm the source before acting on a match:
cohere-ai: we hold no published IP list, so use reverse DNS where the operator documents a hostname and treat an unverifiable Cohere AI request with caution.
Frequently Asked Questions
What user-agent string does Cohere AI use?
cohere-ai. Evidence that the spelling is in real use comes from our own robots.txt study: 542 of the 5,583 valid robots.txt files in the Tranco top 10,000 name it (fetched October 20, 2026), not from Cohere, which has not published crawler documentation naming it. Use it in robots.txt as the best available match, and treat a request carrying it as a clue rather than proof of who sent it.Is Cohere AI safe?
Will blocking Cohere AI hurt my SEO?
User-agent / Disallow: / without any SEO penalty.How do I block Cohere AI in robots.txt?
/robots.txt file:User-agent: cohere-ai Disallow: /
This instructs Cohere AI not to crawl any path on your site. To block only specific sections, replace / with the path (e.g.,
Disallow: /blog/).Does Cohere AI respect robots.txt?
cohere-ai will stop a client that chooses to honour it; for enforcement, add a firewall or CDN rule as well.How do I verify if Cohere AI is crawling my site?
cohere-ai (for example grep -i "cohere-ai" /var/log/nginx/access.log). Because the operator publishes no IP ranges or reverse-DNS pattern, a match shows that something used the name, not that Cohere sent it.