Cohere is one of the most important AI companies that many website owners have never heard of. Unlike ChatGPT or Google, Cohere does not have a consumer product that people use daily. Instead, it builds AI infrastructure used by thousands of businesses worldwide. A user agent called cohere-ai still appears in server logs and AI blocklists, even though Cohere now says it does not currently crawl the web to train its foundation models.
For website owners, the Cohere crawler presents a different kind of decision compared to consumer-facing AI crawlers. There is no direct search benefit or citation traffic from allowing it. But Cohere's models are used by major corporations for customer support, internal search, and document processing, which means your content could influence enterprise AI applications that serve millions of business users.
Start by checking your current settings. Use the AI crawler checker free tool to see if Cohere's crawler can access your website right now.
Correction: what Cohere says its crawlers do today
That leaves a real puzzle for anyone who sees cohere-ai in their logs. The token appears in public blocklists and in our cohere-ai profile, which is rated Reported, not Official. A request carrying it is not proof that Cohere fetched the page: it could be an older client, a third-party integration, or a spoofed string. The same applies to cohere-training-data-crawler, also Reported.
What is Cohere?
Cohere is a Canadian AI company founded in 2019 by former Google Brain researchers, including Aidan Gomez, who co-authored the landmark "Attention Is All You Need" paper that introduced the Transformer architecture (the "T" in GPT). Cohere is headquartered in Toronto and has raised over $1 billion in funding.
Unlike OpenAI (which focuses on consumer products like ChatGPT) or Anthropic (Claude), Cohere focuses primarily on enterprise AI infrastructure. Their products are designed for businesses that need to integrate AI into their own applications and workflows.
Cohere's Key Products
Command
Cohere's flagship large language model for text generation, summarization, and conversational AI. Used by enterprises for customer support, content creation, and data analysis.
Embed
A text embedding model that converts text into numerical representations for search, clustering, and classification. Used by companies to build semantic search engines.
Rerank
A specialized model that re-orders search results by relevance. Used to improve the accuracy of enterprise search systems and RAG (Retrieval-Augmented Generation) applications.
Aya
Cohere's multilingual AI model supporting 100+ languages. Built through a global research collaboration, it enables AI applications in non-English markets.
The Cohere-AI Crawler: Technical Details
Cohere's web crawler identifies itself with the following user agent string in your server logs:
Third-party directories list cohere-ai as a training crawler. Cohere's current policy does not confirm that, so treat the figures below as observed behaviour of requests carrying the string, not operator documentation:
Requests carrying the cohere-ai string are rarely a load problem. Because Cohere documents no active crawler, compliance with robots.txt and Crawl-delay is unconfirmed, so back any block with a server rule if it matters to you.
What Data Does the Cohere Crawler Collect?
Where requests carrying the cohere-ai string are seen, they fetch the same public content any text crawler reads:
Text content: Articles, blog posts, documentation, product descriptions, and other textual content on public web pages.
HTML structure: Page structure, heading hierarchy, and semantic HTML elements that help the AI understand content organization.
Metadata: Title tags, meta descriptions, and structured data that provide content context.
Cohere's policy says it does not currently use such data for foundation-model training and would publish blocking instructions before any training crawler went live. Cohere models reach the live web mainly through search tools that developers connect, and those tools use their own user agents.
How Cohere's Crawler Compares to Other AI Crawlers
| Feature | cohere-ai | GPTBot | ClaudeBot | Google-Extended |
|---|---|---|---|---|
| Company focus | Enterprise | Consumer + API | Consumer + API | Search + AI |
| Search citations | No | No (GPTBot) | No | Yes (AI Overviews) |
| Crawl volume | Low | Medium | Low-Medium | Low |
| User base | Enterprise only | 400M+ (ChatGPT) | 50M+ (Claude) | Billions (Google) |
| Respects robots.txt | Unconfirmed | Yes | Yes | Yes |
| Server impact | Minimal | Moderate | Low | Minimal |
How to Control Cohere Crawler Access
You can control the Cohere crawler through your robots.txt file:
Block Cohere Crawler
Allow Cohere Crawler
Selective Access (Recommended)
Use the Robots.txt Generator for a complete configuration that includes Cohere and all other major AI crawlers.
Should You Allow or Block the Cohere Crawler?
The decision depends on your priorities:
Consider Allowing If:
You publish educational or informational content
You want your expertise in enterprise AI applications
Server load from Cohere is not a concern
You support open AI development
Your B2B clients use Cohere products
Consider Blocking If:
You have premium or licensed content
You want to minimize AI training data use
No direct benefit justifies the access
You are selectively allowing only search crawlers
Content protection is a top priority
For most website owners, Cohere's crawler falls into the "low priority" category. It does not provide direct traffic benefits like AI search crawlers, and it does not create heavy server load like aggressive crawlers. The decision is primarily about your philosophical stance on AI training data usage.
The Bigger Picture: Enterprise AI Crawlers
Cohere is not the only enterprise AI company with a web crawler. The enterprise AI crawler landscape includes several players:
Cohere (cohere-ai): Enterprise AI infrastructure for text generation, search, and classification.
Diffbot: Crawls the web to build a knowledge graph used by enterprise customers for data extraction and analysis.
AI2 (Ai2Bot): The Allen Institute for AI crawls web data for open-source research models.
Webz.io: Crawls web data to provide structured datasets for AI training and business intelligence.
To get a complete picture of which AI crawlers (both consumer and enterprise) can access your website, run a scan with the AI crawler checker online tool. It checks for 97 AI crawlers including Cohere and other enterprise bots.
Cohere tokens compared with documented AI crawlers
| Token | Operator documents it? | Stated purpose | Robots.txt behaviour | What to do |
|---|---|---|---|---|
| cohere-ai | No (Reported) | Not stated by Cohere | Unconfirmed | Block by name if unwanted; verify before attributing |
| cohere-training-data-crawler | No (Reported) | Listed by third parties as training | Unconfirmed | Same as above |
| GPTBot | Yes | Training | Follows robots.txt | Decide on training |
| ClaudeBot | Yes | Training | Follows robots.txt and Crawl-delay | Decide on training |
| Claude-SearchBot | Yes | AI search index | Follows robots.txt | Allow for citations |
Cohere's policy says any future training crawler will be identified and will respect robots.txt. Until one is published, the practical position is simple: a User-agent: cohere-ai group costs nothing, blocks a name you do not need, and has no SEO effect.
Where enterprise AI actually reads the web
Enterprise models increasingly reach web content through retrieval at question time rather than a training crawl. Cohere's documentation has developers connect models to external search tools, so traffic from such a setup carries the identity of that tool, not Cohere's. In practice, your visibility inside enterprise AI depends on the same search indexes that power consumer assistants.
That makes the retrieval tier the one to protect. Run a free AI crawler checker scan to confirm OAI-SearchBot, Claude-SearchBot and PerplexityBot are allowed, and publish an llms.txt file so any retrieval agent gets a clean summary.
Check your enterprise AI crawler settings
- AI crawler checker: tests 257 bots against your robots.txt in one scan.
- AI crawler diagnostic: fetches a page as each AI bot and names the layer that blocks it.
- robots.txt validator: shows allowed, blocked and partial status per bot.
- robots.txt generator: builds a file with presets for search, agents and training.
- batch URL checker: compares up to 20 sites side by side, with CSV export.
- llms.txt generator: writes the file AI assistants read for a site summary.
- schema markup generator: produces JSON-LD for eight common types.
- GEO audit: scores a page for generative engine optimization.
Key Takeaways
Cohere documents no active crawler. Its web crawler policy says it does not currently crawl for foundation-model training. The cohere-ai token is a reported name with no traffic or citation benefit.
Treat the string as unverified. A cohere-ai request is not proof that Cohere fetched the page; verify before attributing it.
Your decision depends on priorities. If content protection is your top priority, block it. If you support open AI development or serve B2B clients using Cohere, allow it.
Prioritize search crawlers first. Focus your AI crawler strategy on search crawlers (ChatGPT-User, PerplexityBot, Google-Extended) that provide direct traffic benefits. Reported enterprise tokens such as cohere-ai are secondary decisions.
Use selective access if unsure. Allow Cohere access to public blog/article content while blocking premium or proprietary sections. This provides some benefit while protecting valuable content.
Check your current Cohere crawler access and all other AI bots. Use the AI crawlers analysis on our homepage to scan your robots.txt and get a complete access report. Then use the Robots.txt Generator to create the right configuration for your website.
