Devin (Cognition)
SafeUser-agent: Devin
Category: AI & LLM Bots
SEO impact if blocked: 2/10
Cognition AI
1 crawler operated by Cognition AI
https://cognition.ai/Who Cognition AI is, for site owners
Cognition AI operates 1 crawler in our directory, filed under AI & LLM Bots. Its crawl behaviour is rated safe.
Blocking Cognition AI carries an SEO impact score of 2/10. Blocking impact is rated: Low - No SEO ranking impact.
Our recommendation for Cognition AI: Generally safe to allow; provides legitimate crawling value.
Cognition AI at a glance
| Operator | Cognition AI |
|---|---|
| Crawler in directory | 1 |
| Categories | AI & LLM Bots |
| SEO impact if blocked | 2/10 |
| Crawl behaviour | 1 rated safe |
| Website | cognition.ai |
Crawler operated by Cognition AI
What Cognition AI's crawlers are for
Cognition AI runs 1 crawler in our database. Grouped by the job each one does:
- a user-triggered agent:
Devin
The highest-priority token is Tier 2. That puts Cognition AI among the operators whose crawlers have a direct effect on whether AI answers or search results can include your pages.
How site owners treat Cognition AI
How often site owners write rules for Cognition AI's tokens, from our own robots.txt study of the Tranco top 10,000:
| Token | Files naming it | Files blocking it |
|---|---|---|
Devin | 77 (1.4%) | Not measured |
A Cognition AI token can be blocked without being named, through a User-agent: * group with Disallow: /, so its blocking share can be higher than its naming share.
All counts come from 5,583 valid files fetched October 20, 2026.
Writing one policy for Cognition AI
A robots.txt file cannot say "Cognition AI"; it can only name Cognition AI's tokens, one group at a time. With a single token, one group is the whole policy.
# One token controls everything Cognition AI fetches through it
User-agent: Devin
Disallow: /private/
Allow: /- Decide per job, not per company. Training, search and user-triggered fetching have different trade-offs, even from the same operator.
- Name every token you mean. A token you leave out follows
User-agent: *. - Repeat shared rules. A named group ignores the
*group, so copy anyDisallowlines you still want it to obey. - Verify from outside. Run the AI visibility checker after any change to see the result for each Cognition AI token, including firewall refusals.
Test yourself: does User-agent: Cognition AI block all of Cognition AI's crawlers?
No. robots.txt matches crawler tokens, not company names. Use the token Devin.
Test yourself: if you block one Cognition AI token, are the others blocked too?
No. Each token is read separately. Blocking one leaves the rest following whatever group applies to them.
Each Cognition AI crawler, explained
A one-paragraph brief for every Cognition AI token in our database, covering what it does, how it treats robots.txt and how often large sites write rules for it:
- Devin (Cognition) (
Devin, Tier 2): a user-triggered agent that fetches a page when a person asks an assistant or app to open it. Rated safe. Devin (Cognition)'s operator does not clearly document how it treats robots.txt. Write the Devin (Cognition) rule anyway, then confirm in your logs that its requests stop. 1.4% of top-site robots.txt files name it.
Common mistakes with Cognition AI's crawlers
The errors we see most often with Cognition AI's crawlers in the files and firewalls we test:
- Assuming a robots.txt rule will stop Cognition AI's crawlers. User-triggered fetches act for a person, and some operators say robots.txt may not apply to them. Check the operator's statement above.
- Blocking Cognition AI's crawlers as if it were a bulk crawler. It fetches one page because a person asked; blocking it means that person's assistant cannot read your page.
- Counting Cognition AI's crawlers visits as crawl volume. Each request maps to a real user question, which makes the paths it requests a useful signal of what people ask about you.
All three hide inside a file that looks correct, so check Cognition AI's crawlers from outside after every change.
What our own data shows about Cognition AI
Everything below comes from the AI Crawler Check bot database and our two most recent robots.txt scans of the Tranco top 10,000, not from Cognition AI's marketing. Cognition AI has one token in our database, so the whole picture fits in one place.
Devin in our records
We first recorded Devin on June 18, 2026. We could not find a page where Cognition AI names it, so treat the token as reported rather than documented. Devin is outside our tracked token set, but 77 of 5,583 top-site files already name it, which is enough to show owners treat it as a real crawler.
Among the 104 AI and LLM crawlers in our database, Devin sits 36th by priority (Tier 2), level with AddSearchBot, AI2Bot, AI2Bot-DeepResearchEval, aiHitBot. 81 of the 104 share its safe rating. Other operators with a crawler that fetches a page when a person asks an assistant or app to open it: ChatGPT Agent (OpenAI, Tier 1), ChatGPT Operator (OpenAI, Tier 1), ChatGPT-User (OpenAI, Tier 1), ChatGPT-User/2.0 (OpenAI, Tier 1), Claude-SearchBot (Anthropic, Tier 1). A policy for Devin is usually a policy for these too.
A decision record for Devin
Teams that review crawlers once a quarter keep one short entry per token, so anyone can see why a rule exists and when it was last checked. Filled in from our data, the Devin entry reads:
- Operator and job: Cognition AI; it fetches a page when a person asks an assistant or app to open it.
- Priority and behaviour: Tier 2, rated safe, blocking impact low.
- Suggested starting point: Allow, and watch it in your logs.
- Enforcement: Devin's operator does not clearly document how it treats robots.txt. Write the Devin rule anyway, then confirm in your logs that its requests stop.
User-agent: Devin
Allow: /Paste the line into the validator with one of your own URLs before it goes live. Revisit the Devin entry whenever Cognition AI announces a change, because a Tier 2 crawler affects visibility directly.
Checking Devin on your own site
Because Cognition AI runs only Devin in our records, one check covers the whole operator. Search your access logs for the string Devin over the last 30 days, note how many requests it made and which paths it asked for, then compare that with what your robots.txt allows. Cognition AI publishes no page about the crawler that we could find, so judge it by the requests you actually see rather than by its name. A Tier 2 crawler rated safe is worth confirming every month, and our log checker does the counting for you.
How top sites treat Cognition AI's crawlers
From our robots.txt study of the Tranco top 10,000 (5,583 valid files, fetched October 20, 2026):
None of the tokens for Cognition AI's crawlers is tracked by name in our study. For scale, 20.1% of top-site files block at least one AI training crawler and 4.3% refuse every unnamed crawler with User-agent: * and Disallow: /, which would catch Cognition AI's crawlers too.
More questions about Cognition AI
- Will blocking Cognition AI affect my Google rankings?
- Only if one of its tokens is a Google search crawler. None of Cognition AI's tokens are Googlebot, so classic Google rankings are unaffected.
- How quickly does a robots.txt change take effect for Cognition AI?
- Crawlers cache robots.txt, commonly for up to a day. Expect a change to apply within about 24 hours, and confirm it in your logs.
- Can I allow Cognition AI on some sections and block it on others?
- Yes. Inside the group for
Devin, addDisallowlines for the paths to keep private andAllowlines for exceptions. The longest matching rule wins. - Does blocking Cognition AI delete content it already collected?
- No. robots.txt only controls future visits. For content already collected, check whether Cognition AI offers a removal or opt-out process.
- Is a firewall block better than robots.txt for Cognition AI?
- A firewall enforces the decision, while robots.txt asks politely. Many operators recommend robots.txt because blocking their IPs can stop them reading your rules at all. Use robots.txt first, and add a firewall rule only if requests continue.
A five-minute Cognition AI review
- Confirm the token. Compare the user-agent in your logs with the tokens listed on this page. A close spelling is not the same crawler.
- Decide the outcome per crawler. Use the briefs above to choose allow, limit or block for each token.
- Write the group. Add or edit one
User-agentgroup per decision and repeat any shared rules inside it. - Publish and wait. Allow about a day for crawlers to fetch the new file.
- Confirm the result. Check your logs, then run the AI crawler access test to see the outcome from outside, including any firewall layer.
How to block or allow Cognition AI in robots.txt
This rule covers the single crawler Cognition AI operates. Add it to the robots.txt file at the root of your domain.
Block Cognition AI completely
User-agent: Devin
Disallow: /
Allow Cognition AI explicitly
User-agent: Devin
Allow: /
These Cognition AI rules ask; they do not enforce. If a Cognition AI token ignores them, block it at the server or CDN. Paths are case-sensitive, and each rule matches the token, not the full user-agent string.
Frequently asked questions about Cognition AI
What crawler does Cognition AI run?
Cognition AI operates 1 crawler in our directory: Devin (Cognition) (user-agent Devin).
Should I block Cognition AI?
Generally safe to allow; provides legitimate crawling value. The SEO impact score for blocking is 2/10, where 10 means blocking costs you the most visibility.
How do I block Cognition AI in robots.txt?
Add the following to robots.txt at your domain root: User-agent: Devin then Disallow: /.
Does blocking Cognition AI hurt my search rankings?
Blocking impact is rated: Low - No SEO ranking impact. Google's own ranking crawler is Googlebot, which this operator does not run, so blocking Cognition AI does not by itself remove you from Google's index. What it can change is whether AI assistants and analytics tools can read your pages.