CCBot
CautionUser-agent: CCBot
Category: Data Scrapers
SEO impact if blocked: 2/10
Common Crawl
1 crawler operated by Common Crawl
https://commoncrawl.org/Who Common Crawl is, for site owners
Common Crawl operates 1 crawler in our directory, filed under Data Scrapers. Its crawl behaviour is rated caution.
Blocking Common Crawl carries an SEO impact score of 2/10. Blocking impact is rated: Low - No SEO ranking impact.
Our recommendation for Common Crawl: Consider blocking based on your content strategy.
Common Crawl at a glance
| Operator | Common Crawl |
|---|---|
| Crawler in directory | 1 |
| Categories | Data Scrapers |
| SEO impact if blocked | 2/10 |
| Crawl behaviour | 1 rated caution |
| Website | commoncrawl.org |
| Official crawler docs | Common Crawl crawler documentation |
Crawler operated by Common Crawl
What Common Crawl's crawlers are for
Common Crawl runs 1 crawler in our database. Grouped by the job each one does:
- a training crawler:
CCBot
The highest-priority token is Tier 6. That makes Common Crawl a lower-priority operator for most sites: worth a deliberate decision, but rarely the one that decides your visibility.
How site owners treat Common Crawl
How often site owners write rules for Common Crawl's tokens, from our own robots.txt study of the Tranco top 10,000:
| Token | Files naming it | Files blocking it |
|---|---|---|
CCBot | 845 (15.1%) | 899 (16.1%) |
A Common Crawl token can be blocked without being named, through a User-agent: * group with Disallow: /, so its blocking share can be higher than its naming share.
All counts come from 5,583 valid files fetched October 20, 2026.
Writing one policy for Common Crawl
A robots.txt file cannot say "Common Crawl"; it can only name Common Crawl's tokens, one group at a time. With a single token, one group is the whole policy.
# One token controls everything Common Crawl fetches through it
User-agent: CCBot
Disallow: /private/
Allow: /- Decide per job, not per company. Training, search and user-triggered fetching have different trade-offs, even from the same operator.
- Name every token you mean. A token you leave out follows
User-agent: *. - Repeat shared rules. A named group ignores the
*group, so copy anyDisallowlines you still want it to obey. - Verify from outside. Run the AI crawlability test after any change to see the result for each Common Crawl token, including firewall refusals.
Test yourself: does User-agent: Common Crawl block all of Common Crawl's crawlers?
No. robots.txt matches crawler tokens, not company names. Use the token CCBot.
Test yourself: if you block one Common Crawl token, are the others blocked too?
No. Each token is read separately. Blocking one leaves the rest following whatever group applies to them.
Each Common Crawl crawler, explained
A one-paragraph brief for every Common Crawl token in our database, covering what it does, how it treats robots.txt and how often large sites write rules for it:
- CCBot (
CCBot, Tier 6): a training crawler that collects pages to train AI models. Rated caution. CCBot's operator says it follows robots.txt, so a correctly written CCBot group is enough to control it. 15.1% of top-site robots.txt files name it and 16.1% block it.
Common mistakes with Common Crawl's crawlers
The errors we see most often with Common Crawl's crawlers in the files and firewalls we test:
- Assuming that blocking Common Crawl's crawlers removes you from AI answers. It is a training crawler; AI answers are fed by retrieval and user-triggered crawlers with different tokens.
- Blocking Common Crawl's crawlers with a broad
User-agent: *rule that also catches search engines. NameCCBotin its own group instead. - Expecting a Common Crawl's crawlers block to remove content already collected. robots.txt only affects future visits.
All three hide inside a file that looks correct, so check Common Crawl's crawlers from outside after every change.
What our own data shows about Common Crawl
Everything below comes from the AI Crawler Check bot database and our two most recent robots.txt scans of the Tranco top 10,000, not from Common Crawl's marketing. Common Crawl has one token in our database, so the whole picture fits in one place.
CCBot in our records
We first recorded CCBot on March 17, 2026. Common Crawl documents it on commoncrawl.org, and we recheck that page before each release. Between our scans of October 17, 2026 and October 20, 2026, files naming CCBot went from 783 to 845 (+62) and files blocking it from 834 to 899 (+65), so more sites wrote a rule for CCBot and more of those rules block it.
Among the 39 data scrapers in our database, CCBot sits eighth by priority (Tier 6), level with Amazon Kendra, ApifyBot, Barkrowler, BLEXBot. 6 of the 39 share its caution rating. Other operators with a crawler that collects pages to train AI models: ClaudeBot (Anthropic, Tier 1), Google-Extended (Google, Tier 1), GPTBot (OpenAI, Tier 1), Meta-ExternalAgent (Meta, Tier 1), AI2Bot (Allen Institute for AI, Tier 2). A policy for CCBot is usually a policy for these too.
A decision record for CCBot
Teams that review crawlers once a quarter keep one short entry per token, so anyone can see why a rule exists and when it was last checked. Filled in from our data, the CCBot entry reads:
- Operator and job: Common Crawl; it collects pages to train AI models.
- Priority and behaviour: Tier 6, rated caution, blocking impact low.
- Suggested starting point: Your licensing call: Disallow to refuse training, Allow to permit it.
- Enforcement: CCBot's operator says it follows robots.txt, so a correctly written CCBot group is enough to control it.
User-agent: CCBot
Allow: /Paste the line into the validator with one of your own URLs before it goes live. Revisit the CCBot entry sooner than most: a caution rating means Common Crawl traffic deserves a monthly look at your logs.
Checking CCBot on your own site
Because Common Crawl runs only CCBot in our records, one check covers the whole operator. Search your access logs for the string CCBot over the last 30 days, note how many requests it made and which paths it asked for, then compare that with what your robots.txt allows. Match the requests against what Common Crawl publishes about the crawler, since that page is the only authority on its behaviour. A Tier 6 crawler rated caution rarely needs more than a quarterly look, and our log checker does the counting for you.
How top sites treat Common Crawl's crawlers
From our robots.txt study of the Tranco top 10,000 (5,583 valid files, fetched October 20, 2026):
CCBot, the only one tracked among the Common Crawl's crawlers tokens: blocked in 16.1% of files (680 with its own group, 219 throughUser-agent: *), more often by the top 1,000 (22.4%) than the long tail (15%).
For scale, 20.1% of all files block at least one AI training crawler. CCBot is one of the tokens driving that figure.
More questions about Common Crawl
- Will blocking Common Crawl affect my Google rankings?
- Only if one of its tokens is a Google search crawler. None of Common Crawl's tokens are Googlebot, so classic Google rankings are unaffected.
- How quickly does a robots.txt change take effect for Common Crawl?
- Crawlers cache robots.txt, commonly for up to a day. Expect a change to apply within about 24 hours, and confirm it in your logs.
- Can I allow Common Crawl on some sections and block it on others?
- Yes. Inside the group for
CCBot, addDisallowlines for the paths to keep private andAllowlines for exceptions. The longest matching rule wins. - Does blocking Common Crawl delete content it already collected?
- No. robots.txt only controls future visits. For content already collected, check whether Common Crawl offers a removal or opt-out process.
- Is a firewall block better than robots.txt for Common Crawl?
- A firewall enforces the decision, while robots.txt asks politely. Many operators recommend robots.txt because blocking their IPs can stop them reading your rules at all. Use robots.txt first, and add a firewall rule only if requests continue.
A five-minute Common Crawl review
- Confirm the token. Compare the user-agent in your logs with the tokens listed on this page. A close spelling is not the same crawler.
- Decide the outcome per crawler. Use the briefs above to choose allow, limit or block for each token.
- Write the group. Add or edit one
User-agentgroup per decision and repeat any shared rules inside it. - Publish and wait. Allow about a day for crawlers to fetch the new file.
- Confirm the result. Check your logs, then run the AI crawler tester to see the outcome from outside, including any firewall layer.
How to block or allow Common Crawl in robots.txt
This rule covers the single crawler Common Crawl operates. Add it to the robots.txt file at the root of your domain.
Block Common Crawl completely
User-agent: CCBot
Disallow: /
Allow Common Crawl explicitly
User-agent: CCBot
Allow: /
These Common Crawl rules ask; they do not enforce. If a Common Crawl token ignores them, block it at the server or CDN. Paths are case-sensitive, and each rule matches the token, not the full user-agent string.
Frequently asked questions about Common Crawl
What crawler does Common Crawl run?
Common Crawl operates 1 crawler in our directory: CCBot (user-agent CCBot).
Should I block Common Crawl?
Consider blocking based on your content strategy. The SEO impact score for blocking is 2/10, where 10 means blocking costs you the most visibility.
How do I block Common Crawl in robots.txt?
Add the following to robots.txt at your domain root: User-agent: CCBot then Disallow: /.
Does blocking Common Crawl hurt my search rankings?
Blocking impact is rated: Low - No SEO ranking impact. Google's own ranking crawler is Googlebot, which this operator does not run, so blocking Common Crawl does not by itself remove you from Google's index. What it can change is whether AI assistants and analytics tools can read your pages.