PerplexityBot vs CCBot: Which AI Crawler Should You Allow or Block?

Mistakes when writing rules for PerplexityBot and CCBot together

PerplexityBot is named in 12.9% of the top-site robots.txt files in our study and blocked in 10.9%; CCBot is named in 15.1% of the top-site robots.txt files in our study and blocked in 16.1%. Large sites block CCBot more often, which usually reflects a decision about training rather than about PerplexityBot.

1. They are different kinds of agent (a retrieval crawler that feeds AI search answers versus a training crawler), so the usual error is giving them the same rule out of habit. Blocking both to stop training only needs the CCBot line; the other one changes something else entirely.
2. They come from different operators (Perplexity AI and Common Crawl), with different documentation and different update cycles. Do not assume a change one of them announces applies to the other.
3. Putting User-agent: PerplexityBot and User-agent: CCBot in separate groups and then editing only one. Two groups drift apart over time; keep a comment above each that says why it differs from the other.

Each crawler's own profile lists the mistakes that apply to it alone; this list is only about combining the two.

Spotting PerplexityBot and CCBot in your logs

robots.txt says what PerplexityBot and CCBot may do; your access logs show what they actually did.

Count requests

On a server with standard Nginx or Apache logs, this counts matching requests per day (adjust the log path for your host):

grep -iE "PerplexityBot|CCBot" /var/log/nginx/access.log \
  | awk '{print $4}' | cut -d: -f1 | sort | uniq -c

And this shows which paths it requested most, which is the quickest way to see whether your Disallow lines are being respected:

grep -iE "PerplexityBot|CCBot" /var/log/nginx/access.log \
  | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

Check the requests are genuine

Anyone can send these user-agents, so confirm the source before acting on a match:

• PerplexityBot: compare each address with the ranges at www.perplexity.com/perplexitybot.json. A request claiming to be PerplexityBot from outside them is not genuine. Check it separately from CCBot.
• CCBot: compare each address with the ranges at index.commoncrawl.org/ccbot.json. A request claiming to be CCBot from outside them is not genuine. Check it separately from PerplexityBot.

How PerplexityBot and CCBot moved in our latest scans

Between our scans of October 17, 2026 and October 20, 2026, files naming PerplexityBot went from 693 to 718 (+25) and files blocking it from 582 to 609 (+27), so more sites wrote a rule for PerplexityBot and more of those rules block it. For comparison, 2026 and October 20, 2026, files naming CCBot went from 783 to 845 (+62) and files blocking it from 834 to 899 (+65), so more sites wrote a rule for CCBot and more of those rules block it.

Related: how AI engines choose their sources

Over the 3 days between those scans, blocking of PerplexityBot moved +27 files and blocking of CCBot moved +65. Owners are acting faster on CCBot, which is the one to recheck first if you have not touched your file this month.

We first recorded PerplexityBot on March 17, 2026. Perplexity AI documents it on docs.perplexity.ai, and we recheck that page before each release. We first recorded CCBot on March 17, 2026. Common Crawl documents it on commoncrawl.org, and we recheck that page before each release.

Among the 104 AI and LLM crawlers in our database, PerplexityBot sits 14th by priority (Tier 1), level with ChatGPT Agent, ChatGPT Operator, ChatGPT-User, ChatGPT-User/2.0. 81 of the 104 share its safe rating. On the CCBot side: Among the 39 data scrapers in our database, CCBot sits eighth by priority (Tier 6), level with Amazon Kendra, ApifyBot, Barkrowler, BLEXBot. 6 of the 39 share its caution rating.

How top sites treat PerplexityBot and CCBot

From our robots.txt study of the Tranco top 10,000 (5,583 valid files, fetched October 20, 2026):

• CCBot, the most blocked of the PerplexityBot and CCBot tokens: blocked in 16.1% of files (680 with its own group, 219 through User-agent: *), more often by the top 1,000 (22.4%) than the long tail (15%).
• PerplexityBot, second of the PerplexityBot and CCBot tokens: blocked in 10.9% of files (406 with its own group, 203 through User-agent: *), more often by the top 1,000 (17%) than the long tail (9.3%).

For scale, 20.1% of all files block at least one AI training crawler. CCBot is one of the tokens driving that figure.

More questions about PerplexityBot and CCBot

Can I allow one and block the other?

Yes. Give each its own User-agent group with its own rules. Each crawler reads only the group that names it.

Which of the two visits more often?

It depends on your site, your audience and how often your content changes. Your access logs are the reliable answer; the commands above count both.

How long until a change applies?

Crawlers commonly cache robots.txt for up to a day, so allow about 24 hours and confirm in your logs.

Do I need to update my rules when either operator launches a new crawler?

Often, yes. A new token is not covered by rules written for the old ones. Check this comparison and both profiles periodically, or rerun the crawlability checker for AI search after any announcement.