Perplexity AI

Common mistakes with Perplexity AI's crawlers

The errors we see most often with Perplexity AI's crawlers in the files and firewalls we test:

1. Assuming a robots.txt rule will stop Perplexity AI's crawlers. User-triggered fetches act for a person, and some operators say robots.txt may not apply to them. Check the operator's statement above.
2. Blocking Perplexity AI's crawlers as if it were a bulk crawler. It fetches one page because a person asked; blocking it means that person's assistant cannot read your page.
3. Counting Perplexity AI's crawlers visits as crawl volume. Each request maps to a real user question, which makes the paths it requests a useful signal of what people ask about you.

All three hide inside a file that looks correct, so check Perplexity AI's crawlers from outside after every change.

Spotting Perplexity AI in your logs

robots.txt says what Perplexity-User/1.0 and PerplexityBot may do; your access logs show what they actually did.

Count requests

On a server with standard Nginx or Apache logs, this counts matching requests per day (adjust the log path for your host):

grep -iE "Perplexity-User|PerplexityBot" /var/log/nginx/access.log \
  | awk '{print $4}' | cut -d: -f1 | sort | uniq -c

And this shows which paths it requested most, which is the quickest way to see whether your Disallow lines are being respected:

grep -iE "Perplexity-User|PerplexityBot" /var/log/nginx/access.log \
  | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

Check the requests are genuine

Anyone can send these user-agents, so confirm the source before acting on a match:

• Perplexity-User: compare each address with the ranges at www.perplexity.com/perplexity-user.json. A request claiming to be Perplexity-User/1.0 from outside them is not genuine. Check it separately from PerplexityBot.
• PerplexityBot: compare each address with the ranges at www.perplexity.com/perplexitybot.json. A request claiming to be PerplexityBot from outside them is not genuine. Check it separately from Perplexity-User/1.0.

What our own data shows about Perplexity AI

Everything below comes from the AI Crawler Check bot database and our two most recent robots.txt scans of the Tranco top 10,000, not from Perplexity AI's marketing. It covers 2 Perplexity AI tokens in turn.

Perplexity-User in our records

We first recorded Perplexity-User on March 17, 2026. Perplexity AI documents it on docs.perplexity.ai, and we recheck that page before each release. Between our scans of October 17, 2026 and October 20, 2026, files naming Perplexity-User went from 345 to 357 (+12) and files blocking it from 428 to 449 (+21), so more sites wrote a rule for Perplexity-User and more of those rules block it.

Among the 104 AI and LLM crawlers in our database, Perplexity-User sits 74th by priority (Tier 2), level with AddSearchBot, AI2Bot, AI2Bot-DeepResearchEval, aiHitBot. 22 of the 104 share its caution rating. Other operators with a crawler that fetches a page when a person asks an assistant or app to open it: ChatGPT Agent (OpenAI, Tier 1), ChatGPT Operator (OpenAI, Tier 1), ChatGPT-User (OpenAI, Tier 1), ChatGPT-User/2.0 (OpenAI, Tier 1), Claude-SearchBot (Anthropic, Tier 1). A policy for Perplexity-User is usually a policy for these too.

A decision record for Perplexity-User

Teams that review crawlers once a quarter keep one short entry per token, so anyone can see why a rule exists and when it was last checked. Filled in from our data, the Perplexity-User entry reads:

• Operator and job: Perplexity AI; it fetches a page when a person asks an assistant or app to open it.
• Priority and behaviour: Tier 2, rated caution, blocking impact low.
• Suggested starting point: Allow, and watch it in your logs.
• Enforcement: Perplexity-User's operator says it does not follow robots.txt for these requests, so a robots.txt rule will not stop Perplexity-User. Use a firewall, CDN or server rule if you need to block it.

User-agent: Perplexity-User
Allow: /

Paste the line into the validator with one of your own URLs before it goes live. Revisit the Perplexity-User entry sooner than most: a caution rating means Perplexity AI traffic deserves a monthly look at your logs.

PerplexityBot in our records

We first recorded PerplexityBot on March 17, 2026. Perplexity AI documents it on docs.perplexity.ai, and we recheck that page before each release. Between our scans of October 17, 2026 and October 20, 2026, files naming PerplexityBot went from 693 to 718 (+25) and files blocking it from 582 to 609 (+27), so more sites wrote a rule for PerplexityBot and more of those rules block it.

Among the 104 AI and LLM crawlers in our database, PerplexityBot sits 14th by priority (Tier 1), level with ChatGPT Agent, ChatGPT Operator, ChatGPT-User, ChatGPT-User/2.0. 81 of the 104 share its safe rating. Other operators with a crawler that collects pages both for training and for search or answers: Andibot (Andi, Tier 2), Bytespider (ByteDance, Tier 2), cohere-ai (Cohere, Tier 2), DeepSeekBot (DeepSeek, Tier 2), DoubaoBot (ByteDance, Tier 2). A policy for PerplexityBot is usually a policy for these too.

A decision record for PerplexityBot

Teams that review crawlers once a quarter keep one short entry per token, so anyone can see why a rule exists and when it was last checked. Filled in from our data, the PerplexityBot entry reads:

• Operator and job: Perplexity AI; it collects pages both for training and for search or answers.
• Priority and behaviour: Tier 1, rated safe, blocking impact low.
• Suggested starting point: Allow, and watch it in your logs.
• Enforcement: PerplexityBot's robots.txt behaviour is documented with exceptions or has been publicly questioned, so pair a PerplexityBot rule in robots.txt with a server-side rule if blocking matters to you.

User-agent: PerplexityBot
Allow: /

Paste the line into the validator with one of your own URLs before it goes live. Revisit the PerplexityBot entry whenever Perplexity AI announces a change, because a Tier 1 crawler affects visibility directly.

How top sites treat Perplexity AI's crawlers

From our robots.txt study of the Tranco top 10,000 (5,583 valid files, fetched October 20, 2026):

• PerplexityBot, the most blocked of the Perplexity AI's crawlers tokens: blocked in 10.9% of files (406 with its own group, 203 through User-agent: *), more often by the top 1,000 (17%) than the long tail (9.3%).
• Perplexity-User, second of the Perplexity AI's crawlers tokens: blocked in 8% of files (225 with its own group, 224 through User-agent: *), more often by the top 1,000 (15%) than the long tail (6.8%).

For scale, 20.1% of all files block at least one AI training crawler. PerplexityBot is one of the tokens driving that figure.

More questions about Perplexity AI

Will blocking Perplexity AI affect my Google rankings?

Only if one of its tokens is a Google search crawler. None of Perplexity AI's tokens are Googlebot, so classic Google rankings are unaffected.

How quickly does a robots.txt change take effect for Perplexity AI?

Crawlers cache robots.txt, commonly for up to a day. Expect a change to apply within about 24 hours, and confirm it in your logs.

Can I allow Perplexity AI on some sections and block it on others?

Yes. Inside the group for Perplexity-User, add Disallow lines for the paths to keep private and Allow lines for exceptions. The longest matching rule wins.

Does blocking Perplexity AI delete content it already collected?

No. robots.txt only controls future visits. For content already collected, check whether Perplexity AI offers a removal or opt-out process.

Is a firewall block better than robots.txt for Perplexity AI?

A firewall enforces the decision, while robots.txt asks politely. Many operators recommend robots.txt because blocking their IPs can stop them reading your rules at all. Use robots.txt first, and add a firewall rule only if requests continue.