Google-Extended vs PerplexityBot: Which AI Crawler Should You Allow or Block?
Complete side-by-side comparison of Google-Extended (Google) and PerplexityBot (Perplexity AI): user-agent strings, safety ratings, SEO impact, and copy-paste blocking rules. Updated August 2026.
Quick Verdict
Both Google-Extended and PerplexityBot carry the same safety rating (Safe). Your decision should come down to whether you want your content used by Google versus Perplexity AI, and whether each bot's purpose (training, search, or user requests) aligns with your content strategy.
| Attribute | Google-Extended | PerplexityBot |
|---|---|---|
| Operator | Perplexity AI | |
| User-agent string | Google-Extended |
PerplexityBot |
| Category | AI & LLM Bots | AI & LLM Bots |
| Safety rating | Safe | Safe |
| SEO impact if blocked | Low - No SEO ranking impact | Low - No SEO ranking impact |
| Recommendation | Generally safe to allow; provides legitimate crawling value. | Generally safe to allow; provides legitimate crawling value. |
| Official docs | Google-Extended docs | PerplexityBot docs |
What is Google-Extended?
Google-Extended is not a crawler. The full profile covers the rest.
What is PerplexityBot?
PerplexityBot is Perplexity's search crawler. The full profile covers the rest.
Google-Extended vs PerplexityBot by the numbers
Google-Extended is an opt-out token rather than a crawler and PerplexityBot is a retrieval crawler that feeds AI search answers. They do different jobs, so a sensible policy often treats them differently.
| Measure | Google-Extended | PerplexityBot |
|---|---|---|
| robots.txt token | Google-Extended | PerplexityBot |
| What it does | collects pages to train AI models | collects pages both for training and for search or answers |
| Crawler type | an opt-out token rather than a crawler | a retrieval crawler that feeds AI search answers |
| Priority tier | Tier 1 | Tier 1 |
| Follows robots.txt | Not applicable (token) | With exceptions |
| Top-10k robots.txt files | 801 named (14.3%), 723 blocked (13%) | 718 named (12.9%), 609 blocked (10.9%) |
| Blocked by top 1k / middle / tail | 18.6% / 13.1% / 11.9% | 17% / 11.6% / 9.3% |
In our robots.txt study of the Tranco top 10,000 (5,583 valid files, fetched October 20, 2026), Google-Extended is blocked more often than PerplexityBot: 723 files against 609. That gap shows many site owners already treat the two differently.
Writing rules for both
Because each crawler reads only the group that names it, you can give Google-Extended and PerplexityBot different answers in the same file. A common pattern, refusing training while allowing search and assistant fetches, looks like this for these two:
User-agent: Google-Extended
Allow: /
User-agent: PerplexityBot
Allow: /- One group each. Do not rely on
User-agent: *to cover either crawler if you have named the other; name both. - Copy shared rules. Any
Disallowlines from the*group that should still apply must be repeated in each named group. - Mind the edge. A bot manager can refuse one and admit the other regardless of robots.txt.
- Verify both. The GPTBot and ClaudeBot access check shows how each is treated on your live site.
Test yourself: if you write User-agent: Google-Extended and User-agent: PerplexityBot on consecutive lines above one Disallow: /, are both blocked?
Yes. Several User-agent lines in a row form one group, and the rules below apply to every name in it.
Test yourself: does blocking Google-Extended also block PerplexityBot?
No. Each token is matched separately.
Which to allow, by goal
| Your goal | Google-Extended | PerplexityBot |
|---|---|---|
| You want AI answers to cite you | Allow | Allow |
| You want to stop AI training | No effect | No effect |
| Your server is under load | Rate-limit | Rate-limit |
| You need search traffic | No effect | No effect |
Set Google-Extended to control how content is used, not whether it is fetched. Allow PerplexityBot if you want AI answers to read and cite your pages.
The rest of each operator's family
Google also runs AdsBot-Google, AdsBot-Google-Mobile, APIs-Google, Chrome-Lighthouse, FeedFetcher-Google. Rules for Google-Extended do not cover them. Perplexity AI also runs Perplexity-User, Perplexity-User-1.0. Rules for PerplexityBot do not cover them.
Read the full profiles for both crawlers, linked in the table above, for documentation links, user-agent strings and server-side rules.
Mistakes when writing rules for Google-Extended and PerplexityBot together
Google-Extended is named in 14.3% of the top-site robots.txt files in our study and blocked in 13%; PerplexityBot is named in 12.9% of the top-site robots.txt files in our study and blocked in 10.9%. Large sites block Google-Extended more often, which usually reflects a decision about training rather than about PerplexityBot.
- They are different kinds of agent (an opt-out token rather than a crawler versus a retrieval crawler that feeds AI search answers), so the usual error is giving them the same rule out of habit. Blocking both to stop training achieves nothing, because neither is a training crawler.
- They come from different operators (Google and Perplexity AI), with different documentation and different update cycles. Do not assume a change one of them announces applies to the other.
- Putting
User-agent: Google-ExtendedandUser-agent: PerplexityBotin separate groups and then editing only one. Two groups drift apart over time; keep a comment above each that says why it differs from the other.
Each crawler's own profile lists the mistakes that apply to it alone; this list is only about combining the two.
Why Google-Extended never shows up next to PerplexityBot in your logs
Google-Extended is a control token from Google, not a crawler: nothing with that name ever requests a page. PerplexityBot is a a retrieval crawler that feeds AI search answers from Perplexity AI that does send requests, so it is the only one of the two you can count, verify or rate-limit.
- A firewall rule can act on PerplexityBot but not on Google-Extended. The token only works as a robots.txt line.
- Blocking Google-Extended does not slow Google's crawling. It changes what Google may do with pages its other crawlers already fetch.
- In our study of top sites, 13% block Google-Extended and 10.9% block PerplexityBot. Opting out by token is the more common choice here.
How Google-Extended and PerplexityBot moved in our latest scans
Between our scans of October 17, 2026 and October 20, 2026, files naming Google-Extended went from 743 to 801 (+58) and files blocking it from 666 to 723 (+57), so more sites wrote a rule for Google-Extended and more of those rules block it. For comparison, 2026 and October 20, 2026, files naming PerplexityBot went from 693 to 718 (+25) and files blocking it from 582 to 609 (+27), so more sites wrote a rule for PerplexityBot and more of those rules block it.
Over the 3 days between those scans, blocking of Google-Extended moved +57 files and blocking of PerplexityBot moved +27. Owners are acting faster on Google-Extended, which is the one to recheck first if you have not touched your file this month.
We first recorded Google-Extended on March 17, 2026. Google documents it on developers.google.com, and we recheck that page before each release. We first recorded PerplexityBot on March 17, 2026. Perplexity AI documents it on docs.perplexity.ai, and we recheck that page before each release.
Among the 104 AI and LLM crawlers in our database, Google-Extended sits eighth by priority (Tier 1), level with ChatGPT Agent, ChatGPT Operator, ChatGPT-User, ChatGPT-User/2.0. 81 of the 104 share its safe rating. On the PerplexityBot side: Among the 104 AI and LLM crawlers in our database, PerplexityBot sits 14th by priority (Tier 1), level with ChatGPT Agent, ChatGPT Operator, ChatGPT-User, ChatGPT-User/2.0. 81 of the 104 share its safe rating.
How top sites treat Google-Extended and PerplexityBot
From our robots.txt study of the Tranco top 10,000 (5,583 valid files, fetched October 20, 2026):
Google-Extended, the most blocked of the Google-Extended and PerplexityBot tokens: blocked in 13% of files (519 with its own group, 204 throughUser-agent: *), more often by the top 1,000 (18.6%) than the long tail (11.9%).PerplexityBot, second of the Google-Extended and PerplexityBot tokens: blocked in 10.9% of files (406 with its own group, 203 throughUser-agent: *), more often by the top 1,000 (17%) than the long tail (9.3%).
For scale, 20.1% of all files block at least one AI training crawler. Google-Extended is one of the tokens driving that figure.
More questions about Google-Extended and PerplexityBot
- Can I allow one and block the other?
- Yes. Give each its own
User-agentgroup with its own rules. Each crawler reads only the group that names it. - Which of the two visits more often?
- It depends on your site, your audience and how often your content changes. Your access logs are the reliable answer; the commands above count both.
- How long until a change applies?
- Crawlers commonly cache robots.txt for up to a day, so allow about 24 hours and confirm in your logs.
- Do I need to update my rules when either operator launches a new crawler?
- Often, yes. A new token is not covered by rules written for the old ones. Check this comparison and both profiles periodically, or rerun the AI crawlability checker after any announcement.
Frequently asked questions
What is the difference between Google-Extended and PerplexityBot?
Google-Extended is operated by Google (Google-Extended is not a crawler. It is a robots.txt token: disallowing it tells Google not to use content crawled by Googlebot for Gemini training and grounding.) while PerplexityBot is operated by Perplexity AI (PerplexityBot is Perplexity's search crawler. Perplexity says it is "designed to surface and link websites in search results".). They use different user-agent strings (Google-Extended vs PerplexityBot) and must be blocked separately in robots.txt.
Should I block Google-Extended or PerplexityBot?
Both Google-Extended and PerplexityBot carry the same safety rating (Safe). Your decision should come down to whether you want your content used by Google versus Perplexity AI, and whether each bot's purpose (training, search, or user requests) aligns with your content strategy.
Does blocking Google-Extended or PerplexityBot hurt SEO?
Google-Extended: Low - No SEO ranking impact. PerplexityBot: Low - No SEO ranking impact. Neither is a traditional search-ranking crawler, so blocking them does not affect Google or Bing rankings, but it may reduce your visibility in AI-generated answers.
How do I block both Google-Extended and PerplexityBot?
Add both user-agent groups to your robots.txt: "User-agent: Google-Extended" with "Disallow: /" and "User-agent: PerplexityBot" with "Disallow: /". Remember robots.txt is voluntary; combine with server-level rules for enforcement.
How to block each bot
Block Google-Extended in robots.txt
User-agent: Google-Extended Disallow: /
Block PerplexityBot in robots.txt
User-agent: PerplexityBot Disallow: /
Setting Google-Extended, PerplexityBot and the rest in one go? Tick both in the robots.txt generator (265+ presets), then confirm the result with the AI crawler checker.
Are Google-Extended and PerplexityBot allowed on YOUR site?
Run a free check to see exactly which AI crawlers can access your website, plus your AI Visibility Score.
Free AI Crawler Access Check