Safe Official docs AI & LLM Bots

Google-Extended

Q: What is the official user-agent string for Google-Extended?

The official user-agent string for Google-Extended is: Google-Extended . This is the exact string you must use in robots.txt, Nginx, Apache, or Cloudflare firewall rules to target this bot. User-agent matching in robots.txt is case-insensitive, but the string must be spelled correctly. You can verify that a request genuinely comes from Google-Extended by performing a reverse-DNS lookup on the source IP - legitimate bots resolve back to their operator's domain.

Q: Is Google-Extended safe?

Yes. Google-Extended is a documented bot: Google publishes a page that names it and explains what it does. Note how it treats robots.txt (below) before relying on a Disallow rule.

Q: How do I block Google-Extended in robots.txt?

Add the following lines to your /robots.txt file: User-agent: Google-Extended Disallow: / This instructs Google-Extended not to crawl any path on your site. The Disallow: / directive covers the entire domain including subfolders. To only block specific sections, replace / with the path (e.g., Disallow: /blog/ ). Note: robots.txt is publicly readable - any bot or human can inspect it at yourdomain.com/robots.txt .

Q: How do I verify if Google-Extended is crawling my site?

Search your web server access logs for the string Google-Extended (case-insensitive grep: grep -i "Google-Extended" /var/log/nginx/access.log ). You can also check Google Search Console → Coverage → Crawl Stats for Googlebot variants. For Google-Extended specifically, filter by user-agent in your log analysis tool (GoAccess, AWStats, etc.).

Operated by Google

Scout's quick take

Google-Extended is not a crawler you will see in your logs. It is a robots.txt token: a Disallow rule for it opts your content out of a specific AI use, while normal crawling carries on.

User-agent: Google-Extended
Disallow: /

Run an AI crawlability test Build a full robots.txt

Quick Facts

User-Agent:: Google-Extended
Category:: AI & LLM Bots
Operator:: Google
Safety:: Safe
Blocking Impact:: Low - No SEO ranking impact
SEO Impact Score:: 2/10
Official docs:: developers.google.com

Confirmed by the operator

Google names Google-Extended on its own documentation page, linked above. The token, purpose and robots.txt behaviour on this profile come from that page, re-checked on October 19, 2026.

What is Google-Extended?

Google-Extended is not a crawler. It is a robots.txt token: disallowing it tells Google not to use content crawled by Googlebot for Gemini training and grounding.

Disallowing Google-Extended opts content out of Gemini training and grounding. It does not remove pages from Google Search. It identifies itself with the token Google-Extended. Google documents it at developers.google.com.

What happens if you block Google-Extended?

No effect on Google Search rankings or AI Overviews eligibility, which Googlebot controls.

Generally safe to allow; provides legitimate crawling value.

Is Google-Extended a bot?

Google-Extended is not a separate bot: it is a robots.txt control token that Google's existing crawlers read. Disallowing the Google-Extended bot token opts your content out of Gemini training and grounding, never appears in your server logs, and has no effect on Google Search rankings.

User-agent: Google-Extended
Disallow: /

Want to see whether Google-Extended actually visits you? Read how do I monitor AI bot traffic to my website?

How to block Google-Extended with robots.txt

<code>User-agent: Google-Extended</code> - Matching is case-insensitive. Robots.txt is fetched from the root of each subdomain separately.

Block completely (robots.txt)

User-agent: Google-Extended
Disallow: /

Allow all (robots.txt)

User-agent: Google-Extended
Allow: /

Block private only (robots.txt)

User-agent: Google-Extended
Disallow: /private/
Disallow: /api/
Disallow: /admin/
Allow: /

Nginx server block

# Nginx: Hard-block Google-Extended
if ($http_user_agent ~* "Google\-Extended") {
    return 403 "Bot blocked";
}

Apache .htaccess

# Apache: Hard-block Google-Extended
SetEnvIfNoCase User-Agent "Google\-Extended" bad_bot
Order Allow,Deny
Allow from all
Deny from env=bad_bot

Meta robots tag

X-Robots-Tag header

X-Robots-Tag: noindex, nofollow

Is Google-Extended safe to allow?

Yes. Google-Extended is a documented bot: Google publishes a page that names it and explains what it does. Note how it treats robots.txt (below) before relying on a Disallow rule.

Verify by reverse-DNS lookup: legitimate Google-Extended requests resolve to google's domain.

What does Google-Extended do?

From Google's own documentation.

Google-Extended is not a crawler.
Content may be used to train AI models
No effect on Google Search rankings or AI Overviews eligibility, which Googlebot controls.

Google-Extended in context

Google-Extended is an opt-out token rather than a crawler operated by Google. In plain terms, it collects pages to train AI models. Our database files it under AI and LLM crawlers and ranks it Tier 1, the small set whose absence you would notice in AI answers or search results. 14 of the 104 AI and LLM crawlers in our database share this tier.

Its safety rating is safe and its blocking impact is low. So the decision about Google-Extended rests on what you want from Google, not on its behaviour. How we rate safety and impact.

How site owners treat Google-Extended

In our own study of robots.txt files across the Tranco top 10,000 sites (fetched October 20, 2026, 5,583 valid files), 801 files (14.3%) name Google-Extended in a User-agent line, and 723 (13%) block it from the homepage. 519 of those blocks come from a group written for this crawler by name, and 204 come from a broad User-agent: * rule that catches it by accident. Larger sites are stricter: 18.6% of the top 1,000 block it, against 13.1% of the middle band and 11.9% of the tail.

Does it follow robots.txt?

Google-Extended is a control token, not a crawler, so it never appears in your logs. You set Google-Extended in robots.txt and the operator reads that setting through its other crawlers.

Google crawlers and close alternatives

Google-Extended is one of 32 Google crawlers in our database. A User-agent: Google-Extended group controls Google-Extended only; the other Google tokens below need their own lines.

Google-Extended: this crawler (Tier 1)
AdsBot-Google: Google Ads Quality (Desktop), checks pages for an SEO, uptime or analytics service (Tier 3)
AdsBot-Google-Mobile: Google Ads Quality (Mobile), checks pages for an SEO, uptime or analytics service (Tier 3)
APIs-Google: APIs-Google, fetches a page when a person asks an assistant or app to open it (Tier 3)
Chrome-Lighthouse: Lighthouse Audit, checks pages for an SEO, uptime or analytics service (Tier 4)
FeedFetcher-Google: Google Feed Fetcher, builds a search or answer index (Tier 3)
Gemini-Deep-Research: Gemini-Deep-Research, fetches a page when a person asks an assistant or app to open it (Tier 3)
Google-Agent: Google Agent (Project Mariner), fetches a page when a person asks an assistant or app to open it (Tier 2)
Google-CloudVertexBot: Google Vertex AI, fetches a page when a person asks an assistant or app to open it (Tier 3)

To give Google-Extended the same answer as its siblings, stack User-agent: Google-Extended with their lines above one set of rules; to treat Google-Extended differently, give it its own group.

Other AI and LLM crawlers that do the same job (collects pages to train AI models) include ClaudeBot from Anthropic, GPTBot from OpenAI, Meta-ExternalAgent from Meta, AI2Bot from Allen Institute for AI, aiHitBot from aiHit, anthropic-ai from Anthropic. Deciding about Google-Extended is usually a decision about the whole group, so check how your file treats these too.

We first recorded Google-Extended on March 17, 2026. Google documents it on developers.google.com, and we recheck that page before each release.

Which rule Google-Extended actually follows

Google-Extended reads exactly one group in robots.txt, the most specific one that names Google-Extended, and never combines rules across groups. This short file shows the trap:

User-agent: *
Disallow: /private/

User-agent: Google-Extended
Disallow: /drafts/

Find the group. Google-Extended looks for a group naming Google-Extended. One exists, so it ignores User-agent: * completely.
Apply only that group. It is refused /drafts/ and nothing else.
The surprise. /private/ is open to Google-Extended, because that rule lives in the * group it no longer reads. To keep it out of both, repeat Disallow: /private/ inside its own group.

Test yourself: does a rule under User-agent: * apply to Google-Extended if the file also has a User-agent: Google-Extended group?

No. Once a group names it, Google-Extended reads only that group. Put every rule you want it to follow inside its own group.

Test yourself: will blocking Google-Extended remove your pages from AI answers?

No. Google-Extended is a token, not a crawler. It controls how content is used, not whether a crawler can fetch it.

Test yourself: is User-agent: google-extended the same as User-agent: Google-Extended?

Yes. RFC 9309 makes user-agent matching case-insensitive. Paths are a different story: Disallow: /Blog/ does not block /blog/.

Should you allow Google-Extended? Decide by situation

Recommended setting for Google-Extended by type of site
If you are...	Recommended setting
You publish content and want AI answers to cite it	Low stakes either way. It does not decide whether AI answers cite you.
You license content and want to stop AI training	Blocking it does not stop training, because Google-Extended is an opt-out token rather than a crawler. Block the operator's training crawler instead.
You run an online shop	Allow it unless your logs show it is costly.
Your server is under load from bots	Rate-limit before blocking. It is rated safe in our database.

Check Google-Extended on your own site

Open /robots.txt on your domain and search for Google-Extended. If it is not named, it follows User-agent: *.
List the Disallow lines in the group Google-Extended follows and test your key URLs against them; the longest match wins, and Allow wins a tie.
Look beyond the file. A firewall or bot manager can refuse Google-Extended even when robots.txt allows it, and that never shows in the file.
Run our AI crawlability checker to see how Google-Extended and every other crawler we track are treated in one pass, including firewall refusals.

Common mistakes with Google-Extended

The errors we see most often with Google-Extended in the files and firewalls we test:

Looking for Google-Extended in your logs. It is a control token, so it never appears there.
Thinking that blocking Google-Extended removes you from search. It controls how content is used, not whether the search crawler can fetch it.
Adding Google-Extended to a firewall rule. There is no traffic to match; the setting only works in robots.txt.

All three hide inside a file that looks correct, so check Google-Extended from outside after every change.

Frequently Asked Questions

What is the official user-agent string for Google-Extended?

The official user-agent string for Google-Extended is: Google-Extended. This is the exact string you must use in robots.txt, Nginx, Apache, or Cloudflare firewall rules to target this bot. User-agent matching in robots.txt is case-insensitive, but the string must be spelled correctly. You can verify that a request genuinely comes from Google-Extended by performing a reverse-DNS lookup on the source IP - legitimate bots resolve back to their operator's domain.

Is Google-Extended safe?

Yes. Google-Extended is a documented bot: Google publishes a page that names it and explains what it does. Note how it treats robots.txt (below) before relying on a Disallow rule.

Will blocking Google-Extended hurt my SEO?

No effect on Google Search rankings or AI Overviews eligibility, which Googlebot controls.

How do I block Google-Extended in robots.txt?

Add the following lines to your /robots.txt file:

User-agent: Google-Extended
Disallow: /

This instructs Google-Extended not to crawl any path on your site. The Disallow: / directive covers the entire domain including subfolders. To only block specific sections, replace / with the path (e.g., Disallow: /blog/). Note: robots.txt is publicly readable - any bot or human can inspect it at yourdomain.com/robots.txt.

Does Google-Extended respect robots.txt?

It is itself a robots.txt directive, so the question does not apply. It never appears in server logs.

How do I verify if Google-Extended is crawling my site?

Search your web server access logs for the string Google-Extended (case-insensitive grep: grep -i "Google-Extended" /var/log/nginx/access.log). You can also check Google Search Console → Coverage → Crawl Stats for Googlebot variants. For Google-Extended specifically, filter by user-agent in your log analysis tool (GoAccess, AWStats, etc.).

What is the crawl frequency of Google-Extended?

Google-Extended crawls at a moderate rate. If you notice excessive traffic in your logs, you can add a Crawl-delay directive:

User-agent: Google-Extended
Crawl-delay: 10

(10 second delay between requests).

Can I block Google-Extended from specific pages only?

Yes. Instead of a global Disallow: / you can restrict Google-Extended to specific paths:

User-agent: Google-Extended
Disallow: /private/
Disallow: /staging/
Allow: /

This allows Google-Extended everywhere except the listed paths. Path matching in robots.txt uses prefix matching - Disallow: /private/ blocks /private/page.html but NOT /public/private/.

Does blocking Google-Extended prevent AI training on my content?

A robots.txt Disallow for Google-Extended is the opt-out Google documents. Robots.txt is voluntary, so for enforcement add a firewall or CDN rule. Disallowing Google-Extended opts content out of Gemini training and grounding. It does not remove pages from Google Search.

Is there an alternative to robots.txt to opt out of Google-Extended?

The opt-out Google documents is robots.txt. A firewall, CDN or server rule that returns 403 for Google-Extended enforces it. Per-bot meta tags and llms.txt are not opt-out mechanisms any major operator documents.