Google-Extended vs Googlebot: What Each Controls in 2026 (and the New AI Overviews Switch)

Googlebot and Google-Extended sound like two Google crawlers. They are not. Googlebot is the crawler that fetches your pages for Google Search, including AI Overviews and AI Mode. Google-Extended is only a name you write in robots.txt to tell Google whether it may use content it already crawled to train and ground Gemini. No request ever arrives with "Google-Extended" in the user agent.

That difference decides what each control can and cannot do. This guide explains both, adds the new Search Console control Google rolled out to every site on 31 August 2026, and shows what the top 10,000 sites actually chose, using our October 2026 robots.txt study. To see how your own file treats both tokens right now, run the free AI crawler checker.

Google-Extended vs Googlebot in one table

If you remember one sentence, make it Google's own: Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search" (Google, common crawlers list). Our Google-Extended profile has the token details on one page.

What Googlebot does in 2026

Googlebot is Google's main crawler. There are two versions, Googlebot Smartphone and Googlebot Desktop, and both use the robots.txt token Googlebot. Google says rules for this token affect "Google Search (including Discover and all Google Search features), as well as other products such as Google Images, Google Video, Google News, and Discover." Google's crawlers "always obey robots.txt rules when crawling automatically".

AI Overviews and AI Mode do not have their own crawler. Google's guide for site owners says: "AI is built into Search and integral to how Search functions, which is why robots.txt directives for Googlebot is the control for site owners to manage access to how their sites are crawled for Search" (Google, AI features). To be shown as a supporting link in those features, "a page must be indexed and eligible to be shown in Google Search with a snippet". There are no extra technical requirements.

Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36
(compatible; Googlebot/2.1; +http://www.google.com/bot.html)

W.X.Y.Z is the Chrome version, which changes over time. Google advises using wildcards for it when you filter logs.

What Google-Extended actually controls

Google describes Google-Extended as "a standalone product token that web publishers can use to manage whether content Google crawls from their sites may be used for training future generations of Gemini models" that power Gemini Apps and the Vertex AI API for Gemini, "and for grounding (providing content from the Google Search index to the model at prompt time to improve factuality and relevancy)" in Gemini Apps and Grounding with Google Search on Vertex AI.

So the token covers two uses, both outside Google Search:

• Training: future Gemini models learning from your pages.
• Grounding in Gemini products: the Gemini app, and developers' apps on Vertex AI that use Grounding with Google Search, pulling your pages in at answer time.

Because there is no Google-Extended crawler, blocking it does not reduce crawling. Googlebot still visits, and the content still powers Search. What changes is what Google may do with that content in Gemini. That is also why you cannot see Google-Extended in your logs, and why a log filter for it will always return nothing.

User-agent: Google-Extended
Disallow: /

User-agent: *
Allow: /

The new option: the Search generative AI control

Until this year, a site that wanted to stay in Google Search but leave AI Overviews had only snippet controls. On 31 August 2026 Google rolled out a dedicated setting to all sites worldwide (Search Console Help, AI control). You find it in Search Console under Settings, then Search generative AI.

Google is clear about what this control does not do. It "isn't used as a ranking or inclusion signal affecting other parts of Search". And it "doesn't affect AI training; to limit training of the models used to generate responses in Search generative AI features, use Google-Extended." Changes generally take effect within one to two days, though caching can delay some content.

That gives you three independent switches for three different questions:

Measure before you switch anything off

Google also rolled out the Generative AI performance report on 31 August 2026 (Search Console Help, AI report). It shows impressions from AI Overviews and AI Mode by page, country, device and date. An impression here means a link to your site was shown inside one of those features.

Open it before you exclude anything. If your best pages earn many AI Overview impressions, excluding them gives up visibility that you cannot get back by other means. If they earn almost none, the choice costs little. Google also notes that clicks from results pages with AI Overviews are "higher quality (meaning, users are more likely to spend more time on the site)", which is worth weighing.

For Gemini itself there is no Google-side report yet. You can watch referral traffic from gemini.google.com in your analytics, and our guide to monitoring AI bot traffic explains how to separate it from other sources.

What the top 10,000 sites chose

In our robots.txt study we parsed 4,962 valid files from the Tranco top 10,000. Here is how often each Google token is blocked from the whole site.

The pattern is clear. Sites that block Google-Extended almost always keep Googlebot, which is exactly how the token is meant to work. Google-Extended is blocked a little less often than GPTBot (14.3%) and ClaudeBot (13.4%). News publishers are an exception: BuzzStream found 46% of 100 top US and UK news sites block Google-Extended, with US publishers at 58% (BuzzStream via Search Engine Journal).

The small number of Googlebot blocks are mostly hosts that block every crawler with User-agent: * and Disallow: /, such as API and CDN domains. Of 125 files that block Googlebot, only 11 do it by name.

Five scenarios and the right setting for each

1. You want every Google surface, including Gemini

Do nothing. Do not name Google-Extended, leave the Search Console control on Include, and allow Googlebot. This is the right choice for most businesses that want to be recommended by AI.

2. You want Search and AI Overviews, but no Gemini training

User-agent: Google-Extended
Disallow: /

Accept that Gemini Apps will not use your pages for grounding either. AI Overviews and AI Mode are not affected.

3. You want Search, but not AI Overviews or AI Mode

Set the Search generative AI control to Exclude in Search Console. Optionally also block Google-Extended if you do not want Gemini training. Do not touch Googlebot.

4. You want to limit how much text Google quotes

<meta name="robots" content="max-snippet:160">
<p data-nosnippet>This paragraph will not be quoted in snippets or AI features.</p>

Google lists nosnippet, data-nosnippet and max-snippet as the way "to limit the information shown from your pages in Search", including AI features.

5. You want out of Google completely

Use noindex on the pages, and keep crawling allowed until Google has seen it. A robots.txt Disallow alone can leave URLs in the index without a snippet.

Common mistakes with the two tokens

The group mistake is easy to make with RFC 9309's grouping rules. These two lines form one group, so both tokens are blocked:

User-agent: Googlebot
User-agent: Google-Extended
Disallow: /

Other Google tokens you may see

Google lists several more common crawlers. Most site owners can leave them in the default group, but it helps to know what each does.

Our Googlebot profile and the bot directory list the full tokens and example groups.

How to verify real Googlebot traffic

Because "Googlebot" is the most copied user agent on the web, fake Googlebot traffic is common. Google publishes two ways to check (Google, Verifying Googlebot). For one-off checks, run a reverse DNS lookup on the IP, confirm the name ends in googlebot.com, google.com or googleusercontent.com, then run a forward lookup and confirm it returns the same IP.

host 66.249.66.1
# 1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.
host crawl-66-249-66-1.googlebot.com
# crawl-66-249-66-1.googlebot.com has address 66.249.66.1

For scale, match IPs against Google's published JSON lists. Common crawlers such as Googlebot use common-crawlers.json, and Google says they reverse-resolve to crawl-***-***-***-***.googlebot.com. There is nothing to verify for Google-Extended, because no request uses that name.

What changes if you use Cloudflare

Cloudflare changed its defaults on 15 September 2026 for new domains: Training and Agent traffic are blocked by default on pages that show ads, and Search stays allowed (Cloudflare, 1 July 2026). It also said that multi-purpose crawlers "such as Googlebot, Applebot, and BingBot will be blocked by customers who have selected to block Training".

If you use Cloudflare and turned on a "block AI training" option, check whether it now blocks Googlebot too. That would remove you from Google Search, which is almost never the goal. Our AI crawler diagnostic fetches a page as Googlebot and shows whether the edge lets it through.

How Google-Extended compares with other AI opt-outs

Google is not the only company that uses a control token instead of a separate crawler. Apple does the same: Applebot-Extended is a robots.txt name that controls whether content Applebot crawls may be used to train Apple's models, while Applebot itself keeps crawling for Siri and Spotlight (Apple). OpenAI and Anthropic took the other route and run separate training crawlers, GPTBot and ClaudeBot.

Apple adds one twist worth knowing. Its documentation says: "If robots instructions don't mention Applebot but mention Googlebot, the Apple robot will follow Googlebot instructions." So a file that blocks Googlebot for a section, and never names Applebot, blocks Applebot there too. Apple also says Applebot "does not follow crawl-delay", and that rules for Applebot-Extended "are not considered in ranking for Search", which mirrors Google's promise about Google-Extended.

The practical difference: with a token-only control you cannot reduce crawling or bandwidth, because the same crawler still visits. With a real training crawler, blocking it also stops those requests. If server load is your concern, the AI crawler rate limiting guide is the better place to start.

A short timeline of Google's AI controls

The direction is clear: Google keeps Search and Gemini training as separate decisions, and keeps adding ways to measure AI visibility before you change anything. We will update this page when the next control arrives.

A checklist for your Google AI settings

• □Googlebot is allowed in robots.txt and at your CDN or firewall.
• □Google-Extended has its own group, never shared with Googlebot.
• □You decided on Gemini grounding, not only Gemini training, before blocking Google-Extended.
• □You checked the Generative AI performance report before changing the Search generative AI control.
• □Snippet controls are used only where you truly want less text quoted.
• □Your CDN's AI training block does not also block Googlebot.
• □You re-ran the AI crawler checker after every change.

Sources and further reading

For Google-Extended vs Googlebot, start with downloadable CSV.

1. Google Search Central: Google's common crawlers (Google-Extended entry)
2. Google Search Central: AI features and your website
3. Search Console Help: Search generative AI control (rolled out 31 August 2026)
4. Search Console Help: Generative AI performance report (31 August 2026)
5. Google Search Central: Verifying Googlebot and other Google crawlers
6. Google Search Central: How Google interprets the robots.txt specification
7. RFC 9309, Robots Exclusion Protocol (IETF, September 2022)
8. Apple: About Applebot
9. Google: An update on web publisher controls (28 September 2023)
10. Cloudflare: The crawl-to-click gap (29 August 2025)
11. Cloudflare: Content Independence Day, new AI options (1 July 2026)
12. BuzzStream via Search Engine Journal: Most major news publishers block AI training and retrieval bots (January 2026)
13. Tranco: a research-oriented top sites ranking

Blocking Google-Extended never affects Googlebot, but a broad rule written by mistake can. After any edit, run a free Googlebot test on a real page to confirm Googlebot is still allowed and the page is indexable.

Related: how AI engines choose their sources