Google now gives site owners three separate switches for AI, and each one does something different. The newest is the Search generative AI control in Search Console, rolled out to every site on 31 August 2026. It decides whether your pages can appear in AI Overviews, AI Mode and Discover's AI features. It does not touch training. For that you still need Google-Extended in robots.txt. And neither one removes you from Google Search; only noindex does that.

Mixing them up is the most common Google AI mistake we see. This guide explains each switch from Google's own documentation, shows what our top-10k data says about how sites use Google-Extended, and gives a decision table you can use with clients.

Key Takeaways

  • Search generative AI control: Search Console, Settings, then Search generative AI. Include, exclude or inherit. Controls AI Overviews, AI Mode and generative AI in Discover.
  • Google-Extended: a robots.txt token. Controls use of your content to train Gemini models and to ground answers in Gemini Apps and Vertex AI. It does not affect Search.
  • noindex: removes a page from Google Search completely, including every AI feature in Search.
  • Not a ranking signal. Google says the new control is not used as a ranking or inclusion signal for other parts of Search.
  • Google-Extended is widely used. 666 of 5,254 valid top-10k robots.txt files block it, and 529 of those keep Googlebot allowed.
Training crawlers versus search crawlers comparison Two-column comparison. Training crawlers like GPTBot and ClaudeBot collect content to train future AI models; blocking them stops training use. Search crawlers like ChatGPT-User and OAI-SearchBot fetch pages in real time to answer user questions; blocking them removes you from live AI answers and citations. TRAINING CRAWLERS GPTBot · ClaudeBot · Google-Extended • Collect content for future AI models • Visit on their own schedule • No user is waiting for the result Block = your content skips future model training SEARCH / LIVE CRAWLERS ChatGPT-User · OAI-SearchBot · Perplexity-User • Fetch your page the moment a user asks • Power citations and live AI answers • Directly drive AI referral traffic Block = you vanish from live AI answers today They are separate user-agents: you can block training while staying visible in AI search.
Training crawlers vs search crawlers: two different jobs, controlled by separate robots.txt rules.

The three switches at a glance

SwitchWhere you set itWhat it changesWhat it does not change
Search generative AI controlSearch Console, Settings, Search generative AIWhether your links and content appear in AI Overviews, AI Mode and Discover AI featuresTraining; classic search results; ranking
Google-Extendedrobots.txt user-agent tokenTraining of Gemini models; grounding in Gemini Apps and Vertex AIGoogle Search, including AI Overviews and AI Mode
noindexMeta robots tag or X-Robots-Tag headerWhether a page is in Google Search at allTraining data already collected

Read that table twice. The control and Google-Extended cover different products. One is about Search; the other is about Gemini and Google Cloud.

The Search generative AI control

Google's help page for the control describes three choices.

  • Include (the default): your content can appear in Search AI features, both as links and to help ground the AI answer. You can receive impressions and traffic from them.
  • Exclude: your links and content are kept out of those features. Google says you "won't receive any traffic or impressions from these features", and content crawled from your site will not be used as input to generate an AI response or preview there.
  • Inherit: the property follows its closest parent property that has a manual setting. This is the default for any property that has a parent.

What excluding does, and does not, do

Google is unusually direct about the limits. Excluding your site does not stop content from other sites appearing in AI answers, "and it may appear similar to yours". Your content may still help Google Search as a whole, for example to understand the language of queries. The control does not override other programmes such as Merchant Center or Google Ads. And it "doesn't affect AI training"; Google points to Google-Extended for that.

Google also says the control "isn't used as a ranking or inclusion signal affecting other parts of Search". In other words, excluding AI features should not, by itself, move your classic blue-link rankings.

How long it takes

Google says content is excluded within one to two days after the control goes live, and some content can take longer because of caching across Google's systems. Expect a few days in practice, and do not judge the effect from a single day of data.

Inheritance: the part that catches agencies

Search Console properties nest. A domain property such as example.com is the parent of https://example.com/blog/ and of blog.example.com. By default each child inherits from its closest parent that has a manual setting. Two consequences follow.

  1. If someone with access to the domain property sets it to exclude, every URL-prefix and subdomain property inherits that choice, even if those owners never touched the setting.
  2. A child property can override its parent. If the domain is excluded but the blog property is set to include, the blog's pages are eligible. The domain owner sees a notification that a child overrides the parent.

For agencies that manage a client's URL-prefix property while the client's IT team owns the domain property, this matters. Check the control on the highest-level property you can see, and ask who owns the ones above it.

Google-Extended

According to Google's crawler documentation, Google-Extended is a standalone product token that lets you manage whether content Google crawls may be used for training future generations of Gemini models, and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI. It has no user agent of its own: crawling is done by the normal Google crawlers, and the token only works as a control in robots.txt. Google says it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal".

robots.txt: no Gemini training, Search unaffected
User-agent: Google-Extended
Disallow: /

User-agent: *
Allow: /

You will not see Google-Extended in your logs

Because there is no Google-Extended user agent, you cannot measure it in server logs. Requests arrive as Googlebot or another Google crawler. The token only tells Google what it may do with what it already fetched.

Decision tree: should you allow or block AI crawlers Decision flowchart. Start: does your business gain from being discovered in AI answers? If yes, allow AI crawlers and add llms.txt. If your content itself is the paid product, consider blocking training bots while keeping search bots allowed, or pursue licensing. Do you gain customers when AI mentions your brand? YES CONTENT IS THE PRODUCT ALLOW + OPTIMIZE • Allow all tier-1 AI crawlers • Add llms.txt + schema markup • Structure content for citations Best for: SaaS, services, B2B, local, ecommerce SELECTIVE CONTROL • Block training bots (GPTBot, CCBot) • Keep search bots for discovery • Explore licensing deals Best for: publishers, media, premium content There is no one right answer. Match your crawler policy to how your business makes money.
Decision tree: whether to allow or block AI crawlers depends on your business model, not a universal rule.

What our top-10k data shows

In our 17 October 2026 scan of the Tranco top 10,000, 743 valid robots.txt files named Google-Extended in a user-agent line and 666 (12.7%) blocked it from the whole site. Of those, 529 kept Googlebot allowed, which is the intended pattern: no Gemini training, full Search. Only 143 files blocked Googlebot itself.

The Search Console control cannot be seen from outside, so we cannot measure how many sites have excluded AI features. If you track it for clients, record it in your audit notes each month.

The other Google tokens people confuse with these

Google runs more than a dozen crawlers and fetchers. Four of them come up often in AI discussions, and none of them is controlled by the Search generative AI control.

TokenWhat it doesReads robots.txt?Effect of blocking it
GoogleOtherGeneral-purpose crawler Google uses for internal research and product workYesNo effect on Search inclusion
Google-CloudVertexBotCrawls sites that Google Cloud customers point Vertex AI agents atYesVertex AI agents cannot use your pages; Search unaffected
Google-GeminiNotebookFetches a URL when a Gemini Notebook user adds it as a sourceGenerally no (user-triggered)Little; user-triggered fetchers generally ignore robots.txt
Google-AgentActs in a browser for a user, for Google's agent featuresGenerally no (user-triggered)Little; handle at the edge if needed

The pattern is the same as with the three main switches: each token covers one product. Blocking GoogleOther does not remove you from AI Overviews, and allowing Googlebot does not opt you in to Vertex AI. When a client asks "how do we stop Google AI", the honest answer starts with "which one?".

Three scenarios from real conversations

"We want to stay out of AI Overviews because they steal our clicks"

Look at the data first. Open the generative AI performance report, then compare clicks for the same pages in the normal report over the same months. If AI impressions are high and clicks have fallen on those pages, excluding may recover some clicks from people who would otherwise have read the AI answer, but you also lose every visitor who clicked through from the AI feature. If you decide to exclude, set it on the domain property, wait a few days, and measure clicks for at least four weeks before deciding it worked.

"We blocked Google-Extended and still see our content in AI Mode"

That is expected. Google-Extended is about Gemini training and grounding in Gemini Apps and Vertex AI. AI Mode is part of Search. Use the Search Console control if you want out of AI Mode, and keep the Google-Extended rule if you also want to limit training.

"Our agency excluded AI features on our blog property, but the main site is included"

That is a valid setup. The blog property overrides the domain, so the blog stays out of AI features while the rest of the site remains eligible. Write the reason down, because the next team to look at the domain property will see a notification about the override and may undo it.

Which switch for which goal

Your goalSearch generative AI controlGoogle-Extendednoindex
Appear everywhere, allow Gemini trainingIncludeAllowNo
Appear in AI Overviews and AI Mode, no Gemini trainingIncludeDisallowNo
Classic results only, no AI Overviews or AI ModeExcludeYour choiceNo
Out of Google Search entirely for a pageNot neededYour choiceYes, on that page

For most businesses the second row is the sensible default: stay visible in AI answers, where customers increasingly ask, and keep the training decision separate. Excluding AI features is a heavy step. Google's own wording says you lose all impressions and traffic from them, while similar content from other sites can still appear. That trade usually makes sense only for publishers whose content is the product and who believe AI answers replace the visit.

Before you exclude, measure

Google points to the generative AI performance report for exactly this reason. Look at how many impressions your pages get in AI features first, and which pages earn them. Our guide to Search Console's generative AI report explains how to read it. If AI impressions are small, excluding gains you little. If they are large, excluding costs you a lot of visibility.

Common mistakes

  • Blocking Google-Extended to leave AI Overviews. It does not work. AI Overviews are Search; use the Search Console control.
  • Excluding AI features to stop training. It does not work either. Google says the control does not affect training.
  • Blocking Googlebot to stop AI. This removes you from Search completely, including classic results. It is almost never what a business wants.
  • Using Disallow instead of noindex. A disallowed page can still be indexed from links, without content. See noindex vs Disallow.
  • Forgetting inheritance. A child property quietly follows a parent that someone else excluded.
  • Forgetting the other engines. None of these switches affects ChatGPT, Claude or Perplexity. Those use their own tokens, such as OAI-SearchBot, Claude-SearchBot and PerplexityBot.

How this changes GEO advice

Until August 2026, advice about Google AI visibility had to say "there is no opt-out short of leaving Search". That is no longer true, and it changes how you talk to clients. The choice to appear in AI Overviews and AI Mode is now explicit, reversible and measurable. A business that stays included is making a decision, not just accepting a default, and it can show the result in the generative AI report.

It also separates two conversations that used to be tangled. "Do we want Google's AI answers to cite us?" is a marketing question about visibility and clicks. "Do we want Google to train Gemini on our content?" is a rights and licensing question. Different people in a company usually own those answers, and now each answer has its own switch. Put both in your quarterly review, record who decided, and revisit them when Google changes the products each switch covers. Google says it expects the list of covered features to change over time, so a setting that was right in October may need another look next year.

A quick audit you can run today

  • □In Search Console, open Settings, then Search generative AI, on the highest-level property you have. Note include, exclude or inherit.
  • □Repeat for each child property you manage and note any overrides.
  • □Fetch your robots.txt and check whether Google-Extended is allowed or disallowed, and whether Googlebot is allowed. Our robots.txt validator shows both.
  • □Run AI Crawler Check on a key page to confirm Googlebot can actually reach it through your CDN and firewall.
  • □Open the generative AI performance report and note last month's AI impressions.
  • □Write the three decisions into your client notes so the next person does not undo them by accident.