Google

Common mistakes with Google's crawlers

The errors we see most often with Google's crawlers in the files and firewalls we test:

1. Assuming a robots.txt rule will stop Google's crawlers. User-triggered fetches act for a person, and some operators say robots.txt may not apply to them. Check the operator's statement above.
2. Blocking Google's crawlers as if it were a bulk crawler. It fetches one page because a person asked; blocking it means that person's assistant cannot read your page.
3. Counting Google's crawlers visits as crawl volume. Each request maps to a real user question, which makes the paths it requests a useful signal of what people ask about you.

All three hide inside a file that looks correct, so check Google's crawlers from outside after every change.

Spotting Google in your logs

robots.txt says what APIs-Google and Chrome Web Store Fetcher and FeedFetcher-Google and Gemini CLI and Gemini Notebook and Gemini URL Context and Gemini-Deep-Research and Google Ads Quality (Desktop) may do; your access logs show what they actually did.

Count requests

On a server with standard Nginx or Apache logs, this counts matching requests per day (adjust the log path for your host):

grep -iE "APIs-Google|Google-CWS|FeedFetcher-Google|Google-Gemini-CLI|Google-GeminiNotebook|GoogleAgent-URLContext|Gemini-Deep-Research|AdsBot-Google" /var/log/nginx/access.log \
  | awk '{print $4}' | cut -d: -f1 | sort | uniq -c

And this shows which paths it requested most, which is the quickest way to see whether your Disallow lines are being respected:

grep -iE "APIs-Google|Google-CWS|FeedFetcher-Google|Google-Gemini-CLI|Google-GeminiNotebook|GoogleAgent-URLContext|Gemini-Deep-Research|AdsBot-Google" /var/log/nginx/access.log \
  | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

Check the requests are genuine

Anyone can send these user-agents, so confirm the source before acting on a match:

• APIs-Google: we hold no published IP list, so use reverse DNS where Google documents a hostname and treat an unverifiable APIs-Google request with caution. Check it separately from Chrome Web Store Fetcher and FeedFetcher-Google and Gemini CLI and Gemini Notebook and Gemini URL Context and Gemini-Deep-Research and Google Ads Quality (Desktop).
• Google-CWS: we hold no published IP list, so use reverse DNS where Google documents a hostname and treat an unverifiable Chrome Web Store Fetcher request with caution. Check it separately from APIs-Google and FeedFetcher-Google and Gemini CLI and Gemini Notebook and Gemini URL Context and Gemini-Deep-Research and Google Ads Quality (Desktop).
• FeedFetcher-Google: we hold no published IP list, so use reverse DNS where Google documents a hostname and treat an unverifiable FeedFetcher-Google request with caution. Check it separately from APIs-Google and Chrome Web Store Fetcher and Gemini CLI and Gemini Notebook and Gemini URL Context and Gemini-Deep-Research and Google Ads Quality (Desktop).
• Google-Gemini-CLI: we hold no published IP list, so use reverse DNS where Google documents a hostname and treat an unverifiable Gemini CLI request with caution. Check it separately from APIs-Google and Chrome Web Store Fetcher and FeedFetcher-Google and Gemini Notebook and Gemini URL Context and Gemini-Deep-Research and Google Ads Quality (Desktop).
• Google-GeminiNotebook: we hold no published IP list, so use reverse DNS where Google documents a hostname and treat an unverifiable Gemini Notebook request with caution. Check it separately from APIs-Google and Chrome Web Store Fetcher and FeedFetcher-Google and Gemini CLI and Gemini URL Context and Gemini-Deep-Research and Google Ads Quality (Desktop).
• GoogleAgent-URLContext: we hold no published IP list, so use reverse DNS where Google documents a hostname and treat an unverifiable Gemini URL Context request with caution. Check it separately from APIs-Google and Chrome Web Store Fetcher and FeedFetcher-Google and Gemini CLI and Gemini Notebook and Gemini-Deep-Research and Google Ads Quality (Desktop).
• Gemini-Deep-Research: we hold no published IP list, so use reverse DNS where Google documents a hostname and treat an unverifiable Gemini-Deep-Research request with caution. Check it separately from APIs-Google and Chrome Web Store Fetcher and FeedFetcher-Google and Gemini CLI and Gemini Notebook and Gemini URL Context and Google Ads Quality (Desktop).
• AdsBot-Google: we hold no published IP list, so use reverse DNS where Google documents a hostname and treat an unverifiable Google Ads Quality (Desktop) request with caution. Check it separately from APIs-Google and Chrome Web Store Fetcher and FeedFetcher-Google and Gemini CLI and Gemini Notebook and Gemini URL Context and Gemini-Deep-Research.

Related: writing content AI engines can cite

What our own data shows about Google

Everything below comes from the AI Crawler Check bot database and our two most recent robots.txt scans of the Tranco top 10,000, not from Google's marketing. It covers 6 Google tokens in turn.

APIs-Google in our records

We first recorded APIs-Google on September 22, 2026. Google documents it on developers.google.com, and we recheck that page before each release. No file in our top-10,000 scans names APIs-Google, so any rule you write for it puts you ahead of almost every large site.

Among the 24 Google crawlers in our database, APIs-Google sits fifth by priority (Tier 3), level with AdsBot-Google, AdsBot-Google-Mobile, FeedFetcher-Google, Gemini-Deep-Research. 24 of the 24 share its safe rating. Other operators with a crawler that fetches a page when a person asks an assistant or app to open it: ChatGPT Agent (OpenAI, Tier 1), ChatGPT Operator (OpenAI, Tier 1), ChatGPT-User (OpenAI, Tier 1), ChatGPT-User/2.0 (OpenAI, Tier 1), Claude-SearchBot (Anthropic, Tier 1). A policy for APIs-Google is usually a policy for these too.

A decision record for APIs-Google

Teams that review crawlers once a quarter keep one short entry per token, so anyone can see why a rule exists and when it was last checked. Filled in from our data, the APIs-Google entry reads:

• Operator and job: Google; it fetches a page when a person asks an assistant or app to open it.
• Priority and behaviour: Tier 3, rated safe, blocking impact not documented.
• Suggested starting point: Allow by default; revisit if it costs you bandwidth.
• Enforcement: APIs-Google's operator does not clearly document how it treats robots.txt. Write the APIs-Google rule anyway, then confirm in your logs that its requests stop.

User-agent: APIs-Google
Allow: /

Paste the line into the validator with one of your own URLs before it goes live. Revisit the APIs-Google entry once a quarter, or when Google publishes a new token next to it.

Google-CWS in our records

We first recorded Google-CWS on September 22, 2026. Google documents it on developers.google.com, and we recheck that page before each release. No file in our top-10,000 scans names Google-CWS, so any rule you write for it puts you ahead of almost every large site.

Among the 24 Google crawlers in our database, Google-CWS sits 23th by priority (Tier 4), alone in its tier. 24 of the 24 share its safe rating. Other operators with a crawler that checks pages for an SEO, uptime or analytics service: OAI-AdsBot (OpenAI, Tier 2), AdIdxBot (Microsoft, Tier 3), Baiduspider-ads (Baidu, Tier 3), Baiduspider-cpro (Baidu, Tier 3), YaDirectFetcher (Yandex, Tier 3). A policy for Google-CWS is usually a policy for these too.

FeedFetcher-Google in our records

We first recorded FeedFetcher-Google on May 31, 2026. Google documents it on developers.google.com, and we recheck that page before each release. No file in our top-10,000 scans names FeedFetcher-Google, so any rule you write for it puts you ahead of almost every large site.

Among the 24 Google crawlers in our database, FeedFetcher-Google sits sixth by priority (Tier 3), level with AdsBot-Google, AdsBot-Google-Mobile, APIs-Google, Gemini-Deep-Research. 24 of the 24 share its safe rating. Other operators with a crawler that builds a search or answer index: AddSearchBot (AddSearch, Tier 2), AIWebIndex (Lyrenth, Tier 2), atlassian-bot (Atlassian, Tier 2), AzureAI-SearchBot (Microsoft, Tier 2), Cloudflare-AI-Search (Cloudflare, Tier 2). A policy for FeedFetcher-Google is usually a policy for these too.

Google-Gemini-CLI in our records

We first recorded Google-Gemini-CLI on June 18, 2026. We could not find a page where Google names it, so treat the token as reported rather than documented. Google-Gemini-CLI is outside our tracked token set, but 8 of 5,583 top-site files already name it, which is enough to show owners treat it as a real crawler.

Among the 104 AI and LLM crawlers in our database, Google-Gemini-CLI sits 44th by priority (Tier 2), level with AddSearchBot, AI2Bot, AI2Bot-DeepResearchEval, aiHitBot. 81 of the 104 share its safe rating. Other operators with a crawler that fetches a page when a person asks an assistant or app to open it: ChatGPT Agent (OpenAI, Tier 1), ChatGPT Operator (OpenAI, Tier 1), ChatGPT-User (OpenAI, Tier 1), ChatGPT-User/2.0 (OpenAI, Tier 1), Claude-SearchBot (Anthropic, Tier 1). A policy for Google-Gemini-CLI is usually a policy for these too.

Google-GeminiNotebook in our records

We first recorded Google-GeminiNotebook on September 22, 2026. Google documents it on developers.google.com, and we recheck that page before each release. No file in our top-10,000 scans names Google-GeminiNotebook, so any rule you write for it puts you ahead of almost every large site.

Among the 24 Google crawlers in our database, Google-GeminiNotebook sits second by priority (Tier 2), level with Google-Agent. 24 of the 24 share its safe rating. Other operators with a crawler that fetches a page when a person asks an assistant or app to open it: ChatGPT Agent (OpenAI, Tier 1), ChatGPT Operator (OpenAI, Tier 1), ChatGPT-User (OpenAI, Tier 1), ChatGPT-User/2.0 (OpenAI, Tier 1), Claude-SearchBot (Anthropic, Tier 1). A policy for Google-GeminiNotebook is usually a policy for these too.

GoogleAgent-URLContext in our records

We first recorded GoogleAgent-URLContext on June 18, 2026. We could not find a page where Google names it, so treat the token as reported rather than documented. GoogleAgent-URLContext is outside our tracked token set, but 10 of 5,583 top-site files already name it, which is enough to show owners treat it as a real crawler.

Among the 104 AI and LLM crawlers in our database, GoogleAgent-URLContext sits 46th by priority (Tier 2), level with AddSearchBot, AI2Bot, AI2Bot-DeepResearchEval, aiHitBot. 81 of the 104 share its safe rating. Other operators with a crawler that fetches a page when a person asks an assistant or app to open it: ChatGPT Agent (OpenAI, Tier 1), ChatGPT Operator (OpenAI, Tier 1), ChatGPT-User (OpenAI, Tier 1), ChatGPT-User/2.0 (OpenAI, Tier 1), Claude-SearchBot (Anthropic, Tier 1). A policy for GoogleAgent-URLContext is usually a policy for these too.

How top sites treat Google's crawlers

From our robots.txt study of the Tranco top 10,000 (5,583 valid files, fetched October 20, 2026):

• Google-Extended, the most blocked of the Google's crawlers tokens: blocked in 13% of files (519 with its own group, 204 through User-agent: *), more often by the top 1,000 (18.6%) than the long tail (11.9%).
• Googlebot, second of the Google's crawlers tokens: blocked in 2.8% of files (13 with its own group, 141 through User-agent: *), more often by the top 1,000 (3.6%) than the long tail (2.4%).

For scale, 20.1% of all files block at least one AI training crawler. Google-Extended is one of the tokens driving that figure.

More questions about Google

Will blocking Google affect my Google rankings?

Only if one of its tokens is a Google search crawler. It is: blocking Googlebot removes pages from Google Search.

How quickly does a robots.txt change take effect for Google?

Crawlers cache robots.txt, commonly for up to a day. Expect a change to apply within about 24 hours, and confirm it in your logs.

Can I allow Google on some sections and block it on others?

Yes. Inside the group for APIs-Google, add Disallow lines for the paths to keep private and Allow lines for exceptions. The longest matching rule wins.

Does blocking Google delete content it already collected?

No. robots.txt only controls future visits. For content already collected, check whether Google offers a removal or opt-out process.

Is a firewall block better than robots.txt for Google?

A firewall enforces the decision, while robots.txt asks politely. Many operators recommend robots.txt because blocking their IPs can stop them reading your rules at all. Use robots.txt first, and add a firewall rule only if requests continue.