Bot directoryCategory

Google Bots Directory

Google's specialized crawlers for Images, Video, News and other services.

Run a live crawler access testBuild a robots.txt

Google runs many fetchers. Googlebot drives Search, while Google-Extended is only a robots.txt token for opting out of Gemini training. Blocking one does not block the other.
  • 24tokens
  • 24safe
  • 0caution
  • 0aggressive

All 24 Google Bots

The Google crawlers category in numbers

Our database lists 24 Google crawlers from 1 operator. 24 are rated safe, 0 caution and 0 aggressive. What they are for:

Google crawlers by purpose
PurposeCrawlers
builds a search or answer index9
checks pages for an SEO, uptime or analytics service7
fetches a page when a person asks an assistant or app to open it7
fetches a page to build a link preview when someone shares it1

Priority tiers

How the 24 Google crawlers spread across our six priority tiers, where Tier 1 is what you would notice losing:

  • Tier 2: 2 crawlers
  • Tier 3: 20 crawlers
  • Tier 4: 1 crawler
  • Tier 5: 1 crawler

The highest-priority Google crawlers here are Google-Agent, Google-GeminiNotebook. Start your review with these.

How to review Google crawlers on your site

  1. List the ones that matter. Start with Tier 1 and 2 in the table above; the rest can usually follow a default.
  2. Choose a default. Decide what User-agent: * should say, because every crawler you do not name follows it.
  3. Name the exceptions. Give each crawler that needs a different answer its own group, and repeat any shared Disallow lines inside it.
  4. Check the edge. Firewalls and bot managers often treat these crawlers differently from robots.txt. Run the website crawlability test to see both layers together.
Test yourself: a crawler in this category is not named anywhere in your robots.txt. Which rules does it follow?

The User-agent: * group. If there is no such group, it may crawl everything.

Test yourself: does a safety rating of "safe" mean you should allow a crawler?

Not necessarily. Safety describes behaviour. Whether to allow it depends on what it does with your content and what you lose by blocking it.

The Google crawlers worth knowing first

Ranked by priority tier, these are the Google crawlers with the biggest effect on a typical site:

  • Google-Agent (Google, Tier 2): a user-triggered agent that fetches a page when a person asks an assistant or app to open it.
  • Google-GeminiNotebook (Google, Tier 2): a user-triggered agent that fetches a page when a person asks an assistant or app to open it.
  • AdsBot-Google (Google, Tier 3): a utility crawler that checks pages for an SEO, uptime or analytics service.
  • AdsBot-Google-Mobile (Google, Tier 3): a utility crawler that checks pages for an SEO, uptime or analytics service.
  • APIs-Google (Google, Tier 3): a user-triggered agent that fetches a page when a person asks an assistant or app to open it.
  • FeedFetcher-Google (Google, Tier 3): a user-triggered agent that builds a search or answer index.
  • Gemini-Deep-Research (Google, Tier 3): a user-triggered agent that fetches a page when a person asks an assistant or app to open it.
  • Google-InspectionTool (Google, Tier 3): a utility crawler that checks pages for an SEO, uptime or analytics service.

Of the 24 Google crawlers we track, 1 are documented as following robots.txt, 9 as not following it for some requests, 0 with exceptions or disputed behaviour, and 14 with no clear statement from the operator. Where behaviour is unclear, back a robots.txt rule with a server-side one.

What to do with Google crawlers, by type of site

Recommended approach to Google crawlers by type of site
If you run...Recommended approach
Any site that wants search trafficAllow the major engines; blocking them removes pages from their results.
Site with a regional audienceAllow the engines your audience uses; regional engines can matter more than you expect.
Staging or private siteBlock everything and protect it with authentication, not robots.txt alone.
Site under heavy crawl loadUse caching and engine crawl-rate settings before blocking.

Common mistakes with Google crawlers

The errors we see most often with Google crawlers in the files and firewalls we test:

  1. Assuming a robots.txt rule will stop Google crawlers. User-triggered fetches act for a person, and some operators say robots.txt may not apply to them. Check the operator's statement above.
  2. Blocking Google crawlers as if it were a bulk crawler. It fetches one page because a person asked; blocking it means that person's assistant cannot read your page.
  3. Counting Google crawlers visits as crawl volume. Each request maps to a real user question, which makes the paths it requests a useful signal of what people ask about you.

All three hide inside a file that looks correct, so check Google crawlers from outside after every change.

Spotting Google crawlers in your logs

robots.txt says what Google-Agent and Google-GeminiNotebook and AdsBot-Google and AdsBot-Google-Mobile and APIs-Google and FeedFetcher-Google may do; your access logs show what they actually did.

Count requests

On a server with standard Nginx or Apache logs, this counts matching requests per day (adjust the log path for your host):

grep -iE "Google-Agent|Google-GeminiNotebook|AdsBot-Google|AdsBot-Google-Mobile|APIs-Google|FeedFetcher-Google" /var/log/nginx/access.log \
  | awk '{print $4}' | cut -d: -f1 | sort | uniq -c

And this shows which paths it requested most, which is the quickest way to see whether your Disallow lines are being respected:

grep -iE "Google-Agent|Google-GeminiNotebook|AdsBot-Google|AdsBot-Google-Mobile|APIs-Google|FeedFetcher-Google" /var/log/nginx/access.log \
  | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

Check the requests are genuine

Anyone can send these user-agents, so confirm the source before acting on a match:

  • Google-Agent: compare each address with the ranges at developers.google.com/static/crawling/ipranges/user-triggered-agents.json. A request claiming to be Google-Agent from outside them is not genuine. Check it separately from Google-GeminiNotebook and AdsBot-Google and AdsBot-Google-Mobile and APIs-Google and FeedFetcher-Google.
  • Google-GeminiNotebook: we hold no published IP list, so use reverse DNS where Google crawlers documents a hostname and treat an unverifiable Google-GeminiNotebook request with caution. Check it separately from Google-Agent and AdsBot-Google and AdsBot-Google-Mobile and APIs-Google and FeedFetcher-Google.
  • AdsBot-Google: we hold no published IP list, so use reverse DNS where Google crawlers documents a hostname and treat an unverifiable AdsBot-Google request with caution. Check it separately from Google-Agent and Google-GeminiNotebook and AdsBot-Google-Mobile and APIs-Google and FeedFetcher-Google.
  • AdsBot-Google-Mobile: we hold no published IP list, so use reverse DNS where Google crawlers documents a hostname and treat an unverifiable AdsBot-Google-Mobile request with caution. Check it separately from Google-Agent and Google-GeminiNotebook and AdsBot-Google and APIs-Google and FeedFetcher-Google.
  • APIs-Google: we hold no published IP list, so use reverse DNS where Google crawlers documents a hostname and treat an unverifiable APIs-Google request with caution. Check it separately from Google-Agent and Google-GeminiNotebook and AdsBot-Google and AdsBot-Google-Mobile and FeedFetcher-Google.
  • FeedFetcher-Google: we hold no published IP list, so use reverse DNS where Google crawlers documents a hostname and treat an unverifiable FeedFetcher-Google request with caution. Check it separately from Google-Agent and Google-GeminiNotebook and AdsBot-Google and AdsBot-Google-Mobile and APIs-Google.

Recent changes among Google crawlers

The newest Google crawlers in our database: Mediapartners-Google (September 22, 2026), GoogleMessages (September 22, 2026), Google-Safety (September 22, 2026), Google-Pinpoint (September 22, 2026), Google-GeminiNotebook (September 22, 2026), Google-CWS (September 22, 2026). A User-agent: * default decides what each of these gets until you name it.

None of the 24 Google crawlers is in our tracked token set yet, so we report first-seen dates only.

How top sites treat Google crawlers

From our robots.txt study of the Tranco top 10,000 (5,583 valid files, fetched October 20, 2026):

None of the tokens for Google crawlers is tracked by name in our study. For scale, 20.1% of top-site files block at least one AI training crawler and 4.3% refuse every unnamed crawler with User-agent: * and Disallow: /, which would catch Google crawlers too.

More questions about Google crawlers

How many Google crawlers should I name in robots.txt?
Only the ones that need a different answer from your default. Every crawler you do not name follows User-agent: *, so a short file with a sensible default is easier to maintain than a long list.
Do new Google crawlers appear often?
Yes. New crawlers launch every month, which is why a deliberate User-agent: * default matters: it decides what happens to crawlers you have never heard of.
Can I copy another site's rules for Google crawlers?
Use them as a pattern, not a template. Each file reflects that site's licensing and infrastructure. Copy the structure, then name the crawlers that matter to you.
Where do I check how my site treats Google crawlers?
Run the AI visibility checker. It reports every crawler in this category in one pass, including firewall refusals that robots.txt never shows.

FAQ

How many Google Bots are tracked?
We currently track 24 Google Bots with user-agent strings, safety ratings, and blocking rules.
Which Google Bots should I never block?
Never block bots with 'critical' impact rating - there are 10 in this category. Blocking them removes your site from their search results.
Are all Google Bots safe?
Out of 24 bots in this category: 24 safe, 0 caution, 0 aggressive.
How do I block all Google Bots at once?
Generate a combined robots.txt block rule for all Google Bots using our bulk robots.txt generator, or manually list each User-agent + Disallow: / directive.
WhatsApp us