Google

32 crawlers operated by Google

https://developers.google.com/

Who Google is, for site owners

Google operates 32 crawlers in our directory, filed under Google Bots and AI & LLM Bots and Cloud Services and Search Engines and SEO Tools. Of those 32, 31 rated safe, 1 rated caution for crawl behaviour.

Blocking Google carries an SEO impact score of 0/10 to 10/10 across its crawlers. 10 of them share the highest blocking impact here: Low - No SEO ranking impact.

Our recommendation for Google, set by its least reassuring crawler (Google-CloudVertexBot, rated caution): Consider blocking based on your content strategy.

Google at a glance

Operator	Google
Crawlers in directory	32
Categories	Google Bots, AI & LLM Bots, Cloud Services, Search Engines, SEO Tools
SEO impact if blocked	0/10 to 10/10
Crawl behaviour	31 rated safe, 1 rated caution
Website	developers.google.com
Official crawler docs	Google crawler documentation

Crawlers operated by Google

How to block or allow Google in robots.txt

These rules cover all 32 crawlers Google operates. Add them to the robots.txt file at the root of your domain.

Block Google completely

User-agent: APIs-Google
User-agent: Google-CWS
User-agent: FeedFetcher-Google
User-agent: Google-Gemini-CLI
User-agent: Google-GeminiNotebook
User-agent: Google-NotebookLM
User-agent: GoogleAgent-URLContext
User-agent: Gemini-Deep-Research
User-agent: AdsBot-Google
User-agent: AdsBot-Google-Mobile
User-agent: Mediapartners-Google
User-agent: Google-Firebase
User-agent: GoogleMessages
User-agent: Google-Pinpoint
User-agent: Google-Safety
User-agent: Google-Agent
User-agent: Google-CloudVertexBot
User-agent: Google-Extended
User-agent: Google-InspectionTool
User-agent: Google-Read-Aloud
User-agent: Google-Site-Verification
User-agent: Googlebot
User-agent: Googlebot-Image
User-agent: Googlebot-News
User-agent: Googlebot-Video
User-agent: GoogleOther
User-agent: GoogleOther-Image
User-agent: GoogleOther-Video
User-agent: GoogleProducer
User-agent: Chrome-Lighthouse
User-agent: GoogleAgent-Mariner
User-agent: Storebot-Google
Disallow: /

Allow Google explicitly

User-agent: APIs-Google
User-agent: Google-CWS
User-agent: FeedFetcher-Google
User-agent: Google-Gemini-CLI
User-agent: Google-GeminiNotebook
User-agent: Google-NotebookLM
User-agent: GoogleAgent-URLContext
User-agent: Gemini-Deep-Research
User-agent: AdsBot-Google
User-agent: AdsBot-Google-Mobile
User-agent: Mediapartners-Google
User-agent: Google-Firebase
User-agent: GoogleMessages
User-agent: Google-Pinpoint
User-agent: Google-Safety
User-agent: Google-Agent
User-agent: Google-CloudVertexBot
User-agent: Google-Extended
User-agent: Google-InspectionTool
User-agent: Google-Read-Aloud
User-agent: Google-Site-Verification
User-agent: Googlebot
User-agent: Googlebot-Image
User-agent: Googlebot-News
User-agent: Googlebot-Video
User-agent: GoogleOther
User-agent: GoogleOther-Image
User-agent: GoogleOther-Video
User-agent: GoogleProducer
User-agent: Chrome-Lighthouse
User-agent: GoogleAgent-Mariner
User-agent: Storebot-Google
Allow: /

robots.txt is a request, not an enforcement mechanism. A crawler that ignores it needs a block at the server or CDN layer. Rules are case-sensitive on the path and matched against the user-agent token, not the full user-agent string.

Frequently asked questions about Google

What crawlers does Google run?

Google operates 32 crawlers in our directory: APIs-Google (user-agent APIs-Google), Chrome Web Store Fetcher (user-agent Google-CWS), FeedFetcher-Google (user-agent FeedFetcher-Google), Gemini CLI (user-agent Google-Gemini-CLI), Gemini Notebook (user-agent Google-GeminiNotebook), Google-NotebookLM (superseded) (user-agent Google-NotebookLM), Gemini URL Context (user-agent GoogleAgent-URLContext), Gemini-Deep-Research (user-agent Gemini-Deep-Research), Google Ads Quality (Desktop) (user-agent AdsBot-Google), Google Ads Quality (Mobile) (user-agent AdsBot-Google-Mobile), Google AdSense (user-agent Mediapartners-Google), Google Firebase AI (user-agent Google-Firebase), Google Messages Preview (user-agent GoogleMessages), Google Pinpoint (user-agent Google-Pinpoint), Google Safety (user-agent Google-Safety), Google-Agent (user-agent Google-Agent), Google-CloudVertexBot (user-agent Google-CloudVertexBot), Google-Extended (user-agent Google-Extended), Google-InspectionTool (user-agent Google-InspectionTool), Google-Read-Aloud (user-agent Google-Read-Aloud), Google-Site-Verification (user-agent Google-Site-Verification), Googlebot (user-agent Googlebot), Googlebot-Image (user-agent Googlebot-Image), Googlebot-News (user-agent Googlebot-News), Googlebot-Video (user-agent Googlebot-Video), GoogleOther (user-agent GoogleOther), GoogleOther-Image (user-agent GoogleOther-Image), GoogleOther-Video (user-agent GoogleOther-Video), GoogleProducer (user-agent GoogleProducer), Lighthouse Audit (user-agent Chrome-Lighthouse), Project Mariner (user-agent GoogleAgent-Mariner), Storebot-Google (user-agent Storebot-Google).

Should I block Google?

Consider blocking based on your content strategy. That guidance follows this operator's least reassuring crawler, Google-CloudVertexBot, rated caution, because allowing an operator allows every crawler it runs. The SEO impact score for blocking is 0/10 to 10/10, where 10 means blocking costs you the most visibility.

How do I block Google in robots.txt?

Add the following to robots.txt at your domain root: User-agent: APIs-Google then User-agent: Google-CWS then User-agent: FeedFetcher-Google then User-agent: Google-Gemini-CLI then User-agent: Google-GeminiNotebook then User-agent: Google-NotebookLM then User-agent: GoogleAgent-URLContext then User-agent: Gemini-Deep-Research then User-agent: AdsBot-Google then User-agent: AdsBot-Google-Mobile then User-agent: Mediapartners-Google then User-agent: Google-Firebase then User-agent: GoogleMessages then User-agent: Google-Pinpoint then User-agent: Google-Safety then User-agent: Google-Agent then User-agent: Google-CloudVertexBot then User-agent: Google-Extended then User-agent: Google-InspectionTool then User-agent: Google-Read-Aloud then User-agent: Google-Site-Verification then User-agent: Googlebot then User-agent: Googlebot-Image then User-agent: Googlebot-News then User-agent: Googlebot-Video then User-agent: GoogleOther then User-agent: GoogleOther-Image then User-agent: GoogleOther-Video then User-agent: GoogleProducer then User-agent: Chrome-Lighthouse then User-agent: GoogleAgent-Mariner then User-agent: Storebot-Google then Disallow: /. One Disallow line covers all 32 user-agent tokens.

Does blocking Google hurt my search rankings?

10 of these share the highest blocking impact here, rated: Low - No SEO ranking impact. Those are search-indexing crawlers, so blocking them can remove your pages from the corresponding search results. Block them only if you intend to leave that index. The remaining 22 crawlers in the list above do not affect search indexing and can be blocked separately.