Bot directoryCategory

Cloud Services Directory

Cloud provider crawlers used for security scanning, CDN warm-up or threat intelligence.

Check if AI can crawl my siteBuild a robots.txt

Traffic from cloud platforms is often someone else's script, not a named crawler. Judge it by behaviour in your logs rather than by name.
  • 3tokens
  • 3safe
  • 0caution
  • 0aggressive

All 3 Cloud Services

The cloud services category in numbers

Our database lists 3 cloud services from 3 operators. 3 are rated safe, 0 caution and 0 aggressive. What they are for:

cloud services by purpose
PurposeCrawlers
checks pages for an SEO, uptime or analytics service1
collects pages to train AI models1
fetches a page when a person asks an assistant or app to open it1

Priority tiers

How the 3 cloud services spread across our six priority tiers, where Tier 1 is what you would notice losing:

  • Tier 2: 1 crawler
  • Tier 3: 1 crawler
  • Tier 6: 1 crawler

The highest-priority cloud services here are Amazonbot. Start your review with these.

How top sites treat this category

From our robots.txt study of the Tranco top 10,000 (5,583 valid files, fetched October 20, 2026):

cloud services named and blocked in top-site robots.txt files
TokenFiles naming itFiles blocking it
Amazonbot10.6%12.2%

Tokens in cloud services can be blocked without being named, through User-agent: * with Disallow: /, which is why some blocking shares in this table exceed the naming shares.

How to review cloud services on your site

  1. List the ones that matter. Start with Tier 1 and 2 in the table above; the rest can usually follow a default.
  2. Choose a default. Decide what User-agent: * should say, because every crawler you do not name follows it.
  3. Name the exceptions. Give each crawler that needs a different answer its own group, and repeat any shared Disallow lines inside it.
  4. Check the edge. Firewalls and bot managers often treat these crawlers differently from robots.txt. Run the AI bot crawlability test to see both layers together.
Test yourself: a crawler in this category is not named anywhere in your robots.txt. Which rules does it follow?

The User-agent: * group. If there is no such group, it may crawl everything.

Test yourself: does a safety rating of "safe" mean you should allow a crawler?

Not necessarily. Safety describes behaviour. Whether to allow it depends on what it does with your content and what you lose by blocking it.

The cloud services worth knowing first

Ranked by priority tier, these are the cloud services with the biggest effect on a typical site:

  • Amazonbot (Amazon, Tier 2): a training crawler that collects pages to train AI models.
  • Google-CloudVertexBot (Google, Tier 3): a retrieval crawler that feeds AI search answers that fetches a page when a person asks an assistant or app to open it.
  • AliyunSecBot (Alibaba Cloud, Tier 6): a utility crawler that checks pages for an SEO, uptime or analytics service.

Of the 3 cloud services we track, 2 are documented as following robots.txt, 0 as not following it for some requests, 0 with exceptions or disputed behaviour, and 1 with no clear statement from the operator. Where behaviour is unclear, back a robots.txt rule with a server-side one.

What to do with cloud services, by type of site

Recommended approach to cloud services by type of site
If you run...Recommended approach
Site that uses this kind of serviceAllow the crawlers for services you actually use; they need access to work.
Site that does not use the serviceBlocking is low risk; it has little effect on search or AI visibility.
Site under heavy bot loadRate-limit or block the noisiest crawlers first.
Privacy-sensitive siteBlock crawlers whose data use you have not reviewed.

Common mistakes with cloud services

The errors we see most often with cloud services in the files and firewalls we test:

  1. Treating cloud services as an AI or search crawler. Blocking it has little effect on visibility; decide based on load and what it does with your data.
  2. Relying on robots.txt alone when cloud services is rated caution or aggressive. Back the rule with a server-side block.
  3. Blocking cloud services by IP alone. Ranges change; match the token and verify with logs.

All three hide inside a file that looks correct, so check cloud services from outside after every change.

Spotting cloud services in your logs

robots.txt says what Amazonbot and Google-CloudVertexBot and AliyunSecBot may do; your access logs show what they actually did.

Count requests

On a server with standard Nginx or Apache logs, this counts matching requests per day (adjust the log path for your host):

grep -iE "Amazonbot|Google-CloudVertexBot|AliyunSecBot" /var/log/nginx/access.log \
  | awk '{print $4}' | cut -d: -f1 | sort | uniq -c

And this shows which paths it requested most, which is the quickest way to see whether your Disallow lines are being respected:

grep -iE "Amazonbot|Google-CloudVertexBot|AliyunSecBot" /var/log/nginx/access.log \
  | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

Check the requests are genuine

Anyone can send these user-agents, so confirm the source before acting on a match:

  • Amazonbot: we hold no published IP list, so use reverse DNS where cloud services documents a hostname and treat an unverifiable Amazonbot request with caution. Check it separately from Google-CloudVertexBot and AliyunSecBot.
  • Google-CloudVertexBot: compare each address with the ranges at developers.google.com/static/crawling/ipranges/common-crawlers.json. A request claiming to be Google-CloudVertexBot from outside them is not genuine. Check it separately from Amazonbot and AliyunSecBot.
  • AliyunSecBot: we hold no published IP list, so use reverse DNS where cloud services documents a hostname and treat an unverifiable AliyunSecBot request with caution. Check it separately from Amazonbot and Google-CloudVertexBot.

Recent changes among cloud services

The newest cloud services in our database: Google-CloudVertexBot (March 17, 2026), Amazonbot (March 17, 2026), AliyunSecBot (March 17, 2026). A User-agent: * default decides what each of these gets until you name it.

Movement among the tracked cloud services: 2026 and October 20, 2026, files naming Amazonbot went from 543 to 593 (+50) and files blocking it from 624 to 680 (+56), so more sites wrote a rule for Amazonbot and more of those rules block it.

Each of the 3 cloud services in one line

  • AliyunSecBot from Alibaba Cloud: it checks pages for an SEO, uptime or analytics service; Tier 6, rated safe, blocking impact not documented.
  • Amazonbot from Amazon: it collects pages to train AI models; Tier 2, rated safe, blocking impact low.
  • Google-CloudVertexBot from Google: it fetches a page when a person asks an assistant or app to open it; Tier 3, rated safe, blocking impact low.

Related: how AI engines choose their sources

With so few cloud services, you can decide about every one of them by name rather than relying on a User-agent: * default.

How top sites treat cloud services

From our robots.txt study of the Tranco top 10,000 (5,583 valid files, fetched October 20, 2026):

  • Amazonbot, the only one tracked among the cloud services tokens: blocked in 12.2% of files (464 with its own group, 216 through User-agent: *), more often by the top 1,000 (15.6%) than the long tail (11.9%).

For scale, 20.1% of all files block at least one AI training crawler. Amazonbot is one of the tokens driving that figure.

More questions about cloud services

How many cloud services should I name in robots.txt?
Only the ones that need a different answer from your default. Every crawler you do not name follows User-agent: *, so a short file with a sensible default is easier to maintain than a long list.
Do new cloud services appear often?
Yes. New crawlers launch every month, which is why a deliberate User-agent: * default matters: it decides what happens to crawlers you have never heard of.
Can I copy another site's rules for cloud services?
Use them as a pattern, not a template. Each file reflects that site's licensing and infrastructure. Copy the structure, then name the crawlers that matter to you.
Where do I check how my site treats cloud services?
Run the AI bot crawlability test. It reports every crawler in this category in one pass, including firewall refusals that robots.txt never shows.

FAQ

How many Cloud Services are tracked?
We currently track 3 Cloud Services with user-agent strings, safety ratings, and blocking rules.
Which Cloud Services should I never block?
Never block bots with 'critical' impact rating - there are 0 in this category. Blocking them removes your site from their search results.
Are all Cloud Services safe?
Out of 3 bots in this category: 3 safe, 0 caution, 0 aggressive.
How do I block all Cloud Services at once?
Generate a combined robots.txt block rule for all Cloud Services using our bulk robots.txt generator, or manually list each User-agent + Disallow: / directive.