Amazon

Common mistakes with Amazon's crawlers

The errors we see most often with Amazon's crawlers in the files and firewalls we test:

1. Assuming a robots.txt rule will stop Amazon's crawlers. User-triggered fetches act for a person, and some operators say robots.txt may not apply to them. Check the operator's statement above.
2. Blocking Amazon's crawlers as if it were a bulk crawler. It fetches one page because a person asked; blocking it means that person's assistant cannot read your page.
3. Counting Amazon's crawlers visits as crawl volume. Each request maps to a real user question, which makes the paths it requests a useful signal of what people ask about you.

All three hide inside a file that looks correct, so check Amazon's crawlers from outside after every change.

Spotting Amazon in your logs

robots.txt says what Amazon Buy For Me and Amazon Kendra and Amazon Nova Act and Amazon Q Business and Amazon User Agent and Amazonbot and Amzn-SearchBot and AWS Bedrock Bot may do; your access logs show what they actually did.

Count requests

On a server with standard Nginx or Apache logs, this counts matching requests per day (adjust the log path for your host):

grep -iE "AmazonBuyForMe|Amazon Kendra|NovaAct|amazon-QBusiness|Amzn-User|Amazonbot|Amzn-SearchBot|bedrockbot" /var/log/nginx/access.log \
  | awk '{print $4}' | cut -d: -f1 | sort | uniq -c

And this shows which paths it requested most, which is the quickest way to see whether your Disallow lines are being respected:

grep -iE "AmazonBuyForMe|Amazon Kendra|NovaAct|amazon-QBusiness|Amzn-User|Amazonbot|Amzn-SearchBot|bedrockbot" /var/log/nginx/access.log \
  | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

Check the requests are genuine

Anyone can send these user-agents, so confirm the source before acting on a match:

• AmazonBuyForMe: we hold no published IP list, so use reverse DNS where Amazon documents a hostname and treat an unverifiable Amazon Buy For Me request with caution. Check it separately from Amazon Kendra and Amazon Nova Act and Amazon Q Business and Amazon User Agent and Amazonbot and Amzn-SearchBot and AWS Bedrock Bot.
• Amazon Kendra: we hold no published IP list, so use reverse DNS where Amazon documents a hostname and treat an unverifiable Amazon Kendra request with caution. Check it separately from Amazon Buy For Me and Amazon Nova Act and Amazon Q Business and Amazon User Agent and Amazonbot and Amzn-SearchBot and AWS Bedrock Bot.
• NovaAct: we hold no published IP list, so use reverse DNS where Amazon documents a hostname and treat an unverifiable Amazon Nova Act request with caution. Check it separately from Amazon Buy For Me and Amazon Kendra and Amazon Q Business and Amazon User Agent and Amazonbot and Amzn-SearchBot and AWS Bedrock Bot.
• amazon-QBusiness: we hold no published IP list, so use reverse DNS where Amazon documents a hostname and treat an unverifiable Amazon Q Business request with caution. Check it separately from Amazon Buy For Me and Amazon Kendra and Amazon Nova Act and Amazon User Agent and Amazonbot and Amzn-SearchBot and AWS Bedrock Bot.
• Amzn-User: we hold no published IP list, so use reverse DNS where Amazon documents a hostname and treat an unverifiable Amazon User Agent request with caution. Check it separately from Amazon Buy For Me and Amazon Kendra and Amazon Nova Act and Amazon Q Business and Amazonbot and Amzn-SearchBot and AWS Bedrock Bot.
• Amazonbot: we hold no published IP list, so use reverse DNS where Amazon documents a hostname and treat an unverifiable Amazonbot request with caution. Check it separately from Amazon Buy For Me and Amazon Kendra and Amazon Nova Act and Amazon Q Business and Amazon User Agent and Amzn-SearchBot and AWS Bedrock Bot.
• Amzn-SearchBot: we hold no published IP list, so use reverse DNS where Amazon documents a hostname and treat an unverifiable Amzn-SearchBot request with caution. Check it separately from Amazon Buy For Me and Amazon Kendra and Amazon Nova Act and Amazon Q Business and Amazon User Agent and Amazonbot and AWS Bedrock Bot.
• bedrockbot: we hold no published IP list, so use reverse DNS where Amazon documents a hostname and treat an unverifiable AWS Bedrock Bot request with caution. Check it separately from Amazon Buy For Me and Amazon Kendra and Amazon Nova Act and Amazon Q Business and Amazon User Agent and Amazonbot and Amzn-SearchBot.

Related: how AI engines choose their sources

What our own data shows about Amazon

Everything below comes from the AI Crawler Check bot database and our two most recent robots.txt scans of the Tranco top 10,000, not from Amazon's marketing. It covers 6 Amazon tokens in turn.

AmazonBuyForMe in our records

We first recorded AmazonBuyForMe on June 18, 2026. We could not find a page where Amazon names it, so treat the token as reported rather than documented. AmazonBuyForMe is outside our tracked token set, but 58 of 5,583 top-site files already name it, which is enough to show owners treat it as a real crawler.

Among the 104 AI and LLM crawlers in our database, AmazonBuyForMe sits 20th by priority (Tier 2), level with AddSearchBot, AI2Bot, AI2Bot-DeepResearchEval, aiHitBot. 81 of the 104 share its safe rating. Other operators with a crawler that fetches a page when a person asks an assistant or app to open it: ChatGPT Agent (OpenAI, Tier 1), ChatGPT Operator (OpenAI, Tier 1), ChatGPT-User (OpenAI, Tier 1), ChatGPT-User/2.0 (OpenAI, Tier 1), Claude-SearchBot (Anthropic, Tier 1). A policy for AmazonBuyForMe is usually a policy for these too.

A decision record for AmazonBuyForMe

Teams that review crawlers once a quarter keep one short entry per token, so anyone can see why a rule exists and when it was last checked. Filled in from our data, the AmazonBuyForMe entry reads:

• Operator and job: Amazon; it fetches a page when a person asks an assistant or app to open it.
• Priority and behaviour: Tier 2, rated safe, blocking impact low.
• Suggested starting point: Allow, and watch it in your logs.
• Enforcement: AmazonBuyForMe's operator does not clearly document how it treats robots.txt. Write the AmazonBuyForMe rule anyway, then confirm in your logs that its requests stop.

User-agent: AmazonBuyForMe
Allow: /

Paste the line into the validator with one of your own URLs before it goes live. Revisit the AmazonBuyForMe entry whenever Amazon announces a change, because a Tier 2 crawler affects visibility directly.

Amazon Kendra in our records

We first recorded Amazon Kendra on March 17, 2026. Amazon documents it on docs.aws.amazon.com, and we recheck that page before each release. No file in our top-10,000 scans names Amazon Kendra, so any rule you write for it puts you ahead of almost every large site.

Among the 39 data scrapers in our database, Amazon Kendra sits third by priority (Tier 6), level with ApifyBot, Barkrowler, BLEXBot, Brightbot. 29 of the 39 share its safe rating. Other operators with a crawler that harvests pages for a dataset or data product: FirecrawlAgent (Firecrawl, Tier 2), iaskspider (iAsk, Tier 2), CloudflareBrowserRenderingCrawler (Cloudflare, Tier 3), ApifyBot (Apify, Tier 6), archive.org_bot (Internet Archive, Tier 6). A policy for Amazon Kendra is usually a policy for these too.

NovaAct in our records

We first recorded NovaAct on March 17, 2026. We could not find a page where Amazon names it, so treat the token as reported rather than documented. NovaAct is outside our tracked token set, but 64 of 5,583 top-site files already name it, which is enough to show owners treat it as a real crawler.

Among the 104 AI and LLM crawlers in our database, NovaAct sits 71th by priority (Tier 2), level with AddSearchBot, AI2Bot, AI2Bot-DeepResearchEval, aiHitBot. 81 of the 104 share its safe rating. Other operators with a crawler that harvests pages for a dataset or data product: FirecrawlAgent (Firecrawl, Tier 2), iaskspider (iAsk, Tier 2), CloudflareBrowserRenderingCrawler (Cloudflare, Tier 3), ApifyBot (Apify, Tier 6), archive.org_bot (Internet Archive, Tier 6). A policy for NovaAct is usually a policy for these too.

amazon-QBusiness in our records

We first recorded amazon-QBusiness on September 22, 2026. We could not find a page where Amazon names it, so treat the token as reported rather than documented. amazon-QBusiness is outside our tracked token set, but 35 of 5,583 top-site files already name it, which is enough to show owners treat it as a real crawler.

Among the 104 AI and LLM crawlers in our database, amazon-QBusiness sits 91th by priority (Tier 3), level with AIWebIndex-Agent, alphalens-bot, Anomura, Channel3Bot. 81 of the 104 share its safe rating. Other operators with a crawler that builds a search or answer index: AddSearchBot (AddSearch, Tier 2), AIWebIndex (Lyrenth, Tier 2), atlassian-bot (Atlassian, Tier 2), AzureAI-SearchBot (Microsoft, Tier 2), Cloudflare-AI-Search (Cloudflare, Tier 2). A policy for amazon-QBusiness is usually a policy for these too.

Amzn-User in our records

We first recorded Amzn-User on June 18, 2026. Amazon documents it on developer.amazon.com, and we recheck that page before each release. No file in our top-10,000 scans names Amzn-User, so any rule you write for it puts you ahead of almost every large site.

Among the 104 AI and LLM crawlers in our database, Amzn-User sits 21th by priority (Tier 2), level with AddSearchBot, AI2Bot, AI2Bot-DeepResearchEval, aiHitBot. 81 of the 104 share its safe rating. Other operators with a crawler that fetches a page when a person asks an assistant or app to open it: ChatGPT Agent (OpenAI, Tier 1), ChatGPT Operator (OpenAI, Tier 1), ChatGPT-User (OpenAI, Tier 1), ChatGPT-User/2.0 (OpenAI, Tier 1), Claude-SearchBot (Anthropic, Tier 1). A policy for Amzn-User is usually a policy for these too.

Amazonbot in our records

We first recorded Amazonbot on March 17, 2026. Amazon documents it on developer.amazon.com, and we recheck that page before each release. Between our scans of October 17, 2026 and October 20, 2026, files naming Amazonbot went from 543 to 593 (+50) and files blocking it from 624 to 680 (+56), so more sites wrote a rule for Amazonbot and more of those rules block it.

Among the 3 cloud services in our database, Amazonbot sits first by priority (Tier 2), alone in its tier. 3 of the 3 share its safe rating. Other operators with a crawler that collects pages to train AI models: ClaudeBot (Anthropic, Tier 1), Google-Extended (Google, Tier 1), GPTBot (OpenAI, Tier 1), Meta-ExternalAgent (Meta, Tier 1), AI2Bot (Allen Institute for AI, Tier 2). A policy for Amazonbot is usually a policy for these too.

How top sites treat Amazon's crawlers

From our robots.txt study of the Tranco top 10,000 (5,583 valid files, fetched October 20, 2026):

• Amazonbot, the only one tracked among the Amazon's crawlers tokens: blocked in 12.2% of files (464 with its own group, 216 through User-agent: *), more often by the top 1,000 (15.6%) than the long tail (11.9%).

For scale, 20.1% of all files block at least one AI training crawler. Amazonbot is one of the tokens driving that figure.

More questions about Amazon

Will blocking Amazon affect my Google rankings?

Only if one of its tokens is a Google search crawler. None of Amazon's tokens are Googlebot, so classic Google rankings are unaffected.

How quickly does a robots.txt change take effect for Amazon?

Crawlers cache robots.txt, commonly for up to a day. Expect a change to apply within about 24 hours, and confirm it in your logs.

Can I allow Amazon on some sections and block it on others?

Yes. Inside the group for AmazonBuyForMe, add Disallow lines for the paths to keep private and Allow lines for exceptions. The longest matching rule wins.

Does blocking Amazon delete content it already collected?

No. robots.txt only controls future visits. For content already collected, check whether Amazon offers a removal or opt-out process.

Is a firewall block better than robots.txt for Amazon?

A firewall enforces the decision, while robots.txt asks politely. Many operators recommend robots.txt because blocking their IPs can stop them reading your rules at all. Use robots.txt first, and add a firewall rule only if requests continue.