Alibaba

Common mistakes with Alibaba's crawlers

The errors we see most often with Alibaba's crawlers in the files and firewalls we test:

1. Assuming that blocking Alibaba's crawlers removes you from AI answers. It is a training crawler; AI answers are fed by retrieval and user-triggered crawlers with different tokens.
2. Blocking Alibaba's crawlers with a broad User-agent: * rule that also catches search engines. Name QwenBot in its own group instead.
3. Expecting a Alibaba's crawlers block to remove content already collected. robots.txt only affects future visits.

All three hide inside a file that looks correct, so check Alibaba's crawlers from outside after every change.

Spotting Alibaba in your logs

robots.txt says what Qwen (Alibaba) and Tongyi Qianwen Fetcher may do; your access logs show what they actually did.

Count requests

On a server with standard Nginx or Apache logs, this counts matching requests per day (adjust the log path for your host):

grep -iE "QwenBot|TongyiBot" /var/log/nginx/access.log \
  | awk '{print $4}' | cut -d: -f1 | sort | uniq -c

And this shows which paths it requested most, which is the quickest way to see whether your Disallow lines are being respected:

grep -iE "QwenBot|TongyiBot" /var/log/nginx/access.log \
  | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

Check the requests are genuine

Anyone can send these user-agents, so confirm the source before acting on a match:

• QwenBot: we hold no published IP list, so use reverse DNS where Alibaba documents a hostname and treat an unverifiable Qwen (Alibaba) request with caution. Check it separately from Tongyi Qianwen Fetcher.
• TongyiBot: we hold no published IP list, so use reverse DNS where Alibaba documents a hostname and treat an unverifiable Tongyi Qianwen Fetcher request with caution. Check it separately from Qwen (Alibaba).

What our own data shows about Alibaba

Everything below comes from the AI Crawler Check bot database and our two most recent robots.txt scans of the Tranco top 10,000, not from Alibaba's marketing. It covers 2 Alibaba tokens in turn.

QwenBot in our records

We first recorded QwenBot on September 22, 2026. We could not find a page where Alibaba names it, so treat the token as reported rather than documented. QwenBot is outside our tracked token set, but 18 of 5,583 top-site files already name it, which is enough to show owners treat it as a real crawler.

Among the 104 AI and LLM crawlers in our database, QwenBot sits 78th by priority (Tier 2), level with AddSearchBot, AI2Bot, AI2Bot-DeepResearchEval, aiHitBot. 22 of the 104 share its caution rating. Other operators with a crawler that collects pages to train AI models: ClaudeBot (Anthropic, Tier 1), Google-Extended (Google, Tier 1), GPTBot (OpenAI, Tier 1), Meta-ExternalAgent (Meta, Tier 1), AI2Bot (Allen Institute for AI, Tier 2). A policy for QwenBot is usually a policy for these too.

A decision record for QwenBot

Teams that review crawlers once a quarter keep one short entry per token, so anyone can see why a rule exists and when it was last checked. Filled in from our data, the QwenBot entry reads:

• Operator and job: Alibaba; it collects pages to train AI models.
• Priority and behaviour: Tier 2, rated caution, blocking impact low.
• Suggested starting point: Your licensing call: Disallow to refuse training, Allow to permit it.
• Enforcement: QwenBot's operator does not clearly document how it treats robots.txt. Write the QwenBot rule anyway, then confirm in your logs that its requests stop.

User-agent: QwenBot
Allow: /

Paste the line into the validator with one of your own URLs before it goes live. Revisit the QwenBot entry sooner than most: a caution rating means Alibaba traffic deserves a monthly look at your logs.

TongyiBot in our records

We first recorded TongyiBot on September 22, 2026. We could not find a page where Alibaba names it, so treat the token as reported rather than documented. TongyiBot is outside our tracked token set, but 22 of 5,583 top-site files already name it, which is enough to show owners treat it as a real crawler.

Among the 104 AI and LLM crawlers in our database, TongyiBot sits 84th by priority (Tier 2), level with AddSearchBot, AI2Bot, AI2Bot-DeepResearchEval, aiHitBot. 22 of the 104 share its caution rating. Other operators with a crawler that fetches a page when a person asks an assistant or app to open it: ChatGPT Agent (OpenAI, Tier 1), ChatGPT Operator (OpenAI, Tier 1), ChatGPT-User (OpenAI, Tier 1), ChatGPT-User/2.0 (OpenAI, Tier 1), Claude-SearchBot (Anthropic, Tier 1). A policy for TongyiBot is usually a policy for these too.

A decision record for TongyiBot

Teams that review crawlers once a quarter keep one short entry per token, so anyone can see why a rule exists and when it was last checked. Filled in from our data, the TongyiBot entry reads:

• Operator and job: Alibaba; it fetches a page when a person asks an assistant or app to open it.
• Priority and behaviour: Tier 2, rated caution, blocking impact low.
• Suggested starting point: Allow, and watch it in your logs.
• Enforcement: TongyiBot's operator does not clearly document how it treats robots.txt. Write the TongyiBot rule anyway, then confirm in your logs that its requests stop.

User-agent: TongyiBot
Allow: /

Paste the line into the validator with one of your own URLs before it goes live. Revisit the TongyiBot entry sooner than most: a caution rating means Alibaba traffic deserves a monthly look at your logs.

How top sites treat Alibaba's crawlers

From our robots.txt study of the Tranco top 10,000 (5,583 valid files, fetched October 20, 2026):

None of the tokens for Alibaba's crawlers is tracked by name in our study. For scale, 20.1% of top-site files block at least one AI training crawler and 4.3% refuse every unnamed crawler with User-agent: * and Disallow: /, which would catch Alibaba's crawlers too.

More questions about Alibaba

Will blocking Alibaba affect my Google rankings?

Only if one of its tokens is a Google search crawler. None of Alibaba's tokens are Googlebot, so classic Google rankings are unaffected.

How quickly does a robots.txt change take effect for Alibaba?

Crawlers cache robots.txt, commonly for up to a day. Expect a change to apply within about 24 hours, and confirm it in your logs.

Can I allow Alibaba on some sections and block it on others?

Yes. Inside the group for QwenBot, add Disallow lines for the paths to keep private and Allow lines for exceptions. The longest matching rule wins.

Does blocking Alibaba delete content it already collected?

No. robots.txt only controls future visits. For content already collected, check whether Alibaba offers a removal or opt-out process.

Is a firewall block better than robots.txt for Alibaba?

A firewall enforces the decision, while robots.txt asks politely. Many operators recommend robots.txt because blocking their IPs can stop them reading your rules at all. Use robots.txt first, and add a firewall rule only if requests continue.