Microsoft

Common mistakes with Microsoft's crawlers

The errors we see most often with Microsoft's crawlers in the files and firewalls we test:

1. Blocking Microsoft's crawlers while trying to stop AI training. Retrieval crawlers fetch pages so answers can cite them; blocking it can remove you from those answers without affecting training.
2. Letting a firewall or bot-fight mode refuse Microsoft's crawlers while robots.txt allows it. The file looks right, but the crawler still gets a 403 or a challenge page.
3. Forgetting that once a User-agent: AzureAI-SearchBot group exists, Microsoft's crawlers ignores every rule under User-agent: *.

All three hide inside a file that looks correct, so check Microsoft's crawlers from outside after every change.

Spotting Microsoft in your logs

robots.txt says what Azure AI Search and Bing Ads Crawler and Bing Page Snapshot and Bing Video Preview and Bingbot and Microsoft Preview may do; your access logs show what they actually did.

Count requests

On a server with standard Nginx or Apache logs, this counts matching requests per day (adjust the log path for your host):

grep -iE "AzureAI-SearchBot|AdIdxBot|BingPreview|BingVideoPreview|bingbot|MicrosoftPreview" /var/log/nginx/access.log \
  | awk '{print $4}' | cut -d: -f1 | sort | uniq -c

And this shows which paths it requested most, which is the quickest way to see whether your Disallow lines are being respected:

grep -iE "AzureAI-SearchBot|AdIdxBot|BingPreview|BingVideoPreview|bingbot|MicrosoftPreview" /var/log/nginx/access.log \
  | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

Check the requests are genuine

Anyone can send these user-agents, so confirm the source before acting on a match:

• AzureAI-SearchBot: we hold no published IP list, so use reverse DNS where Microsoft documents a hostname and treat an unverifiable Azure AI Search request with caution. Check it separately from Bing Ads Crawler and Bing Page Snapshot and Bing Video Preview and Bingbot and Microsoft Preview.
• AdIdxBot: we hold no published IP list, so use reverse DNS where Microsoft documents a hostname and treat an unverifiable Bing Ads Crawler request with caution. Check it separately from Azure AI Search and Bing Page Snapshot and Bing Video Preview and Bingbot and Microsoft Preview.
• BingPreview: we hold no published IP list, so use reverse DNS where Microsoft documents a hostname and treat an unverifiable Bing Page Snapshot request with caution. Check it separately from Azure AI Search and Bing Ads Crawler and Bing Video Preview and Bingbot and Microsoft Preview.
• BingVideoPreview: we hold no published IP list, so use reverse DNS where Microsoft documents a hostname and treat an unverifiable Bing Video Preview request with caution. Check it separately from Azure AI Search and Bing Ads Crawler and Bing Page Snapshot and Bingbot and Microsoft Preview.
• bingbot: compare each address with the ranges at www.bing.com/toolbox/bingbot.json. A request claiming to be Bingbot from outside them is not genuine. Check it separately from Azure AI Search and Bing Ads Crawler and Bing Page Snapshot and Bing Video Preview and Microsoft Preview.
• MicrosoftPreview: we hold no published IP list, so use reverse DNS where Microsoft documents a hostname and treat an unverifiable Microsoft Preview request with caution. Check it separately from Azure AI Search and Bing Ads Crawler and Bing Page Snapshot and Bing Video Preview and Bingbot.

What our own data shows about Microsoft

Everything below comes from the AI Crawler Check bot database and our two most recent robots.txt scans of the Tranco top 10,000, not from Microsoft's marketing. It covers 6 Microsoft tokens in turn.

AzureAI-SearchBot in our records

We first recorded AzureAI-SearchBot on June 18, 2026. We could not find a page where Microsoft names it, so treat the token as reported rather than documented. AzureAI-SearchBot is outside our tracked token set, but 26 of 5,583 top-site files already name it, which is enough to show owners treat it as a real crawler.

Among the 104 AI and LLM crawlers in our database, AzureAI-SearchBot sits 25th by priority (Tier 2), level with AddSearchBot, AI2Bot, AI2Bot-DeepResearchEval, aiHitBot. 81 of the 104 share its safe rating. Other operators with a crawler that builds a search or answer index: AddSearchBot (AddSearch, Tier 2), AIWebIndex (Lyrenth, Tier 2), atlassian-bot (Atlassian, Tier 2), Cloudflare-AI-Search (Cloudflare, Tier 2), Cloudflare-AutoRAG (Cloudflare, Tier 2). A policy for AzureAI-SearchBot is usually a policy for these too.

A decision record for AzureAI-SearchBot

Teams that review crawlers once a quarter keep one short entry per token, so anyone can see why a rule exists and when it was last checked. Filled in from our data, the AzureAI-SearchBot entry reads:

• Operator and job: Microsoft; it builds a search or answer index.
• Priority and behaviour: Tier 2, rated safe, blocking impact low.
• Suggested starting point: Allow, and watch it in your logs.
• Enforcement: AzureAI-SearchBot's operator does not clearly document how it treats robots.txt. Write the AzureAI-SearchBot rule anyway, then confirm in your logs that its requests stop.

User-agent: AzureAI-SearchBot
Allow: /

Paste the line into the validator with one of your own URLs before it goes live. Revisit the AzureAI-SearchBot entry whenever Microsoft announces a change, because a Tier 2 crawler affects visibility directly.

AdIdxBot in our records

We first recorded AdIdxBot on September 22, 2026. Microsoft documents it on bing.com, and we recheck that page before each release. No file in our top-10,000 scans names AdIdxBot, so any rule you write for it puts you ahead of almost every large site.

Among the 63 search engines in our database, AdIdxBot sits second by priority (Tier 3), level with Amzn-SearchBot, Applebot, baidu, Baiduspider. 62 of the 63 share its safe rating. Other operators with a crawler that checks pages for an SEO, uptime or analytics service: OAI-AdsBot (OpenAI, Tier 2), AdsBot-Google (Google, Tier 3), AdsBot-Google-Mobile (Google, Tier 3), Baiduspider-ads (Baidu, Tier 3), Baiduspider-cpro (Baidu, Tier 3). A policy for AdIdxBot is usually a policy for these too.

BingPreview in our records

We first recorded BingPreview on September 22, 2026. Microsoft documents it on bing.com, and we recheck that page before each release. No file in our top-10,000 scans names BingPreview, so any rule you write for it puts you ahead of almost every large site.

Among the 63 search engines in our database, BingPreview sits 14th by priority (Tier 3), level with AdIdxBot, Amzn-SearchBot, Applebot, baidu. 62 of the 63 share its safe rating. Other operators with a crawler that builds a search or answer index: AddSearchBot (AddSearch, Tier 2), AIWebIndex (Lyrenth, Tier 2), atlassian-bot (Atlassian, Tier 2), Cloudflare-AI-Search (Cloudflare, Tier 2), Cloudflare-AutoRAG (Cloudflare, Tier 2). A policy for BingPreview is usually a policy for these too.

BingVideoPreview in our records

We first recorded BingVideoPreview on September 22, 2026. Microsoft documents it on bing.com, and we recheck that page before each release. No file in our top-10,000 scans names BingVideoPreview, so any rule you write for it puts you ahead of almost every large site.

Among the 63 search engines in our database, BingVideoPreview sits 15th by priority (Tier 3), level with AdIdxBot, Amzn-SearchBot, Applebot, baidu. 62 of the 63 share its safe rating. Other operators with a crawler that builds a search or answer index: AddSearchBot (AddSearch, Tier 2), AIWebIndex (Lyrenth, Tier 2), atlassian-bot (Atlassian, Tier 2), Cloudflare-AI-Search (Cloudflare, Tier 2), Cloudflare-AutoRAG (Cloudflare, Tier 2). A policy for BingVideoPreview is usually a policy for these too.

bingbot in our records

We first recorded bingbot on March 17, 2026. Microsoft documents it on bing.com, and we recheck that page before each release. Between our scans of October 17, 2026 and October 20, 2026, files naming bingbot went from 381 to 397 (+16) and files blocking it from 169 to 182 (+13), so more sites wrote a rule for bingbot and more of those rules block it.

Among the 63 search engines in our database, bingbot sits 13th by priority (Tier 3), level with AdIdxBot, Amzn-SearchBot, Applebot, baidu. 62 of the 63 share its safe rating. Other operators with a crawler that builds a search or answer index: AddSearchBot (AddSearch, Tier 2), AIWebIndex (Lyrenth, Tier 2), atlassian-bot (Atlassian, Tier 2), Cloudflare-AI-Search (Cloudflare, Tier 2), Cloudflare-AutoRAG (Cloudflare, Tier 2). A policy for bingbot is usually a policy for these too.

MicrosoftPreview in our records

We first recorded MicrosoftPreview on September 22, 2026. Microsoft documents it on bing.com, and we recheck that page before each release. No file in our top-10,000 scans names MicrosoftPreview, so any rule you write for it puts you ahead of almost every large site.

Among the 63 search engines in our database, MicrosoftPreview sits 20th by priority (Tier 3), level with AdIdxBot, Amzn-SearchBot, Applebot, baidu. 62 of the 63 share its safe rating. Other operators with a crawler that fetches a page to build a link preview when someone shares it: FacebookBot (Meta, Tier 5), facebookexternalhit (Meta, Tier 5), GoogleMessages (Google, Tier 5), LinkedInBot (LinkedIn, Tier 5), Pinterestbot (Pinterest, Tier 5). A policy for MicrosoftPreview is usually a policy for these too.

How top sites treat Microsoft's crawlers

From our robots.txt study of the Tranco top 10,000 (5,583 valid files, fetched October 20, 2026):

• bingbot, the only one tracked among the Microsoft's crawlers tokens: blocked in 3.3% of files (21 with its own group, 161 through User-agent: *), more often by the top 1,000 (3.8%) than the long tail (3%).

For scale, 20.1% of all files block at least one AI training crawler. None of the tokens for Microsoft's crawlers comes close to that rate, so blocking them is still a minority choice.

More questions about Microsoft

Will blocking Microsoft affect my Google rankings?

Only if one of its tokens is a Google search crawler. None of Microsoft's tokens are Googlebot, so classic Google rankings are unaffected.

How quickly does a robots.txt change take effect for Microsoft?

Crawlers cache robots.txt, commonly for up to a day. Expect a change to apply within about 24 hours, and confirm it in your logs.

Can I allow Microsoft on some sections and block it on others?

Yes. Inside the group for AzureAI-SearchBot, add Disallow lines for the paths to keep private and Allow lines for exceptions. The longest matching rule wins.

Does blocking Microsoft delete content it already collected?

No. robots.txt only controls future visits. For content already collected, check whether Microsoft offers a removal or opt-out process.

Is a firewall block better than robots.txt for Microsoft?

A firewall enforces the decision, while robots.txt asks politely. Many operators recommend robots.txt because blocking their IPs can stop them reading your rules at all. Use robots.txt first, and add a firewall rule only if requests continue.