Yandex

Common mistakes with Yandex's crawlers

The errors we see most often with Yandex's crawlers in the files and firewalls we test:

1. Blocking Yandex's crawlers to save bandwidth and then wondering why pages drop out of that search engine. Use crawl-rate settings or caching first.
2. Writing a User-agent: Yandex group to add one rule for Yandex's crawlers and accidentally dropping every rule it used to follow under User-agent: *.
3. Disallowing CSS and JavaScript folders, which can stop Yandex's crawlers rendering your pages properly.

All three hide inside a file that looks correct, so check Yandex's crawlers from outside after every change.

Spotting Yandex in your logs

robots.txt says what Yandex and Yandex Accessibility and Yandex Advertising Network and Yandex Blogs and Yandex Business Directory and Yandex Calendar and Yandex Check and Yandex Direct may do; your access logs show what they actually did.

Count requests

On a server with standard Nginx or Apache logs, this counts matching requests per day (adjust the log path for your host):

grep -iE "Yandex|YandexAccessibilityBot|YandexAdNet|YandexBlogs|YandexSpravBot|YandexCalendar|YandexCheckBot|YandexDirect" /var/log/nginx/access.log \
  | awk '{print $4}' | cut -d: -f1 | sort | uniq -c

And this shows which paths it requested most, which is the quickest way to see whether your Disallow lines are being respected:

grep -iE "Yandex|YandexAccessibilityBot|YandexAdNet|YandexBlogs|YandexSpravBot|YandexCalendar|YandexCheckBot|YandexDirect" /var/log/nginx/access.log \
  | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

Check the requests are genuine

Anyone can send these user-agents, so confirm the source before acting on a match:

• Yandex: we hold no published IP list, so use reverse DNS where the operator documents a hostname and treat an unverifiable Yandex request with caution. Check it separately from Yandex Accessibility and Yandex Advertising Network and Yandex Blogs and Yandex Business Directory and Yandex Calendar and Yandex Check and Yandex Direct.
• YandexAccessibilityBot: we hold no published IP list, so use reverse DNS where Yandex documents a hostname and treat an unverifiable Yandex Accessibility request with caution. Check it separately from Yandex and Yandex Advertising Network and Yandex Blogs and Yandex Business Directory and Yandex Calendar and Yandex Check and Yandex Direct.
• YandexAdNet: we hold no published IP list, so use reverse DNS where Yandex documents a hostname and treat an unverifiable Yandex Advertising Network request with caution. Check it separately from Yandex and Yandex Accessibility and Yandex Blogs and Yandex Business Directory and Yandex Calendar and Yandex Check and Yandex Direct.
• YandexBlogs: we hold no published IP list, so use reverse DNS where Yandex documents a hostname and treat an unverifiable Yandex Blogs request with caution. Check it separately from Yandex and Yandex Accessibility and Yandex Advertising Network and Yandex Business Directory and Yandex Calendar and Yandex Check and Yandex Direct.
• YandexSpravBot: we hold no published IP list, so use reverse DNS where Yandex documents a hostname and treat an unverifiable Yandex Business Directory request with caution. Check it separately from Yandex and Yandex Accessibility and Yandex Advertising Network and Yandex Blogs and Yandex Calendar and Yandex Check and Yandex Direct.
• YandexCalendar: we hold no published IP list, so use reverse DNS where Yandex documents a hostname and treat an unverifiable Yandex Calendar request with caution. Check it separately from Yandex and Yandex Accessibility and Yandex Advertising Network and Yandex Blogs and Yandex Business Directory and Yandex Check and Yandex Direct.
• YandexCheckBot: we hold no published IP list, so use reverse DNS where Yandex documents a hostname and treat an unverifiable Yandex Check request with caution. Check it separately from Yandex and Yandex Accessibility and Yandex Advertising Network and Yandex Blogs and Yandex Business Directory and Yandex Calendar and Yandex Direct.
• YandexDirect: we hold no published IP list, so use reverse DNS where Yandex documents a hostname and treat an unverifiable Yandex Direct request with caution. Check it separately from Yandex and Yandex Accessibility and Yandex Advertising Network and Yandex Blogs and Yandex Business Directory and Yandex Calendar and Yandex Check.

Related: how AI engines choose their sources

What our own data shows about Yandex

Everything below comes from the AI Crawler Check bot database and our two most recent robots.txt scans of the Tranco top 10,000, not from Yandex's marketing. It covers 6 Yandex tokens in turn.

Yandex in our records

We first recorded Yandex on March 17, 2026. Yandex documents it on yandex.com, and we recheck that page before each release. No file in our top-10,000 scans names Yandex, so any rule you write for it puts you ahead of almost every large site.

Among the 63 search engines in our database, Yandex sits 27th by priority (Tier 3), level with AdIdxBot, Amzn-SearchBot, Applebot, baidu. 62 of the 63 share its safe rating. Other operators with a crawler that builds a search or answer index: AddSearchBot (AddSearch, Tier 2), AIWebIndex (Lyrenth, Tier 2), atlassian-bot (Atlassian, Tier 2), AzureAI-SearchBot (Microsoft, Tier 2), Cloudflare-AI-Search (Cloudflare, Tier 2). A policy for Yandex is usually a policy for these too.

A decision record for Yandex

Teams that review crawlers once a quarter keep one short entry per token, so anyone can see why a rule exists and when it was last checked. Filled in from our data, the Yandex entry reads:

• Operator and job: Yandex; it builds a search or answer index.
• Priority and behaviour: Tier 3, rated safe, blocking impact critical.
• Suggested starting point: Allow by default; revisit if it costs you bandwidth.
• Enforcement: Yandex's operator does not clearly document how it treats robots.txt. Write the Yandex rule anyway, then confirm in your logs that its requests stop.

User-agent: Yandex
Allow: /

Paste the line into the validator with one of your own URLs before it goes live. Revisit the Yandex entry once a quarter, or when Yandex publishes a new token next to it.

YandexAccessibilityBot in our records

We first recorded YandexAccessibilityBot on September 22, 2026. Yandex documents it on yandex.com, and we recheck that page before each release. No file in our top-10,000 scans names YandexAccessibilityBot, so any rule you write for it puts you ahead of almost every large site.

Among the 63 search engines in our database, YandexAccessibilityBot sits 28th by priority (Tier 3), level with AdIdxBot, Amzn-SearchBot, Applebot, baidu. 62 of the 63 share its safe rating. Other operators with a crawler that builds a search or answer index: AddSearchBot (AddSearch, Tier 2), AIWebIndex (Lyrenth, Tier 2), atlassian-bot (Atlassian, Tier 2), AzureAI-SearchBot (Microsoft, Tier 2), Cloudflare-AI-Search (Cloudflare, Tier 2). A policy for YandexAccessibilityBot is usually a policy for these too.

YandexAdNet in our records

We first recorded YandexAdNet on September 22, 2026. Yandex documents it on yandex.com, and we recheck that page before each release. No file in our top-10,000 scans names YandexAdNet, so any rule you write for it puts you ahead of almost every large site.

Among the 63 search engines in our database, YandexAdNet sits 31th by priority (Tier 3), level with AdIdxBot, Amzn-SearchBot, Applebot, baidu. 62 of the 63 share its safe rating. Other operators with a crawler that checks pages for an SEO, uptime or analytics service: OAI-AdsBot (OpenAI, Tier 2), AdIdxBot (Microsoft, Tier 3), AdsBot-Google (Google, Tier 3), AdsBot-Google-Mobile (Google, Tier 3), Baiduspider-ads (Baidu, Tier 3). A policy for YandexAdNet is usually a policy for these too.

YandexBlogs in our records

We first recorded YandexBlogs on September 22, 2026. Yandex documents it on yandex.com, and we recheck that page before each release. No file in our top-10,000 scans names YandexBlogs, so any rule you write for it puts you ahead of almost every large site.

Among the 63 search engines in our database, YandexBlogs sits 32th by priority (Tier 3), level with AdIdxBot, Amzn-SearchBot, Applebot, baidu. 62 of the 63 share its safe rating. Other operators with a crawler that builds a search or answer index: AddSearchBot (AddSearch, Tier 2), AIWebIndex (Lyrenth, Tier 2), atlassian-bot (Atlassian, Tier 2), AzureAI-SearchBot (Microsoft, Tier 2), Cloudflare-AI-Search (Cloudflare, Tier 2). A policy for YandexBlogs is usually a policy for these too.

YandexSpravBot in our records

We first recorded YandexSpravBot on September 22, 2026. Yandex documents it on yandex.com, and we recheck that page before each release. No file in our top-10,000 scans names YandexSpravBot, so any rule you write for it puts you ahead of almost every large site.

Among the 63 search engines in our database, YandexSpravBot sits 56th by priority (Tier 3), level with AdIdxBot, Amzn-SearchBot, Applebot, baidu. 62 of the 63 share its safe rating. Other operators with a crawler that builds a search or answer index: AddSearchBot (AddSearch, Tier 2), AIWebIndex (Lyrenth, Tier 2), atlassian-bot (Atlassian, Tier 2), AzureAI-SearchBot (Microsoft, Tier 2), Cloudflare-AI-Search (Cloudflare, Tier 2). A policy for YandexSpravBot is usually a policy for these too.

YandexCalendar in our records

We first recorded YandexCalendar on September 22, 2026. Yandex documents it on yandex.com, and we recheck that page before each release. No file in our top-10,000 scans names YandexCalendar, so any rule you write for it puts you ahead of almost every large site.

Among the 63 search engines in our database, YandexCalendar sits 34th by priority (Tier 3), level with AdIdxBot, Amzn-SearchBot, Applebot, baidu. 62 of the 63 share its safe rating. Other operators with a crawler that fetches a page when a person asks an assistant or app to open it: ChatGPT Agent (OpenAI, Tier 1), ChatGPT Operator (OpenAI, Tier 1), ChatGPT-User (OpenAI, Tier 1), ChatGPT-User/2.0 (OpenAI, Tier 1), Claude-SearchBot (Anthropic, Tier 1). A policy for YandexCalendar is usually a policy for these too.

How top sites treat Yandex's crawlers

From our robots.txt study of the Tranco top 10,000 (5,583 valid files, fetched October 20, 2026):

None of the tokens for Yandex's crawlers is tracked by name in our study. For scale, 20.1% of top-site files block at least one AI training crawler and 4.3% refuse every unnamed crawler with User-agent: * and Disallow: /, which would catch Yandex's crawlers too.

More questions about Yandex

Will blocking Yandex affect my Google rankings?

Only if one of its tokens is a Google search crawler. None of Yandex's tokens are Googlebot, so classic Google rankings are unaffected.

How quickly does a robots.txt change take effect for Yandex?

Crawlers cache robots.txt, commonly for up to a day. Expect a change to apply within about 24 hours, and confirm it in your logs.

Can I allow Yandex on some sections and block it on others?

Yes. Inside the group for Yandex, add Disallow lines for the paths to keep private and Allow lines for exceptions. The longest matching rule wins.

Does blocking Yandex delete content it already collected?

No. robots.txt only controls future visits. For content already collected, check whether Yandex offers a removal or opt-out process.

Is a firewall block better than robots.txt for Yandex?

A firewall enforces the decision, while robots.txt asks politely. Many operators recommend robots.txt because blocking their IPs can stop them reading your rules at all. Use robots.txt first, and add a firewall rule only if requests continue.