Search Engines Directory

Common mistakes with search engines

The errors we see most often with search engines in the files and firewalls we test:

1. Blocking search engines to save bandwidth and then wondering why pages drop out of that search engine. Use crawl-rate settings or caching first.
2. Writing a User-agent: Applebot-Extended group to add one rule for search engines and accidentally dropping every rule it used to follow under User-agent: *.
3. Disallowing CSS and JavaScript folders, which can stop search engines rendering your pages properly.

All three hide inside a file that looks correct, so check search engines from outside after every change.

Spotting search engines in your logs

robots.txt says what AdIdxBot and Amzn-SearchBot and Applebot and baidu and Baiduspider and Baiduspider-ads may do; your access logs show what they actually did.

Count requests

On a server with standard Nginx or Apache logs, this counts matching requests per day (adjust the log path for your host):

grep -iE "AdIdxBot|Amzn-SearchBot|Applebot|baidu|Baiduspider|Baiduspider-ads" /var/log/nginx/access.log \
  | awk '{print $4}' | cut -d: -f1 | sort | uniq -c

And this shows which paths it requested most, which is the quickest way to see whether your Disallow lines are being respected:

grep -iE "AdIdxBot|Amzn-SearchBot|Applebot|baidu|Baiduspider|Baiduspider-ads" /var/log/nginx/access.log \
  | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

Check the requests are genuine

Anyone can send these user-agents, so confirm the source before acting on a match:

• AdIdxBot: we hold no published IP list, so use reverse DNS where search engines documents a hostname and treat an unverifiable AdIdxBot request with caution. Check it separately from Amzn-SearchBot and Applebot and baidu and Baiduspider and Baiduspider-ads.
• Amzn-SearchBot: we hold no published IP list, so use reverse DNS where search engines documents a hostname and treat an unverifiable Amzn-SearchBot request with caution. Check it separately from AdIdxBot and Applebot and baidu and Baiduspider and Baiduspider-ads.
• Applebot: we hold no published IP list, so use reverse DNS where search engines documents a hostname and treat an unverifiable Applebot request with caution. Check it separately from AdIdxBot and Amzn-SearchBot and baidu and Baiduspider and Baiduspider-ads.
• baidu: we hold no published IP list, so use reverse DNS where search engines documents a hostname and treat an unverifiable baidu request with caution. Check it separately from AdIdxBot and Amzn-SearchBot and Applebot and Baiduspider and Baiduspider-ads.
• Baiduspider: we hold no published IP list, so use reverse DNS where search engines documents a hostname and treat an unverifiable Baiduspider request with caution. Check it separately from AdIdxBot and Amzn-SearchBot and Applebot and baidu and Baiduspider-ads.
• Baiduspider-ads: we hold no published IP list, so use reverse DNS where search engines documents a hostname and treat an unverifiable Baiduspider-ads request with caution. Check it separately from AdIdxBot and Amzn-SearchBot and Applebot and baidu and Baiduspider.

Recent changes among search engines

The newest search engines in our database: Yeti (October 19, 2026), YandexWebmaster (September 22, 2026), YandexVideoParser (September 22, 2026), YandexVideo (September 22, 2026), YandexVertis (September 22, 2026), YandexVerticals (September 22, 2026). A User-agent: * default decides what each of these gets until you name it.

Related: how AI engines choose their sources

Movement among the tracked search engines: 2026 and October 20, 2026, files naming Applebot went from 239 to 250 (+11) and files blocking it from 265 to 283 (+18), so more sites wrote a rule for Applebot and more of those rules block it. 2026 and October 20, 2026, files naming Applebot-Extended went from 556 to 600 (+44) and files blocking it from 643 to 694 (+51), so more sites wrote a rule for Applebot-Extended and more of those rules block it. 2026 and October 20, 2026, files naming bingbot went from 381 to 397 (+16) and files blocking it from 169 to 182 (+13), so more sites wrote a rule for bingbot and more of those rules block it. 2026 and October 20, 2026, files naming Googlebot went from 502 to 528 (+26) and files blocking it from 143 to 154 (+11), so more sites wrote a rule for Googlebot and more of those rules block it.

How top sites treat search engines

From our robots.txt study of the Tranco top 10,000 (5,583 valid files, fetched October 20, 2026):

• Applebot-Extended, the most blocked of the search engines tokens: blocked in 12.4% of files (478 with its own group, 216 through User-agent: *), more often by the top 1,000 (17.6%) than the long tail (11.3%).
• Applebot, second of the search engines tokens: blocked in 5.1% of files (90 with its own group, 193 through User-agent: *), more often by the top 1,000 (6.4%) than the long tail (4.7%).
• bingbot, third of the search engines tokens: blocked in 3.3% of files (21 with its own group, 161 through User-agent: *), more often by the top 1,000 (3.8%) than the long tail (3%).
• Googlebot, fourth of the search engines tokens: blocked in 2.8% of files (13 with its own group, 141 through User-agent: *), more often by the top 1,000 (3.6%) than the long tail (2.4%).

Related: writing content AI engines can cite

For scale, 20.1% of all files block at least one AI training crawler. Applebot-Extended is one of the tokens driving that figure.

More questions about search engines

How many search engines should I name in robots.txt?

Only the ones that need a different answer from your default. Every crawler you do not name follows User-agent: *, so a short file with a sensible default is easier to maintain than a long list.

Do new search engines appear often?

Yes. New crawlers launch every month, which is why a deliberate User-agent: * default matters: it decides what happens to crawlers you have never heard of.

Can I copy another site's rules for search engines?

Use them as a pattern, not a template. Each file reflects that site's licensing and infrastructure. Copy the structure, then name the crawlers that matter to you.

Where do I check how my site treats search engines?

Run the AI crawler access test. It reports every crawler in this category in one pass, including firewall refusals that robots.txt never shows.