AI Crawler Check
Free Bot Analysis Tool
Check
Home / Directory / Search Engines

Search Engines Directory

Search engine crawlers that index your content for organic search visibility. Critical to allow.

24 total
23 safe
1 caution
0 aggressive

Amzn-SearchBot

Safe
User-agent: Amzn-SearchBot
Operator: Amazon

Applebot

Safe
User-agent: Applebot
Operator: Apple

Applebot-Extended

Caution
User-agent: Applebot-Extended
Operator: Apple

AspiegelBot

Safe
User-agent: AspiegelBot
Operator: Unknown

Baiduspider

Safe
User-agent: Baiduspider
Operator: Baidu

Bingbot

Safe
User-agent: bingbot
Operator: Microsoft

DuckDuckBot

Safe
User-agent: DuckDuckBot
Operator: DuckDuckGo

Googlebot

Safe
User-agent: Googlebot
Operator: Google

Mojeek

Safe
User-agent: Mojeek
Operator: Mojeek

MojeekBot

Safe
User-agent: MojeekBot
Operator: Mojeek

PetalBot

Safe
User-agent: PetalBot
Operator: Huawei

SeznamBot

Safe
User-agent: SeznamBot
Operator: Seznam

SeznamHomepageCrawler

Safe
User-agent: SeznamHomepageCrawler
Operator: Seznam

Slurp

Safe
User-agent: Slurp
Operator: Yahoo

Teoma

Safe
User-agent: Teoma
Operator: Ask

Yahoo-Blogs

Safe
User-agent: Yahoo-Blogs
Operator: Yahoo

Yahoo-FeedSeeker

Safe
User-agent: Yahoo-FeedSeeker
Operator: Yahoo

Yahoo-MMCrawler

Safe
User-agent: Yahoo-MMCrawler
Operator: Yahoo

YahooSeeker

Safe
User-agent: YahooSeeker
Operator: Yahoo

Yandex

Safe
User-agent: Yandex
Operator: Yandex

YandexAdditional

Safe
User-agent: YandexAdditional
Operator: Yandex

YandexAdditionalBot

Safe
User-agent: YandexAdditionalBot
Operator: Yandex

YandexBot

Safe
User-agent: YandexBot
Operator: Yandex

baidu

Safe
User-agent: baidu
Operator: Baidu

FAQ

How many Search Engines are tracked?
We currently track 24 Search Engines with user-agent strings, safety ratings, and blocking rules.
Which Search Engines should I never block?
Never block bots with 'critical' impact rating — there are 24 in this category. Blocking them removes your site from their search results.
Are all Search Engines safe?
Out of 24 bots in this category: 23 safe, 1 caution, 0 aggressive.
How do I block all Search Engines at once?
Generate a combined robots.txt block rule for all Search Engines using our bulk robots.txt generator, or manually list each User-agent + Disallow: / directive.