Bot directoryCategory

SEO Tools Directory

Commercial SEO platform crawlers (Ahrefs, Semrush etc.). Blocking hides your data from competitors.

Check which AI crawlers get inBuild a robots.txt

SEO platforms crawl to build backlink and audit data. Blocking them does not touch your rankings; it just hides your site from those tools, including competitors who use them.
  • 19tokens
  • 16safe
  • 3caution
  • 0aggressive

Common providers

Pick a provider to see only its crawlers. Each provider also has a full profile under Bot Operators.

All 19 SEO Tools

The SEO tools category in numbers

Our database lists 19 SEO tools from 12 operators. 16 are rated safe, 3 caution and 0 aggressive. What they are for:

SEO tools by purpose
PurposeCrawlers
checks pages for an SEO, uptime or analytics service18
fetches a page when a person asks an assistant or app to open it1

Priority tiers

How the 19 SEO tools spread across our six priority tiers, where Tier 1 is what you would notice losing:

  • Tier 4: 19 crawlers

None of the SEO tools reach Tier 1 or 2, so decisions in this category rarely decide your visibility. They are mostly about load, data use and privacy.

How top sites treat this category

From our robots.txt study of the Tranco top 10,000 (5,583 valid files, fetched October 20, 2026):

SEO tools named and blocked in top-site robots.txt files
TokenFiles naming itFiles blocking it
AhrefsBot6.2%8.2%
DotBot4%7.5%
MJ12bot4.9%8.1%
SemrushBot4.7%7.3%

Tokens in SEO tools can be blocked without being named, through User-agent: * with Disallow: /, which is why some blocking shares in this table exceed the naming shares.

How to review SEO tools on your site

  1. List the ones that matter. Start with Tier 1 and 2 in the table above; the rest can usually follow a default.
  2. Choose a default. Decide what User-agent: * should say, because every crawler you do not name follows it.
  3. Name the exceptions. Give each crawler that needs a different answer its own group, and repeat any shared Disallow lines inside it.
  4. Check the edge. Firewalls and bot managers often treat these crawlers differently from robots.txt. Run the GPTBot and ClaudeBot access check to see both layers together.
Test yourself: a crawler in this category is not named anywhere in your robots.txt. Which rules does it follow?

The User-agent: * group. If there is no such group, it may crawl everything.

Test yourself: does a safety rating of "safe" mean you should allow a crawler?

Not necessarily. Safety describes behaviour. Whether to allow it depends on what it does with your content and what you lose by blocking it.

The SEO tools worth knowing first

Ranked by priority tier, these are the SEO tools with the biggest effect on a typical site:

  • AhrefsBot (Ahrefs, Tier 4): a utility crawler that checks pages for an SEO, uptime or analytics service.
  • AhrefsSiteAudit (Ahrefs, Tier 4): a utility crawler that checks pages for an SEO, uptime or analytics service.
  • AwarioBot (Awario, Tier 4): a utility crawler that checks pages for an SEO, uptime or analytics service.
  • AwarioRssBot (Awario, Tier 4): a utility crawler that checks pages for an SEO, uptime or analytics service.
  • AwarioSmartBot (Awario, Tier 4): a utility crawler that checks pages for an SEO, uptime or analytics service.
  • Chrome-Lighthouse (Google, Tier 4): a utility crawler that checks pages for an SEO, uptime or analytics service.
  • DataForSeoBot (DataForSEO, Tier 4): a utility crawler that checks pages for an SEO, uptime or analytics service.
  • DotBot (Moz, Tier 4): a utility crawler that checks pages for an SEO, uptime or analytics service.

Of the 19 SEO tools we track, 0 are documented as following robots.txt, 0 as not following it for some requests, 0 with exceptions or disputed behaviour, and 19 with no clear statement from the operator. Where behaviour is unclear, back a robots.txt rule with a server-side one.

What to do with SEO tools, by type of site

Recommended approach to SEO tools by type of site
If you run...Recommended approach
Site that uses this kind of serviceAllow the crawlers for services you actually use; they need access to work.
Site that does not use the serviceBlocking is low risk; it has little effect on search or AI visibility.
Site under heavy bot loadRate-limit or block the noisiest crawlers first.
Privacy-sensitive siteBlock crawlers whose data use you have not reviewed.

Common mistakes with SEO tools

The errors we see most often with SEO tools in the files and firewalls we test:

  1. Treating SEO tools as an AI or search crawler. Blocking it has little effect on visibility; decide based on load and what it does with your data.
  2. Relying on robots.txt alone when SEO tools is rated caution or aggressive. Back the rule with a server-side block.
  3. Blocking SEO tools by IP alone. Ranges change; match the token and verify with logs.

All three hide inside a file that looks correct, so check SEO tools from outside after every change.

Spotting SEO tools in your logs

robots.txt says what AhrefsBot and AhrefsSiteAudit and AwarioBot and AwarioRssBot and AwarioSmartBot and Chrome-Lighthouse may do; your access logs show what they actually did.

Count requests

On a server with standard Nginx or Apache logs, this counts matching requests per day (adjust the log path for your host):

grep -iE "AhrefsBot|AhrefsSiteAudit|AwarioBot|AwarioRssBot|AwarioSmartBot|Chrome-Lighthouse" /var/log/nginx/access.log \
  | awk '{print $4}' | cut -d: -f1 | sort | uniq -c

And this shows which paths it requested most, which is the quickest way to see whether your Disallow lines are being respected:

grep -iE "AhrefsBot|AhrefsSiteAudit|AwarioBot|AwarioRssBot|AwarioSmartBot|Chrome-Lighthouse" /var/log/nginx/access.log \
  | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

Check the requests are genuine

Anyone can send these user-agents, so confirm the source before acting on a match:

  • AhrefsBot: we hold no published IP list, so use reverse DNS where SEO tools documents a hostname and treat an unverifiable AhrefsBot request with caution. Check it separately from AhrefsSiteAudit and AwarioBot and AwarioRssBot and AwarioSmartBot and Chrome-Lighthouse.
  • AhrefsSiteAudit: we hold no published IP list, so use reverse DNS where SEO tools documents a hostname and treat an unverifiable AhrefsSiteAudit request with caution. Check it separately from AhrefsBot and AwarioBot and AwarioRssBot and AwarioSmartBot and Chrome-Lighthouse.
  • AwarioBot: we hold no published IP list, so use reverse DNS where SEO tools documents a hostname and treat an unverifiable AwarioBot request with caution. Check it separately from AhrefsBot and AhrefsSiteAudit and AwarioRssBot and AwarioSmartBot and Chrome-Lighthouse.
  • AwarioRssBot: we hold no published IP list, so use reverse DNS where SEO tools documents a hostname and treat an unverifiable AwarioRssBot request with caution. Check it separately from AhrefsBot and AhrefsSiteAudit and AwarioBot and AwarioSmartBot and Chrome-Lighthouse.
  • AwarioSmartBot: we hold no published IP list, so use reverse DNS where SEO tools documents a hostname and treat an unverifiable AwarioSmartBot request with caution. Check it separately from AhrefsBot and AhrefsSiteAudit and AwarioBot and AwarioRssBot and Chrome-Lighthouse.
  • Chrome-Lighthouse: we hold no published IP list, so use reverse DNS where SEO tools documents a hostname and treat an unverifiable Chrome-Lighthouse request with caution. Check it separately from AhrefsBot and AhrefsSiteAudit and AwarioBot and AwarioRssBot and AwarioSmartBot.

Recent changes among SEO tools

The newest SEO tools in our database: YandexTracker (September 22, 2026), YandexMetrika (September 22, 2026), SemrushBot-SWA (September 22, 2026), Rogerbot (September 22, 2026), AwarioBot (September 22, 2026), AhrefsSiteAudit (September 22, 2026). A User-agent: * default decides what each of these gets until you name it.

Related: how AI engines choose their sources

Movement among the tracked SEO tools: 2026 and October 20, 2026, files naming AhrefsBot went from 336 to 347 (+11) and files blocking it from 436 to 456 (+20), so more sites wrote a rule for AhrefsBot and more of those rules block it. 2026 and October 20, 2026, files naming DotBot went from 215 to 224 (+9) and files blocking it from 395 to 419 (+24), so more sites wrote a rule for DotBot and more of those rules block it. 2026 and October 20, 2026, files naming MJ12bot went from 267 to 273 (+6) and files blocking it from 435 to 455 (+20), so more sites wrote a rule for MJ12bot and more of those rules block it. 2026 and October 20, 2026, files naming SemrushBot went from 252 to 262 (+10) and files blocking it from 386 to 409 (+23), so more sites wrote a rule for SemrushBot and more of those rules block it.

How top sites treat SEO tools

From our robots.txt study of the Tranco top 10,000 (5,583 valid files, fetched October 20, 2026):

  • AhrefsBot, the most blocked of the SEO tools tokens: blocked in 8.2% of files (229 with its own group, 227 through User-agent: *), more often by the top 1,000 (11.6%) than the long tail (7.8%).
  • MJ12bot, second of the SEO tools tokens: blocked in 8.1% of files (223 with its own group, 232 through User-agent: *), more often by the top 1,000 (11.6%) than the long tail (7.7%).
  • DotBot, third of the SEO tools tokens: blocked in 7.5% of files (184 with its own group, 235 through User-agent: *), more often by the top 1,000 (10.6%) than the long tail (7.1%).
  • SemrushBot, fourth of the SEO tools tokens: blocked in 7.3% of files (190 with its own group, 219 through User-agent: *), more often by the top 1,000 (10.2%) than the long tail (7.2%).

For scale, 20.1% of all files block at least one AI training crawler. None of the tokens for SEO tools comes close to that rate, so blocking them is still a minority choice.

More questions about SEO tools

How many SEO tools should I name in robots.txt?
Only the ones that need a different answer from your default. Every crawler you do not name follows User-agent: *, so a short file with a sensible default is easier to maintain than a long list.
Do new SEO tools appear often?
Yes. New crawlers launch every month, which is why a deliberate User-agent: * default matters: it decides what happens to crawlers you have never heard of.
Can I copy another site's rules for SEO tools?
Use them as a pattern, not a template. Each file reflects that site's licensing and infrastructure. Copy the structure, then name the crawlers that matter to you.
Where do I check how my site treats SEO tools?
Run the crawlability checker for AI search. It reports every crawler in this category in one pass, including firewall refusals that robots.txt never shows.

FAQ

How many SEO Tools are tracked?
We currently track 19 SEO Tools with user-agent strings, safety ratings, and blocking rules.
Which SEO Tools should I never block?
Never block bots with 'critical' impact rating - there are 0 in this category. Blocking them removes your site from their search results.
Are all SEO Tools safe?
Out of 19 bots in this category: 16 safe, 3 caution, 0 aggressive.
How do I block all SEO Tools at once?
Generate a combined robots.txt block rule for all SEO Tools using our bulk robots.txt generator, or manually list each User-agent + Disallow: / directive.
WhatsApp us