Safe Official docs SEO Tools

SemrushBot

Operated by Semrush

Quick Facts

User-Agent:
SemrushBot
Category:
SEO Tools
Operator:
Semrush
Safety:
Safe
Blocking Impact:
Low - No effect on Google or Bing rankings
SEO Impact Score:
1/10
Official docs:
www.semrush.com

Confirmed by the operator

Semrush names SemrushBot on its own documentation page, linked above. The token, purpose and robots.txt behaviour on this profile come from that page, re-checked on September 22, 2026.

What is SemrushBot?

SemrushBot is the crawler Semrush uses to discover and collect web data, mainly for its public backlink index (Backlink Analytics). Semrush documents it on its own bot page.

Semrush says SemrushBot "is the search bot software that Semrush sends out to discover and collect new and updated web data". The data feeds Backlink Analytics and several other Semrush tools. SemrushBot starts from a list of URLs, saves the links it finds and revisits that "crawl frontier" to track content changes, new pages and dead links.
Semrush runs ten separate user-agent tokens for different products, so a SemrushBot rule does not stop Site Audit, the SEO Writing Assistant or the other tools. Blocking SemrushBot has no effect on Google or Bing, because search engines do not use Semrush data. What it changes is what Semrush users can see about your links.

What happens if you block SemrushBot?

✅ No effect on Google or Bing - Search engines do not use Semrush data, so blocking SemrushBot does not change your rankings. The real effect is inside Semrush: your site's outgoing links stop being refreshed in Semrush's backlink index, so competitors (and you) see older link data. Other Semrush tools use their own tokens and are not affected.
Allow it unless you want to hide your link data from Semrush users. It has no effect on Google.

How to block Semrush bot crawlers

To block Semrush crawlers, add a User-agent: SemrushBot group with Disallow: / to robots.txt; Semrush's own bot page states that its crawlers follow robots.txt. This blocks the main Semrush bot; the audit and on-page tools use separate tokens such as SemrushBot-OCOB, which need their own lines. Blocking Semrush has no effect on Google. To block SEO bots from other vendors as well, read block Semrush and similar tools safely, and compare the Ahrefs bot name and rules.

User-agent: SemrushBot
Disallow: /

The Semrush crawler family compared

"Blocking Semrush" is really ten decisions. The Semrush bot documentation lists a separate token for each product, and a SemrushBot group only stops the backlink crawler.

Semrush user agents and what each one feeds
TokenFeedsBlock if you want to...
SemrushBotBacklink Analytics webgraphHide your links from competitors
SiteAuditBotSite AuditStop audits run by anyone, including you
SemrushBot-BABacklink AuditOpt out of backlink toxicity checks
SemrushBot-SIOn Page SEO Checker and similarStop on-page reports
SemrushBot-SWASEO Writing Assistant URL checksStop writing-tool checks
SplitSignalBotSplitSignal SEO A/B testsOnly if you do not run SplitSignal
SemrushBot-OCOBContent ToolkitStop content reports
SemrushBot-FTPlagiarism CheckerStop plagiarism lookups
RyteBotRyte.com toolsStop Ryte audits
SemrushBot-ESIEnterprise Site IntelligenceStop enterprise audits

What the top 10,000 sites do with Semrush bots

We fetched the robots.txt file of every domain in the Tranco top 10,000 on 5 October 2026 and parsed 4,962 valid files with RFC 9309 rules (full study and method). 353 files (7.1%) block SemrushBot from the whole site. Only 175 do it by naming SemrushBot; the other 178 block every crawler with a wildcard group.

How often each Semrush token is named in top-10,000 robots.txt files
TokenFiles naming itShare of valid files
SemrushBot2364.8%
SemrushBot-SWA921.9%
SemrushBot-OCOB651.3%
SemrushBot-BA250.5%
SemrushBot-SI250.5%
SiteAuditBot210.4%
RyteBot100.2%
SplitSignalBot50.1%
SemrushBot-ESI10.0%
SemrushBot-FT10.0%

Two lessons stand out. First, most sites that write a Semrush rule only name SemrushBot, so Site Audit, the SEO Writing Assistant and the other tools still reach them. Second, 29 files set a Crawl-delay for SemrushBot, and 6 of them use a value above 10 seconds, which Semrush cuts down to 10. Block rates rise with popularity: 9.7% in the top 1,000 against 7.1% in ranks 5,001 to 10,000.

SemrushBot rules that differ from Google and AI crawlers

SemrushBot compared with Googlebot and GPTBot
BehaviourSemrushBotGooglebotGPTBot
Crawl-delayYes, capped at 10 secondsIgnoredNot documented
robots.txt returns 4xxTreated as no restrictionsTreated as no restrictionsNot documented
robots.txt returns 5xxStops crawling the sitePauses crawlingNot documented
Time to see robots.txt changesUp to 1 hour or 100 requestsUsually within 24 hoursNot documented
Block by IP?No: Semrush uses no consecutive IP blocksUse reverse DNSPublished IP list

Coverage context: Hostinger saw SemrushBot on 25% of sites in January 2026, against 60% for AhrefsBot (Hostinger, 20 January 2026).

Share of websites each crawler reached Hostinger coverage data: Googlebot 72%, AhrefsBot 60%, Bingbot 57.67%, Meta-ExternalAgent 57.33%, OAI-SearchBot 55.67%, SemrushBot 25%, Applebot 24.33%, GPTBot 12% (down from 84%). Share of websites each crawler reached Googlebot 72% AhrefsBot 60% Bingbot 57.7% Meta-ExternalAgent 57.3% OAI-SearchBot 55.7% SemrushBot 25% Applebot 24.3% GPTBot 12%was 84% Source: Hostinger, 66.7B bot requests across 5M+ sites, 20 January 2026.
SemrushBot reaches fewer sites than AhrefsBot or the search engines.

SemrushBot use cases

Use cases: what to do with this bot
If you are...Recommended setting and why
An agency that audits client sites in SemrushAllow SiteAuditBot at minimum, or your own audits fail. Block SemrushBot only if you also accept losing backlink data.
A site under crawl loadCrawl-delay: 10 on SemrushBot, which is the maximum Semrush honours.
Hiding your backlink profile from competitorsBlock SemrushBot and AhrefsBot together; one alone leaves the data visible elsewhere.
Running SplitSignal testsNever block SplitSignalBot.
# Keep Semrush audits, slow the backlink crawler
User-agent: SemrushBot
Crawl-delay: 10

User-agent: SiteAuditBot
Allow: /

Run a free AI crawler check to see every Semrush and Ahrefs token at once, and read SemrushBot vs AhrefsBot for the wider trade-off.

Tools for SEO crawler settings

How to block SemrushBot with robots.txt

<code>User-agent: SemrushBot</code> - Matching is case-insensitive. Semrush says you need a robots.txt file on each subdomain, and that it can take up to one hour or 100 requests for SemrushBot to notice a change.

Block completely (robots.txt)
User-agent: SemrushBot Disallow: /
Allow all (robots.txt)
User-agent: SemrushBot Allow: /
Block private only (robots.txt)
User-agent: SemrushBot Disallow: /private/ Disallow: /api/ Disallow: /admin/ Allow: /
Nginx server block
# Nginx: Hard-block SemrushBot if ($http_user_agent ~* "SemrushBot") { return 403 "Bot blocked"; }
Apache .htaccess
# Apache: Hard-block SemrushBot SetEnvIfNoCase User-Agent "SemrushBot" bad_bot Order Allow,Deny Allow from all Deny from env=bad_bot
Meta robots tag
<meta name="robots" content="noindex, nofollow">
X-Robots-Tag header
X-Robots-Tag: noindex, nofollow

Is SemrushBot safe to allow?

Yes. SemrushBot is a documented crawler: Semrush publishes a page that names it, explains what it collects, lists every related token and gives a contact address ([email protected]) for problems. Semrush says SemrushBot follows robots.txt, supports Crawl-delay up to 10 seconds, and will "work quickly to resolve the issue" if you send log entries showing it ignored your rules.
Semrush publishes no IP list and says it does not use consecutive IP blocks, so you cannot verify by IP or reverse DNS. Control it by user agent in robots.txt, and report suspected fakes to [email protected] with log lines.

What does SemrushBot do?

From Semrush's bot page. Each tool below uses its own token unless noted.

  • Builds the public backlink index behind Backlink Analytics (SemrushBot)
  • Runs Site Audit crawls for on-page and technical issues (SiteAuditBot)
  • Checks backlinks for Backlink Audit (SemrushBot-BA) and prospects for Link Building
  • Checks URLs for the SEO Writing Assistant (SemrushBot-SWA) and on-page reports (SemrushBot-SI)
  • Feeds Content Toolkit (SemrushBot-OCOB), the Plagiarism Checker (SemrushBot-FT), SplitSignal tests, Ryte tools and Enterprise Site Intelligence

Frequently Asked Questions

What is SemrushBot?
SemrushBot is the crawler Semrush uses to discover and collect web data, mainly for its public backlink index, Backlink Analytics. Semrush documents it at semrush.com/bot.
Is SemrushBot safe?
Yes. SemrushBot is a documented crawler: Semrush publishes a page that names it, explains what it collects, lists every related token and gives a contact address ([email protected]) for problems. Semrush says SemrushBot follows robots.txt, supports Crawl-delay up to 10 seconds, and will "work quickly to resolve the issue" if you send log entries showing it ignored your rules.
How do I block SemrushBot?
Add this to the robots.txt file at the root of each host:
User-agent: SemrushBot
Disallow: /

Semrush says it can take up to one hour or 100 requests to notice the change. This blocks only the backlink crawler; Site Audit (SiteAuditBot) and the other Semrush tools have their own tokens.
How do I block all Semrush bots?
List every Semrush token in one group:
User-agent: SemrushBot
User-agent: SiteAuditBot
User-agent: SemrushBot-BA
User-agent: SemrushBot-SI
User-agent: SemrushBot-SWA
User-agent: SplitSignalBot
User-agent: SemrushBot-OCOB
User-agent: SemrushBot-FT
User-agent: RyteBot
User-agent: SemrushBot-ESI
Disallow: /
Will blocking SemrushBot hurt my SEO?
No. Google and Bing do not use Semrush data. Blocking it only stops your site's link data being refreshed in Semrush's own backlink index.
Does SemrushBot respect robots.txt and Crawl-delay?
Yes, according to Semrush. It follows robots.txt, supports wildcards, and honours Crawl-delay up to 10 seconds; higher values are cut down to 10. Without a Crawl-delay, it adjusts its rate to your server load.
Can I block SemrushBot by IP address?
Semrush advises against it: "Do not try to block SemrushBot via IP as we do not use any consecutive IP blocks." It publishes no IP list, so use robots.txt or a user-agent rule instead.
What happens if my robots.txt returns an error?
Semrush says a 4xx response is treated as "no robots.txt", so there are no restrictions, while a 5xx response "will prevent SemrushBot from crawling your entire site". It can handle 3xx redirects. Your robots.txt should return 200.
How many sites block SemrushBot?
In our October 2026 study of 4,962 robots.txt files from the Tranco top 10,000, 353 (7.1%) block SemrushBot from the whole site, and 175 of those name it in their own group. Only 21 files name SiteAuditBot.
Why does SemrushBot visit pages that do not exist?
Semrush says its crawler may keep looking for a page "if other sites across the web link to it". Generally it stops crawling pages it finds do not exist.
Does blocking the Semrush bot affect Google?
No. Google does not use Semrush data, so blocking SemrushBot has no effect on Google rankings. To block it, add "User-agent: SemrushBot" and "Disallow: /" to robots.txt; other Semrush tools such as SiteAuditBot and SemrushBot-OCOB use their own tokens and need their own lines.
Does blocking SemrushBot stop Semrush Site Audit?
No. Site Audit uses a separate token, SiteAuditBot. A SemrushBot group only stops the crawler behind Backlink Analytics. Semrush documents ten tokens in total.
What is the maximum Crawl-delay SemrushBot honours?
Ten seconds. Semrush says higher values are cut down to the 10-second limit, and that robots.txt changes are picked up within one hour or 100 requests.
Should I block Semrush bots?
Only if you want to hide your link data from Semrush users or cut crawl load. Blocking has no effect on Google or Bing. If you or your agency use Semrush Site Audit, keep SiteAuditBot allowed.

Related Bots