SemrushBot
Operated by Semrush
Quick Facts
- User-Agent:
- SemrushBot
- Category:
- SEO Tools
- Operator:
- Semrush
- Safety:
- Safe
- Blocking Impact:
- Low - No effect on Google or Bing rankings
- SEO Impact Score:
- 1/10
- Official docs:
- www.semrush.com
Confirmed by the operator
Semrush names SemrushBot on its own documentation page, linked above. The token, purpose and robots.txt behaviour on this profile come from that page, re-checked on October 19, 2026.
What is SemrushBot?
SemrushBot is the crawler Semrush uses to discover and collect web data, mainly for its public backlink index (Backlink Analytics). Semrush documents it on its own bot page.
Semrush says SemrushBot "is the search bot software that Semrush sends out to discover and collect new and updated web data". The data feeds Backlink Analytics and several other Semrush tools. SemrushBot starts from a list of URLs, saves the links it finds and revisits that "crawl frontier" to track content changes, new pages and dead links.
Semrush runs ten separate user-agent tokens for different products, so a SemrushBot rule does not stop Site Audit, the SEO Writing Assistant or the other tools. Blocking SemrushBot has no effect on Google or Bing, because search engines do not use Semrush data. What it changes is what Semrush users can see about your links.
What happens if you block SemrushBot?
How to block Semrush bot crawlers
To block Semrush crawlers, add a User-agent: SemrushBot group with Disallow: / to robots.txt; Semrush's own bot page states that its crawlers follow robots.txt. This blocks the main Semrush bot; the audit and on-page tools use separate tokens such as SemrushBot-OCOB, which need their own lines. Blocking Semrush has no effect on Google. To block SEO bots from other vendors as well, read block Semrush and similar tools safely, and compare the Ahrefs bot name and rules.
User-agent: SemrushBot
Disallow: /The Semrush crawler family compared
"Blocking Semrush" is really ten decisions. The Semrush bot documentation lists a separate token for each product, and a SemrushBot group only stops the backlink crawler.
| Token | Feeds | Block if you want to... |
|---|---|---|
| SemrushBot | Backlink Analytics webgraph | Hide your links from competitors |
| SiteAuditBot | Site Audit | Stop audits run by anyone, including you |
| SemrushBot-BA | Backlink Audit | Opt out of backlink toxicity checks |
| SemrushBot-SI | On Page SEO Checker and similar | Stop on-page reports |
| SemrushBot-SWA | SEO Writing Assistant URL checks | Stop writing-tool checks |
| SplitSignalBot | SplitSignal SEO A/B tests | Only if you do not run SplitSignal |
| SemrushBot-OCOB | Content Toolkit | Stop content reports |
| SemrushBot-FT | Plagiarism Checker | Stop plagiarism lookups |
| RyteBot | Ryte.com tools | Stop Ryte audits |
| SemrushBot-ESI | Enterprise Site Intelligence | Stop enterprise audits |
What the top 10,000 sites do with Semrush bots
We fetched the robots.txt file of every domain in the Tranco top 10,000 on 20 October 2026 and parsed 5,583 valid files with RFC 9309 rules (full study and method). 409 files (7.3%) block SemrushBot from the whole site. Only 190 do it by naming SemrushBot; the other 219 block every crawler with a wildcard group.
| Token | Files naming it | Share of valid files |
|---|---|---|
| SemrushBot | 262 | 4.7% |
| SemrushBot-SWA | 142 | 2.5% |
| SemrushBot-OCOB | 77 | 1.4% |
| SemrushBot-BA | 30 | 0.5% |
| SemrushBot-SI | 27 | 0.5% |
| SiteAuditBot | 29 | 0.5% |
| RyteBot | 12 | 0.2% |
| SplitSignalBot | 8 | 0.1% |
| SemrushBot-ESI | 2 | 0.0% |
| SemrushBot-FT | 2 | 0.0% |
Two lessons stand out. First, most sites that write a Semrush rule only name SemrushBot, so Site Audit, the SEO Writing Assistant and the other tools still reach them. Second, 33 files set a Crawl-delay for SemrushBot, and 5 of them use a value above 10 seconds, which Semrush cuts down to 10. Block rates rise with popularity: 10.2% in the top 1,000 against 7.2% in ranks 5,001 to 10,000.
SemrushBot rules that differ from Google and AI crawlers
| Behaviour | SemrushBot | Googlebot | GPTBot |
|---|---|---|---|
| Crawl-delay | Yes, capped at 10 seconds | Ignored | Not documented |
| robots.txt returns 4xx | Treated as no restrictions | Treated as no restrictions | Not documented |
| robots.txt returns 5xx | Stops crawling the site | Pauses crawling | Not documented |
| Time to see robots.txt changes | Up to 1 hour or 100 requests | Usually within 24 hours | Not documented |
| Block by IP? | No: Semrush uses no consecutive IP blocks | Use reverse DNS | Published IP list |
Coverage context: Hostinger saw SemrushBot on 25% of sites in January 2026, against 60% for AhrefsBot (Hostinger, 20 January 2026).
SemrushBot use cases
| If you are... | Recommended setting and why |
|---|---|
| An agency that audits client sites in Semrush | Allow SiteAuditBot at minimum, or your own audits fail. Block SemrushBot only if you also accept losing backlink data. |
| A site under crawl load | Crawl-delay: 10 on SemrushBot, which is the maximum Semrush honours. |
| Hiding your backlink profile from competitors | Block SemrushBot and AhrefsBot together; one alone leaves the data visible elsewhere. |
| Running SplitSignal tests | Never block SplitSignalBot. |
# Keep Semrush audits, slow the backlink crawler
User-agent: SemrushBot
Crawl-delay: 10
User-agent: SiteAuditBot
Allow: /Run a website crawlability test to see every Semrush and Ahrefs token at once, and read SemrushBot vs AhrefsBot for the wider trade-off.
Tools for SEO crawler settings
- AI crawler checker: tests 265 bots against your robots.txt in one scan.
- AI crawler diagnostic: fetches a page as each AI bot and names the layer that blocks it.
- robots.txt validator: shows allowed, blocked and partial status per bot.
- robots.txt generator: builds a file with presets for search, agents and training.
- batch URL checker: compares up to 20 sites side by side, with CSV export.
- llms.txt generator: writes the file AI assistants read for a site summary.
- schema markup generator: produces JSON-LD for eight common types.
- GEO audit: scores a page for generative engine optimization.
How to block SemrushBot with robots.txt
<code>User-agent: SemrushBot</code> - Matching is case-insensitive. Semrush says you need a robots.txt file on each subdomain, and that it can take up to one hour or 100 requests for SemrushBot to notice a change.
Is SemrushBot safe to allow?
What does SemrushBot do?
From Semrush's bot page. Each tool below uses its own token unless noted.
- Builds the public backlink index behind Backlink Analytics (SemrushBot)
- Runs Site Audit crawls for on-page and technical issues (SiteAuditBot)
- Checks backlinks for Backlink Audit (SemrushBot-BA) and prospects for Link Building
- Checks URLs for the SEO Writing Assistant (SemrushBot-SWA) and on-page reports (SemrushBot-SI)
- Feeds Content Toolkit (SemrushBot-OCOB), the Plagiarism Checker (SemrushBot-FT), SplitSignal tests, Ryte tools and Enterprise Site Intelligence
SemrushBot in context
SemrushBot is a utility crawler operated by Semrush. In plain terms, it checks pages for an SEO, uptime or analytics service. Our database files it under SEO tools and ranks it Tier 4, specialist crawlers that matter to particular sites or markets. 19 of the 19 SEO tools in our database share this tier.
Its safety rating is safe and its blocking impact is not documented. So the decision about SemrushBot rests on what you want from Semrush, not on its behaviour. How we rate safety and impact.
How site owners treat SemrushBot
In our own study of robots.txt files across the Tranco top 10,000 sites (fetched October 20, 2026, 5,583 valid files), 262 files (4.7%) name SemrushBot in a User-agent line, and 409 (7.3%) block it from the homepage. 190 of those blocks come from a group written for this crawler by name, and 219 come from a broad User-agent: * rule that catches it by accident. Larger sites are stricter: 10.2% of the top 1,000 block it, against 6.8% of the middle band and 7.2% of the tail.
Does it follow robots.txt?
SemrushBot's operator does not clearly document how it treats robots.txt. Write the SemrushBot rule anyway, then confirm in your logs that its requests stop.
Semrush crawlers and close alternatives
SemrushBot is one of 3 Semrush crawlers in our database. A User-agent: SemrushBot group controls SemrushBot only; the other Semrush tokens below need their own lines.
SemrushBot: this crawler (Tier 4)SemrushBot-OCOB: SemrushBot-OCOB, checks pages for an SEO, uptime or analytics service (Tier 4)SemrushBot-SWA: Semrush Writing Assistant, checks pages for an SEO, uptime or analytics service (Tier 4)
To give SemrushBot the same answer as its siblings, stack User-agent: SemrushBot with their lines above one set of rules; to treat SemrushBot differently, give it its own group.
Other SEO tools that do the same job (checks pages for an SEO, uptime or analytics service) include AhrefsBot from Ahrefs, AhrefsSiteAudit from Ahrefs, AwarioBot from Awario, AwarioRssBot from Awario, AwarioSmartBot from Awario, Chrome-Lighthouse from Google. Deciding about SemrushBot is usually a decision about the whole group, so check how your file treats these too.
We first recorded SemrushBot on March 17, 2026. Semrush documents it on semrush.com, and we recheck that page before each release.
Which rule SemrushBot actually follows
SemrushBot reads exactly one group in robots.txt, the most specific one that names SemrushBot, and never combines rules across groups. This short file shows the trap:
User-agent: *
Disallow: /private/
User-agent: SemrushBot
Disallow: /drafts/- Find the group. SemrushBot looks for a group naming
SemrushBot. One exists, so it ignoresUser-agent: *completely. - Apply only that group. It is refused
/drafts/and nothing else. - The surprise.
/private/is open to SemrushBot, because that rule lives in the*group it no longer reads. To keep it out of both, repeatDisallow: /private/inside its own group.
Test yourself: does a rule under User-agent: * apply to SemrushBot if the file also has a User-agent: SemrushBot group?
No. Once a group names it, SemrushBot reads only that group. Put every rule you want it to follow inside its own group.
Test yourself: will blocking SemrushBot remove your pages from AI answers?
Often, yes. SemrushBot is a utility crawler, so pages it cannot fetch can drop out of the answers or results it supports.
Test yourself: is User-agent: semrushbot the same as User-agent: SemrushBot?
Yes. RFC 9309 makes user-agent matching case-insensitive. Paths are a different story: Disallow: /Blog/ does not block /blog/.
Related: how AI engines choose their sources
Should you allow SemrushBot? Decide by situation
| If you are... | Recommended setting |
|---|---|
| You publish content and want AI answers to cite it | Low stakes either way. It does not decide whether AI answers cite you. |
| You license content and want to stop AI training | Blocking it does not stop training, because SemrushBot is a utility crawler. Block the operator's training crawler instead. |
| You run an online shop | Allow it unless your logs show it is costly. |
| Your server is under load from bots | Rate-limit before blocking. It is rated safe in our database. |
Check SemrushBot on your own site
- Open
/robots.txton your domain and search forSemrushBot. If it is not named, it followsUser-agent: *. - List the
Disallowlines in the group SemrushBot follows and test your key URLs against them; the longest match wins, andAllowwins a tie. - Look beyond the file. A firewall or bot manager can refuse SemrushBot even when robots.txt allows it, and that never shows in the file.
- Run our AI crawler access test to see how SemrushBot and every other crawler we track are treated in one pass, including firewall refusals.
Common mistakes with SemrushBot
The errors we see most often with SemrushBot in the files and firewalls we test:
- Treating SemrushBot as an AI or search crawler. Blocking it has little effect on visibility; decide based on load and what it does with your data.
- Relying on robots.txt alone when SemrushBot is rated caution or aggressive. Back the rule with a server-side block.
- Blocking SemrushBot by IP alone. Ranges change; match the token and verify with logs.
All three hide inside a file that looks correct, so check SemrushBot from outside after every change.
Spotting SemrushBot in your logs
robots.txt says what SemrushBot may do; your access logs show what it actually did.
Count requests
On a server with standard Nginx or Apache logs, this counts matching requests per day (adjust the log path for your host):
grep -iE "SemrushBot" /var/log/nginx/access.log \
| awk '{print $4}' | cut -d: -f1 | sort | uniq -cAnd this shows which paths it requested most, which is the quickest way to see whether your Disallow lines are being respected:
grep -iE "SemrushBot" /var/log/nginx/access.log \
| awk '{print $7}' | sort | uniq -c | sort -rn | head -20Check the requests are genuine
Anyone can send the SemrushBot user-agent, so confirm the source before acting on a match:
SemrushBot: we hold no published IP list, so use reverse DNS where the operator documents a hostname and treat an unverifiable SemrushBot request with caution.
Frequently Asked Questions
What is SemrushBot?
Is SemrushBot safe?
How do I block SemrushBot?
User-agent: SemrushBot Disallow: /
Semrush says it can take up to one hour or 100 requests to notice the change. This blocks only the backlink crawler; Site Audit (SiteAuditBot) and the other Semrush tools have their own tokens.
How do I block all Semrush bots?
User-agent: SemrushBot User-agent: SiteAuditBot User-agent: SemrushBot-BA User-agent: SemrushBot-SI User-agent: SemrushBot-SWA User-agent: SplitSignalBot User-agent: SemrushBot-OCOB User-agent: SemrushBot-FT User-agent: RyteBot User-agent: SemrushBot-ESI Disallow: /
Will blocking SemrushBot hurt my SEO?
Does SemrushBot respect robots.txt and Crawl-delay?
Can I block SemrushBot by IP address?
What happens if my robots.txt returns an error?
How many sites block SemrushBot?
Why does SemrushBot visit pages that do not exist?
Does blocking the Semrush bot affect Google?
Does blocking SemrushBot stop Semrush Site Audit?
What is the maximum Crawl-delay SemrushBot honours?
Should I block Semrush bots?
Related Bots
- SemrushBot-OCOB Cautionsame_operator
- AhrefsBotSafesame_category
- AwarioRssBotSafesame_category
- AwarioSmartBotSafesame_category