Semrush Writing Assistant
SafeUser-agent: SemrushBot-SWA
Category: SEO Tools
SEO impact if blocked: 2/10
Semrush
3 crawlers operated by Semrush
https://www.semrush.com/Who Semrush is, for site owners
Semrush operates 3 crawlers in our directory, filed under SEO Tools. Of those 3, 2 rated safe, 1 rated caution for crawl behaviour.
Blocking Semrush carries an SEO impact score of 0/10 to 2/10 across its crawlers. 2 of them share the highest blocking impact here: Varies - Evaluate before blocking.
Our recommendation for Semrush, set by its least reassuring crawler (SemrushBot-OCOB, rated caution): Consider blocking based on your content strategy.
Semrush at a glance
| Operator | Semrush |
|---|---|
| Crawlers in directory | 3 |
| Categories | SEO Tools |
| SEO impact if blocked | 0/10 to 2/10 |
| Crawl behaviour | 2 rated safe, 1 rated caution |
| Website | www.semrush.com |
| Official crawler docs | Semrush crawler documentation |
Crawlers operated by Semrush
What Semrush's crawlers are for
Semrush runs 3 crawlers in our database. Grouped by the job each one does:
- a utility crawler:
SemrushBot-SWA,SemrushBot,SemrushBot-OCOB
The highest-priority token is Tier 4. That makes Semrush a lower-priority operator for most sites: worth a deliberate decision, but rarely the one that decides your visibility.
How site owners treat Semrush
How often site owners write rules for Semrush's tokens, from our own robots.txt study of the Tranco top 10,000:
| Token | Files naming it | Files blocking it |
|---|---|---|
SemrushBot | 262 (4.7%) | 409 (7.3%) |
A Semrush token can be blocked without being named, through a User-agent: * group with Disallow: /, so its blocking share can be higher than its naming share.
All counts come from 5,583 valid files fetched October 20, 2026.
Writing one policy for Semrush
A robots.txt file cannot say "Semrush"; it can only name Semrush's tokens, one group at a time. So "block Semrush" really means naming each of its 3 tokens, and you can give them different answers.
# Allow Semrush search and assistants, refuse training
User-agent: SemrushBot-SWA
Disallow: /
User-agent: SemrushBot-SWA
User-agent: SemrushBot
User-agent: SemrushBot-OCOB
Allow: /- Decide per job, not per company. Training, search and user-triggered fetching have different trade-offs, even from the same operator.
- Name every token you mean. A token you leave out follows
User-agent: *. - Repeat shared rules. A named group ignores the
*group, so copy anyDisallowlines you still want it to obey. - Verify from outside. Run the AI crawler checker after any change to see the result for each Semrush token, including firewall refusals.
Test yourself: does User-agent: Semrush block all of Semrush's crawlers?
No. robots.txt matches crawler tokens, not company names. Name each token: SemrushBot-SWA, SemrushBot, SemrushBot-OCOB.
Test yourself: if you block one Semrush token, are the others blocked too?
No. Each token is read separately. Blocking one leaves the rest following whatever group applies to them.
Each Semrush crawler, explained
A one-paragraph brief for every Semrush token in our database, covering what it does, how it treats robots.txt and how often large sites write rules for it:
- Semrush Writing Assistant (
SemrushBot-SWA, Tier 4): a utility crawler that checks pages for an SEO, uptime or analytics service. Rated safe. Semrush Writing Assistant's operator does not clearly document how it treats robots.txt. Write the Semrush Writing Assistant rule anyway, then confirm in your logs that its requests stop. Not tracked in our robots.txt study. - SemrushBot (
SemrushBot, Tier 4): a utility crawler that checks pages for an SEO, uptime or analytics service. Rated safe. SemrushBot's operator does not clearly document how it treats robots.txt. Write the SemrushBot rule anyway, then confirm in your logs that its requests stop. 4.7% of top-site robots.txt files name it and 7.3% block it. - SemrushBot-OCOB (
SemrushBot-OCOB, Tier 4): a utility crawler that checks pages for an SEO, uptime or analytics service. Rated caution. SemrushBot-OCOB's operator does not clearly document how it treats robots.txt. Write the SemrushBot-OCOB rule anyway, then confirm in your logs that its requests stop. Not tracked in our robots.txt study.
Common mistakes with Semrush's crawlers
The errors we see most often with Semrush's crawlers in the files and firewalls we test:
- Treating Semrush's crawlers as an AI or search crawler. Blocking it has little effect on visibility; decide based on load and what it does with your data.
- Relying on robots.txt alone when Semrush's crawlers is rated caution or aggressive. Back the rule with a server-side block.
- Blocking Semrush's crawlers by IP alone. Ranges change; match the token and verify with logs.
All three hide inside a file that looks correct, so check Semrush's crawlers from outside after every change.
Spotting Semrush in your logs
robots.txt says what Semrush Writing Assistant and SemrushBot and SemrushBot-OCOB may do; your access logs show what they actually did.
Count requests
On a server with standard Nginx or Apache logs, this counts matching requests per day (adjust the log path for your host):
grep -iE "SemrushBot-SWA|SemrushBot|SemrushBot-OCOB" /var/log/nginx/access.log \
| awk '{print $4}' | cut -d: -f1 | sort | uniq -cAnd this shows which paths it requested most, which is the quickest way to see whether your Disallow lines are being respected:
grep -iE "SemrushBot-SWA|SemrushBot|SemrushBot-OCOB" /var/log/nginx/access.log \
| awk '{print $7}' | sort | uniq -c | sort -rn | head -20Check the requests are genuine
Anyone can send these user-agents, so confirm the source before acting on a match:
SemrushBot-SWA: we hold no published IP list, so use reverse DNS where Semrush documents a hostname and treat an unverifiable Semrush Writing Assistant request with caution. Check it separately from SemrushBot and SemrushBot-OCOB.SemrushBot: we hold no published IP list, so use reverse DNS where Semrush documents a hostname and treat an unverifiable SemrushBot request with caution. Check it separately from Semrush Writing Assistant and SemrushBot-OCOB.SemrushBot-OCOB: we hold no published IP list, so use reverse DNS where Semrush documents a hostname and treat an unverifiable SemrushBot-OCOB request with caution. Check it separately from Semrush Writing Assistant and SemrushBot.
What our own data shows about Semrush
Everything below comes from the AI Crawler Check bot database and our two most recent robots.txt scans of the Tranco top 10,000, not from Semrush's marketing. It covers 3 Semrush tokens in turn.
SemrushBot-SWA in our records
We first recorded SemrushBot-SWA on September 22, 2026. Semrush documents it on semrush.com, and we recheck that page before each release. No file in our top-10,000 scans names SemrushBot-SWA, so any rule you write for it puts you ahead of almost every large site.
Among the 19 SEO tools in our database, SemrushBot-SWA sits 16th by priority (Tier 4), level with AhrefsBot, AhrefsSiteAudit, AwarioBot, AwarioRssBot. 16 of the 19 share its safe rating. Other operators with a crawler that checks pages for an SEO, uptime or analytics service: OAI-AdsBot (OpenAI, Tier 2), AdIdxBot (Microsoft, Tier 3), AdsBot-Google (Google, Tier 3), AdsBot-Google-Mobile (Google, Tier 3), Baiduspider-ads (Baidu, Tier 3). A policy for SemrushBot-SWA is usually a policy for these too.
A decision record for SemrushBot-SWA
Teams that review crawlers once a quarter keep one short entry per token, so anyone can see why a rule exists and when it was last checked. Filled in from our data, the SemrushBot-SWA entry reads:
- Operator and job: Semrush; it checks pages for an SEO, uptime or analytics service.
- Priority and behaviour: Tier 4, rated safe, blocking impact not documented.
- Suggested starting point: Allow by default; revisit if it costs you bandwidth.
- Enforcement: SemrushBot-SWA's operator does not clearly document how it treats robots.txt. Write the SemrushBot-SWA rule anyway, then confirm in your logs that its requests stop.
User-agent: SemrushBot-SWA
Allow: /Paste the line into the validator with one of your own URLs before it goes live. Revisit the SemrushBot-SWA entry once a quarter, or when Semrush publishes a new token next to it.
SemrushBot in our records
We first recorded SemrushBot on March 17, 2026. Semrush documents it on semrush.com, and we recheck that page before each release. Between our scans of October 17, 2026 and October 20, 2026, files naming SemrushBot went from 252 to 262 (+10) and files blocking it from 386 to 409 (+23), so more sites wrote a rule for SemrushBot and more of those rules block it.
Among the 19 SEO tools in our database, SemrushBot sits 14th by priority (Tier 4), level with AhrefsBot, AhrefsSiteAudit, AwarioBot, AwarioRssBot. 16 of the 19 share its safe rating. Other operators with a crawler that checks pages for an SEO, uptime or analytics service: OAI-AdsBot (OpenAI, Tier 2), AdIdxBot (Microsoft, Tier 3), AdsBot-Google (Google, Tier 3), AdsBot-Google-Mobile (Google, Tier 3), Baiduspider-ads (Baidu, Tier 3). A policy for SemrushBot is usually a policy for these too.
SemrushBot-OCOB in our records
We first recorded SemrushBot-OCOB on March 17, 2026. Semrush documents it on semrush.com, and we recheck that page before each release. No file in our top-10,000 scans names SemrushBot-OCOB, so any rule you write for it puts you ahead of almost every large site.
Among the 19 SEO tools in our database, SemrushBot-OCOB sits 15th by priority (Tier 4), level with AhrefsBot, AhrefsSiteAudit, AwarioBot, AwarioRssBot. 3 of the 19 share its caution rating. Other operators with a crawler that checks pages for an SEO, uptime or analytics service: OAI-AdsBot (OpenAI, Tier 2), AdIdxBot (Microsoft, Tier 3), AdsBot-Google (Google, Tier 3), AdsBot-Google-Mobile (Google, Tier 3), Baiduspider-ads (Baidu, Tier 3). A policy for SemrushBot-OCOB is usually a policy for these too.
How top sites treat Semrush's crawlers
From our robots.txt study of the Tranco top 10,000 (5,583 valid files, fetched October 20, 2026):
SemrushBot, the only one tracked among the Semrush's crawlers tokens: blocked in 7.3% of files (190 with its own group, 219 throughUser-agent: *), more often by the top 1,000 (10.2%) than the long tail (7.2%).
For scale, 20.1% of all files block at least one AI training crawler. None of the tokens for Semrush's crawlers comes close to that rate, so blocking them is still a minority choice.
More questions about Semrush
- Will blocking Semrush affect my Google rankings?
- Only if one of its tokens is a Google search crawler. None of Semrush's tokens are Googlebot, so classic Google rankings are unaffected.
- How quickly does a robots.txt change take effect for Semrush?
- Crawlers cache robots.txt, commonly for up to a day. Expect a change to apply within about 24 hours, and confirm it in your logs.
- Can I allow Semrush on some sections and block it on others?
- Yes. Inside the group for
SemrushBot-SWA, addDisallowlines for the paths to keep private andAllowlines for exceptions. The longest matching rule wins. - Does blocking Semrush delete content it already collected?
- No. robots.txt only controls future visits. For content already collected, check whether Semrush offers a removal or opt-out process.
- Is a firewall block better than robots.txt for Semrush?
- A firewall enforces the decision, while robots.txt asks politely. Many operators recommend robots.txt because blocking their IPs can stop them reading your rules at all. Use robots.txt first, and add a firewall rule only if requests continue.
A five-minute Semrush review
- Confirm the token. Compare the user-agent in your logs with the tokens listed on this page. A close spelling is not the same crawler.
- Decide the outcome per crawler. Use the briefs above to choose allow, limit or block for each token.
- Write the group. Add or edit one
User-agentgroup per decision and repeat any shared rules inside it. - Publish and wait. Allow about a day for crawlers to fetch the new file.
- Confirm the result. Check your logs, then run the AI visibility checker to see the outcome from outside, including any firewall layer.
How to block or allow Semrush in robots.txt
These rules cover all 3 crawlers Semrush operates. Add them to the robots.txt file at the root of your domain.
Block Semrush completely
User-agent: SemrushBot-SWA
User-agent: SemrushBot
User-agent: SemrushBot-OCOB
Disallow: /
Allow Semrush explicitly
User-agent: SemrushBot-SWA
User-agent: SemrushBot
User-agent: SemrushBot-OCOB
Allow: /
These Semrush rules ask; they do not enforce. If a Semrush token ignores them, block it at the server or CDN. Paths are case-sensitive, and each rule matches the token, not the full user-agent string.
Frequently asked questions about Semrush
What crawlers does Semrush run?
Semrush operates 3 crawlers in our directory: Semrush Writing Assistant (user-agent SemrushBot-SWA), SemrushBot (user-agent SemrushBot), SemrushBot-OCOB (user-agent SemrushBot-OCOB).
Should I block Semrush?
Consider blocking based on your content strategy. That guidance follows this operator's least reassuring crawler, SemrushBot-OCOB, rated caution, because allowing an operator allows every crawler it runs. The SEO impact score for blocking is 0/10 to 2/10, where 10 means blocking costs you the most visibility.
How do I block Semrush in robots.txt?
Add the following to robots.txt at your domain root: User-agent: SemrushBot-SWA then User-agent: SemrushBot then User-agent: SemrushBot-OCOB then Disallow: /. One Disallow line covers all 3 user-agent tokens.
Does blocking Semrush hurt my search rankings?
2 of these share the highest blocking impact here, rated: Varies - Evaluate before blocking. Google's own ranking crawler is Googlebot, which this operator does not run, so blocking Semrush does not by itself remove you from Google's index. What it can change is whether AI assistants and analytics tools can read your pages.