Ahrefs Site Audit
SafeUser-agent: AhrefsSiteAudit
Category: SEO Tools
SEO impact if blocked: 2/10
Ahrefs
2 crawlers operated by Ahrefs
https://ahrefs.com/Who Ahrefs is, for site owners
Ahrefs operates 2 crawlers in our directory, filed under SEO Tools. Of those 2, 2 rated safe for crawl behaviour.
Blocking Ahrefs carries an SEO impact score of 0/10 to 2/10 across its crawlers. Every crawler here carries the same blocking impact: Low - No SEO ranking impact.
Our recommendation for Ahrefs, set by its least reassuring crawler (Ahrefs Site Audit, rated safe): Generally safe to allow; provides legitimate crawling value.
Ahrefs at a glance
| Operator | Ahrefs |
|---|---|
| Crawlers in directory | 2 |
| Categories | SEO Tools |
| SEO impact if blocked | 0/10 to 2/10 |
| Crawl behaviour | 2 rated safe |
| Website | ahrefs.com |
| Official crawler docs | Ahrefs crawler documentation |
Crawlers operated by Ahrefs
AhrefsBot
SafeUser-agent:
AhrefsBotCategory: SEO Tools
SEO impact if blocked: 0/10
What Ahrefs's crawlers are for
Ahrefs runs 2 crawlers in our database. Grouped by the job each one does:
- a utility crawler:
AhrefsSiteAudit,AhrefsBot
The highest-priority token is Tier 4. That makes Ahrefs a lower-priority operator for most sites: worth a deliberate decision, but rarely the one that decides your visibility.
How site owners treat Ahrefs
How often site owners write rules for Ahrefs's tokens, from our own robots.txt study of the Tranco top 10,000:
| Token | Files naming it | Files blocking it |
|---|---|---|
AhrefsBot | 347 (6.2%) | 456 (8.2%) |
A Ahrefs token can be blocked without being named, through a User-agent: * group with Disallow: /, so its blocking share can be higher than its naming share.
All counts come from 5,583 valid files fetched October 20, 2026.
Writing one policy for Ahrefs
A robots.txt file cannot say "Ahrefs"; it can only name Ahrefs's tokens, one group at a time. So "block Ahrefs" really means naming each of its 2 tokens, and you can give them different answers.
# Allow Ahrefs search and assistants, refuse training
User-agent: AhrefsSiteAudit
Disallow: /
User-agent: AhrefsSiteAudit
User-agent: AhrefsBot
Allow: /- Decide per job, not per company. Training, search and user-triggered fetching have different trade-offs, even from the same operator.
- Name every token you mean. A token you leave out follows
User-agent: *. - Repeat shared rules. A named group ignores the
*group, so copy anyDisallowlines you still want it to obey. - Verify from outside. Run the AI crawlability test after any change to see the result for each Ahrefs token, including firewall refusals.
Test yourself: does User-agent: Ahrefs block all of Ahrefs's crawlers?
No. robots.txt matches crawler tokens, not company names. Name each token: AhrefsSiteAudit, AhrefsBot.
Test yourself: if you block one Ahrefs token, are the others blocked too?
No. Each token is read separately. Blocking one leaves the rest following whatever group applies to them.
Each Ahrefs crawler, explained
A one-paragraph brief for every Ahrefs token in our database, covering what it does, how it treats robots.txt and how often large sites write rules for it:
- Ahrefs Site Audit (
AhrefsSiteAudit, Tier 4): a utility crawler that checks pages for an SEO, uptime or analytics service. Rated safe. Ahrefs Site Audit's operator does not clearly document how it treats robots.txt. Write the Ahrefs Site Audit rule anyway, then confirm in your logs that its requests stop. Not tracked in our robots.txt study. - AhrefsBot (
AhrefsBot, Tier 4): a utility crawler that checks pages for an SEO, uptime or analytics service. Rated safe. AhrefsBot's operator does not clearly document how it treats robots.txt. Write the AhrefsBot rule anyway, then confirm in your logs that its requests stop. 6.2% of top-site robots.txt files name it and 8.2% block it.
Common mistakes with Ahrefs's crawlers
The errors we see most often with Ahrefs's crawlers in the files and firewalls we test:
- Treating Ahrefs's crawlers as an AI or search crawler. Blocking it has little effect on visibility; decide based on load and what it does with your data.
- Relying on robots.txt alone when Ahrefs's crawlers is rated caution or aggressive. Back the rule with a server-side block.
- Blocking Ahrefs's crawlers by IP alone. Ranges change; match the token and verify with logs.
All three hide inside a file that looks correct, so check Ahrefs's crawlers from outside after every change.
Spotting Ahrefs in your logs
robots.txt says what Ahrefs Site Audit and AhrefsBot may do; your access logs show what they actually did.
Count requests
On a server with standard Nginx or Apache logs, this counts matching requests per day (adjust the log path for your host):
grep -iE "AhrefsSiteAudit|AhrefsBot" /var/log/nginx/access.log \
| awk '{print $4}' | cut -d: -f1 | sort | uniq -cAnd this shows which paths it requested most, which is the quickest way to see whether your Disallow lines are being respected:
grep -iE "AhrefsSiteAudit|AhrefsBot" /var/log/nginx/access.log \
| awk '{print $7}' | sort | uniq -c | sort -rn | head -20Check the requests are genuine
Anyone can send these user-agents, so confirm the source before acting on a match:
AhrefsSiteAudit: we hold no published IP list, so use reverse DNS where Ahrefs documents a hostname and treat an unverifiable Ahrefs Site Audit request with caution. Check it separately from AhrefsBot.AhrefsBot: we hold no published IP list, so use reverse DNS where Ahrefs documents a hostname and treat an unverifiable AhrefsBot request with caution. Check it separately from Ahrefs Site Audit.
What our own data shows about Ahrefs
Everything below comes from the AI Crawler Check bot database and our two most recent robots.txt scans of the Tranco top 10,000, not from Ahrefs's marketing. It covers 2 Ahrefs tokens in turn.
AhrefsSiteAudit in our records
We first recorded AhrefsSiteAudit on September 22, 2026. Ahrefs documents it on ahrefs.com, and we recheck that page before each release. No file in our top-10,000 scans names AhrefsSiteAudit, so any rule you write for it puts you ahead of almost every large site.
Among the 19 SEO tools in our database, AhrefsSiteAudit sits second by priority (Tier 4), level with AhrefsBot, AwarioBot, AwarioRssBot, AwarioSmartBot. 16 of the 19 share its safe rating. Other operators with a crawler that checks pages for an SEO, uptime or analytics service: OAI-AdsBot (OpenAI, Tier 2), AdIdxBot (Microsoft, Tier 3), AdsBot-Google (Google, Tier 3), AdsBot-Google-Mobile (Google, Tier 3), Baiduspider-ads (Baidu, Tier 3). A policy for AhrefsSiteAudit is usually a policy for these too.
A decision record for AhrefsSiteAudit
Teams that review crawlers once a quarter keep one short entry per token, so anyone can see why a rule exists and when it was last checked. Filled in from our data, the AhrefsSiteAudit entry reads:
- Operator and job: Ahrefs; it checks pages for an SEO, uptime or analytics service.
- Priority and behaviour: Tier 4, rated safe, blocking impact not documented.
- Suggested starting point: Allow by default; revisit if it costs you bandwidth.
- Enforcement: AhrefsSiteAudit's operator does not clearly document how it treats robots.txt. Write the AhrefsSiteAudit rule anyway, then confirm in your logs that its requests stop.
User-agent: AhrefsSiteAudit
Allow: /Paste the line into the validator with one of your own URLs before it goes live. Revisit the AhrefsSiteAudit entry once a quarter, or when Ahrefs publishes a new token next to it.
AhrefsBot in our records
We first recorded AhrefsBot on March 17, 2026. Ahrefs documents it on ahrefs.com, and we recheck that page before each release. Between our scans of October 17, 2026 and October 20, 2026, files naming AhrefsBot went from 336 to 347 (+11) and files blocking it from 436 to 456 (+20), so more sites wrote a rule for AhrefsBot and more of those rules block it.
Among the 19 SEO tools in our database, AhrefsBot sits first by priority (Tier 4), level with AhrefsSiteAudit, AwarioBot, AwarioRssBot, AwarioSmartBot. 16 of the 19 share its safe rating. Other operators with a crawler that checks pages for an SEO, uptime or analytics service: OAI-AdsBot (OpenAI, Tier 2), AdIdxBot (Microsoft, Tier 3), AdsBot-Google (Google, Tier 3), AdsBot-Google-Mobile (Google, Tier 3), Baiduspider-ads (Baidu, Tier 3). A policy for AhrefsBot is usually a policy for these too.
A decision record for AhrefsBot
Teams that review crawlers once a quarter keep one short entry per token, so anyone can see why a rule exists and when it was last checked. Filled in from our data, the AhrefsBot entry reads:
- Operator and job: Ahrefs; it checks pages for an SEO, uptime or analytics service.
- Priority and behaviour: Tier 4, rated safe, blocking impact not documented.
- Suggested starting point: Allow by default; revisit if it costs you bandwidth.
- Enforcement: AhrefsBot's operator does not clearly document how it treats robots.txt. Write the AhrefsBot rule anyway, then confirm in your logs that its requests stop.
User-agent: AhrefsBot
Allow: /Paste the line into the validator with one of your own URLs before it goes live. Revisit the AhrefsBot entry once a quarter, or when Ahrefs publishes a new token next to it.
How top sites treat Ahrefs's crawlers
From our robots.txt study of the Tranco top 10,000 (5,583 valid files, fetched October 20, 2026):
AhrefsBot, the only one tracked among the Ahrefs's crawlers tokens: blocked in 8.2% of files (229 with its own group, 227 throughUser-agent: *), more often by the top 1,000 (11.6%) than the long tail (7.8%).
For scale, 20.1% of all files block at least one AI training crawler. None of the tokens for Ahrefs's crawlers comes close to that rate, so blocking them is still a minority choice.
More questions about Ahrefs
- Will blocking Ahrefs affect my Google rankings?
- Only if one of its tokens is a Google search crawler. None of Ahrefs's tokens are Googlebot, so classic Google rankings are unaffected.
- How quickly does a robots.txt change take effect for Ahrefs?
- Crawlers cache robots.txt, commonly for up to a day. Expect a change to apply within about 24 hours, and confirm it in your logs.
- Can I allow Ahrefs on some sections and block it on others?
- Yes. Inside the group for
AhrefsSiteAudit, addDisallowlines for the paths to keep private andAllowlines for exceptions. The longest matching rule wins. - Does blocking Ahrefs delete content it already collected?
- No. robots.txt only controls future visits. For content already collected, check whether Ahrefs offers a removal or opt-out process.
- Is a firewall block better than robots.txt for Ahrefs?
- A firewall enforces the decision, while robots.txt asks politely. Many operators recommend robots.txt because blocking their IPs can stop them reading your rules at all. Use robots.txt first, and add a firewall rule only if requests continue.
A five-minute Ahrefs review
- Confirm the token. Compare the user-agent in your logs with the tokens listed on this page. A close spelling is not the same crawler.
- Decide the outcome per crawler. Use the briefs above to choose allow, limit or block for each token.
- Write the group. Add or edit one
User-agentgroup per decision and repeat any shared rules inside it. - Publish and wait. Allow about a day for crawlers to fetch the new file.
- Confirm the result. Check your logs, then run the AI visibility checker to see the outcome from outside, including any firewall layer.
How to block or allow Ahrefs in robots.txt
These rules cover all 2 crawlers Ahrefs operates. Add them to the robots.txt file at the root of your domain.
Block Ahrefs completely
User-agent: AhrefsSiteAudit
User-agent: AhrefsBot
Disallow: /
Allow Ahrefs explicitly
User-agent: AhrefsSiteAudit
User-agent: AhrefsBot
Allow: /
These Ahrefs rules ask; they do not enforce. If a Ahrefs token ignores them, block it at the server or CDN. Paths are case-sensitive, and each rule matches the token, not the full user-agent string.
Frequently asked questions about Ahrefs
What crawlers does Ahrefs run?
Ahrefs operates 2 crawlers in our directory: Ahrefs Site Audit (user-agent AhrefsSiteAudit), AhrefsBot (user-agent AhrefsBot).
Should I block Ahrefs?
Generally safe to allow; provides legitimate crawling value. That guidance follows this operator's least reassuring crawler, Ahrefs Site Audit, rated safe, because allowing an operator allows every crawler it runs. The SEO impact score for blocking is 0/10 to 2/10, where 10 means blocking costs you the most visibility.
How do I block Ahrefs in robots.txt?
Add the following to robots.txt at your domain root: User-agent: AhrefsSiteAudit then User-agent: AhrefsBot then Disallow: /. One Disallow line covers all 2 user-agent tokens.
Does blocking Ahrefs hurt my search rankings?
Every crawler here carries the same blocking impact: Low - No SEO ranking impact. Google's own ranking crawler is Googlebot, which this operator does not run, so blocking Ahrefs does not by itself remove you from Google's index. What it can change is whether AI assistants and analytics tools can read your pages.