ia_archiver
Operated by Internet Archive
Quick Facts
- User-Agent:
- ia_archiver
- Category:
- Other Agents
- Operator:
- Internet Archive
- Safety:
- Use Caution
- Blocking Impact:
- Low - No SEO ranking impact
- SEO Impact Score:
- 2/10
- Official docs:
- None published by Internet Archive
- Named by:
- Internet Archive help centre
Reported, not operator documented
Internet Archive has not published a crawler page that names ia_archiver. It is listed here because an independent primary source does: Internet Archive help centre. Legacy Alexa / Internet Archive crawler token, still referenced by the Internet Archive help centre for exclusion rules. Archive.org documents archive.org_bot as its current crawler.
What that means for you: the ia_archiver rule on this page is the best available spelling, but Internet Archive has made no statement of purpose, crawl rate or robots.txt compliance. Treat a ia_archiver request in your logs as a clue, not proof of who sent it, and add a firewall rule if that traffic matters to you. If the operator publishes documentation, this profile moves to Official docs.
What is ia_archiver?
ia_archiver is a developer-facing extraction agent operated by Internet Archive. It fetches and structures web content on behalf of applications and AI agents built on Internet Archive's platform. Traffic volume depends on how third-party developers configure it, so monitor your logs if you see heavy activity.
ia_archiver is a developer-facing extraction agent operated by Internet Archive. It fetches and structures web content on behalf of applications and AI agents built on Internet Archive's platform. Traffic volume depends on how third-party developers configure it, so monitor your logs if you see heavy activity.
ia_archiver uses the user-agent token ia_archiver. You can control it via robots.txt, meta tags (noai), or the emerging llms.txt standard. Robots.txt is voluntary; for hard enforcement, combine it with server-level IP blocking.
What happens if you block ia_archiver?
How to block ia_archiver with robots.txt
<code>User-agent: ia_archiver</code> - Matching is case-insensitive. Robots.txt is fetched from the root of each subdomain separately.
Is ia_archiver safe to allow?
What does ia_archiver do?
Understanding ia_archiver's purpose helps you decide whether to allow or block it.
- Extracts and structures web content for developer applications
- Powers AI agents built on the operator platform
- Volume depends on third-party developer configuration
- Monitor logs if you notice heavy activity
- Can be rate-limited via robots.txt or server rules
ia_archiver in context
ia_archiver is a utility crawler operated by Internet Archive. In plain terms, it harvests pages for a dataset or data product. Our database files it under other agents and ranks it Tier 6, the lowest priority: scrapers and tools that offer most sites little or nothing. 4 of the 4 other agents in our database share this tier.
Its safety rating is caution and its blocking impact is not documented. So the decision about ia_archiver rests on what you want from Internet Archive, not on its behaviour. How we rate safety and impact.
How site owners treat ia_archiver
In our own scan of the Tranco top 10,000 (5,583 valid robots.txt files), 262 files (4.7%) name ia_archiver in a User-agent line. That shows site owners treat the token as a real crawler worth a rule. It does not, on its own, prove who operates it.
Does it follow robots.txt?
ia_archiver's operator does not clearly document how it treats robots.txt. Write the ia_archiver rule anyway, then confirm in your logs that its requests stop.
Internet Archive crawlers and close alternatives
ia_archiver is one of 2 Internet Archive crawlers in our database. A User-agent: ia_archiver group controls ia_archiver only; the other Internet Archive tokens below need their own lines.
ia_archiver: this crawler (Tier 6)archive.org_bot: archive.org_bot, harvests pages for a dataset or data product (Tier 6)
To give ia_archiver the same answer as its siblings, stack User-agent: ia_archiver with their lines above one set of rules; to treat ia_archiver differently, give it its own group.
Other other agents that do the same job (harvests pages for a dataset or data product) include Arquivo-web-crawler from Arquivo.pt, TurnitinBot from Turnitin. Deciding about ia_archiver is usually a decision about the whole group, so check how your file treats these too.
We first recorded ia_archiver on March 17, 2026. We could not find a page where Internet Archive names it, so treat the token as reported rather than documented.
Related: how AI engines choose their sources
Which rule ia_archiver actually follows
ia_archiver reads exactly one group in robots.txt, the most specific one that names ia_archiver, and never combines rules across groups. This short file shows the trap:
User-agent: *
Disallow: /private/
User-agent: ia_archiver
Disallow: /drafts/- Find the group. ia_archiver looks for a group naming
ia_archiver. One exists, so it ignoresUser-agent: *completely. - Apply only that group. It is refused
/drafts/and nothing else. - The surprise.
/private/is open to ia_archiver, because that rule lives in the*group it no longer reads. To keep it out of both, repeatDisallow: /private/inside its own group.
Test yourself: does a rule under User-agent: * apply to ia_archiver if the file also has a User-agent: ia_archiver group?
No. Once a group names it, ia_archiver reads only that group. Put every rule you want it to follow inside its own group.
Test yourself: will blocking ia_archiver remove your pages from AI answers?
Often, yes. ia_archiver is a utility crawler, so pages it cannot fetch can drop out of the answers or results it supports.
Test yourself: is User-agent: ia_archiver the same as User-agent: ia_archiver?
Yes. RFC 9309 makes user-agent matching case-insensitive. Paths are a different story: Disallow: /Blog/ does not block /blog/.
Should you allow ia_archiver? Decide by situation
| If you are... | Recommended setting |
|---|---|
| You publish content and want AI answers to cite it | Low stakes either way. It does not decide whether AI answers cite you. |
| You license content and want to stop AI training | Blocking it does not stop training, because ia_archiver is a utility crawler. Block the operator's training crawler instead. |
| You run an online shop | Allow it unless your logs show it is costly. |
| Your server is under load from bots | Rate-limit before blocking. It is rated caution in our database. |
Check ia_archiver on your own site
- Open
/robots.txton your domain and search foria_archiver. If it is not named, it followsUser-agent: *. - List the
Disallowlines in the group ia_archiver follows and test your key URLs against them; the longest match wins, andAllowwins a tie. - Look beyond the file. A firewall or bot manager can refuse ia_archiver even when robots.txt allows it, and that never shows in the file.
- Run our free AI crawler check to see how ia_archiver and every other crawler we track are treated in one pass, including firewall refusals.
Common mistakes with ia_archiver
The errors we see most often with ia_archiver in the files and firewalls we test:
- Treating ia_archiver as an AI or search crawler. Blocking it has little effect on visibility; decide based on load and what it does with your data.
- Relying on robots.txt alone when ia_archiver is rated caution or aggressive. Back the rule with a server-side block.
- Blocking ia_archiver by IP alone. Ranges change; match the token and verify with logs.
All three hide inside a file that looks correct, so check ia_archiver from outside after every change.
Spotting ia_archiver in your logs
robots.txt says what ia_archiver may do; your access logs show what it actually did.
Count requests
On a server with standard Nginx or Apache logs, this counts matching requests per day (adjust the log path for your host):
grep -iE "ia_archiver" /var/log/nginx/access.log \
| awk '{print $4}' | cut -d: -f1 | sort | uniq -cAnd this shows which paths it requested most, which is the quickest way to see whether your Disallow lines are being respected:
grep -iE "ia_archiver" /var/log/nginx/access.log \
| awk '{print $7}' | sort | uniq -c | sort -rn | head -20Check the requests are genuine
Anyone can send the ia_archiver user-agent, so confirm the source before acting on a match:
ia_archiver: we hold no published IP list, so use reverse DNS where the operator documents a hostname and treat an unverifiable ia_archiver request with caution.
Frequently Asked Questions
What user-agent string does ia_archiver use?
ia_archiver. Evidence that the spelling is in real use comes from Internet Archive help centre, not from Internet Archive, which has not published crawler documentation naming it. Use it in robots.txt as the best available match, and treat a request carrying it as a clue rather than proof of who sent it.Is ia_archiver safe?
Will blocking ia_archiver hurt my SEO?
How do I block ia_archiver in robots.txt?
/robots.txt file:User-agent: ia_archiver Disallow: /
This instructs ia_archiver not to crawl any path on your site. To block only specific sections, replace / with the path (e.g.,
Disallow: /blog/).Does ia_archiver respect robots.txt?
ia_archiver will stop a client that chooses to honour it; for enforcement, add a firewall or CDN rule as well.How do I verify if ia_archiver is crawling my site?
ia_archiver (for example grep -i "ia_archiver" /var/log/nginx/access.log). Because the operator publishes no IP ranges or reverse-DNS pattern, a match shows that something used the name, not that Internet Archive sent it.