archive.org_bot
CautionUser-agent: archive.org_bot
Operator: Internet Archive
Added:
Bot directoryCategory
Other Agents Directory
Miscellaneous crawlers not fitting other categories.
- 4tokens
- 2safe
- 2caution
- 0aggressive
All 4 Other Agents
Arquivo.pt Web Archive
SafeUser-agent:
Arquivo-web-crawlerOperator: Arquivo.pt
Added:
ia_archiver
CautionUser-agent:
ia_archiverOperator: Internet Archive
Added:
TurnitinBot
SafeUser-agent:
TurnitinBotOperator: Turnitin
Added:
No bots match those filters. Try a shorter search or browse every category. Missing a crawler? Tell us and we will research it.
The other agents category in numbers
Our database lists 4 other agents from 3 operators. 2 are rated safe, 2 caution and 0 aggressive. What they are for:
| Purpose | Crawlers |
|---|---|
| harvests pages for a dataset or data product | 4 |
Priority tiers
How the 4 other agents spread across our six priority tiers, where Tier 1 is what you would notice losing:
- Tier 6: 4 crawlers
None of the other agents reach Tier 1 or 2, so decisions in this category rarely decide your visibility. They are mostly about load, data use and privacy.
How to review other agents on your site
- List the ones that matter. Start with Tier 1 and 2 in the table above; the rest can usually follow a default.
- Choose a default. Decide what
User-agent: *should say, because every crawler you do not name follows it. - Name the exceptions. Give each crawler that needs a different answer its own group, and repeat any shared
Disallowlines inside it. - Check the edge. Firewalls and bot managers often treat these crawlers differently from robots.txt. Run the live AI crawler test to see both layers together.
Test yourself: a crawler in this category is not named anywhere in your robots.txt. Which rules does it follow?
The User-agent: * group. If there is no such group, it may crawl everything.
Test yourself: does a safety rating of "safe" mean you should allow a crawler?
Not necessarily. Safety describes behaviour. Whether to allow it depends on what it does with your content and what you lose by blocking it.
The other agents worth knowing first
Ranked by priority tier, these are the other agents with the biggest effect on a typical site:
archive.org_bot(Internet Archive, Tier 6): a utility crawler that harvests pages for a dataset or data product.Arquivo-web-crawler(Arquivo.pt, Tier 6): a utility crawler that harvests pages for a dataset or data product.ia_archiver(Internet Archive, Tier 6): a utility crawler that harvests pages for a dataset or data product.TurnitinBot(Turnitin, Tier 6): a utility crawler that harvests pages for a dataset or data product.
Of the 4 other agents we track, 0 are documented as following robots.txt, 0 as not following it for some requests, 0 with exceptions or disputed behaviour, and 4 with no clear statement from the operator. Where behaviour is unclear, back a robots.txt rule with a server-side one.
What to do with other agents, by type of site
| If you run... | Recommended approach |
|---|---|
| Site that uses this kind of service | Allow the crawlers for services you actually use; they need access to work. |
| Site that does not use the service | Blocking is low risk; it has little effect on search or AI visibility. |
| Site under heavy bot load | Rate-limit or block the noisiest crawlers first. |
| Privacy-sensitive site | Block crawlers whose data use you have not reviewed. |
Common mistakes with other agents
The errors we see most often with other agents in the files and firewalls we test:
- Treating other agents as an AI or search crawler. Blocking it has little effect on visibility; decide based on load and what it does with your data.
- Relying on robots.txt alone when other agents is rated caution or aggressive. Back the rule with a server-side block.
- Blocking other agents by IP alone. Ranges change; match the token and verify with logs.
All three hide inside a file that looks correct, so check other agents from outside after every change.
Spotting other agents in your logs
robots.txt says what archive.org_bot and Arquivo-web-crawler and ia_archiver and TurnitinBot may do; your access logs show what they actually did.
Count requests
On a server with standard Nginx or Apache logs, this counts matching requests per day (adjust the log path for your host):
grep -iE "archive\.org_bot|Arquivo-web-crawler|ia_archiver|TurnitinBot" /var/log/nginx/access.log \
| awk '{print $4}' | cut -d: -f1 | sort | uniq -cAnd this shows which paths it requested most, which is the quickest way to see whether your Disallow lines are being respected:
grep -iE "archive\.org_bot|Arquivo-web-crawler|ia_archiver|TurnitinBot" /var/log/nginx/access.log \
| awk '{print $7}' | sort | uniq -c | sort -rn | head -20Check the requests are genuine
Anyone can send these user-agents, so confirm the source before acting on a match:
archive.org_bot: we hold no published IP list, so use reverse DNS where other agents documents a hostname and treat an unverifiable archive.org_bot request with caution. Check it separately from Arquivo-web-crawler and ia_archiver and TurnitinBot.Arquivo-web-crawler: we hold no published IP list, so use reverse DNS where other agents documents a hostname and treat an unverifiable Arquivo-web-crawler request with caution. Check it separately from archive.org_bot and ia_archiver and TurnitinBot.ia_archiver: we hold no published IP list, so use reverse DNS where other agents documents a hostname and treat an unverifiable ia_archiver request with caution. Check it separately from archive.org_bot and Arquivo-web-crawler and TurnitinBot.TurnitinBot: we hold no published IP list, so use reverse DNS where other agents documents a hostname and treat an unverifiable TurnitinBot request with caution. Check it separately from archive.org_bot and Arquivo-web-crawler and ia_archiver.
Recent changes among other agents
The newest other agents in our database: TurnitinBot (March 17, 2026), ia_archiver (March 17, 2026), Arquivo-web-crawler (March 17, 2026), archive.org_bot (March 17, 2026). A User-agent: * default decides what each of these gets until you name it.
None of the 4 other agents is in our tracked token set yet, so we report first-seen dates only.
Each of the 4 other agents in one line
archive.org_botfrom Internet Archive: it harvests pages for a dataset or data product; Tier 6, rated caution, blocking impact not documented.Arquivo-web-crawlerfrom Arquivo.pt: it harvests pages for a dataset or data product; Tier 6, rated safe, blocking impact low.ia_archiverfrom Internet Archive: it harvests pages for a dataset or data product; Tier 6, rated caution, blocking impact not documented.TurnitinBotfrom Turnitin: it harvests pages for a dataset or data product; Tier 6, rated safe, blocking impact not documented.
Related: how AI engines choose their sources
With so few other agents, you can decide about every one of them by name rather than relying on a User-agent: * default.
How top sites treat other agents
From our robots.txt study of the Tranco top 10,000 (5,583 valid files, fetched October 20, 2026):
None of the tokens for other agents is tracked by name in our study. For scale, 20.1% of top-site files block at least one AI training crawler and 4.3% refuse every unnamed crawler with User-agent: * and Disallow: /, which would catch other agents too.
More questions about other agents
- How many other agents should I name in robots.txt?
- Only the ones that need a different answer from your default. Every crawler you do not name follows
User-agent: *, so a short file with a sensible default is easier to maintain than a long list. - Do new other agents appear often?
- Yes. New crawlers launch every month, which is why a deliberate
User-agent: *default matters: it decides what happens to crawlers you have never heard of. - Can I copy another site's rules for other agents?
- Use them as a pattern, not a template. Each file reflects that site's licensing and infrastructure. Copy the structure, then name the crawlers that matter to you.
- Where do I check how my site treats other agents?
- Run the crawlability checker for AI search. It reports every crawler in this category in one pass, including firewall refusals that robots.txt never shows.
FAQ
How many Other Agents are tracked?
We currently track 4 Other Agents with user-agent strings, safety ratings, and blocking rules.
Which Other Agents should I never block?
Never block bots with 'critical' impact rating - there are 0 in this category. Blocking them removes your site from their search results.
Are all Other Agents safe?
Out of 4 bots in this category: 2 safe, 2 caution, 0 aggressive.
How do I block all Other Agents at once?
Generate a combined robots.txt block rule for all Other Agents using our bulk robots.txt generator, or manually list each User-agent + Disallow: / directive.