Safe Official docs Other Agents

Arquivo.pt Web Archive

Q: Is Arquivo.pt Web Archive safe?

Yes, Arquivo.pt Web Archive is a safe and legitimate crawler operated by Arquivo.pt. It follows the Robots Exclusion Protocol (RFC 9309) and can be controlled with standard robots.txt rules.

Q: Will blocking Arquivo.pt Web Archive hurt my SEO?

✅ No SEO Impact - Blocking Arquivo.pt Web Archive does not affect your rankings in Google, Bing, or any other search engine. Arquivo.pt Web Archive is an AI crawler, not a traditional search indexer. You can freely block it via User-agent / Disallow: / without any SEO penalty.

Q: How do I block Arquivo.pt Web Archive in robots.txt?

Add the following lines to your /robots.txt file: User-agent: Arquivo-web-crawler Disallow: / This instructs Arquivo.pt Web Archive not to crawl any path on your site. To block only specific sections, replace / with the path (e.g., Disallow: /blog/ ).

Q: How do I verify if Arquivo.pt Web Archive is crawling my site?

Search your web server access logs for the string Arquivo-web-crawler (case-insensitive: grep -i "Arquivo-web-crawler" /var/log/nginx/access.log ). Filter by user-agent in your log analytics tool (GoAccess, AWStats, etc.).

Operated by Arquivo.pt

Scout's quick take

Arquivo-web-crawler (Arquivo.pt Web Archive) collects content for datasets or resale. If you do not want that, block it, and back the rule with a firewall in case it ignores robots.txt.

User-agent: Arquivo-web-crawler
Disallow: /

Check AI bot crawlability Build a full robots.txt

Quick Facts

User-Agent:: Arquivo-web-crawler
Category:: Other Agents
Operator:: Arquivo.pt
Safety:: Safe
Blocking Impact:: Low - No SEO ranking impact
SEO Impact Score:: 2/10
Official docs:: arquivo.pt

Confirmed by the operator

Arquivo.pt names Arquivo-web-crawler on its own documentation page, linked above. The token, purpose and robots.txt behaviour on this profile come from that page, re-checked on October 19, 2026.

What is Arquivo.pt Web Archive?

Arquivo.pt Web Archive is a developer-facing extraction agent operated by Arquivo.pt. It fetches and structures web content on behalf of applications and AI agents built on Arquivo.pt's platform. Traffic volume depends on how third-party developers configure it, so monitor your logs if you see heavy activity.
Arquivo.pt Web Archive uses the user-agent token Arquivo-web-crawler. You can control it via robots.txt, meta tags (noai), or the emerging llms.txt standard. Robots.txt is voluntary; for hard enforcement, combine it with server-level IP blocking.

What happens if you block Arquivo.pt Web Archive?

✅ No SEO Impact - Blocking Arquivo.pt Web Archive does not affect your rankings in Google, Bing, or any other search engine. Arquivo.pt Web Archive is an AI crawler, not a traditional search indexer. You can freely block it via User-agent / Disallow: / without any SEO penalty.

Generally safe to allow; provides legitimate crawling value.

How to block Arquivo.pt Web Archive with robots.txt

<code>User-agent: Arquivo-web-crawler</code> - Matching is case-insensitive. Robots.txt is fetched from the root of each subdomain separately.

Block completely (robots.txt)

User-agent: Arquivo-web-crawler
Disallow: /

Allow all (robots.txt)

User-agent: Arquivo-web-crawler
Allow: /

Block private only (robots.txt)

User-agent: Arquivo-web-crawler
Disallow: /private/
Disallow: /api/
Disallow: /admin/
Allow: /

Nginx server block

# Nginx: Hard-block Arquivo.pt Web Archive if ($http_user_agent ~* "Arquivo-web-crawler") { return 403 "Bot blocked"; }

Apache .htaccess

# Apache: Hard-block Arquivo.pt Web Archive
SetEnvIfNoCase User-Agent "Arquivo-web-crawler" bad_bot
Order Allow,Deny
Allow from all
Deny from env=bad_bot

Meta robots tag

X-Robots-Tag header

X-Robots-Tag: noindex, nofollow

Is Arquivo.pt Web Archive safe to allow?

Yes, Arquivo.pt Web Archive is a safe and legitimate crawler operated by Arquivo.pt. It follows the Robots Exclusion Protocol (RFC 9309) and can be controlled with standard robots.txt rules.

Verify by reverse-DNS lookup: legitimate Arquivo.pt Web Archive requests resolve to Arquivo.pt's domain.

What does Arquivo.pt Web Archive do?

Understanding Arquivo.pt Web Archive's purpose helps you decide whether to allow or block it.

Extracts and structures web content for developer applications
Powers AI agents built on the operator platform
Volume depends on third-party developer configuration
Monitor logs if you notice heavy activity
Can be rate-limited via robots.txt or server rules

Arquivo.pt Web Archive in context

Arquivo.pt Web Archive is a utility crawler operated by Arquivo.pt. In plain terms, it harvests pages for a dataset or data product. Our database files it under other agents and ranks it Tier 6, the lowest priority: scrapers and tools that offer most sites little or nothing. 4 of the 4 other agents in our database share this tier.

Its safety rating is safe and its blocking impact is low. So the decision about Arquivo.pt Web Archive rests on what you want from Arquivo.pt, not on its behaviour. How we rate safety and impact.

How site owners treat Arquivo.pt Web Archive

Our robots.txt study of the Tranco top 10,000 did not track Arquivo-web-crawler as a named token, so we do not publish an adoption figure for it. If you run a large site, your own logs are the best evidence of how often it visits.

Does it follow robots.txt?

Arquivo.pt Web Archive's operator does not clearly document how it treats robots.txt. Write the Arquivo.pt Web Archive rule anyway, then confirm in your logs that its requests stop.

Arquivo.pt crawlers and close alternatives

Arquivo.pt runs only this one crawler in our database, so a single User-agent: Arquivo-web-crawler group controls everything Arquivo.pt fetches through it.

Other other agents that do the same job (harvests pages for a dataset or data product) include archive.org_bot from Internet Archive, ia_archiver from Internet Archive, TurnitinBot from Turnitin. Deciding about Arquivo.pt Web Archive is usually a decision about the whole group, so check how your file treats these too.

We first recorded Arquivo-web-crawler on March 17, 2026. Arquivo.pt documents it on arquivo.pt, and we recheck that page before each release.

Which rule Arquivo.pt Web Archive actually follows

Arquivo.pt Web Archive reads exactly one group in robots.txt, the most specific one that names Arquivo-web-crawler, and never combines rules across groups. This short file shows the trap:

User-agent: *
Disallow: /private/

User-agent: Arquivo-web-crawler
Disallow: /drafts/

Find the group. Arquivo.pt Web Archive looks for a group naming Arquivo-web-crawler. One exists, so it ignores User-agent: * completely.
Apply only that group. It is refused /drafts/ and nothing else.
The surprise. /private/ is open to Arquivo.pt Web Archive, because that rule lives in the * group it no longer reads. To keep it out of both, repeat Disallow: /private/ inside its own group.

Test yourself: does a rule under User-agent: * apply to Arquivo.pt Web Archive if the file also has a User-agent: Arquivo-web-crawler group?

No. Once a group names it, Arquivo.pt Web Archive reads only that group. Put every rule you want it to follow inside its own group.

Test yourself: will blocking Arquivo.pt Web Archive remove your pages from AI answers?

Often, yes. Arquivo.pt Web Archive is a utility crawler, so pages it cannot fetch can drop out of the answers or results it supports.

Test yourself: is User-agent: arquivo-web-crawler the same as User-agent: Arquivo-web-crawler?

Yes. RFC 9309 makes user-agent matching case-insensitive. Paths are a different story: Disallow: /Blog/ does not block /blog/.

Should you allow Arquivo.pt Web Archive? Decide by situation

Recommended setting for Arquivo.pt Web Archive by type of site
If you are...	Recommended setting
You publish content and want AI answers to cite it	Low stakes either way. It does not decide whether AI answers cite you.
You license content and want to stop AI training	Blocking it does not stop training, because Arquivo.pt Web Archive is a utility crawler. Block the operator's training crawler instead.
You run an online shop	Allow it unless your logs show it is costly.
Your server is under load from bots	Rate-limit before blocking. It is rated safe in our database.

Check Arquivo.pt Web Archive on your own site

Open /robots.txt on your domain and search for Arquivo-web-crawler. If it is not named, it follows User-agent: *.
List the Disallow lines in the group Arquivo.pt Web Archive follows and test your key URLs against them; the longest match wins, and Allow wins a tie.
Look beyond the file. A firewall or bot manager can refuse Arquivo.pt Web Archive even when robots.txt allows it, and that never shows in the file.
Run our blocked AI crawler check to see how Arquivo.pt Web Archive and every other crawler we track are treated in one pass, including firewall refusals.

Common mistakes with Arquivo.pt Web Archive

The errors we see most often with Arquivo.pt Web Archive in the files and firewalls we test:

Treating Arquivo.pt Web Archive as an AI or search crawler. Blocking it has little effect on visibility; decide based on load and what it does with your data.
Relying on robots.txt alone when Arquivo.pt Web Archive is rated caution or aggressive. Back the rule with a server-side block.
Blocking Arquivo.pt Web Archive by IP alone. Ranges change; match the token and verify with logs.

All three hide inside a file that looks correct, so check Arquivo.pt Web Archive from outside after every change.

Spotting Arquivo.pt Web Archive in your logs

robots.txt says what Arquivo.pt Web Archive may do; your access logs show what it actually did.

Count requests

On a server with standard Nginx or Apache logs, this counts matching requests per day (adjust the log path for your host):

grep -iE "Arquivo-web-crawler" /var/log/nginx/access.log \
  | awk '{print $4}' | cut -d: -f1 | sort | uniq -c

And this shows which paths it requested most, which is the quickest way to see whether your Disallow lines are being respected:

grep -iE "Arquivo-web-crawler" /var/log/nginx/access.log \
  | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

Check the requests are genuine

Anyone can send the Arquivo.pt Web Archive user-agent, so confirm the source before acting on a match:

Arquivo-web-crawler: we hold no published IP list, so use reverse DNS where the operator documents a hostname and treat an unverifiable Arquivo.pt Web Archive request with caution.

Frequently Asked Questions

What is the official user-agent string for Arquivo.pt Web Archive?

The official user-agent string for Arquivo.pt Web Archive is: Arquivo-web-crawler. Use this exact string in robots.txt, Nginx, Apache, or Cloudflare firewall rules to target this bot. Matching in robots.txt is case-insensitive. Verify a request genuinely comes from Arquivo.pt Web Archive by performing a reverse-DNS lookup on the source IP.

Is Arquivo.pt Web Archive safe?

Yes, Arquivo.pt Web Archive is a safe and legitimate crawler operated by Arquivo.pt. It follows the Robots Exclusion Protocol (RFC 9309) and can be controlled with standard robots.txt rules.

Will blocking Arquivo.pt Web Archive hurt my SEO?

How do I block Arquivo.pt Web Archive in robots.txt?

Add the following lines to your /robots.txt file:

User-agent: Arquivo-web-crawler
Disallow: /

This instructs Arquivo.pt Web Archive not to crawl any path on your site. To block only specific sections, replace / with the path (e.g., Disallow: /blog/).

Does Arquivo.pt Web Archive respect robots.txt?

Arquivo.pt Web Archive is operated by Arquivo.pt and is expected to fetch and parse /robots.txt before crawling, following RFC 9309. For hard enforcement, combine robots.txt with server-level IP or user-agent blocking.

How do I verify if Arquivo.pt Web Archive is crawling my site?

Search your web server access logs for the string Arquivo-web-crawler (case-insensitive: grep -i "Arquivo-web-crawler" /var/log/nginx/access.log). Filter by user-agent in your log analytics tool (GoAccess, AWStats, etc.).