Arquivo.pt

Common mistakes with Arquivo.pt's crawlers

The errors we see most often with Arquivo.pt's crawlers in the files and firewalls we test:

1. Treating Arquivo.pt's crawlers as an AI or search crawler. Blocking it has little effect on visibility; decide based on load and what it does with your data.
2. Relying on robots.txt alone when Arquivo.pt's crawlers is rated caution or aggressive. Back the rule with a server-side block.
3. Blocking Arquivo.pt's crawlers by IP alone. Ranges change; match the token and verify with logs.

All three hide inside a file that looks correct, so check Arquivo.pt's crawlers from outside after every change.

What our own data shows about Arquivo.pt

Everything below comes from the AI Crawler Check bot database and our two most recent robots.txt scans of the Tranco top 10,000, not from Arquivo.pt's marketing. Arquivo.pt has one token in our database, so the whole picture fits in one place.

Arquivo-web-crawler in our records

We first recorded Arquivo-web-crawler on March 17, 2026. Arquivo.pt documents it on arquivo.pt, and we recheck that page before each release. No file in our top-10,000 scans names Arquivo-web-crawler, so any rule you write for it puts you ahead of almost every large site.

Among the 4 other agents in our database, Arquivo-web-crawler sits second by priority (Tier 6), level with archive.org_bot, ia_archiver, TurnitinBot. 2 of the 4 share its safe rating. Other operators with a crawler that harvests pages for a dataset or data product: FirecrawlAgent (Firecrawl, Tier 2), iaskspider (iAsk, Tier 2), NovaAct (Amazon, Tier 2), CloudflareBrowserRenderingCrawler (Cloudflare, Tier 3), Amazon Kendra (Amazon, Tier 6). A policy for Arquivo-web-crawler is usually a policy for these too.

A decision record for Arquivo-web-crawler

Teams that review crawlers once a quarter keep one short entry per token, so anyone can see why a rule exists and when it was last checked. Filled in from our data, the Arquivo-web-crawler entry reads:

• Operator and job: Arquivo.pt; it harvests pages for a dataset or data product.
• Priority and behaviour: Tier 6, rated safe, blocking impact low.
• Suggested starting point: Allow by default; revisit if it costs you bandwidth.
• Enforcement: Arquivo-web-crawler's operator does not clearly document how it treats robots.txt. Write the Arquivo-web-crawler rule anyway, then confirm in your logs that its requests stop.

User-agent: Arquivo-web-crawler
Allow: /

Paste the line into the validator with one of your own URLs before it goes live. Revisit the Arquivo-web-crawler entry once a quarter, or when Arquivo.pt publishes a new token next to it.

Checking Arquivo-web-crawler on your own site

Because Arquivo.pt runs only Arquivo-web-crawler in our records, one check covers the whole operator. Search your access logs for the string Arquivo-web-crawler over the last 30 days, note how many requests it made and which paths it asked for, then compare that with what your robots.txt allows. Match the requests against what Arquivo.pt publishes about the crawler, since that page is the only authority on its behaviour. A Tier 6 crawler rated safe rarely needs more than a quarterly look, and our log checker does the counting for you.

How top sites treat Arquivo.pt's crawlers

From our robots.txt study of the Tranco top 10,000 (5,583 valid files, fetched October 20, 2026):

None of the tokens for Arquivo.pt's crawlers is tracked by name in our study. For scale, 20.1% of top-site files block at least one AI training crawler and 4.3% refuse every unnamed crawler with User-agent: * and Disallow: /, which would catch Arquivo.pt's crawlers too.

More questions about Arquivo.pt

Will blocking Arquivo.pt affect my Google rankings?

Only if one of its tokens is a Google search crawler. None of Arquivo.pt's tokens are Googlebot, so classic Google rankings are unaffected.

How quickly does a robots.txt change take effect for Arquivo.pt?

Crawlers cache robots.txt, commonly for up to a day. Expect a change to apply within about 24 hours, and confirm it in your logs.

Can I allow Arquivo.pt on some sections and block it on others?

Yes. Inside the group for Arquivo-web-crawler, add Disallow lines for the paths to keep private and Allow lines for exceptions. The longest matching rule wins.

Does blocking Arquivo.pt delete content it already collected?

No. robots.txt only controls future visits. For content already collected, check whether Arquivo.pt offers a removal or opt-out process.

Is a firewall block better than robots.txt for Arquivo.pt?

A firewall enforces the decision, while robots.txt asks politely. Many operators recommend robots.txt because blocking their IPs can stop them reading your rules at all. Use robots.txt first, and add a firewall rule only if requests continue.