Open Source

Common mistakes with Open Source's crawlers

The errors we see most often with Open Source's crawlers in the files and firewalls we test:

1. Treating Open Source's crawlers as an AI or search crawler. Blocking it has little effect on visibility; decide based on load and what it does with your data.
2. Relying on robots.txt alone when Open Source's crawlers is rated caution or aggressive. Back the rule with a server-side block.
3. Blocking Open Source's crawlers by IP alone. Ranges change; match the token and verify with logs.

All three hide inside a file that looks correct, so check Open Source's crawlers from outside after every change.

Spotting Open Source in your logs

robots.txt says what imgproxy and news-please may do; your access logs show what they actually did.

Count requests

On a server with standard Nginx or Apache logs, this counts matching requests per day (adjust the log path for your host):

grep -iE "imgproxy|news-please" /var/log/nginx/access.log \
  | awk '{print $4}' | cut -d: -f1 | sort | uniq -c

And this shows which paths it requested most, which is the quickest way to see whether your Disallow lines are being respected:

grep -iE "imgproxy|news-please" /var/log/nginx/access.log \
  | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

Check the requests are genuine

Anyone can send these user-agents, so confirm the source before acting on a match:

• imgproxy: we hold no published IP list, so use reverse DNS where Open Source documents a hostname and treat an unverifiable imgproxy request with caution. Check it separately from news-please.
• news-please: we hold no published IP list, so use reverse DNS where Open Source documents a hostname and treat an unverifiable news-please request with caution. Check it separately from imgproxy.

What our own data shows about Open Source

Everything below comes from the AI Crawler Check bot database and our two most recent robots.txt scans of the Tranco top 10,000, not from Open Source's marketing. It covers 2 Open Source tokens in turn.

imgproxy in our records

We first recorded imgproxy on March 17, 2026. Open Source documents it on docs.imgproxy.net, and we recheck that page before each release. No file in our top-10,000 scans names imgproxy, so any rule you write for it puts you ahead of almost every large site.

Among the 39 data scrapers in our database, imgproxy sits 21th by priority (Tier 6), level with Amazon Kendra, ApifyBot, Barkrowler, BLEXBot. 29 of the 39 share its safe rating. Other operators with a crawler that harvests pages for a dataset or data product: FirecrawlAgent (Firecrawl, Tier 2), iaskspider (iAsk, Tier 2), NovaAct (Amazon, Tier 2), CloudflareBrowserRenderingCrawler (Cloudflare, Tier 3), Amazon Kendra (Amazon, Tier 6). A policy for imgproxy is usually a policy for these too.

A decision record for imgproxy

Teams that review crawlers once a quarter keep one short entry per token, so anyone can see why a rule exists and when it was last checked. Filled in from our data, the imgproxy entry reads:

• Operator and job: Open Source; it harvests pages for a dataset or data product.
• Priority and behaviour: Tier 6, rated safe, blocking impact low.
• Suggested starting point: Allow by default; revisit if it costs you bandwidth.
• Enforcement: imgproxy's operator does not clearly document how it treats robots.txt. Write the imgproxy rule anyway, then confirm in your logs that its requests stop.

User-agent: imgproxy
Allow: /

Paste the line into the validator with one of your own URLs before it goes live. Revisit the imgproxy entry once a quarter, or when Open Source publishes a new token next to it.

news-please in our records

We first recorded news-please on March 17, 2026. Open Source documents it on github.com, and we recheck that page before each release. No file in our top-10,000 scans names news-please, so any rule you write for it puts you ahead of almost every large site.

Among the 39 data scrapers in our database, news-please sits 27th by priority (Tier 6), level with Amazon Kendra, ApifyBot, Barkrowler, BLEXBot. 4 of the 39 share its aggressive rating. Other operators with a crawler that harvests pages for a dataset or data product: FirecrawlAgent (Firecrawl, Tier 2), iaskspider (iAsk, Tier 2), NovaAct (Amazon, Tier 2), CloudflareBrowserRenderingCrawler (Cloudflare, Tier 3), Amazon Kendra (Amazon, Tier 6). A policy for news-please is usually a policy for these too.

A decision record for news-please

Teams that review crawlers once a quarter keep one short entry per token, so anyone can see why a rule exists and when it was last checked. Filled in from our data, the news-please entry reads:

• Operator and job: Open Source; it harvests pages for a dataset or data product.
• Priority and behaviour: Tier 6, rated aggressive, blocking impact low.
• Suggested starting point: Disallow, and back it with a server rule.
• Enforcement: news-please's operator does not clearly document how it treats robots.txt. Write the news-please rule anyway, then confirm in your logs that its requests stop.

User-agent: news-please
Disallow: /

Paste the line into the validator with one of your own URLs before it goes live. Revisit the news-please entry sooner than most: a aggressive rating means Open Source traffic deserves a monthly look at your logs.

How top sites treat Open Source's crawlers

From our robots.txt study of the Tranco top 10,000 (5,583 valid files, fetched October 20, 2026):

None of the tokens for Open Source's crawlers is tracked by name in our study. For scale, 20.1% of top-site files block at least one AI training crawler and 4.3% refuse every unnamed crawler with User-agent: * and Disallow: /, which would catch Open Source's crawlers too.

More questions about Open Source

Will blocking Open Source affect my Google rankings?

Only if one of its tokens is a Google search crawler. None of Open Source's tokens are Googlebot, so classic Google rankings are unaffected.

How quickly does a robots.txt change take effect for Open Source?

Crawlers cache robots.txt, commonly for up to a day. Expect a change to apply within about 24 hours, and confirm it in your logs.

Can I allow Open Source on some sections and block it on others?

Yes. Inside the group for imgproxy, add Disallow lines for the paths to keep private and Allow lines for exceptions. The longest matching rule wins.

Does blocking Open Source delete content it already collected?

No. robots.txt only controls future visits. For content already collected, check whether Open Source offers a removal or opt-out process.

Is a firewall block better than robots.txt for Open Source?

A firewall enforces the decision, while robots.txt asks politely. Many operators recommend robots.txt because blocking their IPs can stop them reading your rules at all. Use robots.txt first, and add a firewall rule only if requests continue.