Caution Official docs Data Scrapers

ICC-Crawler

Operated by ICC

Quick Facts

User-Agent:
ICC-Crawler
Category:
Data Scrapers
Operator:
ICC
Safety:
Caution
Blocking Impact:
Low - No SEO ranking impact
SEO Impact Score:
2/10
Official docs:
ucri.nict.go.jp

Confirmed by the operator

ICC names ICC-Crawler on its own documentation page, linked above. The token, purpose and robots.txt behaviour on this profile come from that page, re-checked on October 19, 2026.

What is ICC-Crawler?

ICC-Crawler is run by Japan's National Institute of Information and Communications Technology (NICT). NICT uses the pages for research on translation, information analysis and AI, and may share them with research partners.

NICT lists AI research among its purposes. It identifies itself with the token ICC-Crawler. ICC documents it at ucri.nict.go.jp.

What happens if you block ICC-Crawler?

No effect on Google or Bing.
Consider blocking based on your content strategy.

How to block ICC-Crawler with robots.txt

<code>User-agent: ICC-Crawler</code> - Matching is case-insensitive. Robots.txt is fetched from the root of each subdomain separately.

Block completely (robots.txt)
User-agent: ICC-Crawler Disallow: /
Allow all (robots.txt)
User-agent: ICC-Crawler Allow: /
Block private only (robots.txt)
User-agent: ICC-Crawler Disallow: /private/ Disallow: /api/ Disallow: /admin/ Allow: /
Nginx server block
# Nginx: Hard-block ICC-Crawler if ($http_user_agent ~* "ICC\-Crawler") { return 403 "Bot blocked"; }
Apache .htaccess
# Apache: Hard-block ICC-Crawler SetEnvIfNoCase User-Agent "ICC\-Crawler" bad_bot Order Allow,Deny Allow from all Deny from env=bad_bot
Meta robots tag
<meta name="robots" content="noindex, nofollow">
X-Robots-Tag header
X-Robots-Tag: noindex, nofollow

Is ICC-Crawler safe to allow?

ICC-Crawler is rated Caution. ICC documents it, so it is not a spoofing risk in itself, but your content may be used for AI training. Decide on its purpose.

What does ICC-Crawler do?

From ICC's own documentation.

  • ICC-Crawler is run by Japan's National Institute of Information and Communications Technology (NICT).
  • Content may be used to train AI models
  • No effect on Google or Bing.

ICC-Crawler in context

ICC-Crawler is a utility crawler operated by ICC. In plain terms, it harvests pages for a dataset or data product. Our database files it under data scrapers and ranks it Tier 6, the lowest priority: scrapers and tools that offer most sites little or nothing. 37 of the 39 data scrapers in our database share this tier.

Its safety rating is caution and its blocking impact is low. So the decision about ICC-Crawler rests on what you want from ICC, not on its behaviour. How we rate safety and impact.

How site owners treat ICC-Crawler

Our robots.txt study of the Tranco top 10,000 did not track ICC-Crawler as a named token, so we do not publish an adoption figure for it. If you run a large site, your own logs are the best evidence of how often it visits.

Does it follow robots.txt?

ICC-Crawler's operator does not clearly document how it treats robots.txt. Write the ICC-Crawler rule anyway, then confirm in your logs that its requests stop.

ICC crawlers and close alternatives

ICC runs only this one crawler in our database, so a single User-agent: ICC-Crawler group controls everything ICC fetches through it.

Other data scrapers that do the same job (harvests pages for a dataset or data product) include Amazon Kendra from Amazon, ApifyBot from Apify, Barkrowler from Babbar, BLEXBot from WebMeUp, coccocbot-web from Unknown, Crawl4AI from Unknown. Deciding about ICC-Crawler is usually a decision about the whole group, so check how your file treats these too.

We first recorded ICC-Crawler on March 17, 2026. ICC documents it on ucri.nict.go.jp, and we recheck that page before each release.

Which rule ICC-Crawler actually follows

ICC-Crawler reads exactly one group in robots.txt, the most specific one that names ICC-Crawler, and never combines rules across groups. This short file shows the trap:

User-agent: *
Disallow: /private/

User-agent: ICC-Crawler
Disallow: /drafts/
  1. Find the group. ICC-Crawler looks for a group naming ICC-Crawler. One exists, so it ignores User-agent: * completely.
  2. Apply only that group. It is refused /drafts/ and nothing else.
  3. The surprise. /private/ is open to ICC-Crawler, because that rule lives in the * group it no longer reads. To keep it out of both, repeat Disallow: /private/ inside its own group.
Test yourself: does a rule under User-agent: * apply to ICC-Crawler if the file also has a User-agent: ICC-Crawler group?

No. Once a group names it, ICC-Crawler reads only that group. Put every rule you want it to follow inside its own group.

Test yourself: will blocking ICC-Crawler remove your pages from AI answers?

Often, yes. ICC-Crawler is a utility crawler, so pages it cannot fetch can drop out of the answers or results it supports.

Test yourself: is User-agent: icc-crawler the same as User-agent: ICC-Crawler?

Yes. RFC 9309 makes user-agent matching case-insensitive. Paths are a different story: Disallow: /Blog/ does not block /blog/.

Related: how AI engines choose their sources

Should you allow ICC-Crawler? Decide by situation

Recommended setting for ICC-Crawler by type of site
If you are...Recommended setting
You publish content and want AI answers to cite itLow stakes either way. It does not decide whether AI answers cite you.
You license content and want to stop AI trainingBlocking it does not stop training, because ICC-Crawler is a utility crawler. Block the operator's training crawler instead.
You run an online shopConsider blocking it, especially on price and stock pages that competitors scrape.
Your server is under load from botsRate-limit before blocking. It is rated caution in our database.

Check ICC-Crawler on your own site

  1. Open /robots.txt on your domain and search for ICC-Crawler. If it is not named, it follows User-agent: *.
  2. List the Disallow lines in the group ICC-Crawler follows and test your key URLs against them; the longest match wins, and Allow wins a tie.
  3. Look beyond the file. A firewall or bot manager can refuse ICC-Crawler even when robots.txt allows it, and that never shows in the file.
  4. Run our AI bot crawlability test to see how ICC-Crawler and every other crawler we track are treated in one pass, including firewall refusals.

Common mistakes with ICC-Crawler

The errors we see most often with ICC-Crawler in the files and firewalls we test:

  1. Treating ICC-Crawler as an AI or search crawler. Blocking it has little effect on visibility; decide based on load and what it does with your data.
  2. Relying on robots.txt alone when ICC-Crawler is rated caution or aggressive. Back the rule with a server-side block.
  3. Blocking ICC-Crawler by IP alone. Ranges change; match the token and verify with logs.

All three hide inside a file that looks correct, so check ICC-Crawler from outside after every change.

Spotting ICC-Crawler in your logs

robots.txt says what ICC-Crawler may do; your access logs show what it actually did.

Count requests

On a server with standard Nginx or Apache logs, this counts matching requests per day (adjust the log path for your host):

grep -iE "ICC-Crawler" /var/log/nginx/access.log \
  | awk '{print $4}' | cut -d: -f1 | sort | uniq -c

And this shows which paths it requested most, which is the quickest way to see whether your Disallow lines are being respected:

grep -iE "ICC-Crawler" /var/log/nginx/access.log \
  | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

Check the requests are genuine

Anyone can send the ICC-Crawler user-agent, so confirm the source before acting on a match:

  • ICC-Crawler: we hold no published IP list, so use reverse DNS where the operator documents a hostname and treat an unverifiable ICC-Crawler request with caution.

Frequently Asked Questions

What is the official user-agent string for ICC-Crawler?
The official user-agent string for ICC-Crawler is: ICC-Crawler. This is the exact string you must use in robots.txt, Nginx, Apache, or Cloudflare firewall rules to target this bot. User-agent matching in robots.txt is case-insensitive, but the string must be spelled correctly. You can verify that a request genuinely comes from ICC-Crawler by performing a reverse-DNS lookup on the source IP - legitimate bots resolve back to their operator's domain.
Is ICC-Crawler safe?
ICC-Crawler is rated Caution. ICC documents it, so it is not a spoofing risk in itself, but your content may be used for AI training. Decide on its purpose.
Will blocking ICC-Crawler hurt my SEO?
No effect on Google or Bing.
How do I block ICC-Crawler in robots.txt?
Add the following lines to your /robots.txt file:
User-agent: ICC-Crawler
Disallow: /

This instructs ICC-Crawler not to crawl any path on your site. The Disallow: / directive covers the entire domain including subfolders. To only block specific sections, replace / with the path (e.g., Disallow: /blog/). Note: robots.txt is publicly readable - any bot or human can inspect it at yourdomain.com/robots.txt.
Does ICC-Crawler respect robots.txt?
Yes. NICT says it complies with robots.txt (RFC 9309) and Crawl-delay, and will stop crawling a host on request.
How do I verify if ICC-Crawler is crawling my site?
Search your web server access logs for the string ICC-Crawler (case-insensitive grep: grep -i "ICC-Crawler" /var/log/nginx/access.log). You can also check Google Search Console → Coverage → Crawl Stats for Googlebot variants. For ICC-Crawler specifically, filter by user-agent in your log analysis tool (GoAccess, AWStats, etc.).
What is the crawl frequency of ICC-Crawler?
Crawl frequency data for ICC-Crawler is not publicly documented. Monitor your logs to understand actual visit patterns.
Can I block ICC-Crawler from specific pages only?
Yes. Instead of a global Disallow: / you can restrict ICC-Crawler to specific paths:
User-agent: ICC-Crawler
Disallow: /private/
Disallow: /staging/
Allow: /

This allows ICC-Crawler everywhere except the listed paths. Path matching in robots.txt uses prefix matching - Disallow: /private/ blocks /private/page.html but NOT /public/private/.
Is ICC-Crawler causing high server load?
If ICC-Crawler is generating excessive requests, you can:
1. Add Crawl-delay: 30 below the User-agent directive in robots.txt.
2. Rate-limit the user-agent via Nginx's limit_req_zone or Apache's mod_ratelimit.
3. Block it outright at Cloudflare WAF with rule: http.user_agent contains "ICC-Crawler".
4. Use fail2ban to auto-block IPs exceeding request thresholds.

Related Bots

WhatsApp us