yacy
SafeUser-agent: yacy
Category: Data Scrapers
SEO impact if blocked: 2/10
YaCy Community
2 crawlers operated by YaCy Community
Who YaCy Community is, for site owners
YaCy Community operates 2 crawlers in our directory, filed under Data Scrapers. Of those 2, 2 rated safe for crawl behaviour.
Blocking YaCy Community carries an SEO impact score of 2/10 across its crawlers. Every crawler here carries the same blocking impact: Low - No SEO ranking impact.
Our recommendation for YaCy Community, set by its least reassuring crawler (yacy, rated safe): Generally safe to allow; provides legitimate crawling value.
YaCy Community at a glance
| Operator | YaCy Community |
|---|---|
| Crawlers in directory | 2 |
| Categories | Data Scrapers |
| SEO impact if blocked | 2/10 |
| Crawl behaviour | 2 rated safe |
| Official crawler docs | YaCy Community crawler documentation |
Crawlers operated by YaCy Community
What YaCy Community's crawlers are for
YaCy Community runs 2 crawlers in our database. Grouped by the job each one does:
- a retrieval crawler that feeds AI search answers:
yacy,yacybot
The highest-priority token is Tier 6. That makes YaCy Community a lower-priority operator for most sites: worth a deliberate decision, but rarely the one that decides your visibility.
How site owners treat YaCy Community
None of YaCy Community's tokens were among the names tracked in our robots.txt study of the Tranco top 10,000, which usually means few large sites write rules for them yet. Your own logs are the best evidence of how often they visit.
Writing one policy for YaCy Community
A robots.txt file cannot say "YaCy Community"; it can only name YaCy Community's tokens, one group at a time. So "block YaCy Community" really means naming each of its 2 tokens, and you can give them different answers.
# Allow YaCy Community search and assistants, refuse training
User-agent: yacy
Disallow: /
User-agent: yacy
User-agent: yacybot
Allow: /- Decide per job, not per company. Training, search and user-triggered fetching have different trade-offs, even from the same operator.
- Name every token you mean. A token you leave out follows
User-agent: *. - Repeat shared rules. A named group ignores the
*group, so copy anyDisallowlines you still want it to obey. - Verify from outside. Run the GPTBot and ClaudeBot access check after any change to see the result for each YaCy Community token, including firewall refusals.
Test yourself: does User-agent: YaCy Community block all of YaCy Community's crawlers?
No. robots.txt matches crawler tokens, not company names. Name each token: yacy, yacybot.
Test yourself: if you block one YaCy Community token, are the others blocked too?
No. Each token is read separately. Blocking one leaves the rest following whatever group applies to them.
Each YaCy Community crawler, explained
A one-paragraph brief for every YaCy Community token in our database, covering what it does, how it treats robots.txt and how often large sites write rules for it:
- yacy (
yacy, Tier 6): a retrieval crawler that feeds AI search answers that builds a search or answer index. Rated safe. yacy's operator does not clearly document how it treats robots.txt. Write the yacy rule anyway, then confirm in your logs that its requests stop. Not tracked in our robots.txt study. - yacybot (
yacybot, Tier 6): a retrieval crawler that feeds AI search answers that builds a search or answer index. Rated safe. yacybot's operator does not clearly document how it treats robots.txt. Write the yacybot rule anyway, then confirm in your logs that its requests stop. Not tracked in our robots.txt study.
Common mistakes with YaCy Community's crawlers
The errors we see most often with YaCy Community's crawlers in the files and firewalls we test:
- Blocking YaCy Community's crawlers while trying to stop AI training. Retrieval crawlers fetch pages so answers can cite them; blocking it can remove you from those answers without affecting training.
- Letting a firewall or bot-fight mode refuse YaCy Community's crawlers while robots.txt allows it. The file looks right, but the crawler still gets a 403 or a challenge page.
- Forgetting that once a
User-agent: yacygroup exists, YaCy Community's crawlers ignores every rule underUser-agent: *.
All three hide inside a file that looks correct, so check YaCy Community's crawlers from outside after every change.
Spotting YaCy Community in your logs
robots.txt says what yacy and yacybot may do; your access logs show what they actually did.
Count requests
On a server with standard Nginx or Apache logs, this counts matching requests per day (adjust the log path for your host):
grep -iE "yacy|yacybot" /var/log/nginx/access.log \
| awk '{print $4}' | cut -d: -f1 | sort | uniq -cAnd this shows which paths it requested most, which is the quickest way to see whether your Disallow lines are being respected:
grep -iE "yacy|yacybot" /var/log/nginx/access.log \
| awk '{print $7}' | sort | uniq -c | sort -rn | head -20Check the requests are genuine
Anyone can send these user-agents, so confirm the source before acting on a match:
yacy: we hold no published IP list, so use reverse DNS where YaCy Community documents a hostname and treat an unverifiable yacy request with caution. Check it separately from yacybot.yacybot: we hold no published IP list, so use reverse DNS where YaCy Community documents a hostname and treat an unverifiable yacybot request with caution. Check it separately from yacy.
What our own data shows about YaCy Community
Everything below comes from the AI Crawler Check bot database and our two most recent robots.txt scans of the Tranco top 10,000, not from YaCy Community's marketing. It covers 2 YaCy Community tokens in turn.
yacy in our records
We first recorded yacy on March 17, 2026. YaCy Community documents it on yacy.net, and we recheck that page before each release. No file in our top-10,000 scans names yacy, so any rule you write for it puts you ahead of almost every large site.
Among the 39 data scrapers in our database, yacy sits 38th by priority (Tier 6), level with Amazon Kendra, ApifyBot, Barkrowler, BLEXBot. 29 of the 39 share its safe rating. Other operators with a crawler that builds a search or answer index: AddSearchBot (AddSearch, Tier 2), AIWebIndex (Lyrenth, Tier 2), atlassian-bot (Atlassian, Tier 2), AzureAI-SearchBot (Microsoft, Tier 2), Cloudflare-AI-Search (Cloudflare, Tier 2). A policy for yacy is usually a policy for these too.
A decision record for yacy
Teams that review crawlers once a quarter keep one short entry per token, so anyone can see why a rule exists and when it was last checked. Filled in from our data, the yacy entry reads:
- Operator and job: YaCy Community; it builds a search or answer index.
- Priority and behaviour: Tier 6, rated safe, blocking impact low.
- Suggested starting point: Allow by default; revisit if it costs you bandwidth.
- Enforcement: yacy's operator does not clearly document how it treats robots.txt. Write the yacy rule anyway, then confirm in your logs that its requests stop.
User-agent: yacy
Allow: /Paste the line into the validator with one of your own URLs before it goes live. Revisit the yacy entry once a quarter, or when YaCy Community publishes a new token next to it.
yacybot in our records
We first recorded yacybot on March 17, 2026. YaCy Community documents it on yacy.net, and we recheck that page before each release. No file in our top-10,000 scans names yacybot, so any rule you write for it puts you ahead of almost every large site.
Among the 39 data scrapers in our database, yacybot sits 39th by priority (Tier 6), level with Amazon Kendra, ApifyBot, Barkrowler, BLEXBot. 29 of the 39 share its safe rating. Other operators with a crawler that builds a search or answer index: AddSearchBot (AddSearch, Tier 2), AIWebIndex (Lyrenth, Tier 2), atlassian-bot (Atlassian, Tier 2), AzureAI-SearchBot (Microsoft, Tier 2), Cloudflare-AI-Search (Cloudflare, Tier 2). A policy for yacybot is usually a policy for these too.
A decision record for yacybot
Teams that review crawlers once a quarter keep one short entry per token, so anyone can see why a rule exists and when it was last checked. Filled in from our data, the yacybot entry reads:
- Operator and job: YaCy Community; it builds a search or answer index.
- Priority and behaviour: Tier 6, rated safe, blocking impact low.
- Suggested starting point: Allow by default; revisit if it costs you bandwidth.
- Enforcement: yacybot's operator does not clearly document how it treats robots.txt. Write the yacybot rule anyway, then confirm in your logs that its requests stop.
User-agent: yacybot
Allow: /Paste the line into the validator with one of your own URLs before it goes live. Revisit the yacybot entry once a quarter, or when YaCy Community publishes a new token next to it.
How top sites treat YaCy Community's crawlers
From our robots.txt study of the Tranco top 10,000 (5,583 valid files, fetched October 20, 2026):
None of the tokens for YaCy Community's crawlers is tracked by name in our study. For scale, 20.1% of top-site files block at least one AI training crawler and 4.3% refuse every unnamed crawler with User-agent: * and Disallow: /, which would catch YaCy Community's crawlers too.
More questions about YaCy Community
- Will blocking YaCy Community affect my Google rankings?
- Only if one of its tokens is a Google search crawler. None of YaCy Community's tokens are Googlebot, so classic Google rankings are unaffected.
- How quickly does a robots.txt change take effect for YaCy Community?
- Crawlers cache robots.txt, commonly for up to a day. Expect a change to apply within about 24 hours, and confirm it in your logs.
- Can I allow YaCy Community on some sections and block it on others?
- Yes. Inside the group for
yacy, addDisallowlines for the paths to keep private andAllowlines for exceptions. The longest matching rule wins. - Does blocking YaCy Community delete content it already collected?
- No. robots.txt only controls future visits. For content already collected, check whether YaCy Community offers a removal or opt-out process.
- Is a firewall block better than robots.txt for YaCy Community?
- A firewall enforces the decision, while robots.txt asks politely. Many operators recommend robots.txt because blocking their IPs can stop them reading your rules at all. Use robots.txt first, and add a firewall rule only if requests continue.
A five-minute YaCy Community review
- Confirm the token. Compare the user-agent in your logs with the tokens listed on this page. A close spelling is not the same crawler.
- Decide the outcome per crawler. Use the briefs above to choose allow, limit or block for each token.
- Write the group. Add or edit one
User-agentgroup per decision and repeat any shared rules inside it. - Publish and wait. Allow about a day for crawlers to fetch the new file.
- Confirm the result. Check your logs, then run the crawler access checker to see the outcome from outside, including any firewall layer.
How to block or allow YaCy Community in robots.txt
These rules cover all 2 crawlers YaCy Community operates. Add them to the robots.txt file at the root of your domain.
Block YaCy Community completely
User-agent: yacy
User-agent: yacybot
Disallow: /
Allow YaCy Community explicitly
User-agent: yacy
User-agent: yacybot
Allow: /
These YaCy Community rules ask; they do not enforce. If a YaCy Community token ignores them, block it at the server or CDN. Paths are case-sensitive, and each rule matches the token, not the full user-agent string.
Frequently asked questions about YaCy Community
What crawlers does YaCy Community run?
YaCy Community operates 2 crawlers in our directory: yacy (user-agent yacy), yacybot (user-agent yacybot).
Should I block YaCy Community?
Generally safe to allow; provides legitimate crawling value. That guidance follows this operator's least reassuring crawler, yacy, rated safe, because allowing an operator allows every crawler it runs. The SEO impact score for blocking is 2/10, where 10 means blocking costs you the most visibility.
How do I block YaCy Community in robots.txt?
Add the following to robots.txt at your domain root: User-agent: yacy then User-agent: yacybot then Disallow: /. One Disallow line covers all 2 user-agent tokens.
Does blocking YaCy Community hurt my search rankings?
Every crawler here carries the same blocking impact: Low - No SEO ranking impact. Google's own ranking crawler is Googlebot, which this operator does not run, so blocking YaCy Community does not by itself remove you from Google's index. What it can change is whether AI assistants and analytics tools can read your pages.