Caution Official docs AI & LLM Bots

Parallel Web Crawler

Operated by Parallel

Quick Facts

User-Agent:
ShapBot
Category:
AI & LLM Bots
Operator:
Parallel
Safety:
Use Caution
Blocking Impact:
Low - No SEO ranking impact
SEO Impact Score:
2/10
Official docs:
docs.parallel.ai

Confirmed by the operator

Parallel names ShapBot on its own documentation page, linked above. The token, purpose and robots.txt behaviour on this profile come from that page, re-checked on October 19, 2026.

What is Parallel Web Crawler?

Parallel Web Crawler is an AI search-index crawler operated by Parallel. It collects and indexes web content so it can be surfaced as citations or answers inside Parallel's AI search experience. Allowing Parallel Web Crawler can increase your visibility in AI-powered search results, which is why most sites choose to permit it.

Parallel Web Crawler is an AI search-index crawler operated by Parallel. It collects and indexes web content so it can be surfaced as citations or answers inside Parallel's AI search experience. Allowing Parallel Web Crawler can increase your visibility in AI-powered search results, which is why most sites choose to permit it.
Parallel Web Crawler uses the user-agent token ShapBot. You can control it via robots.txt, meta tags (noai), or the emerging llms.txt standard. Robots.txt is voluntary; for hard enforcement, combine it with server-level IP blocking.

What happens if you block Parallel Web Crawler?

✅ No SEO Impact - Blocking Parallel Web Crawler does not affect your rankings in Google, Bing, or any other search engine. Parallel Web Crawler is an AI crawler, not a traditional search indexer. You can freely block it via User-agent / Disallow: / without any SEO penalty.
Generally safe to allow; provides legitimate crawling value.

Want to see whether Parallel Web Crawler actually visits you? Read how do I monitor AI bot traffic to my website?

How to block Parallel Web Crawler with robots.txt

<code>User-agent: ShapBot</code> - Matching is case-insensitive. Robots.txt is fetched from the root of each subdomain separately.

Block completely (robots.txt)
User-agent: ShapBot Disallow: /
Allow all (robots.txt)
User-agent: ShapBot Allow: /
Block private only (robots.txt)
User-agent: ShapBot Disallow: /private/ Disallow: /api/ Disallow: /admin/ Allow: /
Nginx server block
# Nginx: Hard-block Parallel Web Crawler if ($http_user_agent ~* "ShapBot") { return 403 "Bot blocked"; }
Apache .htaccess
# Apache: Hard-block Parallel Web Crawler SetEnvIfNoCase User-Agent "ShapBot" bad_bot Order Allow,Deny Allow from all Deny from env=bad_bot
Meta robots tag
<meta name="robots" content="noindex, nofollow">
X-Robots-Tag header
X-Robots-Tag: noindex, nofollow

Is Parallel Web Crawler safe to allow?

Parallel Web Crawler is generally legitimate but warrants caution. It is operated by Parallel. Review its crawl behaviour in your logs and apply robots.txt or rate-limiting if its activity is heavier than you expect.
Verify by reverse-DNS lookup: legitimate Parallel Web Crawler requests resolve to Parallel's domain.

What does Parallel Web Crawler do?

Understanding Parallel Web Crawler's purpose helps you decide whether to allow or block it.

  • Indexes web pages for AI-powered search answers
  • Surfaces your content as citations inside AI search results
  • Builds a knowledge index used to answer user questions
  • Allowing it can increase visibility in AI search
  • Respects robots.txt directives

Parallel Web Crawler in context

Parallel Web Crawler is a retrieval crawler that feeds AI search answers operated by Parallel. In plain terms, it builds a search or answer index. Our database files it under AI and LLM crawlers and ranks it Tier 3, mid-priority crawlers with a clear purpose but a smaller audience. 15 of the 104 AI and LLM crawlers in our database share this tier.

Its safety rating is caution and its blocking impact is low. So the decision about Parallel Web Crawler rests on what you want from Parallel, not on its behaviour. How we rate safety and impact.

How site owners treat Parallel Web Crawler

Our robots.txt study of the Tranco top 10,000 did not track ShapBot as a named token, so we do not publish an adoption figure for it. If you run a large site, your own logs are the best evidence of how often it visits.

Does it follow robots.txt?

Parallel Web Crawler's operator does not clearly document how it treats robots.txt. Write the Parallel Web Crawler rule anyway, then confirm in your logs that its requests stop.

Parallel crawlers and close alternatives

Parallel Web Crawler is one of 2 Parallel crawlers in our database. A User-agent: ShapBot group controls Parallel Web Crawler only; the other Parallel tokens below need their own lines.

  • ShapBot: this crawler (Tier 3)
  • Shap-User: Parallel Fetcher, fetches a page when a person asks an assistant or app to open it (Tier 3)

To give Parallel Web Crawler the same answer as its siblings, stack User-agent: ShapBot with their lines above one set of rules; to treat Parallel Web Crawler differently, give it its own group.

Other AI and LLM crawlers that do the same job (builds a search or answer index) include AddSearchBot from AddSearch, AIWebIndex from Lyrenth, atlassian-bot from Atlassian, AzureAI-SearchBot from Microsoft, Cloudflare-AI-Search from Cloudflare, Cloudflare-AutoRAG from Cloudflare. Deciding about Parallel Web Crawler is usually a decision about the whole group, so check how your file treats these too.

We first recorded ShapBot on September 22, 2026. Parallel documents it on docs.parallel.ai, and we recheck that page before each release.

Which rule Parallel Web Crawler actually follows

Parallel Web Crawler reads exactly one group in robots.txt, the most specific one that names ShapBot, and never combines rules across groups. This short file shows the trap:

User-agent: *
Disallow: /private/

User-agent: ShapBot
Disallow: /drafts/
  1. Find the group. Parallel Web Crawler looks for a group naming ShapBot. One exists, so it ignores User-agent: * completely.
  2. Apply only that group. It is refused /drafts/ and nothing else.
  3. The surprise. /private/ is open to Parallel Web Crawler, because that rule lives in the * group it no longer reads. To keep it out of both, repeat Disallow: /private/ inside its own group.
Test yourself: does a rule under User-agent: * apply to Parallel Web Crawler if the file also has a User-agent: ShapBot group?

No. Once a group names it, Parallel Web Crawler reads only that group. Put every rule you want it to follow inside its own group.

Test yourself: will blocking Parallel Web Crawler remove your pages from AI answers?

Often, yes. Parallel Web Crawler is a retrieval crawler that feeds AI search answers, so pages it cannot fetch can drop out of the answers or results it supports.

Test yourself: is User-agent: shapbot the same as User-agent: ShapBot?

Yes. RFC 9309 makes user-agent matching case-insensitive. Paths are a different story: Disallow: /Blog/ does not block /blog/.

Should you allow Parallel Web Crawler? Decide by situation

Recommended setting for Parallel Web Crawler by type of site
If you are...Recommended setting
You publish content and want AI answers to cite itAllow it. Parallel Web Crawler fetches pages for answers, so blocking it can remove you from them.
You license content and want to stop AI trainingBlocking it does not stop training, because Parallel Web Crawler is a retrieval crawler that feeds AI search answers. Block the operator's training crawler instead.
You run an online shopAllow it on product and category pages, which are the pages shoppers ask assistants about.
Your server is under load from botsRate-limit before blocking. It is rated caution in our database.

Check Parallel Web Crawler on your own site

  1. Open /robots.txt on your domain and search for ShapBot. If it is not named, it follows User-agent: *.
  2. List the Disallow lines in the group Parallel Web Crawler follows and test your key URLs against them; the longest match wins, and Allow wins a tie.
  3. Look beyond the file. A firewall or bot manager can refuse Parallel Web Crawler even when robots.txt allows it, and that never shows in the file.
  4. Run our GPTBot and ClaudeBot access check to see how Parallel Web Crawler and every other crawler we track are treated in one pass, including firewall refusals.

Common mistakes with Parallel Web Crawler

The errors we see most often with Parallel Web Crawler in the files and firewalls we test:

  1. Blocking Parallel Web Crawler while trying to stop AI training. Retrieval crawlers fetch pages so answers can cite them; blocking it can remove you from those answers without affecting training.
  2. Letting a firewall or bot-fight mode refuse Parallel Web Crawler while robots.txt allows it. The file looks right, but the crawler still gets a 403 or a challenge page.
  3. Forgetting that once a User-agent: ShapBot group exists, Parallel Web Crawler ignores every rule under User-agent: *.

All three hide inside a file that looks correct, so check Parallel Web Crawler from outside after every change.

Spotting Parallel Web Crawler in your logs

robots.txt says what Parallel Web Crawler may do; your access logs show what it actually did.

Count requests

On a server with standard Nginx or Apache logs, this counts matching requests per day (adjust the log path for your host):

grep -iE "ShapBot" /var/log/nginx/access.log \
  | awk '{print $4}' | cut -d: -f1 | sort | uniq -c

And this shows which paths it requested most, which is the quickest way to see whether your Disallow lines are being respected:

grep -iE "ShapBot" /var/log/nginx/access.log \
  | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

Check the requests are genuine

Anyone can send the Parallel Web Crawler user-agent, so confirm the source before acting on a match:

  • ShapBot: we hold no published IP list, so use reverse DNS where the operator documents a hostname and treat an unverifiable Parallel Web Crawler request with caution.

Frequently Asked Questions

What is the official user-agent string for Parallel Web Crawler?
The official user-agent string for Parallel Web Crawler is: ShapBot. Use this exact string in robots.txt, Nginx, Apache, or Cloudflare firewall rules to target this bot. Matching in robots.txt is case-insensitive. Verify a request genuinely comes from Parallel Web Crawler by performing a reverse-DNS lookup on the source IP.
Is Parallel Web Crawler safe?
Parallel Web Crawler is generally legitimate but warrants caution. It is operated by Parallel. Review its crawl behaviour in your logs and apply robots.txt or rate-limiting if its activity is heavier than you expect.
Will blocking Parallel Web Crawler hurt my SEO?
✅ No SEO Impact - Blocking Parallel Web Crawler does not affect your rankings in Google, Bing, or any other search engine. Parallel Web Crawler is an AI crawler, not a traditional search indexer. You can freely block it via User-agent / Disallow: / without any SEO penalty.
How do I block Parallel Web Crawler in robots.txt?
Add the following lines to your /robots.txt file:
User-agent: ShapBot
Disallow: /

This instructs Parallel Web Crawler not to crawl any path on your site. To block only specific sections, replace / with the path (e.g., Disallow: /blog/).
Does Parallel Web Crawler respect robots.txt?
Parallel Web Crawler is operated by Parallel and is expected to fetch and parse /robots.txt before crawling, following RFC 9309. For hard enforcement, combine robots.txt with server-level IP or user-agent blocking.
How do I verify if Parallel Web Crawler is crawling my site?
Search your web server access logs for the string ShapBot (case-insensitive: grep -i "ShapBot" /var/log/nginx/access.log). Filter by user-agent in your log analytics tool (GoAccess, AWStats, etc.).

Related Bots

WhatsApp us