Caution Official docs AI & LLM Bots

Parallel Web Crawler

Operated by Parallel

Quick Facts

User-Agent:
ShapBot
Category:
AI & LLM Bots
Operator:
Parallel
Safety:
Use Caution
Blocking Impact:
Low - No SEO ranking impact
SEO Impact Score:
2/10
Official docs:
docs.parallel.ai

Confirmed by the operator

Parallel names ShapBot on its own documentation page, linked above. The token, purpose and robots.txt behaviour on this profile come from that page, re-checked on September 22, 2026.

What is Parallel Web Crawler?

Parallel Web Crawler is an AI search-index crawler operated by Parallel. It collects and indexes web content so it can be surfaced as citations or answers inside Parallel's AI search experience. Allowing Parallel Web Crawler can increase your visibility in AI-powered search results, which is why most sites choose to permit it.

Parallel Web Crawler is an AI search-index crawler operated by Parallel. It collects and indexes web content so it can be surfaced as citations or answers inside Parallel's AI search experience. Allowing Parallel Web Crawler can increase your visibility in AI-powered search results, which is why most sites choose to permit it.
Parallel Web Crawler uses the user-agent token ShapBot. You can control it via robots.txt, meta tags (noai), or the emerging llms.txt standard. Robots.txt is voluntary; for hard enforcement, combine it with server-level IP blocking.

What happens if you block Parallel Web Crawler?

✅ No SEO Impact - Blocking Parallel Web Crawler does not affect your rankings in Google, Bing, or any other search engine. Parallel Web Crawler is an AI crawler, not a traditional search indexer. You can freely block it via User-agent / Disallow: / without any SEO penalty.
Generally safe to allow; provides legitimate crawling value.

Want to see whether Parallel Web Crawler actually visits you? Read how do I monitor AI bot traffic to my website?

How to block Parallel Web Crawler with robots.txt

<code>User-agent: ShapBot</code> - Matching is case-insensitive. Robots.txt is fetched from the root of each subdomain separately.

Block completely (robots.txt)
User-agent: ShapBot Disallow: /
Allow all (robots.txt)
User-agent: ShapBot Allow: /
Block private only (robots.txt)
User-agent: ShapBot Disallow: /private/ Disallow: /api/ Disallow: /admin/ Allow: /
Nginx server block
# Nginx: Hard-block Parallel Web Crawler if ($http_user_agent ~* "ShapBot") { return 403 "Bot blocked"; }
Apache .htaccess
# Apache: Hard-block Parallel Web Crawler SetEnvIfNoCase User-Agent "ShapBot" bad_bot Order Allow,Deny Allow from all Deny from env=bad_bot
Meta robots tag
<meta name="robots" content="noindex, nofollow">
X-Robots-Tag header
X-Robots-Tag: noindex, nofollow

Is Parallel Web Crawler safe to allow?

Parallel Web Crawler is generally legitimate but warrants caution. It is operated by Parallel. Review its crawl behaviour in your logs and apply robots.txt or rate-limiting if its activity is heavier than you expect.
Verify by reverse-DNS lookup: legitimate Parallel Web Crawler requests resolve to Parallel's domain.

What does Parallel Web Crawler do?

Understanding Parallel Web Crawler's purpose helps you decide whether to allow or block it.

  • Indexes web pages for AI-powered search answers
  • Surfaces your content as citations inside AI search results
  • Builds a knowledge index used to answer user questions
  • Allowing it can increase visibility in AI search
  • Respects robots.txt directives

Frequently Asked Questions

What is the official user-agent string for Parallel Web Crawler?
The official user-agent string for Parallel Web Crawler is: ShapBot. Use this exact string in robots.txt, Nginx, Apache, or Cloudflare firewall rules to target this bot. Matching in robots.txt is case-insensitive. Verify a request genuinely comes from Parallel Web Crawler by performing a reverse-DNS lookup on the source IP.
Is Parallel Web Crawler safe?
Parallel Web Crawler is generally legitimate but warrants caution. It is operated by Parallel. Review its crawl behaviour in your logs and apply robots.txt or rate-limiting if its activity is heavier than you expect.
Will blocking Parallel Web Crawler hurt my SEO?
✅ No SEO Impact - Blocking Parallel Web Crawler does not affect your rankings in Google, Bing, or any other search engine. Parallel Web Crawler is an AI crawler, not a traditional search indexer. You can freely block it via User-agent / Disallow: / without any SEO penalty.
How do I block Parallel Web Crawler in robots.txt?
Add the following lines to your /robots.txt file:
User-agent: ShapBot
Disallow: /

This instructs Parallel Web Crawler not to crawl any path on your site. To block only specific sections, replace / with the path (e.g., Disallow: /blog/).
Does Parallel Web Crawler respect robots.txt?
Parallel Web Crawler is operated by Parallel and is expected to fetch and parse /robots.txt before crawling, following RFC 9309. For hard enforcement, combine robots.txt with server-level IP or user-agent blocking.
How do I verify if Parallel Web Crawler is crawling my site?
Search your web server access logs for the string ShapBot (case-insensitive: grep -i "ShapBot" /var/log/nginx/access.log). Filter by user-agent in your log analytics tool (GoAccess, AWStats, etc.).

Related Bots