Caution Reported Scrapers

Apify Crawler

Operated by Apify

Quick Facts

User-Agent:
ApifyBot
Category:
Scrapers
Operator:
Apify
Safety:
Use Caution
Blocking Impact:
Low - No SEO ranking impact
SEO Impact Score:
2/10
Official docs:
None published by Apify
Named by:
ai.robots.txt

Reported, not operator documented

Apify has not published a crawler page that names ApifyBot. It is listed here because a credible public source does: ai.robots.txt. Token used by crawlers run on the Apify platform by its customers.

What that means for you: the robots.txt rule on this page is the best available spelling, and a well-behaved crawler using this string will honour it, but there is no vendor statement of purpose, crawl rate or robots.txt compliance to rely on. Treat a matching request in your logs as a detection clue, not proof of who sent it, and combine the rule with a firewall rule if the traffic matters to you. If the operator publishes documentation, this profile moves to Official docs.

What is Apify Crawler?

Apify Crawler is a developer-facing extraction agent operated by Apify. It fetches and structures web content on behalf of applications and AI agents built on Apify's platform. Traffic volume depends on how third-party developers configure it, so monitor your logs if you see heavy activity.

Apify Crawler is a developer-facing extraction agent operated by Apify. It fetches and structures web content on behalf of applications and AI agents built on Apify's platform. Traffic volume depends on how third-party developers configure it, so monitor your logs if you see heavy activity.
Apify Crawler uses the user-agent token ApifyBot. You can control it via robots.txt, meta tags (noai), or the emerging llms.txt standard. Robots.txt is voluntary; for hard enforcement, combine it with server-level IP blocking.

What happens if you block Apify Crawler?

✅ No SEO Impact - Blocking Apify Crawler does not affect your rankings in Google, Bing, or any other search engine. Apify Crawler is an AI crawler, not a traditional search indexer. You can freely block it via User-agent / Disallow: / without any SEO penalty.
Generally safe to allow; provides legitimate crawling value.

How to block Apify Crawler with robots.txt

<code>User-agent: ApifyBot</code> - Matching is case-insensitive. Robots.txt is fetched from the root of each subdomain separately.

Block completely (robots.txt)
User-agent: ApifyBot Disallow: /
Allow all (robots.txt)
User-agent: ApifyBot Allow: /
Block private only (robots.txt)
User-agent: ApifyBot Disallow: /private/ Disallow: /api/ Disallow: /admin/ Allow: /
Nginx server block
# Nginx: Hard-block Apify Crawler if ($http_user_agent ~* "ApifyBot") { return 403 "Bot blocked"; }
Apache .htaccess
# Apache: Hard-block Apify Crawler SetEnvIfNoCase User-Agent "ApifyBot" bad_bot Order Allow,Deny Allow from all Deny from env=bad_bot
Meta robots tag
<meta name="robots" content="noindex, nofollow">
X-Robots-Tag header
X-Robots-Tag: noindex, nofollow

Is Apify Crawler safe to allow?

Apify Crawler is generally legitimate but warrants caution. It is operated by Apify. Review its crawl behaviour in your logs and apply robots.txt or rate-limiting if its activity is heavier than you expect.
Verify by reverse-DNS lookup: legitimate Apify Crawler requests resolve to Apify's domain.

What does Apify Crawler do?

Understanding Apify Crawler's purpose helps you decide whether to allow or block it.

  • Extracts and structures web content for developer applications
  • Powers AI agents built on the operator platform
  • Volume depends on third-party developer configuration
  • Monitor logs if you notice heavy activity
  • Can be rate-limited via robots.txt or server rules

Frequently Asked Questions

What is the official user-agent string for Apify Crawler?
The official user-agent string for Apify Crawler is: ApifyBot. Use this exact string in robots.txt, Nginx, Apache, or Cloudflare firewall rules to target this bot. Matching in robots.txt is case-insensitive. Verify a request genuinely comes from Apify Crawler by performing a reverse-DNS lookup on the source IP.
Is Apify Crawler safe?
Apify Crawler is generally legitimate but warrants caution. It is operated by Apify. Review its crawl behaviour in your logs and apply robots.txt or rate-limiting if its activity is heavier than you expect.
Will blocking Apify Crawler hurt my SEO?
✅ No SEO Impact - Blocking Apify Crawler does not affect your rankings in Google, Bing, or any other search engine. Apify Crawler is an AI crawler, not a traditional search indexer. You can freely block it via User-agent / Disallow: / without any SEO penalty.
How do I block Apify Crawler in robots.txt?
Add the following lines to your /robots.txt file:
User-agent: ApifyBot
Disallow: /

This instructs Apify Crawler not to crawl any path on your site. To block only specific sections, replace / with the path (e.g., Disallow: /blog/).
Does Apify Crawler respect robots.txt?
Apify Crawler is operated by Apify and is expected to fetch and parse /robots.txt before crawling, following RFC 9309. For hard enforcement, combine robots.txt with server-level IP or user-agent blocking.
How do I verify if Apify Crawler is crawling my site?
Search your web server access logs for the string ApifyBot (case-insensitive: grep -i "ApifyBot" /var/log/nginx/access.log). Filter by user-agent in your log analytics tool (GoAccess, AWStats, etc.).

Related Bots