Caution Official docs AI & LLM Bots

Firecrawl Agent

Operated by Firecrawl

Quick Facts

User-Agent:
FirecrawlAgent
Category:
AI & LLM Bots
Operator:
Firecrawl
Safety:
Use Caution
Blocking Impact:
Medium - Minor visibility trade-off
SEO Impact Score:
5/10
Official docs:
docs.firecrawl.dev

Confirmed by the operator

Firecrawl names FirecrawlAgent on its own documentation page, linked above. The token, purpose and robots.txt behaviour on this profile come from that page, re-checked on October 19, 2026.

What is Firecrawl Agent?

Firecrawl Agent is a developer-facing extraction agent operated by Firecrawl. It fetches and structures web content on behalf of applications and AI agents built on Firecrawl's platform. Traffic volume depends on how third-party developers configure it, so monitor your logs if you see heavy activity.

Firecrawl Agent is a developer-facing extraction agent operated by Firecrawl. It fetches and structures web content on behalf of applications and AI agents built on Firecrawl's platform. Traffic volume depends on how third-party developers configure it, so monitor your logs if you see heavy activity.
Firecrawl Agent uses the user-agent token FirecrawlAgent. You can control it via robots.txt, meta tags (noai), or the emerging llms.txt standard. Robots.txt is voluntary; for hard enforcement, combine it with server-level IP blocking.

What happens if you block Firecrawl Agent?

⚠️ Minor Trade-off - Blocking Firecrawl Agent has no effect on classic search rankings, but you may reduce your visibility inside AI-powered search and assistant answers. Decide based on whether you want your content cited by Firecrawl Agent.
Review your goals before blocking; weigh AI visibility against crawl load.

Want to see whether Firecrawl Agent actually visits you? Read how do I monitor AI bot traffic to my website?

How to block Firecrawl Agent with robots.txt

<code>User-agent: FirecrawlAgent</code> - Matching is case-insensitive. Robots.txt is fetched from the root of each subdomain separately.

Block completely (robots.txt)
User-agent: FirecrawlAgent Disallow: /
Allow all (robots.txt)
User-agent: FirecrawlAgent Allow: /
Block private only (robots.txt)
User-agent: FirecrawlAgent Disallow: /private/ Disallow: /api/ Disallow: /admin/ Allow: /
Nginx server block
# Nginx: Hard-block Firecrawl Agent if ($http_user_agent ~* "FirecrawlAgent") { return 403 "Bot blocked"; }
Apache .htaccess
# Apache: Hard-block Firecrawl Agent SetEnvIfNoCase User-Agent "FirecrawlAgent" bad_bot Order Allow,Deny Allow from all Deny from env=bad_bot
Meta robots tag
<meta name="robots" content="noindex, nofollow">
X-Robots-Tag header
X-Robots-Tag: noindex, nofollow

Is Firecrawl Agent safe to allow?

Firecrawl Agent is generally legitimate but warrants caution. It is operated by Firecrawl. Review its crawl behaviour in your logs and apply robots.txt or rate-limiting if its activity is heavier than you expect.
Verify by reverse-DNS lookup: legitimate Firecrawl Agent requests resolve to Firecrawl's domain.

What does Firecrawl Agent do?

Understanding Firecrawl Agent's purpose helps you decide whether to allow or block it.

  • Extracts and structures web content for developer applications
  • Powers AI agents built on the operator platform
  • Volume depends on third-party developer configuration
  • Monitor logs if you notice heavy activity
  • Can be rate-limited via robots.txt or server rules

Firecrawl Agent in context

Firecrawl Agent is a user-triggered agent operated by Firecrawl. In plain terms, it harvests pages for a dataset or data product. Our database files it under AI and LLM crawlers and ranks it Tier 2, important crawlers from established AI and search operators. 74 of the 104 AI and LLM crawlers in our database share this tier.

Its safety rating is caution and its blocking impact is medium. So the decision about Firecrawl Agent rests on what you want from Firecrawl, not on its behaviour. How we rate safety and impact.

How site owners treat Firecrawl Agent

Our robots.txt study of the Tranco top 10,000 did not track FirecrawlAgent as a named token, so we do not publish an adoption figure for it. If you run a large site, your own logs are the best evidence of how often it visits.

Does it follow robots.txt?

Firecrawl Agent's operator says it follows robots.txt, so a correctly written Firecrawl Agent group is enough to control it.

Firecrawl crawlers and close alternatives

Firecrawl runs only this one crawler in our database, so a single User-agent: FirecrawlAgent group controls everything Firecrawl fetches through it.

Other AI and LLM crawlers that do the same job (harvests pages for a dataset or data product) include iaskspider from iAsk, NovaAct from Amazon, CloudflareBrowserRenderingCrawler from Cloudflare. Deciding about Firecrawl Agent is usually a decision about the whole group, so check how your file treats these too.

We first recorded FirecrawlAgent on June 18, 2026. Firecrawl documents it on docs.firecrawl.dev, and we recheck that page before each release.

Which rule Firecrawl Agent actually follows

Firecrawl Agent reads exactly one group in robots.txt, the most specific one that names FirecrawlAgent, and never combines rules across groups. This short file shows the trap:

User-agent: *
Disallow: /private/

User-agent: FirecrawlAgent
Disallow: /drafts/
  1. Find the group. Firecrawl Agent looks for a group naming FirecrawlAgent. One exists, so it ignores User-agent: * completely.
  2. Apply only that group. It is refused /drafts/ and nothing else.
  3. The surprise. /private/ is open to Firecrawl Agent, because that rule lives in the * group it no longer reads. To keep it out of both, repeat Disallow: /private/ inside its own group.
Test yourself: does a rule under User-agent: * apply to Firecrawl Agent if the file also has a User-agent: FirecrawlAgent group?

No. Once a group names it, Firecrawl Agent reads only that group. Put every rule you want it to follow inside its own group.

Test yourself: will blocking Firecrawl Agent remove your pages from AI answers?

Often, yes. Firecrawl Agent is a user-triggered agent, so pages it cannot fetch can drop out of the answers or results it supports.

Test yourself: is User-agent: firecrawlagent the same as User-agent: FirecrawlAgent?

Yes. RFC 9309 makes user-agent matching case-insensitive. Paths are a different story: Disallow: /Blog/ does not block /blog/.

Should you allow Firecrawl Agent? Decide by situation

Recommended setting for Firecrawl Agent by type of site
If you are...Recommended setting
You publish content and want AI answers to cite itAllow it. Firecrawl Agent fetches pages for answers, so blocking it can remove you from them.
You license content and want to stop AI trainingBlocking it does not stop training, because Firecrawl Agent is a user-triggered agent. Block the operator's training crawler instead.
You run an online shopAllow it on product and category pages, which are the pages shoppers ask assistants about.
Your server is under load from botsRate-limit before blocking. It is rated caution in our database.

Check Firecrawl Agent on your own site

  1. Open /robots.txt on your domain and search for FirecrawlAgent. If it is not named, it follows User-agent: *.
  2. List the Disallow lines in the group Firecrawl Agent follows and test your key URLs against them; the longest match wins, and Allow wins a tie.
  3. Look beyond the file. A firewall or bot manager can refuse Firecrawl Agent even when robots.txt allows it, and that never shows in the file.
  4. Run our AI crawler tester to see how Firecrawl Agent and every other crawler we track are treated in one pass, including firewall refusals.

Common mistakes with Firecrawl Agent

The errors we see most often with Firecrawl Agent in the files and firewalls we test:

  1. Assuming a robots.txt rule will stop Firecrawl Agent. User-triggered fetches act for a person, and some operators say robots.txt may not apply to them. Check the operator's statement above.
  2. Blocking Firecrawl Agent as if it were a bulk crawler. It fetches one page because a person asked; blocking it means that person's assistant cannot read your page.
  3. Counting Firecrawl Agent visits as crawl volume. Each request maps to a real user question, which makes the paths it requests a useful signal of what people ask about you.

All three hide inside a file that looks correct, so check Firecrawl Agent from outside after every change.

Spotting Firecrawl Agent in your logs

robots.txt says what Firecrawl Agent may do; your access logs show what it actually did.

Count requests

On a server with standard Nginx or Apache logs, this counts matching requests per day (adjust the log path for your host):

grep -iE "FirecrawlAgent" /var/log/nginx/access.log \
  | awk '{print $4}' | cut -d: -f1 | sort | uniq -c

And this shows which paths it requested most, which is the quickest way to see whether your Disallow lines are being respected:

grep -iE "FirecrawlAgent" /var/log/nginx/access.log \
  | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

Check the requests are genuine

Anyone can send the Firecrawl Agent user-agent, so confirm the source before acting on a match:

  • FirecrawlAgent: we hold no published IP list, so use reverse DNS where the operator documents a hostname and treat an unverifiable Firecrawl Agent request with caution.

Frequently Asked Questions

What is the official user-agent string for Firecrawl Agent?
The official user-agent string for Firecrawl Agent is: FirecrawlAgent. Use this exact string in robots.txt, Nginx, Apache, or Cloudflare firewall rules to target this bot. Matching in robots.txt is case-insensitive. Verify a request genuinely comes from Firecrawl Agent by performing a reverse-DNS lookup on the source IP.
Is Firecrawl Agent safe?
Firecrawl Agent is generally legitimate but warrants caution. It is operated by Firecrawl. Review its crawl behaviour in your logs and apply robots.txt or rate-limiting if its activity is heavier than you expect.
Will blocking Firecrawl Agent hurt my SEO?
⚠️ Minor Trade-off - Blocking Firecrawl Agent has no effect on classic search rankings, but you may reduce your visibility inside AI-powered search and assistant answers. Decide based on whether you want your content cited by Firecrawl Agent.
How do I block Firecrawl Agent in robots.txt?
Add the following lines to your /robots.txt file:
User-agent: FirecrawlAgent
Disallow: /

This instructs Firecrawl Agent not to crawl any path on your site. To block only specific sections, replace / with the path (e.g., Disallow: /blog/).
Does Firecrawl Agent respect robots.txt?
Firecrawl Agent is operated by Firecrawl and is expected to fetch and parse /robots.txt before crawling, following RFC 9309. For hard enforcement, combine robots.txt with server-level IP or user-agent blocking.
How do I verify if Firecrawl Agent is crawling my site?
Search your web server access logs for the string FirecrawlAgent (case-insensitive: grep -i "FirecrawlAgent" /var/log/nginx/access.log). Filter by user-agent in your log analytics tool (GoAccess, AWStats, etc.).

Related Bots

WhatsApp us