Firecrawl Agent
CautionUser-agent: FirecrawlAgent
Category: AI & LLM Bots
SEO impact if blocked: 5/10
Firecrawl
1 crawler operated by Firecrawl
https://www.firecrawl.dev/Who Firecrawl is, for site owners
Firecrawl operates 1 crawler in our directory, filed under AI & LLM Bots. Its crawl behaviour is rated caution.
Blocking Firecrawl carries an SEO impact score of 5/10. Blocking impact is rated: Medium - Minor visibility trade-off.
Our recommendation for Firecrawl: Review your goals before blocking; weigh AI visibility against crawl load.
Firecrawl at a glance
| Operator | Firecrawl |
|---|---|
| Crawler in directory | 1 |
| Categories | AI & LLM Bots |
| SEO impact if blocked | 5/10 |
| Crawl behaviour | 1 rated caution |
| Website | www.firecrawl.dev |
| Official crawler docs | Firecrawl crawler documentation |
Crawler operated by Firecrawl
What Firecrawl's crawlers are for
Firecrawl runs 1 crawler in our database. Grouped by the job each one does:
- a user-triggered agent:
FirecrawlAgent
The highest-priority token is Tier 2. That puts Firecrawl among the operators whose crawlers have a direct effect on whether AI answers or search results can include your pages.
How site owners treat Firecrawl
None of Firecrawl's tokens were among the names tracked in our robots.txt study of the Tranco top 10,000, which usually means few large sites write rules for them yet. Your own logs are the best evidence of how often they visit.
Writing one policy for Firecrawl
A robots.txt file cannot say "Firecrawl"; it can only name Firecrawl's tokens, one group at a time. With a single token, one group is the whole policy.
# One token controls everything Firecrawl fetches through it
User-agent: FirecrawlAgent
Disallow: /private/
Allow: /- Decide per job, not per company. Training, search and user-triggered fetching have different trade-offs, even from the same operator.
- Name every token you mean. A token you leave out follows
User-agent: *. - Repeat shared rules. A named group ignores the
*group, so copy anyDisallowlines you still want it to obey. - Verify from outside. Run the AI bot checker after any change to see the result for each Firecrawl token, including firewall refusals.
Test yourself: does User-agent: Firecrawl block all of Firecrawl's crawlers?
No. robots.txt matches crawler tokens, not company names. Use the token FirecrawlAgent.
Test yourself: if you block one Firecrawl token, are the others blocked too?
No. Each token is read separately. Blocking one leaves the rest following whatever group applies to them.
Each Firecrawl crawler, explained
A one-paragraph brief for every Firecrawl token in our database, covering what it does, how it treats robots.txt and how often large sites write rules for it:
- Firecrawl Agent (
FirecrawlAgent, Tier 2): a user-triggered agent that harvests pages for a dataset or data product. Rated caution. Firecrawl Agent's operator says it follows robots.txt, so a correctly written Firecrawl Agent group is enough to control it. Not tracked in our robots.txt study.
Common mistakes with Firecrawl's crawlers
The errors we see most often with Firecrawl's crawlers in the files and firewalls we test:
- Assuming a robots.txt rule will stop Firecrawl's crawlers. User-triggered fetches act for a person, and some operators say robots.txt may not apply to them. Check the operator's statement above.
- Blocking Firecrawl's crawlers as if it were a bulk crawler. It fetches one page because a person asked; blocking it means that person's assistant cannot read your page.
- Counting Firecrawl's crawlers visits as crawl volume. Each request maps to a real user question, which makes the paths it requests a useful signal of what people ask about you.
All three hide inside a file that looks correct, so check Firecrawl's crawlers from outside after every change.
What our own data shows about Firecrawl
Everything below comes from the AI Crawler Check bot database and our two most recent robots.txt scans of the Tranco top 10,000, not from Firecrawl's marketing. Firecrawl has one token in our database, so the whole picture fits in one place.
FirecrawlAgent in our records
We first recorded FirecrawlAgent on June 18, 2026. Firecrawl documents it on docs.firecrawl.dev, and we recheck that page before each release. No file in our top-10,000 scans names FirecrawlAgent, so any rule you write for it puts you ahead of almost every large site.
Among the 104 AI and LLM crawlers in our database, FirecrawlAgent sits 42th by priority (Tier 2), level with AddSearchBot, AI2Bot, AI2Bot-DeepResearchEval, aiHitBot. 22 of the 104 share its caution rating. Other operators with a crawler that harvests pages for a dataset or data product: iaskspider (iAsk, Tier 2), NovaAct (Amazon, Tier 2), CloudflareBrowserRenderingCrawler (Cloudflare, Tier 3), Amazon Kendra (Amazon, Tier 6), ApifyBot (Apify, Tier 6). A policy for FirecrawlAgent is usually a policy for these too.
A decision record for FirecrawlAgent
Teams that review crawlers once a quarter keep one short entry per token, so anyone can see why a rule exists and when it was last checked. Filled in from our data, the FirecrawlAgent entry reads:
- Operator and job: Firecrawl; it harvests pages for a dataset or data product.
- Priority and behaviour: Tier 2, rated caution, blocking impact medium.
- Suggested starting point: Allow, and watch it in your logs.
- Enforcement: FirecrawlAgent's operator says it follows robots.txt, so a correctly written FirecrawlAgent group is enough to control it.
User-agent: FirecrawlAgent
Allow: /Paste the line into the validator with one of your own URLs before it goes live. Revisit the FirecrawlAgent entry sooner than most: a caution rating means Firecrawl traffic deserves a monthly look at your logs.
Checking FirecrawlAgent on your own site
Because Firecrawl runs only FirecrawlAgent in our records, one check covers the whole operator. Search your access logs for the string FirecrawlAgent over the last 30 days, note how many requests it made and which paths it asked for, then compare that with what your robots.txt allows. Match the requests against what Firecrawl publishes about the crawler, since that page is the only authority on its behaviour. A Tier 2 crawler rated caution is worth confirming every month, and our log checker does the counting for you.
How top sites treat Firecrawl's crawlers
From our robots.txt study of the Tranco top 10,000 (5,583 valid files, fetched October 20, 2026):
None of the tokens for Firecrawl's crawlers is tracked by name in our study. For scale, 20.1% of top-site files block at least one AI training crawler and 4.3% refuse every unnamed crawler with User-agent: * and Disallow: /, which would catch Firecrawl's crawlers too.
More questions about Firecrawl
- Will blocking Firecrawl affect my Google rankings?
- Only if one of its tokens is a Google search crawler. None of Firecrawl's tokens are Googlebot, so classic Google rankings are unaffected.
- How quickly does a robots.txt change take effect for Firecrawl?
- Crawlers cache robots.txt, commonly for up to a day. Expect a change to apply within about 24 hours, and confirm it in your logs.
- Can I allow Firecrawl on some sections and block it on others?
- Yes. Inside the group for
FirecrawlAgent, addDisallowlines for the paths to keep private andAllowlines for exceptions. The longest matching rule wins. - Does blocking Firecrawl delete content it already collected?
- No. robots.txt only controls future visits. For content already collected, check whether Firecrawl offers a removal or opt-out process.
- Is a firewall block better than robots.txt for Firecrawl?
- A firewall enforces the decision, while robots.txt asks politely. Many operators recommend robots.txt because blocking their IPs can stop them reading your rules at all. Use robots.txt first, and add a firewall rule only if requests continue.
A five-minute Firecrawl review
- Confirm the token. Compare the user-agent in your logs with the tokens listed on this page. A close spelling is not the same crawler.
- Decide the outcome per crawler. Use the briefs above to choose allow, limit or block for each token.
- Write the group. Add or edit one
User-agentgroup per decision and repeat any shared rules inside it. - Publish and wait. Allow about a day for crawlers to fetch the new file.
- Confirm the result. Check your logs, then run the AI crawler checker to see the outcome from outside, including any firewall layer.
How to block or allow Firecrawl in robots.txt
This rule covers the single crawler Firecrawl operates. Add it to the robots.txt file at the root of your domain.
Block Firecrawl completely
User-agent: FirecrawlAgent
Disallow: /
Allow Firecrawl explicitly
User-agent: FirecrawlAgent
Allow: /
These Firecrawl rules ask; they do not enforce. If a Firecrawl token ignores them, block it at the server or CDN. Paths are case-sensitive, and each rule matches the token, not the full user-agent string.
Frequently asked questions about Firecrawl
What crawler does Firecrawl run?
Firecrawl operates 1 crawler in our directory: Firecrawl Agent (user-agent FirecrawlAgent).
Should I block Firecrawl?
Review your goals before blocking; weigh AI visibility against crawl load. The SEO impact score for blocking is 5/10, where 10 means blocking costs you the most visibility.
How do I block Firecrawl in robots.txt?
Add the following to robots.txt at your domain root: User-agent: FirecrawlAgent then Disallow: /.
Does blocking Firecrawl hurt my search rankings?
Blocking impact is rated: Medium - Minor visibility trade-off. Google's own ranking crawler is Googlebot, which this operator does not run, so blocking Firecrawl does not by itself remove you from Google's index. What it can change is whether AI assistants and analytics tools can read your pages.