Cloudflare AI Search
SafeUser-agent: Cloudflare-AI-Search
Category: AI & LLM Bots
SEO impact if blocked: 1/10
Cloudflare
4 crawlers operated by Cloudflare
https://www.cloudflare.com/Who Cloudflare is, for site owners
Cloudflare operates 4 crawlers in our directory, filed under AI & LLM Bots. Of those 4, 4 rated safe for crawl behaviour.
Blocking Cloudflare carries an SEO impact score of 1/10 across its crawlers. Every crawler here carries the same blocking impact: Low - No effect on Google or Bing rankings.
Our recommendation for Cloudflare, set by its least reassuring crawler (Cloudflare AI Search, rated safe): Allow it if you want your pages in Cloudflare AI Search indexes; block it if you do not.
Cloudflare at a glance
| Operator | Cloudflare |
|---|---|
| Crawlers in directory | 4 |
| Categories | AI & LLM Bots |
| SEO impact if blocked | 1/10 |
| Crawl behaviour | 4 rated safe |
| Website | www.cloudflare.com |
| Official crawler docs | Cloudflare crawler documentation |
Crawlers operated by Cloudflare
Cloudflare AI Search (external sites)
SafeUser-agent:
Cloudflare-AI-Search-ExternalCategory: AI & LLM Bots
SEO impact if blocked: 1/10
Cloudflare AutoRAG
SafeUser-agent:
Cloudflare-AutoRAGCategory: AI & LLM Bots
SEO impact if blocked: 1/10
Cloudflare Crawler (Browser Run)
SafeUser-agent:
CloudflareBrowserRenderingCrawlerCategory: AI & LLM Bots
SEO impact if blocked: 1/10
What Cloudflare's crawlers are for
Cloudflare runs 4 crawlers in our database. Grouped by the job each one does:
- a retrieval crawler that feeds AI search answers:
Cloudflare-AI-Search,Cloudflare-AI-Search-External,Cloudflare-AutoRAG - a utility crawler:
CloudflareBrowserRenderingCrawler
The highest-priority token is Tier 2. That puts Cloudflare among the operators whose crawlers have a direct effect on whether AI answers or search results can include your pages.
How site owners treat Cloudflare
None of Cloudflare's tokens were among the names tracked in our robots.txt study of the Tranco top 10,000, which usually means few large sites write rules for them yet. Your own logs are the best evidence of how often they visit.
Writing one policy for Cloudflare
A robots.txt file cannot say "Cloudflare"; it can only name Cloudflare's tokens, one group at a time. So "block Cloudflare" really means naming each of its 4 tokens, and you can give them different answers.
# Allow Cloudflare search and assistants, refuse training
User-agent: Cloudflare-AI-Search
Disallow: /
User-agent: Cloudflare-AI-Search
User-agent: Cloudflare-AI-Search-External
User-agent: Cloudflare-AutoRAG
User-agent: CloudflareBrowserRenderingCrawler
Allow: /- Decide per job, not per company. Training, search and user-triggered fetching have different trade-offs, even from the same operator.
- Name every token you mean. A token you leave out follows
User-agent: *. - Repeat shared rules. A named group ignores the
*group, so copy anyDisallowlines you still want it to obey. - Verify from outside. Run the AI bot access checker after any change to see the result for each Cloudflare token, including firewall refusals.
Test yourself: does User-agent: Cloudflare block all of Cloudflare's crawlers?
No. robots.txt matches crawler tokens, not company names. Name each token: Cloudflare-AI-Search, Cloudflare-AI-Search-External, Cloudflare-AutoRAG, CloudflareBrowserRenderingCrawler.
Test yourself: if you block one Cloudflare token, are the others blocked too?
No. Each token is read separately. Blocking one leaves the rest following whatever group applies to them.
Each Cloudflare crawler, explained
A one-paragraph brief for every Cloudflare token in our database, covering what it does, how it treats robots.txt and how often large sites write rules for it:
- Cloudflare AI Search (
Cloudflare-AI-Search, Tier 2): a retrieval crawler that feeds AI search answers that builds a search or answer index. Rated safe. Cloudflare AI Search's operator says it follows robots.txt, so a correctly written Cloudflare AI Search group is enough to control it. Not tracked in our robots.txt study. - Cloudflare AI Search (external sites) (
Cloudflare-AI-Search-External, Tier 3): a retrieval crawler that feeds AI search answers that builds a search or answer index. Rated safe. Cloudflare AI Search (external sites)'s operator says it follows robots.txt, so a correctly written Cloudflare AI Search (external sites) group is enough to control it. Not tracked in our robots.txt study. - Cloudflare AutoRAG (
Cloudflare-AutoRAG, Tier 2): a retrieval crawler that feeds AI search answers that builds a search or answer index. Rated safe. Cloudflare AutoRAG's operator says it follows robots.txt, so a correctly written Cloudflare AutoRAG group is enough to control it. Not tracked in our robots.txt study. - Cloudflare Crawler (Browser Run) (
CloudflareBrowserRenderingCrawler, Tier 3): a utility crawler that harvests pages for a dataset or data product. Rated safe. Cloudflare Crawler (Browser Run)'s operator says it follows robots.txt, so a correctly written Cloudflare Crawler (Browser Run) group is enough to control it. Not tracked in our robots.txt study.
Common mistakes with Cloudflare's crawlers
The errors we see most often with Cloudflare's crawlers in the files and firewalls we test:
- Blocking Cloudflare's crawlers while trying to stop AI training. Retrieval crawlers fetch pages so answers can cite them; blocking it can remove you from those answers without affecting training.
- Letting a firewall or bot-fight mode refuse Cloudflare's crawlers while robots.txt allows it. The file looks right, but the crawler still gets a 403 or a challenge page.
- Forgetting that once a
User-agent: Cloudflare-AI-Searchgroup exists, Cloudflare's crawlers ignores every rule underUser-agent: *.
All three hide inside a file that looks correct, so check Cloudflare's crawlers from outside after every change.
Spotting Cloudflare in your logs
robots.txt says what Cloudflare AI Search and Cloudflare AI Search (external sites) and Cloudflare AutoRAG and Cloudflare Crawler (Browser Run) may do; your access logs show what they actually did.
Count requests
On a server with standard Nginx or Apache logs, this counts matching requests per day (adjust the log path for your host):
grep -iE "Cloudflare-AI-Search|Cloudflare-AI-Search-External|Cloudflare-AutoRAG|CloudflareBrowserRenderingCrawler" /var/log/nginx/access.log \
| awk '{print $4}' | cut -d: -f1 | sort | uniq -cAnd this shows which paths it requested most, which is the quickest way to see whether your Disallow lines are being respected:
grep -iE "Cloudflare-AI-Search|Cloudflare-AI-Search-External|Cloudflare-AutoRAG|CloudflareBrowserRenderingCrawler" /var/log/nginx/access.log \
| awk '{print $7}' | sort | uniq -c | sort -rn | head -20Check the requests are genuine
Anyone can send these user-agents, so confirm the source before acting on a match:
Cloudflare-AI-Search: we hold no published IP list, so use reverse DNS where Cloudflare documents a hostname and treat an unverifiable Cloudflare AI Search request with caution. Check it separately from Cloudflare AI Search (external sites) and Cloudflare AutoRAG and Cloudflare Crawler (Browser Run).Cloudflare-AI-Search-External: we hold no published IP list, so use reverse DNS where Cloudflare documents a hostname and treat an unverifiable Cloudflare AI Search (external sites) request with caution. Check it separately from Cloudflare AI Search and Cloudflare AutoRAG and Cloudflare Crawler (Browser Run).Cloudflare-AutoRAG: we hold no published IP list, so use reverse DNS where Cloudflare documents a hostname and treat an unverifiable Cloudflare AutoRAG request with caution. Check it separately from Cloudflare AI Search and Cloudflare AI Search (external sites) and Cloudflare Crawler (Browser Run).CloudflareBrowserRenderingCrawler: we hold no published IP list, so use reverse DNS where Cloudflare documents a hostname and treat an unverifiable Cloudflare Crawler (Browser Run) request with caution. Check it separately from Cloudflare AI Search and Cloudflare AI Search (external sites) and Cloudflare AutoRAG.
What our own data shows about Cloudflare
Everything below comes from the AI Crawler Check bot database and our two most recent robots.txt scans of the Tranco top 10,000, not from Cloudflare's marketing. It covers 4 Cloudflare tokens in turn.
Cloudflare-AI-Search in our records
We first recorded Cloudflare-AI-Search on October 19, 2026. Cloudflare documents it on developers.cloudflare.com, and we recheck that page before each release. No file in our top-10,000 scans names Cloudflare-AI-Search, so any rule you write for it puts you ahead of almost every large site.
Among the 104 AI and LLM crawlers in our database, Cloudflare-AI-Search sits 31th by priority (Tier 2), level with AddSearchBot, AI2Bot, AI2Bot-DeepResearchEval, aiHitBot. 81 of the 104 share its safe rating. Other operators with a crawler that builds a search or answer index: AddSearchBot (AddSearch, Tier 2), AIWebIndex (Lyrenth, Tier 2), atlassian-bot (Atlassian, Tier 2), AzureAI-SearchBot (Microsoft, Tier 2), ExaBot (Exa, Tier 2). A policy for Cloudflare-AI-Search is usually a policy for these too.
A decision record for Cloudflare-AI-Search
Teams that review crawlers once a quarter keep one short entry per token, so anyone can see why a rule exists and when it was last checked. Filled in from our data, the Cloudflare-AI-Search entry reads:
- Operator and job: Cloudflare; it builds a search or answer index.
- Priority and behaviour: Tier 2, rated safe, blocking impact low.
- Suggested starting point: Allow, and watch it in your logs.
- Enforcement: Cloudflare-AI-Search's operator says it follows robots.txt, so a correctly written Cloudflare-AI-Search group is enough to control it.
User-agent: Cloudflare-AI-Search
Allow: /Paste the line into the validator with one of your own URLs before it goes live. Revisit the Cloudflare-AI-Search entry whenever Cloudflare announces a change, because a Tier 2 crawler affects visibility directly.
Cloudflare-AI-Search-External in our records
We first recorded Cloudflare-AI-Search-External on October 19, 2026. Cloudflare documents it on developers.cloudflare.com, and we recheck that page before each release. No file in our top-10,000 scans names Cloudflare-AI-Search-External, so any rule you write for it puts you ahead of almost every large site.
Among the 104 AI and LLM crawlers in our database, Cloudflare-AI-Search-External sits 94th by priority (Tier 3), level with AIWebIndex-Agent, alphalens-bot, amazon-QBusiness, Anomura. 81 of the 104 share its safe rating. Other operators with a crawler that builds a search or answer index: AddSearchBot (AddSearch, Tier 2), AIWebIndex (Lyrenth, Tier 2), atlassian-bot (Atlassian, Tier 2), AzureAI-SearchBot (Microsoft, Tier 2), ExaBot (Exa, Tier 2). A policy for Cloudflare-AI-Search-External is usually a policy for these too.
Cloudflare-AutoRAG in our records
We first recorded Cloudflare-AutoRAG on June 18, 2026. Cloudflare documents it on developers.cloudflare.com, and we recheck that page before each release. No file in our top-10,000 scans names Cloudflare-AutoRAG, so any rule you write for it puts you ahead of almost every large site.
Among the 104 AI and LLM crawlers in our database, Cloudflare-AutoRAG sits 32th by priority (Tier 2), level with AddSearchBot, AI2Bot, AI2Bot-DeepResearchEval, aiHitBot. 81 of the 104 share its safe rating. Other operators with a crawler that builds a search or answer index: AddSearchBot (AddSearch, Tier 2), AIWebIndex (Lyrenth, Tier 2), atlassian-bot (Atlassian, Tier 2), AzureAI-SearchBot (Microsoft, Tier 2), ExaBot (Exa, Tier 2). A policy for Cloudflare-AutoRAG is usually a policy for these too.
CloudflareBrowserRenderingCrawler in our records
We first recorded CloudflareBrowserRenderingCrawler on October 19, 2026. Cloudflare documents it on developers.cloudflare.com, and we recheck that page before each release. No file in our top-10,000 scans names CloudflareBrowserRenderingCrawler, so any rule you write for it puts you ahead of almost every large site.
Among the 104 AI and LLM crawlers in our database, CloudflareBrowserRenderingCrawler sits 95th by priority (Tier 3), level with AIWebIndex-Agent, alphalens-bot, amazon-QBusiness, Anomura. 81 of the 104 share its safe rating. Other operators with a crawler that harvests pages for a dataset or data product: FirecrawlAgent (Firecrawl, Tier 2), iaskspider (iAsk, Tier 2), NovaAct (Amazon, Tier 2), Amazon Kendra (Amazon, Tier 6), ApifyBot (Apify, Tier 6). A policy for CloudflareBrowserRenderingCrawler is usually a policy for these too.
How top sites treat Cloudflare's crawlers
From our robots.txt study of the Tranco top 10,000 (5,583 valid files, fetched October 20, 2026):
None of the tokens for Cloudflare's crawlers is tracked by name in our study. For scale, 20.1% of top-site files block at least one AI training crawler and 4.3% refuse every unnamed crawler with User-agent: * and Disallow: /, which would catch Cloudflare's crawlers too.
More questions about Cloudflare
- Will blocking Cloudflare affect my Google rankings?
- Only if one of its tokens is a Google search crawler. None of Cloudflare's tokens are Googlebot, so classic Google rankings are unaffected.
- How quickly does a robots.txt change take effect for Cloudflare?
- Crawlers cache robots.txt, commonly for up to a day. Expect a change to apply within about 24 hours, and confirm it in your logs.
- Can I allow Cloudflare on some sections and block it on others?
- Yes. Inside the group for
Cloudflare-AI-Search, addDisallowlines for the paths to keep private andAllowlines for exceptions. The longest matching rule wins. - Does blocking Cloudflare delete content it already collected?
- No. robots.txt only controls future visits. For content already collected, check whether Cloudflare offers a removal or opt-out process.
- Is a firewall block better than robots.txt for Cloudflare?
- A firewall enforces the decision, while robots.txt asks politely. Many operators recommend robots.txt because blocking their IPs can stop them reading your rules at all. Use robots.txt first, and add a firewall rule only if requests continue.
A five-minute Cloudflare review
- Confirm the token. Compare the user-agent in your logs with the tokens listed on this page. A close spelling is not the same crawler.
- Decide the outcome per crawler. Use the briefs above to choose allow, limit or block for each token.
- Write the group. Add or edit one
User-agentgroup per decision and repeat any shared rules inside it. - Publish and wait. Allow about a day for crawlers to fetch the new file.
- Confirm the result. Check your logs, then run the AI crawler tester to see the outcome from outside, including any firewall layer.
How to block or allow Cloudflare in robots.txt
These rules cover all 4 crawlers Cloudflare operates. Add them to the robots.txt file at the root of your domain.
Block Cloudflare completely
User-agent: Cloudflare-AI-Search
User-agent: Cloudflare-AI-Search-External
User-agent: Cloudflare-AutoRAG
User-agent: CloudflareBrowserRenderingCrawler
Disallow: /
Allow Cloudflare explicitly
User-agent: Cloudflare-AI-Search
User-agent: Cloudflare-AI-Search-External
User-agent: Cloudflare-AutoRAG
User-agent: CloudflareBrowserRenderingCrawler
Allow: /
These Cloudflare rules ask; they do not enforce. If a Cloudflare token ignores them, block it at the server or CDN. Paths are case-sensitive, and each rule matches the token, not the full user-agent string.
Frequently asked questions about Cloudflare
What crawlers does Cloudflare run?
Cloudflare operates 4 crawlers in our directory: Cloudflare AI Search (user-agent Cloudflare-AI-Search), Cloudflare AI Search (external sites) (user-agent Cloudflare-AI-Search-External), Cloudflare AutoRAG (user-agent Cloudflare-AutoRAG), Cloudflare Crawler (Browser Run) (user-agent CloudflareBrowserRenderingCrawler).
Should I block Cloudflare?
Allow it if you want your pages in Cloudflare AI Search indexes; block it if you do not. That guidance follows this operator's least reassuring crawler, Cloudflare AI Search, rated safe, because allowing an operator allows every crawler it runs. The SEO impact score for blocking is 1/10, where 10 means blocking costs you the most visibility.
How do I block Cloudflare in robots.txt?
Add the following to robots.txt at your domain root: User-agent: Cloudflare-AI-Search then User-agent: Cloudflare-AI-Search-External then User-agent: Cloudflare-AutoRAG then User-agent: CloudflareBrowserRenderingCrawler then Disallow: /. One Disallow line covers all 4 user-agent tokens.
Does blocking Cloudflare hurt my search rankings?
Every crawler here carries the same blocking impact: Low - No effect on Google or Bing rankings. Google's own ranking crawler is Googlebot, which this operator does not run, so blocking Cloudflare does not by itself remove you from Google's index. What it can change is whether AI assistants and analytics tools can read your pages.