Cloudflare AutoRAG: User Agent, robots.txt Rules & Safety

Q: What user-agent string does Cloudflare AutoRAG use?

Cloudflare AutoRAG is reported to identify itself with Cloudflare-AutoRAG. That spelling comes from ai.robots.txt, not from Cloudflare, which has not published crawler documentation naming it. Use it in robots.txt as the best available match, and treat a request carrying it as a clue rather than proof of who sent it.

Q: Is Cloudflare AutoRAG safe?

Cloudflare AutoRAG is rated Safe in this directory, meaning there are no credible public reports of abusive crawling under this name. It is a Reported token: Cloudflare has not published documentation for it, so its purpose, crawl rate and robots.txt behaviour are not confirmed by the operator, and anyone can send the same user-agent string. Allow or block it on its stated purpose, and back the robots.txt rule with a firewall rule if the traffic matters to you.

Q: Will blocking Cloudflare AutoRAG hurt my SEO?

✅ No SEO Impact - Blocking Cloudflare AutoRAG does not affect your rankings in Google, Bing, or any other search engine. Cloudflare AutoRAG is an AI crawler, not a traditional search indexer. You can freely block it via User-agent / Disallow: / without any SEO penalty.

Q: How do I block Cloudflare AutoRAG in robots.txt?

Add the following lines to your /robots.txt file: User-agent: Cloudflare-AutoRAG Disallow: / This instructs Cloudflare AutoRAG not to crawl any path on your site. To block only specific sections, replace / with the path (e.g., Disallow: /blog/).

Q: How do I verify if Cloudflare AutoRAG is crawling my site?

Search your access logs for Cloudflare-AutoRAG (for example grep -i "Cloudflare-AutoRAG" /var/log/nginx/access.log). Because the operator publishes no IP ranges or reverse-DNS pattern, a match shows that something used the name, not that Cloudflare sent it.

Quick Facts

User-Agent:: Cloudflare-AutoRAG
Category:: AI & LLM Bots
Operator:: Cloudflare
Safety:: Safe
Blocking Impact:: Low - No SEO ranking impact
SEO Impact Score:: 2/10
Official docs:: None published by Cloudflare
Named by:: ai.robots.txt

Reported, not operator documented

Cloudflare has not published a crawler page that names Cloudflare-AutoRAG. It is listed here because a credible public source does: ai.robots.txt. Cloudflare AutoRAG (now AI Search) website crawler; the Cloudflare docs describe the product without printing the UA.

What that means for you: the robots.txt rule on this page is the best available spelling, and a well-behaved crawler using this string will honour it, but there is no vendor statement of purpose, crawl rate or robots.txt compliance to rely on. Treat a matching request in your logs as a detection clue, not proof of who sent it, and combine the rule with a firewall rule if the traffic matters to you. If the operator publishes documentation, this profile moves to Official docs.

What is Cloudflare AutoRAG?

Cloudflare AutoRAG is an AI search-index crawler operated by Cloudflare. It collects and indexes web content so it can be surfaced as citations or answers inside Cloudflare's AI search experience. Allowing Cloudflare AutoRAG can increase your visibility in AI-powered search results, which is why most sites choose to permit it.

Cloudflare AutoRAG is an AI search-index crawler operated by Cloudflare. It collects and indexes web content so it can be surfaced as citations or answers inside Cloudflare's AI search experience. Allowing Cloudflare AutoRAG can increase your visibility in AI-powered search results, which is why most sites choose to permit it.
Cloudflare AutoRAG uses the user-agent token Cloudflare-AutoRAG. You can control it via robots.txt, meta tags (noai), or the emerging llms.txt standard. Robots.txt is voluntary; for hard enforcement, combine it with server-level IP blocking.

What happens if you block Cloudflare AutoRAG?

✅ No SEO Impact - Blocking Cloudflare AutoRAG does not affect your rankings in Google, Bing, or any other search engine. Cloudflare AutoRAG is an AI crawler, not a traditional search indexer. You can freely block it via User-agent / Disallow: / without any SEO penalty.

Generally safe to allow; provides legitimate crawling value.

How to block Cloudflare AutoRAG with robots.txt

<code>User-agent: Cloudflare-AutoRAG</code> - Matching is case-insensitive. Robots.txt is fetched from the root of each subdomain separately.

Block completely (robots.txt)

User-agent: Cloudflare-AutoRAG
Disallow: /

Allow all (robots.txt)

User-agent: Cloudflare-AutoRAG
Allow: /

Block private only (robots.txt)

User-agent: Cloudflare-AutoRAG
Disallow: /private/
Disallow: /api/
Disallow: /admin/
Allow: /

Nginx server block

# Nginx: Hard-block Cloudflare AutoRAG
if ($http_user_agent ~* "Cloudflare-AutoRAG") {
    return 403 "Bot blocked";
}

Apache .htaccess

# Apache: Hard-block Cloudflare AutoRAG
SetEnvIfNoCase User-Agent "Cloudflare-AutoRAG" bad_bot
Order Allow,Deny
Allow from all
Deny from env=bad_bot

Meta robots tag

X-Robots-Tag header

X-Robots-Tag: noindex, nofollow

Is Cloudflare AutoRAG safe to allow?

Cloudflare AutoRAG is rated Safe in this directory, meaning there are no credible public reports of abusive crawling under this name. It is a Reported token: Cloudflare has not published documentation for it, so its purpose, crawl rate and robots.txt behaviour are not confirmed by the operator, and anyone can send the same user-agent string. Allow or block it on its stated purpose, and back the robots.txt rule with a firewall rule if the traffic matters to you.

Cloudflare publishes no IP ranges or reverse-DNS pattern for Cloudflare AutoRAG, so the user-agent string cannot be verified. Rate-limit unusual volume instead of trusting the name.

What does Cloudflare AutoRAG do?

Understanding Cloudflare AutoRAG's purpose helps you decide whether to allow or block it.

Indexes web pages for AI-powered search answers
Surfaces your content as citations inside AI search results
Builds a knowledge index used to answer user questions
Allowing it can increase visibility in AI search
robots.txt behaviour not documented by the operator

Frequently Asked Questions

What user-agent string does Cloudflare AutoRAG use?

Cloudflare AutoRAG is reported to identify itself with Cloudflare-AutoRAG. That spelling comes from ai.robots.txt, not from Cloudflare, which has not published crawler documentation naming it. Use it in robots.txt as the best available match, and treat a request carrying it as a clue rather than proof of who sent it.

Is Cloudflare AutoRAG safe?

Cloudflare AutoRAG is rated Safe in this directory, meaning there are no credible public reports of abusive crawling under this name. It is a Reported token: Cloudflare has not published documentation for it, so its purpose, crawl rate and robots.txt behaviour are not confirmed by the operator, and anyone can send the same user-agent string. Allow or block it on its stated purpose, and back the robots.txt rule with a firewall rule if the traffic matters to you.

Will blocking Cloudflare AutoRAG hurt my SEO?

✅ No SEO Impact - Blocking Cloudflare AutoRAG does not affect your rankings in Google, Bing, or any other search engine. Cloudflare AutoRAG is an AI crawler, not a traditional search indexer. You can freely block it via User-agent / Disallow: / without any SEO penalty.

How do I block Cloudflare AutoRAG in robots.txt?

Add the following lines to your /robots.txt file:

User-agent: Cloudflare-AutoRAG
Disallow: /

This instructs Cloudflare AutoRAG not to crawl any path on your site. To block only specific sections, replace / with the path (e.g., Disallow: /blog/).

Does Cloudflare AutoRAG respect robots.txt?

Not documented. Cloudflare has published no statement that Cloudflare AutoRAG reads robots.txt. A rule for Cloudflare-AutoRAG will stop a client that chooses to honour it; for enforcement, add a firewall or CDN rule as well.

How do I verify if Cloudflare AutoRAG is crawling my site?

Search your access logs for Cloudflare-AutoRAG (for example grep -i "Cloudflare-AutoRAG" /var/log/nginx/access.log). Because the operator publishes no IP ranges or reverse-DNS pattern, a match shows that something used the name, not that Cloudflare sent it.