AI Crawler Rate Limiting: What Each Bot Honors and How to Set Limits (2026)

Rate limiting lets an AI crawler visit your site without letting it take over your server. It is often a better answer than a full block: you keep the bots that can cite you, you cap the ones that mostly take, and you protect real visitors from slowdowns. The hard part is that each crawler responds to different signals. Some read Crawl-delay in robots.txt, some ignore it, and some only slow down when your server returns errors.

This guide shows which crawlers honor which controls, according to their own documentation, how to measure before you throttle, and how to set limits in robots.txt, Nginx, Apache and Cloudflare without hurting AI search. It also uses our study of the top 10,000 sites to show what others do. Before changing anything, run the free AI crawler checker so you know which bots can reach you today.

AI crawler analytics: measure before you throttle

AI crawler analytics tells you which bots are costing you bandwidth before you rate limit any of them. Count requests and bytes per AI user agent from your server logs over a week, note the status codes each one gets, then compare that with the visitors each platform sends you. Throttle only the bots that are both heavy and not sending you citations. Crawl-delay is ignored by several major crawlers, so see crawl delay for AI bots for what each one actually reads.

awk -F'"' '{ua=$6; split($3,a," "); b=a[2]}
  ua ~ /GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|PerplexityBot|Bytespider|Meta-ExternalAgent|Amazonbot|CCBot/ {
    match(ua,/GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|PerplexityBot|Bytespider|Meta-ExternalAgent|Amazonbot|CCBot/)
    k=substr(ua,RSTART,RLENGTH); n[k]++; s[k]+=b }
  END { for (k in n) printf "%-20s %8d req %9.1f MB\n", k, n[k], s[k]/1048576 }' /var/log/nginx/access.log

grep "Meta-ExternalAgent" /var/log/nginx/access.log | awk '{print substr($4,2,17)}' \
  | sort | uniq -c | sort -rn | head -5

The second command shows the peak requests per minute, which is the number your rate limit needs to handle. Our AI bot monitoring guide explains how to verify each bot's IP and how to add the referral side from Google Analytics.

Cloudflare's network data gives useful context for the "worth it" question. In July 2025 it measured how many pages each operator crawled for every visitor it referred: Anthropic about 38,065, OpenAI about 1,091, Perplexity about 195, Microsoft about 41 and Google about 5 (Cloudflare, August 2025). Training crawling made up nearly 80% of AI bot activity. Those are the requests a rate limit can trim with little loss.

Which crawlers honor Crawl-delay

Crawl-delay is not part of the robots.txt standard, RFC 9309, so each operator decides. Here is what each one says in its own documentation.

The rule of thumb: if an operator does not document Crawl-delay, assume it does not read it, and enforce the limit somewhere it cannot ignore.

# Anthropic documents Crawl-delay support
User-agent: ClaudeBot
Crawl-delay: 5

# Semrush caps at 10 seconds
User-agent: SemrushBot
Crawl-delay: 10

User-agent: bingbot
Crawl-delay: 2

What the top 10,000 sites set

In our study, 573 of 4,962 valid robots.txt files (11.5%) contain at least one Crawl-delay line. These are the values they use most.

Ten seconds wins by a wide margin. At one request every ten seconds a crawler can fetch at most 8,640 URLs a day. For a site with 500 pages that is plenty; for a store with 200,000 product URLs, it means a full recrawl takes more than three weeks. Pick a value from your page count, not from habit. We also found 23 files that set Crawl-delay for Googlebot, which Google ignores.

Server-level rate limiting with Nginx

Nginx's limit_req module counts requests per key and rejects the excess. The trick for bots is the key: Nginx does not count requests whose key is empty. So you map the user agent to a key that is empty for normal visitors and set for the bots you want to slow down.

# http {} block
map $http_user_agent $ai_train_key {
    default                                                   "";
    ~*(GPTBot|ClaudeBot|Meta-ExternalAgent|Bytespider|CCBot|Amazonbot)  $binary_remote_addr;
}
limit_req_zone $ai_train_key zone=ai_train:10m rate=30r/m;
limit_req_status 429;

# server {} block
location / {
    limit_req zone=ai_train burst=20 nodelay;
    # ... your normal config
}

This allows each training crawler IP 30 requests a minute with a burst of 20, and returns 429 beyond that. Search and user-fetch agents such as OAI-SearchBot, Claude-SearchBot, ChatGPT-User and PerplexityBot are not in the map, so their key is empty and they are never limited by this zone.

Add a Retry-After header so polite crawlers know when to come back:

error_page 429 = @too_many;
location @too_many {
    add_header Retry-After 120 always;
    return 429;
}

Rate by IP has a weakness: large crawlers spread requests over many addresses. If one bot still overloads you, key the zone on the bot name instead, so all its IPs share one budget. Replace $binary_remote_addr in the map with a fixed string per bot, for example "gptbot".

Apache and other servers

Apache has no built-in per-user-agent rate limiter as simple as Nginx's, but modules such as mod_evasive, mod_qos or mod_security can do it. If you cannot add a module, a blunt but effective option is to return 429 to a crawler during your peak hours:

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (Meta-ExternalAgent|Bytespider) [NC]
RewriteCond %{TIME_HOUR} >08
RewriteCond %{TIME_HOUR} <18
RewriteRule .* - [R=429,L]

On managed hosting, ask your host first. Many have rate limiting at their edge already and can apply it per user agent for you.

Cloudflare and other CDNs

A CDN rate limit stops traffic before it reaches your server, which is where you want it. On Cloudflare, create a rate limiting rule with an expression that matches the bots you want to slow down, a request threshold per period, and an action of Block or Managed Challenge for a short time.

(http.user_agent contains "GPTBot") or
(http.user_agent contains "Meta-ExternalAgent") or
(http.user_agent contains "Bytespider")

Be careful with Cloudflare's broader AI settings. On 1 July 2026 Cloudflare split AI traffic into Search, Agent and Training, and from 15 September 2026 new domains block Training and Agent traffic by default on pages that show ads. Cloudflare warned that multi-purpose crawlers "such as Googlebot, Applebot, and BingBot will be blocked by customers who have selected to block Training" (Cloudflare, 1 July 2026). A rate limit is often safer than that switch, because it slows a crawler instead of removing it.

How to pick the right limit for your site

A rate limit that is too tight blocks useful crawling. One that is too loose does nothing. You can work out a sensible number from three facts about your own site.

1. How many URLs you want crawled. Count the pages in your sitemap, say 20,000.
2. How often they should be refreshed. If a full pass every 7 days is enough, each crawler needs about 20,000 / 7 = 2,857 requests a day.
3. What your server can spare. Load-test or check your monitoring. If the server stays healthy up to 10 extra requests per second, that is 864,000 a day in total for all bots.

In this example, 2,857 requests a day is about 2 requests a minute, on average. Crawlers are bursty, so allow a few times the average. A limit of 10 requests a minute with a burst of 20 lets a training crawler finish its weekly pass easily while keeping its peak far below what your server can take. For a 500-page brochure site, even 2 requests a minute is generous.

These are starting points, not rules. Raise the limit if your logs show a crawler never finishing a pass; lower it if peak minutes still cause slow pages for people. Revisit the numbers whenever your site grows a lot or you move hosts.

No server access? Options for WordPress and managed hosting

Many site owners cannot edit an Nginx config. You still have options, from easiest to strongest.

• robots.txt Crawl-delay for the bots that read it: ClaudeBot, Bingbot and SemrushBot. Most CMSs and SEO plugins let you edit robots.txt from the dashboard.
• Your CDN. If your site uses Cloudflare, even the free plan has rate limiting rules and AI traffic settings, and they act before your host is touched.
• A security plugin with rate limiting by user agent. Test carefully: many such plugins block all "bots" by default, including the search bots you need.
• Your host's support team. Managed WordPress and e-commerce hosts often have edge rules and can rate limit one user agent for you.
• Caching. A full-page cache plugin makes each bot request cheap, which often solves the problem without any limit.

Whatever you choose, check the result with the AI crawler diagnostic from outside your network. Plugin rules sometimes block bots on some URLs and not others.

Common mistakes

Slowing down Googlebot the right way

Google ignores Crawl-delay, so the only fast way to slow Googlebot is to return errors. Google's own guidance: "return 500, 503, or 429 HTTP response status code instead of 200 to the crawl requests", for "a couple of hours, or 1-2 days" (Google, crawl rate guide). Its infrastructure then lowers the crawl rate for the whole hostname and raises it again when the errors stop.

Do not do this for long. Errors that last days can cause pages to drop from the index. If the cause is structural, such as faceted navigation creating millions of URLs, fix that instead; Google names it as one of the most common reasons for a crawling spike. Because Gemini and AI Overviews use Googlebot, anything that slows Googlebot slows Google's AI features too.

Limits that keep you visible in AI search

The biggest risk of rate limiting is hitting the wrong bots. Use three tiers.

User-triggered agents arrive in bursts when a page becomes popular, which can look like an attack. Each request, though, is one person. OpenAI says ChatGPT-User "is not used for crawling the web in an automatic fashion".

Reduce crawl load without any limit

Some of the best rate limiting is making crawling cheaper. These changes help every bot at once:

• Fix 404s and redirect chains. Vercel and MERJ found OpenAI's crawler spent 34.82% of its fetches on 404 pages and 14.36% on redirects in their sample (Vercel and MERJ).
• Block infinite URL spaces such as calendar pages, internal search results and filter combinations in robots.txt.
• Cache HTML for bots. A cached page costs almost nothing to serve, whoever asks for it.
• Support conditional requests. ETag and Last-Modified let well-built crawlers skip unchanged pages.
• Keep your sitemap accurate with real lastmod dates, so crawlers can prioritize changed pages.

Where AI crawler traffic control is heading

Three changes are making rate limits smarter. First, purpose-based controls: Cloudflare now groups AI traffic as Search, Agent and Training, so you can treat each class differently instead of naming every bot. Second, signed requests: OpenAI's Cloud browser signs its requests with Web Bot Auth, based on RFC 9421 (OpenAI Help). A signature lets you give a verified agent a generous limit and a stranger using the same name a strict one. Third, preferences in robots.txt: Cloudflare's Content-Signal line and the IETF AI Preferences work let you say how content may be used, not just whether it may be fetched.

None of these replaces a rate limit yet. They make it easier to aim one at the right traffic.

How to test your limits

1. Deploy on staging first and send a burst of requests with each bot's user agent.
2. Confirm training crawler requests get 429 after the threshold.
3. Confirm search and user-fetch agents still get 200. The AI crawler diagnostic fetches pages as each bot.
4. Deploy to production, then watch 429 counts per bot in your logs for a week.
5. Re-run the AI crawler checker to make sure robots.txt and your new rules agree.

for i in $(seq 1 60); do
  curl -s -o /dev/null -w "%{http_code} " -A "Mozilla/5.0 (compatible; GPTBot/1.4)" https://staging.example.com/
done; echo

Checklist

• □You measured requests, bytes and peak minute per bot for a week.
• □Crawl-delay is set only for crawlers that document support for it.
• □Bots with their own Crawl-delay group still get the Disallow rules they need.
• □Server or CDN limits cover the training crawlers that ignore Crawl-delay.
• □Search bots and user-triggered agents are excluded from strict limits.
• □429 responses include a Retry-After header.
• □Your CDN's AI training setting does not also block Googlebot or Bingbot.
• □You fixed 404s and infinite URL spaces to cut crawl cost for everyone.

Sources and further reading

Every claim above links to the page it came from. These are the primary sources, all read or re-read on 5 October 2026. Where we give a figure from our own data, it comes from our robots.txt study of the top 10,000 sites, and the raw counts are in the downloadable CSV.

1. Anthropic: Does Anthropic crawl data from the web? (7 April 2026)
2. Semrush: SemrushBot
3. Bing Webmaster Blog: To crawl or not to crawl, that is BingBot's question (May 2012)
4. Apple: About Applebot
5. Google Search Central: How Google interprets the robots.txt specification
6. Google Search Central: Reduce the Google crawl rate
7. OpenAI: Overview of OpenAI crawlers
8. RFC 9309, Robots Exclusion Protocol (IETF, September 2022)
9. Cloudflare: The crawl-to-click gap (29 August 2025)
10. Cloudflare: Content Independence Day, new AI options (1 July 2026)
11. OpenAI Help: ChatGPT Work's Cloud browser allowlisting (Web Bot Auth)
12. Vercel and MERJ: The rise of the AI crawler (December 2024)
13. Tranco: a research-oriented top sites ranking