AI Crawl-to-Referral Ratios: What They Mean and How to Measure Your Own

Every AI company that crawls your site takes something: bandwidth, server time and, above all, your content. The question site owners keep asking is what they get back. The cleanest single number for that is the crawl-to-referral ratio: how many pages a company's bots fetch for every visitor its products send to you. Cloudflare made the metric popular in 2025, and its numbers were striking. In one week of June 2025, Anthropic's ratio was about 70,900 to 1.

Industry averages are interesting, but they say nothing about your site. This guide explains how the metric works, what its blind spots are, and how to calculate your own ratio from server logs and analytics in under an hour, so your crawler policy rests on your data rather than a headline.

How the metric works

Cloudflare's crawl-to-refer method post defines the ratio precisely. For each platform, it counts requests for HTML pages (responses with Content-Type: text/html) from user agents associated with that platform, and divides that by the number of HTML page requests whose Referer header contained a hostname associated with the same platform. The result is normalised to one referral, so "38,000:1" means 38,000 bot page requests for every human visit sent back.

The Referer part is the key. When someone clicks a link inside an AI answer, their browser usually tells your server where the click came from, for example chatgpt.com or perplexity.ai. That is how both Cloudflare and your own analytics can attribute the visit.

What the published numbers showed

These figures come from Cloudflare's crawl-to-refer method post and Cloudflare's follow-up analysis. They are snapshots from 2025 across Cloudflare's network, not current figures for any single site. Cloudflare publishes an ongoing view on its Radar service; check it before you quote a number.

The blind spots

The metric is useful, but it is easy to read too much into it. Four limits matter.

1. Missing Referer headers. Cloudflare says traffic from Claude's native app does not include a Referer, and believes the same holds for other native apps. Those visits look like direct traffic, so the ratio is overstated by an unknown amount.
2. Mixed crawler purposes. A company's total includes training crawlers, search crawlers and user fetchers. Training crawlers never send visitors by design, so a company that trains heavily will always look worse.
3. Value is not just clicks. Being cited in an AI answer can build brand awareness and later direct visits that no Referer will ever show.
4. Averages hide your site. A network-wide ratio says nothing about whether a given company sends you visitors. Some sites get far more AI referrals than average; many get almost none.

How to calculate your own ratio

You need two numbers per company for the same period, ideally 30 days: bot page requests from your server logs, and referred visits from your analytics. Here is the method.

Step 1: count bot page requests

Filter your access logs to successful HTML responses (status 200, HTML content type, or simply paths that are pages rather than images and scripts). Then count requests whose user agent contains each company's tokens. Group them by job, because you will want to treat them differently.

# Requests for pages (not assets) that returned 200, by OpenAI token
grep -E '"GET [^ ]*(/|\.html)[ ?]' access.log | grep '" 200 ' \
  | grep -oE 'GPTBot|OAI-SearchBot|ChatGPT-User' | sort | uniq -c

Verify the IPs before you trust the counts. A share of "GPTBot" traffic on most sites is someone else using the name. Check addresses against each company's published lists, as explained in verifying AI bots with reverse DNS, or paste a sample into our AI agent log checker, which does the matching for you.

Step 2: count referred visits

In your analytics, filter sessions by referrer or source to the company's hostnames for the same period. Typical sources include chatgpt.com, claude.ai, perplexity.ai and gemini.google.com. Google's AI Overviews and AI Mode clicks arrive as Google search traffic, so you cannot separate them this way; use the Search Console generative AI report for impressions instead. Our guide to measuring AI referral traffic lists the sources and filters.

Step 3: divide, three ways

Calculate three ratios per company, because each answers a different question.

• All bots to referrals. Comparable with Cloudflare's published figures.
• Search crawler plus fetcher to referrals. The "cost of being findable". This is the ratio that matters for AI search decisions.
• Training crawler requests alone. There is no referral side; this is pure cost, and the decision is about rights, not traffic.

A worked example

The figures below are illustrative, for a mid-sized publisher over 30 days. They are not real data, but the arithmetic is exactly what you would do.

Read it like this. Company A looks expensive overall, but almost all of that is training. Its search side costs three page requests per visitor, which is cheap. Blocking its training crawler and keeping its search crawler would cut its load by more than 90% and keep the visitors. Company B is expensive even on the search side; it may be worth allowing only the fetcher and reviewing again next quarter. Company C does no separate training crawl on this site, and its ratio is good either way.

What a ratio cannot see: the value of a citation

A referral is the easiest benefit to count, but not the only one. When an AI answer names your brand, recommends your product or quotes your data, some readers never click. They remember the name and search for it later, or type your address directly. Those visits arrive as branded search or direct traffic, and no crawl-to-referral ratio will ever credit them to the AI company.

You can get a rough sense of this effect in three ways. Watch branded search impressions in Search Console over the months after you start appearing in AI answers. Ask new customers how they heard of you, and add "an AI assistant" as an option. And test a handful of prompts in each assistant monthly to see whether you are named. None of these is precise, but together they stop you from blocking a company that is building your brand even when it sends few clicks. Our guide to AI visibility KPIs sets out a simple scorecard.

Server cost: when the ratio becomes an operations problem

For most sites, AI crawling is a policy question. For some, it becomes an operations problem: a crawler fetches tens of thousands of pages a day, faceted navigation multiplies the URLs, and the hosting bill or response times suffer. In that case the ratio tells you which company to talk to first, but the fix is technical.

• Cut the crawlable URL space. Disallow filter and sort parameters that create endless near-duplicate pages. That reduces load from every crawler at once.
• Serve cheap responses. Make sure bot requests hit your cache. A cached HTML page costs almost nothing to serve.
• Rate limit by verified identity. Limit requests per minute for a verified crawler rather than blocking it outright.
• Block only what you must. If one training crawler causes most of the load, a single robots.txt group fixes it without touching AI search.

Turning the ratio into a policy

A ratio is only useful if it changes a decision. These rules of thumb work for most sites.

Two cautions. Do not block a search crawler because the ratio looks bad in one month; AI search use is growing fast, and referrals tend to lag crawling. And do not use the ratio to judge user fetchers too harshly: each fetch answers a real person's question, even when the app does not send a Referer.

Reading Cloudflare's numbers fairly

Headlines built on these ratios often say an AI company "takes 38,000 pages for every visitor". That is accurate for the measured period and network, but three points keep the comparison fair. First, Google's ratio is not comparable on its face, because Googlebot feeds both classic search and AI features, while referrals from Google include every kind of search click. Second, a company with a new search product starts with a high ratio and improves as users adopt it; Anthropic's fall from 286,000:1 to 38,000:1 in six months is that effect in action. Third, the ratio says nothing about whether the crawling followed your rules. A company with a high ratio that honours your robots.txt is easier to manage than one with a low ratio that does not. Use the ratio as one input next to compliance, verification and the value of citations, not as a verdict on its own.

Before you start: two quick checks

Two things distort a home-made ratio more than anything else. One is a crawl block you did not know about: if a firewall already turns away a company's search crawler, your ratio will look clean simply because nothing gets through. Run AI Crawler Check first to see which crawlers can actually reach you. The other is analytics that drops referrers, for example because of a strict referrer policy on a redirect page or a consent banner that blocks tracking until a click. Check that a test visit from chatgpt.com shows up with the right source before you trust a month of data.

Make it a monthly number

• □Export 30 days of access logs and analytics referral data on the same date range.
• □Count verified page requests per token and group them by job.
• □Count referred sessions per company hostname.
• □Calculate the three ratios per company and keep a simple table month by month.
• □Re-check crawl access with AI Crawler Check after any robots.txt or CDN change, so a block does not show up as a sudden "improvement".
• □Review the policy each quarter, and add the numbers to your AI visibility KPIs dashboard.
• □Fold the review into your monthly AI crawler audit.