How Do I Monitor AI Bot Traffic to My Website? Logs, GA4 and the New AI Reports (2026)

AI bot traffic comes in two kinds, and you need two different tools to see them. The bots themselves (GPTBot, ClaudeBot, OAI-SearchBot and the rest) show up only in your server or CDN logs, because they do not run the JavaScript that analytics tools depend on. The people those bots send you, when someone clicks a citation in ChatGPT, Perplexity or Copilot, show up in Google Analytics as referrals. Most monitoring setups see one kind and miss the other.

This guide shows how to monitor both, from a one-line log command to a monthly report. It uses each AI company's documentation, Google's and Bing's new AI reports, and network data from Cloudflare and Hostinger. If you have not checked which bots can reach your site at all, start with the free AI crawler checker; monitoring a bot you are blocking will only show you zeros.

How do I monitor AI bot traffic to my website?

Monitor AI bot traffic by reading your server access logs, because AI crawlers do not run the JavaScript that Google Analytics depends on. Filter the logs for known tokens such as GPTBot, ClaudeBot, PerplexityBot and OAI-SearchBot, verify the IPs against each operator's published ranges, then compare that crawl volume with the referral visits each AI platform sends you in analytics. Google Search Console and Bing Webmaster Tools add a third view: how often your pages appear in their AI answers.

The methods below go from quickest to most complete. For step-by-step log parsing on different servers, see AI bot traffic in server logs. For alerts that tell you when something changes, see AI crawler monitoring. And if a bot is costing you too much, read AI crawler rate limiting.

Method 1: server access logs

Your web server writes a line for every request. Each line includes the client IP, the time, the URL, the status code, the bytes sent and the user agent. That is everything you need. On Nginx the default file is /var/log/nginx/access.log; on Apache it is often /var/log/apache2/access.log. Managed hosts usually offer a download in the control panel.

Three log commands that answer most questions

Start with these three commands. The first counts requests per AI bot, the second lists the pages one bot reads most, and the third shows the status codes a bot receives. Together they tell you who visits, what they want, and whether they get it. Replace the bot name in the second and third commands with any token from the table below.

grep -oiE "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|Googlebot|bingbot|Applebot|Amazonbot|Bytespider|Meta-ExternalAgent|CCBot" \
  /var/log/nginx/access.log | sort | uniq -c | sort -rn

grep "OAI-SearchBot" /var/log/nginx/access.log | awk '{print $7}' \
  | sort | uniq -c | sort -rn | head -20

grep "ClaudeBot" /var/log/nginx/access.log | awk '{print $9}' | sort | uniq -c

If you see many 403, 429 or 5xx responses for a bot you want, something is blocking it. Our guide to why AI bots cannot crawl a site walks through each layer.

Which user agents to look for

Search for the product token, not the full string, because version numbers change. These are the tokens the major operators document, grouped by what they do.

For anything not on this list, our bot directory has the token, operator and evidence level for 257 bots.

Verify that a bot is really who it says

User agents are easy to fake, and scrapers often pretend to be Googlebot or GPTBot because sites tend to allow them. Before you trust a number, or block an IP, check it against the operator's own list.

import ipaddress, json, urllib.request

def load(url):
    data = json.load(urllib.request.urlopen(url))
    return [ipaddress.ip_network(p.get("ipv4Prefix") or p.get("ipv6Prefix"))
            for p in data.get("prefixes", [])]

nets = load("https://openai.com/searchbot.json")
for line in open("searchbot_ips.txt"):
    ip = ipaddress.ip_address(line.strip())
    print(ip, "OK" if any(ip in n for n in nets) else "NOT OpenAI")

The same pattern works for GPTBot, ChatGPT-User and Perplexity's lists. Anthropic says that "if a crawler has a source IP address on this list, it indicates that the crawler is coming from Anthropic", linking to one list for all three of its bots (Anthropic). For Googlebot, Google offers both a reverse DNS check and JSON lists (Google, verifying crawlers).

Some operators publish nothing to verify against. xAI documents no crawler, and we could not find a ByteDance IP list for Bytespider. For those, treat the user agent as a clue, not proof.

Method 2: Google Analytics 4 for AI referrals

GA4 will not show you AI bots. Google says: "In Google Analytics properties, traffic from known bots and spiders is automatically excluded" and "you cannot disable known bot traffic exclusion or see how much known bot traffic was excluded" (Google Analytics Help). That is good for your human numbers and bad for bot monitoring, which is why logs come first.

What GA4 does show is people. When someone clicks a link in an AI answer, they arrive as a referral. OpenAI says ChatGPT "automatically includes the UTM parameter utm_source=chatgpt.com in referral URLs" (OpenAI publishers FAQ).

1. Go to Reports, then Acquisition, then Traffic acquisition.
2. Set the primary dimension to Session source.
3. Add a filter: Session source matches regex chatgpt|openai|perplexity|copilot|gemini|claude.
4. Add Landing page as a secondary dimension.
5. Save it as a report or an exploration so you can compare months.

.*(chatgpt\.com|chat\.openai\.com|perplexity\.ai|copilot\.microsoft\.com|gemini\.google\.com|claude\.ai).*

Some AI apps open links without a referrer, so part of this traffic lands in Direct. Treat your AI referral number as a floor, not a total.

Method 3: Google and Bing AI visibility reports

Two new reports show how often you appear in AI answers, which neither logs nor GA4 can tell you.

Search Console: Generative AI performance report

Rolled out to all sites on 31 August 2026, this report shows impressions for your pages in AI Overviews and AI Mode, by page, country, device and date (Search Console Help, AI report). An impression means a link to your site was shown in one of those features. It uses the same 1,000-row limit as the main Performance report, and the newest data can be preliminary for a few hours.

Bing Webmaster Tools: AI Performance

Bing launched AI Performance in public preview on 10 February 2026. It shows "how often your content is cited in generative answers" across Microsoft Copilot, AI summaries in Bing and select partners, with total citations, average cited pages per day, page-level citation counts and grounding queries, which are "the key phrases the AI used when retrieving content" (Bing Webmaster Blog). Bing added Intents, Topics, Citation Share and Compare views in June 2026.

Together these reports answer the question logs cannot: is the crawling turning into visibility? A page that search bots fetch every week but that never earns an impression or citation is a content problem, not an access problem.

Method 4: CDN bot analytics

If your site sits behind Cloudflare or a similar CDN, its dashboard may already group bot traffic by category, including AI crawlers, without you touching a log file. CDN numbers are a good quick check, but they may sample, they classify bots in their own way, and they rarely let you export per-page detail. Use them to spot trends, and logs to investigate.

Cloudflare's July 2026 update also changed what it lets through by default. From 15 September 2026, new domains block Training and Agent traffic on pages that show ads (Cloudflare, 1 July 2026). If a bot vanishes from your logs after that date, check the dashboard before you assume the bot stopped coming.

Turn raw logs into three numbers per bot

Counting hits is a start, but it does not tell you whether a bot is worth its cost. For each AI platform, track three numbers every month:

1. Requests from its crawlers, from your logs.
2. Bytes served to those crawlers, from your logs.
3. Visitors its platform sent you, from GA4.

awk -F'"' '{ua=$6; split($3,a," "); bytes=a[2]}
  ua ~ /GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|PerplexityBot|Bytespider|Meta-ExternalAgent|CCBot/ {
    match(ua,/GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|PerplexityBot|Bytespider|Meta-ExternalAgent|CCBot/)
    bot=substr(ua,RSTART,RLENGTH); n[bot]++; b[bot]+=bytes }
  END { for (k in n) printf "%-20s %8d req %10.1f MB\n", k, n[k], b[k]/1048576 }' /var/log/nginx/access.log

Dividing requests by visitors gives you your own crawl-to-refer ratio, the same measure Cloudflare uses for its network. In July 2025 Cloudflare measured about 1,091 crawls per referred visitor for OpenAI, 38,065 for Anthropic and 195 for Perplexity (Cloudflare, August 2025). Your numbers will differ, but the comparison shows which platforms give back and which mostly take.

AI bot traffic benchmarks for 2026

It helps to know what other sites see before you decide something is wrong. Here is what public data says.

• Training dominates AI crawling. Cloudflare found training drove nearly 80% of AI bot activity by July 2025, up from 72% a year earlier, with search at about 17% and user actions about 3% (Cloudflare, August 2025).
• Coverage, not just volume. Hostinger's January 2026 study of 66.7 billion requests found Meta-ExternalAgent on 57.33% of sites, GPTBot on 55.67% and OpenAI's search bot also on 55.67%, with very different request counts (Hostinger).
• Unnamed bots are a big share. Hostinger counted 34.6% of bot requests from scripts, empty user agents and generic bots.
• Blocking is common at the top. In our study of 4,962 top-10,000 robots.txt files, 14.3% block GPTBot, so your competitors may see far fewer training crawler hits than you do.

Read crawls and user fetches differently

Do not add every AI request into one number. A crawler such as GPTBot or ClaudeBot works through your site on its own schedule, so its volume tells you about cost. A user-triggered agent such as ChatGPT-User, Claude-User or Perplexity-User arrives because a person asked about your page, so its volume tells you about demand. OpenAI says ChatGPT-User "is not used for crawling the web in an automatic fashion" (OpenAI crawler docs).

That makes user fetches one of the most useful signals in your logs. A page that gets many ChatGPT-User or Perplexity-User requests is a page people are asking AI assistants about right now. Compare that list with your GA4 landing pages from chatgpt.com and perplexity.ai. Pages that are fetched often but rarely clicked may need a clearer answer near the top, so the person has a reason to visit.

Set up alerts so you do not have to watch

Monitoring is only useful if you notice changes. A simple daily job can compare today's numbers with yesterday's and email you when something jumps.

#!/bin/sh
LOG=/var/log/nginx/access.log
T=$(date +%d/%b/%Y); Y=$(date -d yesterday +%d/%b/%Y)
for bot in GPTBot ClaudeBot Meta-ExternalAgent Bytespider PerplexityBot OAI-SearchBot; do
  t=$(grep "$T" $LOG | grep -c "$bot")
  y=$(grep "$Y" $LOG | grep -c "$bot")
  if [ "$y" -gt 100 ] && [ "$t" -gt $((y*2)) ]; then
    echo "$bot: $t requests today vs $y yesterday" | mail -s "AI bot spike" [email protected]
  fi
done

Alert on drops too. If OAI-SearchBot or Claude-SearchBot suddenly stops visiting, check your robots.txt and CDN before you lose citations. Our AI crawler alerts guide has more ready-made rules.

When monitoring shows a problem

A monthly AI bot report template

Copy this into a spreadsheet and fill it in once a month. After three months you will see trends that a single snapshot hides.

End each report with one decision: allow, rate limit or block, for each bot that changed. Then re-run the AI crawler checker to confirm your settings match the decision.

Privacy and retention

Server logs contain IP addresses, which privacy laws such as the GDPR can treat as personal data when they belong to people. Crawler IPs belong to companies, but your logs mix both. Keep raw logs only as long as you need them, restrict who can read them, and when you share bot reports, share the per-bot totals rather than raw lines. If you use a third-party log tool, check where it stores data. None of this stops you monitoring bots; it just keeps the human visitors in the same file protected.

Checklist

• □You can read raw server or CDN logs for at least 30 days.
• □You count requests and bytes per AI bot each month.
• □You verify bot IPs against each operator's published list.
• □GA4 has an AI assistants segment or channel group.
• □Search Console and Bing Webmaster Tools are verified, and you check their AI reports.
• □You have an alert for spikes and for drops in search bot visits.
• □Each month ends with an allow, limit or block decision per bot.

Sources and further reading

Every claim above links to the page it came from. These are the primary sources, all read or re-read on 5 October 2026. Where we give a figure from our own data, it comes from our robots.txt study of the top 10,000 sites, and the raw counts are in the downloadable CSV.

1. Google Analytics Help: Known bot-traffic exclusion
2. OpenAI: Overview of OpenAI crawlers
3. OpenAI Help: Publishers and developers FAQ
4. Anthropic: Does Anthropic crawl data from the web? (7 April 2026)
5. Perplexity: Perplexity crawlers
6. Google Search Central: Verifying Googlebot and other Google crawlers
7. Search Console Help: Generative AI performance report (31 August 2026)
8. Bing Webmaster Blog: Introducing AI Performance in Bing Webmaster Tools (10 February 2026)
9. Cloudflare: The crawl-to-click gap (29 August 2025)
10. Cloudflare: Content Independence Day, new AI options (1 July 2026)
11. Hostinger: 66 billion bot requests analysis (20 January 2026)
12. Tranco: a research-oriented top sites ranking