Free, runs in your browser

Grok log checker GrokBot and xAI-SearchBot

Paste lines from your server access log to find requests that use the reported xAI crawler names, GrokBot and xAI-SearchBot. See which IPs sent them and how often, so you can tell a steady crawler from someone borrowing the name.

  • 2reported xAI names
  • by IPgrouped results
  • 0lines uploaded

Check log lines

Paste raw access log lines. Everything runs on this page, so your logs never leave your machine.

Tip: grep -iE "grok|xai" /var/log/nginx/access.log | tail -5000 gives you a manageable sample.

What is known about xAI's crawlers, and what is not

xAI runs Grok, but it has not published crawler documentation for the names site owners write rules for, GrokBot and xAI-SearchBot, and it publishes no IP ranges or reverse DNS pattern for them. That is why our directory labels both Reported rather than Official. Site owners still treat them as real: in our robots.txt study of the Tranco top 10,000 (5,583 valid files), 59 files name GrokBot and 6 name xAI-SearchBot, and the top 1,000 sites block GrokBot at 11.4% against 3.8% in the long tail.

Without published ranges nobody can prove a request is genuine. What you can do is read its pattern, which is what this checker is for.

The xAI names at a glance

Names the checker reports
Name in the logWhat it isEvidence
GrokBotReported xAI crawler nameReported: not in xAI documentation
xAI-SearchBotReported xAI search crawler nameReported: not in xAI documentation
GrokAgentGrok Bot, xAI's agent, acting for a userListed in Cloudflare's verified-bot registry
Other xAI or Grok mentionAny user agent that mentions Grok, xAI or x.ai but matches neither crawler nameNone; investigate
TwitterbotX's link-preview fetcherDocumented by X; not Grok

Getting a useful sample

grep -iE "grok|xai" /var/log/nginx/access.log | tail -5000
  1. Run the command on your server, or export matching lines from your hosting dashboard. On Apache the log is usually /var/log/apache2/access.log.
  2. Paste the lines into the checker. They are processed on this page and never uploaded.
  3. Read the summary, then the table by name and IP.

Reading one log line

203.0.113.7 - - [01/Oct/2026:10:00:00 +0000] "GET /pricing HTTP/1.1" 200 5120 "-" "Mozilla/5.0 (compatible; xAI-SearchBot/1.0; +https://x.ai)"
203.0.113.7
The client IP, the first field. The checker groups requests by it.
[01/Oct/2026:10:00:00 +0000]
When the request arrived. Steady spacing over days suggests a crawler; a burst in one minute suggests a script.
"GET /pricing HTTP/1.1" 200
The path and the status you returned. A 403 here means your firewall already refused it.
The last quoted field
The user agent. The checker matches the xAI names here and ignores the rest of the line.

The checker reads Nginx and Apache combined logs, and any other format where the client IP comes first and the user agent is the last quoted field.

Four tests for a borrowed name

  1. How many IPs? A real crawler tends to come from a small, stable set of addresses. Dozens of unrelated IPs using one name is a warning sign.
  2. How fast? Crawlers pace themselves. Hundreds of requests a minute from one name looks like a scraper.
  3. Which paths? Crawlers spread across your site and usually fetch robots.txt. Requests that hammer one search or API endpoint do not look like crawling.
  4. What status did you return? If the requests already get 403, your firewall has made the decision for you; check that it was the one you wanted.

Where a pattern looks wrong, rate-limit by behaviour instead of trusting or blocking the name. On Nginx, for example:

map $http_user_agent $xai_key {
  default "";
  ~*(grokbot|xai-searchbot) $binary_remote_addr;
}
limit_req_zone $xai_key zone=xai:10m rate=30r/m;

server {
  location / {
    limit_req zone=xai burst=20 nodelay;
  }
}

Requests without an xAI name get an empty key and are not limited.

A worked example: 1,200 lines, three patterns

A shop pastes 1,200 lines from a week of logs. The checker groups them like this:

Example result (illustrative values)
NameIPsRequestsReading
xAI-SearchBot3410Steady over seven days, spread across product pages, robots.txt fetched daily: consistent with a crawler
GrokBot57690Fifty-seven unrelated IPs, most requests on the search endpoint in two bursts: someone borrowing the name
GrokAgent235A few product and contact pages at a time: user-triggered agent sessions
Other xAI or Grok mention165A user agent mentioning "grok" that matches no known name: investigate before writing rules

The shop keeps xAI-SearchBot allowed, rate-limits the GrokBot traffic by behaviour with the rule above rather than blocking the name, and leaves GrokAgent alone because each session is a customer. The unknown name gets a closer look in the CDN log, where the request headers are visible.

Checking your CDN as well as your logs

Your origin log only shows requests that got past the CDN. If the CDN challenged or blocked a request, the origin never saw it. Look in both places:

  • CDN bot or security analytics show challenged and blocked requests by user agent, including names the origin never logged.
  • Verified-bot categories in some CDNs list agents such as GrokAgent separately from unverified traffic that only claims a name.
  • Firewall event logs record which rule acted on a request, which tells you whether a block was deliberate.

Compare the counts: if the CDN blocks most GrokBot requests and your log shows a few, the few may be the ones your rules let through on purpose.

How to read the result

Traffic patterns and what they suggest
PatternWhat it suggestsWhat to do
A few IPs, steady requests over daysConsistent with a real crawler, though still unconfirmedAllow or block by name in robots.txt as you prefer
Many unrelated IPsSomeone borrowing the name, or a distributed scraperRate-limit by behaviour rather than trusting the name
Bursts on one URLA tool or script, not a crawlerRate-limit or block the IPs
"Other xAI or Grok mention"A user agent that mentions Grok, xAI or x.ai but is neither reported nameInvestigate before writing rules; no xAI source documents it
GrokAgentGrok Bot, xAI's agent, as listed in Cloudflare's verified-bot registryTreat as user-triggered traffic, not crawling
TwitterbotX's link-preview fetcher, not GrokKeep it allowed, or shared links lose their previews

Spoofed crawler names are common; spoofed AI crawlers explains how to tell them apart.

Allowing or blocking the xAI names

If you decide to block, name both in one group, because a crawler obeys only the group that names it:

User-agent: GrokBot
User-agent: xAI-SearchBot
Disallow: /

The robots.txt generator has Grok presets that write this for you. Because xAI documents neither name, treat the rule as a request and add a firewall rule if you need it enforced. The Grok crawler user agent guide covers every xAI name we track.

A routine for watching xAI traffic

  1. Once a month, export a week of lines that mention grok, xai or x.ai and paste them here.
  2. Write down three numbers per name: requests, distinct IPs and the share answered with 403 or 429.
  3. Compare with last month. A sudden jump in IPs for one name is the clearest sign of someone borrowing it.
  4. After any robots.txt change for the xAI names, check that the crawler stopped fetching the paths you refused. A name that keeps fetching refused paths is either ignoring the rule or is not who it says it is.
  5. Re-read our GrokBot and xAI-SearchBot profiles when xAI publishes documentation; if official IP ranges appear, verification by IP replaces pattern reading.

Grok Bot the agent is different

Grok Bot, the agent, opens websites in a cloud browser for its users. It is not a crawler, robots.txt is not documented as applying to it, and it does not use the GrokBot name. Some sessions route through a member's own computer and look like that person's normal browser. To see whether it can use your pages, run the AI Agent Readiness Check; to find every personal agent at once, use the AI Agent Log Checker.

Grok log checker questions

Are my logs uploaded anywhere?

No. The checker runs entirely in your browser. Nothing you paste is sent to our servers or stored.

Can this prove a request came from xAI?

No. xAI publishes no IP ranges or reverse DNS pattern for GrokBot or xAI-SearchBot, so anyone can send those names. The checker shows which IPs used them and how often, which helps you spot impostors, but it cannot confirm the sender.

Will it find Grok Bot, the agent?

Yes, if the request carries the GrokAgent token. Cloudflare's verified-bot registry lists GrokAgent as the user agent of Grok Bot, xAI's AI agent, and the checker reports it separately. xAI's own documentation does not print the string, and some Grok Bot sessions may route through a member's own computer, so absence from your logs is not proof that Grok Bot never visited. For every personal agent at once, use the AI agent log checker.

What log formats does it read?

Any line that contains the user agent, including the common Nginx and Apache combined formats, where the user agent is the last quoted field and the client IP is the first field.

Where to go next

WhatsApp us