Free, runs in your browser
Grok log checker GrokBot and xAI-SearchBot
Paste lines from your server access log to find requests that use the reported xAI crawler names, GrokBot and xAI-SearchBot. See which IPs sent them and how often, so you can tell a steady crawler from someone borrowing the name.
- 2reported xAI names
- by IPgrouped results
- 0lines uploaded
Summary
| Name | Client IP | Requests |
|---|
What is known about xAI's crawlers, and what is not
xAI runs Grok, but it has not published crawler documentation for the names site owners write rules for, GrokBot and xAI-SearchBot, and it publishes no IP ranges or reverse DNS pattern for them. That is why our directory labels both Reported rather than Official. Site owners still treat them as real: in our robots.txt study of the Tranco top 10,000 (5,583 valid files), 59 files name GrokBot and 6 name xAI-SearchBot, and the top 1,000 sites block GrokBot at 11.4% against 3.8% in the long tail.
Without published ranges nobody can prove a request is genuine. What you can do is read its pattern, which is what this checker is for.
The xAI names at a glance
| Name in the log | What it is | Evidence |
|---|---|---|
| GrokBot | Reported xAI crawler name | Reported: not in xAI documentation |
| xAI-SearchBot | Reported xAI search crawler name | Reported: not in xAI documentation |
| GrokAgent | Grok Bot, xAI's agent, acting for a user | Listed in Cloudflare's verified-bot registry |
| Other xAI or Grok mention | Any user agent that mentions Grok, xAI or x.ai but matches neither crawler name | None; investigate |
| Twitterbot | X's link-preview fetcher | Documented by X; not Grok |
Getting a useful sample
grep -iE "grok|xai" /var/log/nginx/access.log | tail -5000- Run the command on your server, or export matching lines from your hosting dashboard. On Apache the log is usually
/var/log/apache2/access.log. - Paste the lines into the checker. They are processed on this page and never uploaded.
- Read the summary, then the table by name and IP.
Reading one log line
203.0.113.7 - - [01/Oct/2026:10:00:00 +0000] "GET /pricing HTTP/1.1" 200 5120 "-" "Mozilla/5.0 (compatible; xAI-SearchBot/1.0; +https://x.ai)"- 203.0.113.7
- The client IP, the first field. The checker groups requests by it.
- [01/Oct/2026:10:00:00 +0000]
- When the request arrived. Steady spacing over days suggests a crawler; a burst in one minute suggests a script.
- "GET /pricing HTTP/1.1" 200
- The path and the status you returned. A 403 here means your firewall already refused it.
- The last quoted field
- The user agent. The checker matches the xAI names here and ignores the rest of the line.
The checker reads Nginx and Apache combined logs, and any other format where the client IP comes first and the user agent is the last quoted field.
Four tests for a borrowed name
- How many IPs? A real crawler tends to come from a small, stable set of addresses. Dozens of unrelated IPs using one name is a warning sign.
- How fast? Crawlers pace themselves. Hundreds of requests a minute from one name looks like a scraper.
- Which paths? Crawlers spread across your site and usually fetch robots.txt. Requests that hammer one search or API endpoint do not look like crawling.
- What status did you return? If the requests already get 403, your firewall has made the decision for you; check that it was the one you wanted.
Where a pattern looks wrong, rate-limit by behaviour instead of trusting or blocking the name. On Nginx, for example:
map $http_user_agent $xai_key {
default "";
~*(grokbot|xai-searchbot) $binary_remote_addr;
}
limit_req_zone $xai_key zone=xai:10m rate=30r/m;
server {
location / {
limit_req zone=xai burst=20 nodelay;
}
}Requests without an xAI name get an empty key and are not limited.
A worked example: 1,200 lines, three patterns
A shop pastes 1,200 lines from a week of logs. The checker groups them like this:
| Name | IPs | Requests | Reading |
|---|---|---|---|
| xAI-SearchBot | 3 | 410 | Steady over seven days, spread across product pages, robots.txt fetched daily: consistent with a crawler |
| GrokBot | 57 | 690 | Fifty-seven unrelated IPs, most requests on the search endpoint in two bursts: someone borrowing the name |
| GrokAgent | 2 | 35 | A few product and contact pages at a time: user-triggered agent sessions |
| Other xAI or Grok mention | 1 | 65 | A user agent mentioning "grok" that matches no known name: investigate before writing rules |
The shop keeps xAI-SearchBot allowed, rate-limits the GrokBot traffic by behaviour with the rule above rather than blocking the name, and leaves GrokAgent alone because each session is a customer. The unknown name gets a closer look in the CDN log, where the request headers are visible.
Checking your CDN as well as your logs
Your origin log only shows requests that got past the CDN. If the CDN challenged or blocked a request, the origin never saw it. Look in both places:
- CDN bot or security analytics show challenged and blocked requests by user agent, including names the origin never logged.
- Verified-bot categories in some CDNs list agents such as GrokAgent separately from unverified traffic that only claims a name.
- Firewall event logs record which rule acted on a request, which tells you whether a block was deliberate.
Compare the counts: if the CDN blocks most GrokBot requests and your log shows a few, the few may be the ones your rules let through on purpose.
How to read the result
| Pattern | What it suggests | What to do |
|---|---|---|
| A few IPs, steady requests over days | Consistent with a real crawler, though still unconfirmed | Allow or block by name in robots.txt as you prefer |
| Many unrelated IPs | Someone borrowing the name, or a distributed scraper | Rate-limit by behaviour rather than trusting the name |
| Bursts on one URL | A tool or script, not a crawler | Rate-limit or block the IPs |
| "Other xAI or Grok mention" | A user agent that mentions Grok, xAI or x.ai but is neither reported name | Investigate before writing rules; no xAI source documents it |
| GrokAgent | Grok Bot, xAI's agent, as listed in Cloudflare's verified-bot registry | Treat as user-triggered traffic, not crawling |
| Twitterbot | X's link-preview fetcher, not Grok | Keep it allowed, or shared links lose their previews |
Spoofed crawler names are common; spoofed AI crawlers explains how to tell them apart.
Allowing or blocking the xAI names
If you decide to block, name both in one group, because a crawler obeys only the group that names it:
User-agent: GrokBot
User-agent: xAI-SearchBot
Disallow: /The robots.txt generator has Grok presets that write this for you. Because xAI documents neither name, treat the rule as a request and add a firewall rule if you need it enforced. The Grok crawler user agent guide covers every xAI name we track.
A routine for watching xAI traffic
- Once a month, export a week of lines that mention grok, xai or x.ai and paste them here.
- Write down three numbers per name: requests, distinct IPs and the share answered with 403 or 429.
- Compare with last month. A sudden jump in IPs for one name is the clearest sign of someone borrowing it.
- After any robots.txt change for the xAI names, check that the crawler stopped fetching the paths you refused. A name that keeps fetching refused paths is either ignoring the rule or is not who it says it is.
- Re-read our GrokBot and xAI-SearchBot profiles when xAI publishes documentation; if official IP ranges appear, verification by IP replaces pattern reading.
Grok Bot the agent is different
Grok Bot, the agent, opens websites in a cloud browser for its users. It is not a crawler, robots.txt is not documented as applying to it, and it does not use the GrokBot name. Some sessions route through a member's own computer and look like that person's normal browser. To see whether it can use your pages, run the AI Agent Readiness Check; to find every personal agent at once, use the AI Agent Log Checker.
Grok log checker questions
Are my logs uploaded anywhere?
No. The checker runs entirely in your browser. Nothing you paste is sent to our servers or stored.
Can this prove a request came from xAI?
No. xAI publishes no IP ranges or reverse DNS pattern for GrokBot or xAI-SearchBot, so anyone can send those names. The checker shows which IPs used them and how often, which helps you spot impostors, but it cannot confirm the sender.
Will it find Grok Bot, the agent?
Yes, if the request carries the GrokAgent token. Cloudflare's verified-bot registry lists GrokAgent as the user agent of Grok Bot, xAI's AI agent, and the checker reports it separately. xAI's own documentation does not print the string, and some Grok Bot sessions may route through a member's own computer, so absence from your logs is not proof that Grok Bot never visited. For every personal agent at once, use the AI agent log checker.
What log formats does it read?
Any line that contains the user agent, including the common Nginx and Apache combined formats, where the user agent is the last quoted field and the client IP is the first field.