xAI

Common mistakes with xAI's crawlers

The errors we see most often with xAI's crawlers in the files and firewalls we test:

1. Assuming a robots.txt rule will stop xAI's crawlers. User-triggered fetches act for a person, and some operators say robots.txt may not apply to them. Check the operator's statement above.
2. Blocking xAI's crawlers as if it were a bulk crawler. It fetches one page because a person asked; blocking it means that person's assistant cannot read your page.
3. Counting xAI's crawlers visits as crawl volume. Each request maps to a real user question, which makes the paths it requests a useful signal of what people ask about you.

All three hide inside a file that looks correct, so check xAI's crawlers from outside after every change.

Spotting xAI in your logs

robots.txt says what Grok Bot (GrokAgent) and Grok Search and GrokBot (xAI) may do; your access logs show what they actually did.

Count requests

On a server with standard Nginx or Apache logs, this counts matching requests per day (adjust the log path for your host):

grep -iE "GrokAgent|xAI-SearchBot|GrokBot" /var/log/nginx/access.log \
  | awk '{print $4}' | cut -d: -f1 | sort | uniq -c

And this shows which paths it requested most, which is the quickest way to see whether your Disallow lines are being respected:

grep -iE "GrokAgent|xAI-SearchBot|GrokBot" /var/log/nginx/access.log \
  | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

Check the requests are genuine

Anyone can send these user-agents, so confirm the source before acting on a match:

• GrokAgent: we hold no published IP list, so use reverse DNS where xAI documents a hostname and treat an unverifiable Grok Bot (GrokAgent) request with caution. Check it separately from Grok Search and GrokBot (xAI).
• xAI-SearchBot: we hold no published IP list, so use reverse DNS where xAI documents a hostname and treat an unverifiable Grok Search request with caution. Check it separately from Grok Bot (GrokAgent) and GrokBot (xAI).
• GrokBot: we hold no published IP list, so use reverse DNS where xAI documents a hostname and treat an unverifiable GrokBot (xAI) request with caution. Check it separately from Grok Bot (GrokAgent) and Grok Search.

What our own data shows about xAI

Everything below comes from the AI Crawler Check bot database and our two most recent robots.txt scans of the Tranco top 10,000, not from xAI's marketing. It covers 3 xAI tokens in turn.

GrokAgent in our records

We first recorded GrokAgent on October 11, 2026. We could not find a page where xAI names it, so treat the token as reported rather than documented. GrokAgent is outside our tracked token set, but 0 of 5,583 top-site files already name it, which is enough to show owners treat it as a real crawler.

Among the 104 AI and LLM crawlers in our database, GrokAgent sits tenth by priority (Tier 1), level with ChatGPT Agent, ChatGPT Operator, ChatGPT-User, ChatGPT-User/2.0. 22 of the 104 share its caution rating. Other operators with a crawler that fetches a page when a person asks an assistant or app to open it: ChatGPT Agent (OpenAI, Tier 1), ChatGPT Operator (OpenAI, Tier 1), ChatGPT-User (OpenAI, Tier 1), ChatGPT-User/2.0 (OpenAI, Tier 1), Claude-SearchBot (Anthropic, Tier 1). A policy for GrokAgent is usually a policy for these too.

A decision record for GrokAgent

Teams that review crawlers once a quarter keep one short entry per token, so anyone can see why a rule exists and when it was last checked. Filled in from our data, the GrokAgent entry reads:

• Operator and job: xAI; it fetches a page when a person asks an assistant or app to open it.
• Priority and behaviour: Tier 1, rated caution, blocking impact low.
• Suggested starting point: Allow, and watch it in your logs.
• Enforcement: GrokAgent's operator does not clearly document how it treats robots.txt. Write the GrokAgent rule anyway, then confirm in your logs that its requests stop.

User-agent: GrokAgent
Allow: /

Paste the line into the validator with one of your own URLs before it goes live. Revisit the GrokAgent entry sooner than most: a caution rating means xAI traffic deserves a monthly look at your logs.

xAI-SearchBot in our records

We first recorded xAI-SearchBot on September 22, 2026. We could not find a page where xAI names it, so treat the token as reported rather than documented. Between our scans of October 17, 2026 and October 20, 2026, files naming xAI-SearchBot went from 6 to 6 (0) and files blocking it from 232 to 246 (+14), so blocking of xAI-SearchBot grew even though naming did not.

Among the 104 AI and LLM crawlers in our database, xAI-SearchBot sits 86th by priority (Tier 2), level with AddSearchBot, AI2Bot, AI2Bot-DeepResearchEval, aiHitBot. 22 of the 104 share its caution rating. Other operators with a crawler that builds a search or answer index: AddSearchBot (AddSearch, Tier 2), AIWebIndex (Lyrenth, Tier 2), atlassian-bot (Atlassian, Tier 2), AzureAI-SearchBot (Microsoft, Tier 2), Cloudflare-AI-Search (Cloudflare, Tier 2). A policy for xAI-SearchBot is usually a policy for these too.

GrokBot in our records

We first recorded GrokBot on March 17, 2026. We could not find a page where xAI names it, so treat the token as reported rather than documented. Between our scans of October 17, 2026 and October 20, 2026, files naming GrokBot went from 57 to 59 (+2) and files blocking it from 275 to 289 (+14), so more sites wrote a rule for GrokBot and more of those rules block it.

Among the 104 AI and LLM crawlers in our database, GrokBot sits 47th by priority (Tier 2), level with AddSearchBot, AI2Bot, AI2Bot-DeepResearchEval, aiHitBot. 81 of the 104 share its safe rating. Other operators with a crawler that collects pages both for training and for search or answers: PerplexityBot (Perplexity AI, Tier 1), Andibot (Andi, Tier 2), Bytespider (ByteDance, Tier 2), cohere-ai (Cohere, Tier 2), DeepSeekBot (DeepSeek, Tier 2). A policy for GrokBot is usually a policy for these too.

How top sites treat xAI's crawlers

From our robots.txt study of the Tranco top 10,000 (5,583 valid files, fetched October 20, 2026):

• GrokBot, the most blocked of the xAI's crawlers tokens: blocked in 5.2% of files (52 with its own group, 237 through User-agent: *), more often by the top 1,000 (11.4%) than the long tail (3.8%).
• xAI-SearchBot, second of the xAI's crawlers tokens: blocked in 4.4% of files (6 with its own group, 240 through User-agent: *), more often by the top 1,000 (8.8%) than the long tail (3.5%).

For scale, 20.1% of all files block at least one AI training crawler. None of the tokens for xAI's crawlers comes close to that rate, so blocking them is still a minority choice.

More questions about xAI

Will blocking xAI affect my Google rankings?

Only if one of its tokens is a Google search crawler. None of xAI's tokens are Googlebot, so classic Google rankings are unaffected.

How quickly does a robots.txt change take effect for xAI?

Crawlers cache robots.txt, commonly for up to a day. Expect a change to apply within about 24 hours, and confirm it in your logs.

Can I allow xAI on some sections and block it on others?

Yes. Inside the group for GrokAgent, add Disallow lines for the paths to keep private and Allow lines for exceptions. The longest matching rule wins.

Does blocking xAI delete content it already collected?

No. robots.txt only controls future visits. For content already collected, check whether xAI offers a removal or opt-out process.

Is a firewall block better than robots.txt for xAI?

A firewall enforces the decision, while robots.txt asks politely. Many operators recommend robots.txt because blocking their IPs can stop them reading your rules at all. Use robots.txt first, and add a firewall rule only if requests continue.