Safe Reported AI & LLM Bots

GrokBot (xAI)

Operated by xAI

Quick Facts

User-Agent:
GrokBot
Category:
AI & LLM Bots
Operator:
xAI
Safety:
Safe
Blocking Impact:
Low - No SEO ranking impact
SEO Impact Score:
2/10
Official docs:
None published by xAI
Evidence:
Verified by AI Crawler Check: in 59 top-10k robots.txt files

Reported, not operator documented

xAI has not published a crawler page that names GrokBot. It is listed here on our own evidence. Verified by AI Crawler Check: 59 of the 5,583 valid robots.txt files in the Tranco top 10,000 carry a rule for GrokBot, so site owners treat it as a real crawler name (how we measure this). That proves the spelling is in use, not who sends it. xAI Grok crawler token. xAI publishes no crawler documentation.

What that means for you: the GrokBot rule on this page is the best available spelling, but xAI has made no statement of purpose, crawl rate or robots.txt compliance. Treat a GrokBot request in your logs as a clue, not proof of who sent it, and add a firewall rule if that traffic matters to you. If the operator publishes documentation, this profile moves to Official docs.

What is GrokBot?

GrokBot is a user-agent name widely linked to xAI's Grok assistant: 57 of the 5,254 valid robots.txt files in the Tranco top 10,000 carry a rule for it (AI Crawler Check study, October 17, 2026). xAI has not published a page that documents it, so its exact purpose is not confirmed.

It is commonly described as fetching pages for Grok answers. xAI documents a web search tool that lets Grok "search the web in real-time and browse web pages", but that page does not name the user agent the tool sends. Two other names cause confusion:
• xAI-SearchBot is a second Reported xAI name, seen as Mozilla/5.0 (compatible; xAI-SearchBot/1.0; +https://x.ai).
• Grok Bot (two words) is a different thing: an AI agent product xAI launched on 11 August 2026. It browses websites in a cloud browser on behalf of a user, so robots.txt does not govern it.

What happens if you block GrokBot?

✅ No SEO Impact - Blocking GrokBot does not affect your rankings in Google, Bing, or any other search engine. GrokBot is an AI crawler name, not a search indexer. You can block it with User-agent: GrokBot / Disallow: / without any SEO penalty. The trade-off is visibility inside Grok answers, if xAI does use this name.
Generally safe to allow; provides legitimate crawling value.

Is GrokBot safe?

Yes, GrokBot is safe to allow: there are no credible reports of it crawling abusively, and it is rated Safe in this directory. The caveat is proof, not behaviour. xAI has not published a crawler page, user-agent string or IP ranges for it, so anyone can send the name and robots.txt rules for it are best effort. If you want Grok to be able to cite you, allow it; if you do not, block it in robots.txt and at the server. For monitoring, see how do I monitor AI bot traffic to my website?

GrokBot user agent: is there an official xAI crawler string?

No. xAI has not published an official crawler user agent. The strings in circulation come from public registries: GrokBot, and Mozilla/5.0 (compatible; xAI-SearchBot/1.0; +https://x.ai) for xAI-SearchBot. Name both in robots.txt. The Grok crawler is not the same as Grok Bot, xAI's browsing agent, which robots.txt does not control.

User-agent: GrokBot
User-agent: xAI-SearchBot
Allow: /

GrokBot vs xAI-SearchBot vs the Grok Bot agent

xAI-related names compared
NameWhat it isEvidence tierDoes robots.txt apply?
GrokBotReported crawler name linked to Grok answersReportedYes, if the client honours it
xAI-SearchBotReported search crawler, xAI-SearchBot/1.0ReportedYes, if the client honours it
Grok Bot (agent)AI teammate that browses websites in a cloud browser for a userDocumented by xAINo, it acts like a person
TwitterbotX link previews, not GrokOfficialYes
OAI-SearchBotOpenAI's documented equivalent, for comparisonOfficialYes

Because neither xAI crawler name is documented, blocking them by name is a preference that compliant clients will follow, not a guarantee. For enforcement, see spoofed AI crawler user agents.

What is Grok Bot, and can it visit my site?

Grok Bot is an AI agent product that xAI launched on 11 August 2026 (xAI launch post, 11 August 2026). According to the xAI Grok Bot documentation, each Bot works on "a persistent cloud computer with a browser, filesystem, and terminal" and "takes on multi-step work across apps and websites". So yes, it can visit your site, but as a browser acting for a person, not as a crawler.

  • robots.txt does not govern it. Like ChatGPT Agent, it opens pages the way a person would, so a User-agent: GrokBot rule does not stop it.
  • It comes from shared static IPs. The xAI Grok Bot security page says hosted computers use "shared static egress IP addresses", shared across all Grok Bot customers. xAI gives the ranges to customers through its account team; they are not published.
  • It stops at human checks. xAI tells users to complete passwords, two-factor codes and CAPTCHAs themselves, and notes that some services block it because they "flag datacenter IP addresses".

Practical meaning: if you want agents like Grok Bot to complete bookings, sign-ups or purchases on your site, avoid putting a CAPTCHA or firewall challenge on those flows. If you want to stop them, use a firewall or bot-management rule, not robots.txt. The full comparison is in Grok Bot vs GrokBot; test a page with the AI Agent Readiness Check, or read can you block Grok Bot?.

GrokBot use cases

Use cases: what to do with this bot
If you are...Recommended setting and why
You want to appear in Grok answersAllow GrokBot and xAI-SearchBot, and keep public pages server-rendered.
You block AI but want X link previewsBlock the xAI tokens; X link previews use Twitterbot, a separate token.
You only allow documented botsBlock both; neither has an operator specification yet.
User-agent: GrokBot
User-agent: xAI-SearchBot
Disallow: /

User-agent: Twitterbot
Allow: /

Confirm with a website crawlability test scan and read the Grok crawler user agent guide and the xAI operator page. Seeing these names in your logs? Paste them into the Grok log checker. Want Grok to cite you? Read how to get cited by Grok.

Test xAI crawler access

  • AI crawler checker: tests 265 bots against your robots.txt in one scan.
  • AI crawler diagnostic: fetches a page as each AI bot and names the layer that blocks it.
  • robots.txt validator: shows allowed, blocked and partial status per bot.
  • robots.txt generator: builds a file with presets for search, agents and training.
  • batch URL checker: compares up to 20 sites side by side, with CSV export.
  • llms.txt generator: writes the file AI assistants read for a site summary.
  • schema markup generator: produces JSON-LD for eight common types.
  • GEO audit: scores a page for generative engine optimization.

How to block GrokBot with robots.txt

<code>User-agent: GrokBot</code> - Matching is case-insensitive. Robots.txt is fetched from the root of each subdomain separately.

Block completely (robots.txt)
User-agent: GrokBot Disallow: /
Allow all (robots.txt)
User-agent: GrokBot Allow: /
Block private only (robots.txt)
User-agent: GrokBot Disallow: /private/ Disallow: /api/ Disallow: /admin/ Allow: /
Nginx server block
# Nginx: Hard-block GrokBot if ($http_user_agent ~* "GrokBot") { return 403 "Bot blocked"; }
Apache .htaccess
# Apache: Hard-block GrokBot SetEnvIfNoCase User-Agent "GrokBot" bad_bot Order Allow,Deny Allow from all Deny from env=bad_bot
Meta robots tag
<meta name="robots" content="noindex, nofollow">
X-Robots-Tag header
X-Robots-Tag: noindex, nofollow

Is GrokBot safe to allow?

GrokBot is rated Safe in this directory: there are no credible public reports of it crawling abusively, and allowing it is the only way Grok can read your pages if it does use this name. The caveat is evidence, not behaviour. xAI has not documented GrokBot, publishes no IP ranges for it, and does not state that it reads robots.txt, so you cannot prove that a request carrying the name came from xAI. Allow it if you want to be cited in Grok, block it if you do not, and treat log hits as a clue rather than proof.
There is nothing to verify against yet: xAI publishes no IP ranges or reverse-DNS pattern for GrokBot. Rate-limit unusual volume instead of trusting the name.

What does GrokBot do?

As commonly described by site owners and bot trackers; xAI has not confirmed any of this.

  • Reported to fetch pages that Grok uses in its answers
  • May supply web results for Grok search and DeepSearch
  • Purpose, crawl rate and robots.txt behaviour are not documented by xAI
  • Not the same as Grok Bot, xAI's browsing agent, which robots.txt does not control
  • Not used for X (Twitter) link previews, which use Twitterbot

GrokBot (xAI) in context

GrokBot (xAI) is a training crawler operated by xAI. In plain terms, it collects pages both for training and for search or answers. Our database files it under AI and LLM crawlers and ranks it Tier 2, important crawlers from established AI and search operators. 74 of the 104 AI and LLM crawlers in our database share this tier.

Its safety rating is safe and its blocking impact is low. So the decision about GrokBot (xAI) rests on what you want from xAI, not on its behaviour. How we rate safety and impact.

How site owners treat GrokBot (xAI)

In our own study of robots.txt files across the Tranco top 10,000 sites (fetched October 20, 2026, 5,583 valid files), 59 files (1.1%) name GrokBot in a User-agent line, and 289 (5.2%) block it from the homepage. 52 of those blocks come from a group written for this crawler by name, and 237 come from a broad User-agent: * rule that catches it by accident. Larger sites are stricter: 11.4% of the top 1,000 block it, against 5.7% of the middle band and 3.8% of the tail.

Does it follow robots.txt?

GrokBot (xAI)'s operator does not clearly document how it treats robots.txt. Write the GrokBot (xAI) rule anyway, then confirm in your logs that its requests stop.

xAI crawlers and close alternatives

GrokBot (xAI) is one of 3 xAI crawlers in our database. A User-agent: GrokBot group controls GrokBot (xAI) only; the other xAI tokens below need their own lines.

  • GrokBot: this crawler (Tier 2)
  • GrokAgent: Grok Bot (xAI agent), fetches a page when a person asks an assistant or app to open it (Tier 1)
  • xAI-SearchBot: Grok Search, builds a search or answer index (Tier 2)

To give GrokBot (xAI) the same answer as its siblings, stack User-agent: GrokBot with their lines above one set of rules; to treat GrokBot (xAI) differently, give it its own group.

Other AI and LLM crawlers that do the same job (collects pages both for training and for search or answers) include PerplexityBot from Perplexity AI, Andibot from Andi, cohere-ai from Cohere, DeepSeekBot from DeepSeek, DoubaoBot from ByteDance, iAskBot from iAsk. Deciding about GrokBot (xAI) is usually a decision about the whole group, so check how your file treats these too.

We first recorded GrokBot on March 17, 2026. We could not find a page where xAI names it, so treat the token as reported rather than documented.

Which rule GrokBot (xAI) actually follows

GrokBot (xAI) reads exactly one group in robots.txt, the most specific one that names GrokBot, and never combines rules across groups. This short file shows the trap:

User-agent: *
Disallow: /private/

User-agent: GrokBot
Disallow: /drafts/
  1. Find the group. GrokBot (xAI) looks for a group naming GrokBot. One exists, so it ignores User-agent: * completely.
  2. Apply only that group. It is refused /drafts/ and nothing else.
  3. The surprise. /private/ is open to GrokBot (xAI), because that rule lives in the * group it no longer reads. To keep it out of both, repeat Disallow: /private/ inside its own group.
Test yourself: does a rule under User-agent: * apply to GrokBot (xAI) if the file also has a User-agent: GrokBot group?

No. Once a group names it, GrokBot (xAI) reads only that group. Put every rule you want it to follow inside its own group.

Test yourself: will blocking GrokBot (xAI) remove your pages from AI answers?

No. GrokBot (xAI) collects training data. Blocking it changes what future models learn, not which pages AI answers cite today.

Test yourself: is User-agent: grokbot the same as User-agent: GrokBot?

Yes. RFC 9309 makes user-agent matching case-insensitive. Paths are a different story: Disallow: /Blog/ does not block /blog/.

Should you allow GrokBot (xAI)? Decide by situation

Recommended setting for GrokBot (xAI) by type of site
If you are...Recommended setting
You publish content and want AI answers to cite itYour call. GrokBot (xAI) is a training crawler, so blocking it does not remove you from AI answers; retrieval crawlers do that job.
You license content and want to stop AI trainingBlock it by name with User-agent: GrokBot and Disallow: /.
You run an online shopAllow it unless your logs show it is costly.
Your server is under load from botsRate-limit before blocking. It is rated safe in our database.

Check GrokBot (xAI) on your own site

  1. Open /robots.txt on your domain and search for GrokBot. If it is not named, it follows User-agent: *.
  2. List the Disallow lines in the group GrokBot (xAI) follows and test your key URLs against them; the longest match wins, and Allow wins a tie.
  3. Look beyond the file. A firewall or bot manager can refuse GrokBot (xAI) even when robots.txt allows it, and that never shows in the file.
  4. Run our website crawlability test to see how GrokBot (xAI) and every other crawler we track are treated in one pass, including firewall refusals.

Common mistakes with GrokBot (xAI)

The errors we see most often with GrokBot (xAI) in the files and firewalls we test:

  1. Assuming that blocking GrokBot (xAI) removes you from AI answers. It is a training crawler; AI answers are fed by retrieval and user-triggered crawlers with different tokens.
  2. Blocking GrokBot (xAI) with a broad User-agent: * rule that also catches search engines. Name GrokBot in its own group instead.
  3. Expecting a GrokBot (xAI) block to remove content already collected. robots.txt only affects future visits.

All three hide inside a file that looks correct, so check GrokBot (xAI) from outside after every change.

Spotting GrokBot (xAI) in your logs

robots.txt says what GrokBot (xAI) may do; your access logs show what it actually did.

Count requests

On a server with standard Nginx or Apache logs, this counts matching requests per day (adjust the log path for your host):

grep -iE "GrokBot" /var/log/nginx/access.log \
  | awk '{print $4}' | cut -d: -f1 | sort | uniq -c

And this shows which paths it requested most, which is the quickest way to see whether your Disallow lines are being respected:

grep -iE "GrokBot" /var/log/nginx/access.log \
  | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

Check the requests are genuine

Anyone can send the GrokBot (xAI) user-agent, so confirm the source before acting on a match:

  • GrokBot: we hold no published IP list, so use reverse DNS where the operator documents a hostname and treat an unverifiable GrokBot (xAI) request with caution.

Frequently Asked Questions

Is GrokBot safe?
GrokBot is rated Safe in this directory: there are no credible public reports of it crawling abusively, and allowing it is the only way Grok can read your pages if it does use this name. The caveat is evidence, not behaviour. xAI has not documented GrokBot, publishes no IP ranges for it, and does not state that it reads robots.txt, so you cannot prove that a request carrying the name came from xAI. Allow it if you want to be cited in Grok, block it if you do not, and treat log hits as a clue rather than proof.
What is the official GrokBot user agent?
There is no official one. xAI has not published a user-agent string for any crawler. GrokBot is the name site owners most often write rules for, so use it in robots.txt as the best available match. A second reported xAI name, xAI-SearchBot, should get its own rule.
Is GrokBot the same as Grok Bot?
No. GrokBot is a reported crawler name. Grok Bot is an AI agent product xAI launched on 11 August 2026: it signs into tools and browses websites in a cloud browser on a user's behalf. Robots.txt rules for GrokBot do not apply to it.
How do I block GrokBot in robots.txt?
Add a group naming both reported xAI tokens:
User-agent: GrokBot
User-agent: xAI-SearchBot
Disallow: /

A client that honours robots.txt will stop. Because xAI has not documented either name, add a firewall or CDN rule if you need enforcement.
Will blocking GrokBot hurt my SEO?
No. GrokBot has no role in Google or Bing rankings. The only cost is that Grok may not be able to read or cite your pages, if it uses this name.
Does GrokBot respect robots.txt?
xAI has not said. There is no xAI statement about robots.txt for GrokBot or xAI-SearchBot, so a robots.txt rule is a request that a well-behaved client will follow, not a guarantee.
How do I see if GrokBot visits my site?
Search your access logs: grep -iE "GrokBot|xAI-SearchBot" /var/log/nginx/access.log. A match shows that something used the name. xAI publishes no IP ranges for it, so you cannot confirm the sender.

Related Bots

WhatsApp us