Caution Reported AI & LLM Bots

Grok Search

Q: What user-agent string does Grok Search use?

Grok Search is reported to identify itself with xAI-SearchBot . Evidence that the spelling is in real use comes from our own robots.txt study: 6 of the 5,583 valid robots.txt files in the Tranco top 10,000 name it (fetched October 20, 2026), not from xAI, which has not published crawler documentation naming it. Use it in robots.txt as the best available match, and treat a request carrying it as a clue rather than proof of who sent it.

Q: Is Grok Search safe?

Grok Search is rated Caution in this directory. It is a Reported token: xAI has not published documentation for it, so its purpose, crawl rate and robots.txt behaviour are not confirmed by the operator, and anyone can send the same user-agent string. Allow or block it on its stated purpose, and back the robots.txt rule with a firewall rule if the traffic matters to you.

Q: Will blocking Grok Search hurt my SEO?

✅ No SEO Impact - Blocking Grok Search does not affect your rankings in Google, Bing, or any other search engine. Grok Search is an AI crawler, not a traditional search indexer. You can freely block it via User-agent / Disallow: / without any SEO penalty.

Q: How do I block Grok Search in robots.txt?

Add the following lines to your /robots.txt file: User-agent: xAI-SearchBot Disallow: / This instructs Grok Search not to crawl any path on your site. To block only specific sections, replace / with the path (e.g., Disallow: /blog/ ).

Q: How do I verify if Grok Search is crawling my site?

Search your access logs for xAI-SearchBot (for example grep -i "xAI-SearchBot" /var/log/nginx/access.log ). Because the operator publishes no IP ranges or reverse-DNS pattern, a match shows that something used the name, not that xAI sent it.

Operated by xAI

Scout's quick take

Grok Search fetches pages so xAI's AI search can answer and cite them. Block it and you can drop out of those answers, so most sites keep it allowed. This profile is built from reported sightings, not vendor documentation.

# Block only if you really mean to
User-agent: xAI-SearchBot
Disallow: /

Check my AI search visibility Build a full robots.txt

Quick Facts

User-Agent:: xAI-SearchBot
Category:: AI & LLM Bots
Operator:: xAI
Safety:: Use Caution
Blocking Impact:: Low - No SEO ranking impact
SEO Impact Score:: 2/10
Official docs:: None published by xAI
Evidence:: Verified by AI Crawler Check: in 6 top-10k robots.txt files

Reported, not operator documented

xAI has not published a crawler page that names xAI-SearchBot. It is listed here on our own evidence. Verified by AI Crawler Check: 6 of the 5,583 valid robots.txt files in the Tranco top 10,000 carry a rule for xAI-SearchBot, so site owners treat it as a real crawler name (how we measure this). That proves the spelling is in use, not who sends it. Observed UA for xAI Grok search retrieval (Mozilla/5.0 (compatible; xAI-SearchBot/1.0; +https://x.ai)). xAI publishes no crawler page.

What that means for you: the xAI-SearchBot rule on this page is the best available spelling, but xAI has made no statement of purpose, crawl rate or robots.txt compliance. Treat a xAI-SearchBot request in your logs as a clue, not proof of who sent it, and add a firewall rule if that traffic matters to you. If the operator publishes documentation, this profile moves to Official docs.

What is Grok Search?

xAI-SearchBot is a user-agent name reported for xAI's Grok search retrieval. xAI has not published a page that documents it.

The reported string is Mozilla/5.0 (compatible; xAI-SearchBot/1.0; +https://x.ai). The link in it points at xAI's homepage, not at a crawler page, and xAI has published no crawler documentation, IP ranges or robots.txt policy. Write robots.txt rules for it alongside GrokBot, the other reported xAI name.

What happens if you block Grok Search?

✅ No SEO Impact - Blocking Grok Search does not affect your rankings in Google, Bing, or any other search engine. Grok Search is an AI crawler, not a traditional search indexer. You can freely block it via User-agent / Disallow: / without any SEO penalty.

Generally safe to allow; provides legitimate crawling value.

Want to see whether Grok Search actually visits you? Read how do I monitor AI bot traffic to my website?

How to block Grok Search with robots.txt

<code>User-agent: xAI-SearchBot</code> - Matching is case-insensitive. Robots.txt is fetched from the root of each subdomain separately.

Block completely (robots.txt)

User-agent: xAI-SearchBot
Disallow: /

Allow all (robots.txt)

User-agent: xAI-SearchBot
Allow: /

Block private only (robots.txt)

User-agent: xAI-SearchBot
Disallow: /private/
Disallow: /api/
Disallow: /admin/
Allow: /

Nginx server block

# Nginx: Hard-block Grok Search
if ($http_user_agent ~* "xAI-SearchBot") {
    return 403 "Bot blocked";
}

Apache .htaccess

# Apache: Hard-block Grok Search
SetEnvIfNoCase User-Agent "xAI-SearchBot" bad_bot
Order Allow,Deny
Allow from all
Deny from env=bad_bot

Meta robots tag

X-Robots-Tag header

X-Robots-Tag: noindex, nofollow

Is Grok Search safe to allow?

Grok Search is rated Caution in this directory. It is a Reported token: xAI has not published documentation for it, so its purpose, crawl rate and robots.txt behaviour are not confirmed by the operator, and anyone can send the same user-agent string. Allow or block it on its stated purpose, and back the robots.txt rule with a firewall rule if the traffic matters to you.

xAI publishes no IP ranges or reverse-DNS pattern for Grok Search, so the user-agent string cannot be verified. Rate-limit unusual volume instead of trusting the name.

What does Grok Search do?

Understanding Grok Search's purpose helps you decide whether to allow or block it.

Indexes web pages for AI-powered search answers
Surfaces your content as citations inside AI search results
Builds a knowledge index used to answer user questions
Allowing it can increase visibility in AI search
robots.txt behaviour not documented by the operator

Grok Search in context

Grok Search is a retrieval crawler that feeds AI search answers operated by xAI. In plain terms, it builds a search or answer index. Our database files it under AI and LLM crawlers and ranks it Tier 2, important crawlers from established AI and search operators. 74 of the 104 AI and LLM crawlers in our database share this tier.

Its safety rating is caution and its blocking impact is low. So the decision about Grok Search rests on what you want from xAI, not on its behaviour. How we rate safety and impact.

How site owners treat Grok Search

In our own study of robots.txt files across the Tranco top 10,000 sites (fetched October 20, 2026, 5,583 valid files), 6 files (0.1%) name xAI-SearchBot in a User-agent line, and 246 (4.4%) block it from the homepage. 6 of those blocks come from a group written for this crawler by name, and 240 come from a broad User-agent: * rule that catches it by accident. Larger sites are stricter: 8.8% of the top 1,000 block it, against 4.6% of the middle band and 3.5% of the tail.

Does it follow robots.txt?

Grok Search's operator does not clearly document how it treats robots.txt. Write the Grok Search rule anyway, then confirm in your logs that its requests stop.

xAI crawlers and close alternatives

Grok Search is one of 3 xAI crawlers in our database. A User-agent: xAI-SearchBot group controls Grok Search only; the other xAI tokens below need their own lines.

xAI-SearchBot: this crawler (Tier 2)
GrokAgent: Grok Bot (xAI agent), fetches a page when a person asks an assistant or app to open it (Tier 1)
GrokBot: GrokBot (xAI), collects pages both for training and for search or answers (Tier 2)

To give Grok Search the same answer as its siblings, stack User-agent: xAI-SearchBot with their lines above one set of rules; to treat Grok Search differently, give it its own group.

Other AI and LLM crawlers that do the same job (builds a search or answer index) include AddSearchBot from AddSearch, AIWebIndex from Lyrenth, atlassian-bot from Atlassian, AzureAI-SearchBot from Microsoft, Cloudflare-AI-Search from Cloudflare, Cloudflare-AutoRAG from Cloudflare. Deciding about Grok Search is usually a decision about the whole group, so check how your file treats these too.

We first recorded xAI-SearchBot on September 22, 2026. We could not find a page where xAI names it, so treat the token as reported rather than documented.

Which rule Grok Search actually follows

Grok Search reads exactly one group in robots.txt, the most specific one that names xAI-SearchBot, and never combines rules across groups. This short file shows the trap:

User-agent: *
Disallow: /private/

User-agent: xAI-SearchBot
Disallow: /drafts/

Find the group. Grok Search looks for a group naming xAI-SearchBot. One exists, so it ignores User-agent: * completely.
Apply only that group. It is refused /drafts/ and nothing else.
The surprise. /private/ is open to Grok Search, because that rule lives in the * group it no longer reads. To keep it out of both, repeat Disallow: /private/ inside its own group.

Test yourself: does a rule under User-agent: * apply to Grok Search if the file also has a User-agent: xAI-SearchBot group?

No. Once a group names it, Grok Search reads only that group. Put every rule you want it to follow inside its own group.

Test yourself: will blocking Grok Search remove your pages from AI answers?

Often, yes. Grok Search is a retrieval crawler that feeds AI search answers, so pages it cannot fetch can drop out of the answers or results it supports.

Test yourself: is User-agent: xai-searchbot the same as User-agent: xAI-SearchBot?

Yes. RFC 9309 makes user-agent matching case-insensitive. Paths are a different story: Disallow: /Blog/ does not block /blog/.

Should you allow Grok Search? Decide by situation

Recommended setting for Grok Search by type of site
If you are...	Recommended setting
You publish content and want AI answers to cite it	Allow it. Grok Search fetches pages for answers, so blocking it can remove you from them.
You license content and want to stop AI training	Blocking it does not stop training, because Grok Search is a retrieval crawler that feeds AI search answers. Block the operator's training crawler instead.
You run an online shop	Allow it on product and category pages, which are the pages shoppers ask assistants about.
Your server is under load from bots	Rate-limit before blocking. It is rated caution in our database.

Check Grok Search on your own site

Open /robots.txt on your domain and search for xAI-SearchBot. If it is not named, it follows User-agent: *.
List the Disallow lines in the group Grok Search follows and test your key URLs against them; the longest match wins, and Allow wins a tie.
Look beyond the file. A firewall or bot manager can refuse Grok Search even when robots.txt allows it, and that never shows in the file.
Run our AI crawler access test to see how Grok Search and every other crawler we track are treated in one pass, including firewall refusals.

Common mistakes with Grok Search

The errors we see most often with Grok Search in the files and firewalls we test:

Blocking Grok Search while trying to stop AI training. Retrieval crawlers fetch pages so answers can cite them; blocking it can remove you from those answers without affecting training.
Letting a firewall or bot-fight mode refuse Grok Search while robots.txt allows it. The file looks right, but the crawler still gets a 403 or a challenge page.
Forgetting that once a User-agent: xAI-SearchBot group exists, Grok Search ignores every rule under User-agent: *.

All three hide inside a file that looks correct, so check Grok Search from outside after every change.

Spotting Grok Search in your logs

robots.txt says what Grok Search may do; your access logs show what it actually did.

Count requests

On a server with standard Nginx or Apache logs, this counts matching requests per day (adjust the log path for your host):

grep -iE "xAI-SearchBot" /var/log/nginx/access.log \
  | awk '{print $4}' | cut -d: -f1 | sort | uniq -c

And this shows which paths it requested most, which is the quickest way to see whether your Disallow lines are being respected:

grep -iE "xAI-SearchBot" /var/log/nginx/access.log \
  | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

Check the requests are genuine

Anyone can send the Grok Search user-agent, so confirm the source before acting on a match:

xAI-SearchBot: we hold no published IP list, so use reverse DNS where the operator documents a hostname and treat an unverifiable Grok Search request with caution.

Frequently Asked Questions

What user-agent string does Grok Search use?

Grok Search is reported to identify itself with xAI-SearchBot. Evidence that the spelling is in real use comes from our own robots.txt study: 6 of the 5,583 valid robots.txt files in the Tranco top 10,000 name it (fetched October 20, 2026), not from xAI, which has not published crawler documentation naming it. Use it in robots.txt as the best available match, and treat a request carrying it as a clue rather than proof of who sent it.

Is Grok Search safe?

Will blocking Grok Search hurt my SEO?

How do I block Grok Search in robots.txt?

Add the following lines to your /robots.txt file:

User-agent: xAI-SearchBot
Disallow: /

This instructs Grok Search not to crawl any path on your site. To block only specific sections, replace / with the path (e.g., Disallow: /blog/).

Does Grok Search respect robots.txt?

Not documented. xAI has published no statement that Grok Search reads robots.txt. A rule for xAI-SearchBot will stop a client that chooses to honour it; for enforcement, add a firewall or CDN rule as well.

How do I verify if Grok Search is crawling my site?

Search your access logs for xAI-SearchBot (for example grep -i "xAI-SearchBot" /var/log/nginx/access.log). Because the operator publishes no IP ranges or reverse-DNS pattern, a match shows that something used the name, not that xAI sent it.