Grok Search: User Agent, robots.txt Rules & Safety Rating

Q: What user-agent string does Grok Search use?

Grok Search is reported to identify itself with xAI-SearchBot. That spelling comes from Trakkr AI Crawler Observatory, not from xAI, which has not published crawler documentation naming it. Use it in robots.txt as the best available match, and treat a request carrying it as a clue rather than proof of who sent it.

Q: Is Grok Search safe?

Grok Search is rated Caution in this directory. It is a Reported token: xAI has not published documentation for it, so its purpose, crawl rate and robots.txt behaviour are not confirmed by the operator, and anyone can send the same user-agent string. Allow or block it on its stated purpose, and back the robots.txt rule with a firewall rule if the traffic matters to you.

Q: Will blocking Grok Search hurt my SEO?

✅ No SEO Impact - Blocking Grok Search does not affect your rankings in Google, Bing, or any other search engine. Grok Search is an AI crawler, not a traditional search indexer. You can freely block it via User-agent / Disallow: / without any SEO penalty.

Q: How do I block Grok Search in robots.txt?

Add the following lines to your /robots.txt file: User-agent: xAI-SearchBot Disallow: / This instructs Grok Search not to crawl any path on your site. To block only specific sections, replace / with the path (e.g., Disallow: /blog/).

Q: How do I verify if Grok Search is crawling my site?

Search your access logs for xAI-SearchBot (for example grep -i "xAI-SearchBot" /var/log/nginx/access.log). Because the operator publishes no IP ranges or reverse-DNS pattern, a match shows that something used the name, not that xAI sent it.

Quick Facts

User-Agent:: xAI-SearchBot
Category:: AI & LLM Bots
Operator:: xAI
Safety:: Use Caution
Blocking Impact:: Low - No SEO ranking impact
SEO Impact Score:: 2/10
Official docs:: None published by xAI
Named by:: Trakkr AI Crawler Observatory

Reported, not operator documented

xAI has not published a crawler page that names xAI-SearchBot. It is listed here because a credible public source does: Trakkr AI Crawler Observatory. Observed UA for xAI Grok search retrieval (Mozilla/5.0 (compatible; xAI-SearchBot/1.0; +https://x.ai)). xAI publishes no crawler page.

What that means for you: the robots.txt rule on this page is the best available spelling, and a well-behaved crawler using this string will honour it, but there is no vendor statement of purpose, crawl rate or robots.txt compliance to rely on. Treat a matching request in your logs as a detection clue, not proof of who sent it, and combine the rule with a firewall rule if the traffic matters to you. If the operator publishes documentation, this profile moves to Official docs.

What is Grok Search?

xAI-SearchBot is a user-agent name reported for xAI's Grok search retrieval. xAI has not published a page that documents it.

The reported string is Mozilla/5.0 (compatible; xAI-SearchBot/1.0; +https://x.ai). The link in it points at xAI's homepage, not at a crawler page, and xAI has published no crawler documentation, IP ranges or robots.txt policy. Write robots.txt rules for it alongside GrokBot, the other reported xAI name.

What happens if you block Grok Search?

✅ No SEO Impact - Blocking Grok Search does not affect your rankings in Google, Bing, or any other search engine. Grok Search is an AI crawler, not a traditional search indexer. You can freely block it via User-agent / Disallow: / without any SEO penalty.

Generally safe to allow; provides legitimate crawling value.

How to block Grok Search with robots.txt

<code>User-agent: xAI-SearchBot</code> - Matching is case-insensitive. Robots.txt is fetched from the root of each subdomain separately.

Block completely (robots.txt)

User-agent: xAI-SearchBot
Disallow: /

Allow all (robots.txt)

User-agent: xAI-SearchBot
Allow: /

Block private only (robots.txt)

User-agent: xAI-SearchBot
Disallow: /private/
Disallow: /api/
Disallow: /admin/
Allow: /

Nginx server block

# Nginx: Hard-block Grok Search
if ($http_user_agent ~* "xAI-SearchBot") {
    return 403 "Bot blocked";
}

Apache .htaccess

# Apache: Hard-block Grok Search
SetEnvIfNoCase User-Agent "xAI-SearchBot" bad_bot
Order Allow,Deny
Allow from all
Deny from env=bad_bot

Meta robots tag

X-Robots-Tag header

X-Robots-Tag: noindex, nofollow

Is Grok Search safe to allow?

Grok Search is rated Caution in this directory. It is a Reported token: xAI has not published documentation for it, so its purpose, crawl rate and robots.txt behaviour are not confirmed by the operator, and anyone can send the same user-agent string. Allow or block it on its stated purpose, and back the robots.txt rule with a firewall rule if the traffic matters to you.

xAI publishes no IP ranges or reverse-DNS pattern for Grok Search, so the user-agent string cannot be verified. Rate-limit unusual volume instead of trusting the name.

What does Grok Search do?

Understanding Grok Search's purpose helps you decide whether to allow or block it.

Indexes web pages for AI-powered search answers
Surfaces your content as citations inside AI search results
Builds a knowledge index used to answer user questions
Allowing it can increase visibility in AI search
robots.txt behaviour not documented by the operator

Frequently Asked Questions

What user-agent string does Grok Search use?

Grok Search is reported to identify itself with xAI-SearchBot. That spelling comes from Trakkr AI Crawler Observatory, not from xAI, which has not published crawler documentation naming it. Use it in robots.txt as the best available match, and treat a request carrying it as a clue rather than proof of who sent it.

Is Grok Search safe?

Grok Search is rated Caution in this directory. It is a Reported token: xAI has not published documentation for it, so its purpose, crawl rate and robots.txt behaviour are not confirmed by the operator, and anyone can send the same user-agent string. Allow or block it on its stated purpose, and back the robots.txt rule with a firewall rule if the traffic matters to you.

Will blocking Grok Search hurt my SEO?

✅ No SEO Impact - Blocking Grok Search does not affect your rankings in Google, Bing, or any other search engine. Grok Search is an AI crawler, not a traditional search indexer. You can freely block it via User-agent / Disallow: / without any SEO penalty.

How do I block Grok Search in robots.txt?

Add the following lines to your /robots.txt file:

User-agent: xAI-SearchBot
Disallow: /

This instructs Grok Search not to crawl any path on your site. To block only specific sections, replace / with the path (e.g., Disallow: /blog/).

Does Grok Search respect robots.txt?

Not documented. xAI has published no statement that Grok Search reads robots.txt. A rule for xAI-SearchBot will stop a client that chooses to honour it; for enforcement, add a firewall or CDN rule as well.

How do I verify if Grok Search is crawling my site?

Search your access logs for xAI-SearchBot (for example grep -i "xAI-SearchBot" /var/log/nginx/access.log). Because the operator publishes no IP ranges or reverse-DNS pattern, a match shows that something used the name, not that xAI sent it.