Grok Crawler User Agent: The Official Answer and robots.txt Rules

If you have searched for the Grok crawler, you have probably found three different answers. Some guides give you a user agent called GrokBot. Others say xAI-SearchBot. A few tell you to allow something called xAI-Bot. And since August 2026 there is also a product called Grok Bot, which sounds like a crawler but is not one.

This guide sorts it out using only what xAI and other primary sources actually publish, checked on 1 October 2026. Short version: xAI has not documented any crawler. You can still write sensible rules, and you should, but you need to know what they can and cannot do. Before you start, run a free AI crawler check on your site to see how it currently treats the 257 bots we track.

Is there an official Grok crawler user agent?

No. This is the most important fact in the whole topic, so it is worth being precise about what we checked.

• xAI's developer documentation lists every page in its llms.txt index. None of them is a crawler page, and none names a user agent that xAI sends when it fetches websites.
• xAI's Web Search tool documentation says the tool lets Grok "search the web in real-time and browse web pages". It does not say which user agent those requests carry, or whether they read robots.txt.
• Cloudflare's AI crawler reference lists crawlers from OpenAI, Anthropic, Perplexity, Google, Microsoft, ByteDance, Common Crawl, Meta, Apple, Amazon, DuckDuckGo and Mistral. It lists nothing from xAI.

Compare that with OpenAI, Anthropic or Google, which each publish a page naming every crawler, what it is for, and how to block it. For xAI, every user-agent string you will find comes from someone watching their logs, not from xAI.

The Grok user-agent strings that are reported

Two names come up repeatedly in public crawler registries such as Trakkr. We list both in our directory with a Reported badge, which means a credible public source names the token but the operator does not.

Notice that the xAI-SearchBot string links to https://x.ai, the company homepage, rather than to a page about the crawler. Documented crawlers usually link to a page that explains them, which is one more sign that xAI has not published one.

robots.txt matching is case-insensitive under RFC 9309, so grokbot and GrokBot are the same rule. Spelling still matters, though. A rule for xAI or Grok matches nothing that registries have reported.

Grok crawler vs Grok Bot: two different things

On 11 August 2026 xAI launched Grok Bot (launch post). The name is so close to GrokBot that most of the recent confusion comes from here. They are not the same thing, and they need different controls.

According to the Grok Bot documentation, each Bot "works on a persistent cloud computer with a browser, filesystem, and terminal" and "takes on multi-step work across apps and websites". In other words, it is a browser operated by an AI for a person: it signs into tools, fills in forms and reads pages, the way a human assistant would.

The Grok Bot security page adds a useful detail for website owners: hosted computers reach the internet through "shared static egress IP addresses", shared across every Grok Bot customer. xAI hands out the ranges through its account team rather than publishing them. And the Grok Bot security FAQ admits that some websites block Bots because they "flag datacenter IP addresses".

If your concern is agents rather than crawlers, read our guide to building a website that works for AI agents, and see how ChatGPT Agent handles the same problem with signed requests.

How to allow or block the Grok crawler in robots.txt

Because two names are in circulation, write one group that names both. A group with several User-agent lines applies its rules to all of them.

Allow Grok to read and cite your public pages

User-agent: GrokBot
User-agent: xAI-SearchBot
Allow: /
Disallow: /account/
Disallow: /cart/

Sitemap: https://example.com/sitemap.xml

Naming the bots explicitly matters if your User-agent: * group is strict. A crawler obeys only the most specific group that matches it, so without its own group it inherits whatever you wrote for everyone else.

Block Grok completely

User-agent: GrokBot
User-agent: xAI-SearchBot
Disallow: /

Block Grok but keep X link previews

Link previews on X come from Twitterbot, which is a separate, documented token. Blocking Grok does not affect it, but it is worth saying so in the file so nobody "fixes" it later.

User-agent: GrokBot
User-agent: xAI-SearchBot
Disallow: /

User-agent: Twitterbot
Allow: /

After editing, paste the file into the robots.txt validator to confirm each group does what you meant, or build it with the Grok presets in the robots.txt generator.

Does the Grok crawler respect robots.txt?

Nobody can answer this with certainty, because xAI has never said. Here is how we would reason about it.

• A rule costs nothing. If the crawler honours robots.txt, your rule works. If it does not, you have lost nothing.
• A rule is not enforcement. If keeping Grok out matters, add a firewall or CDN rule that refuses requests carrying the two names. See Cloudflare bot blocking vs robots.txt.
• Agents ignore it by design. Grok Bot behaves like a person in a browser, so robots.txt was never meant to apply to it.

Some log analysts report Grok-related traffic arriving without any xAI name at all, using ordinary browser user agents. We cannot confirm those reports, and xAI has not addressed them. If you see heavy traffic you cannot identify, the safe response is rate limiting, which slows abuse without blocking real visitors. Our AI crawler rate limiting guide covers the options.

Enforcing a Grok block at the server or CDN

robots.txt asks. A server or CDN rule refuses. If you have decided Grok should not read your content, pair the robots.txt group above with one of these. Each refuses requests whose user agent contains either reported xAI name, and leaves everyone else alone.

Cloudflare (WAF custom rule)

In the Cloudflare dashboard, go to Security, then WAF, then Custom rules, and create a rule with this expression and the action Block. Cloudflare's own AI crawler reference has no xAI entry, so its managed AI-bot toggles do not cover Grok by name. A custom rule is the reliable route.

(lower(http.user_agent) contains "grokbot") or (lower(http.user_agent) contains "xai-searchbot")

Nginx

Inside the server block for your site:

if ($http_user_agent ~* "(GrokBot|xAI-SearchBot)") {
    return 403;
}

Apache (.htaccess)

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (GrokBot|xAI-SearchBot) [NC]
RewriteRule .* - [F,L]

Keep the robots.txt group even when you add one of these. It is the only way to tell a cooperative crawler what you want before it sends a request, and it documents your decision for whoever maintains the site next.

One caution: none of these rules stop Grok Bot, the agent, because it does not send either name. Cloudflare Radar lists it under a different pattern, GrokAgent, covered on its own profile. Grok Bot can only be filtered by its IP addresses, which xAI gives to its customers rather than publishing, or by the usual human checks such as logins and CAPTCHAs. Our guide to firewalls that block AI crawlers by accident covers the opposite problem, where a strict rule stops bots you wanted.

Common mistakes with Grok rules

These are the errors we see most often when we audit robots.txt files for Grok. Each one makes a rule look right while doing nothing.

1. Writing User-agent: xAI or User-agent: Grok. Neither has been reported as a user agent. The rule reads sensibly to a person and matches no crawler.
2. Naming only one of the two tokens. If you block GrokBot but not xAI-SearchBot, the second name falls through to your User-agent: * group, which is often wide open.
3. Copying an "xAI-Bot" rule from a guide. We found no source for that name. It costs nothing to include, but it is not a substitute for the two reported names.
4. Expecting robots.txt to stop Grok Bot. The agent browses like a person and was never covered by robots.txt. If you need to control it, the tools are your firewall and your login and checkout flows.
5. Blocking Twitterbot to stop Grok. Twitterbot builds link previews on X. Blocking it breaks how your links look when people share them, and has nothing to do with Grok.
6. Allowing Grok in robots.txt while the firewall blocks it. Bot-management products often challenge unfamiliar crawlers. The robots.txt allow then has no effect, and nothing tells you.

If you manage several sites, the batch URL checker runs the same test across up to 20 of them at once, which is the quickest way to find the one property that still has an old rule. For a broader list of errors, see robots.txt mistakes that block AI.

How Grok uses what it reads

It helps to know where your pages could end up once Grok reads them, because that is what the allow or block decision is really about. xAI documents three relevant capabilities, none of which name a crawler.

• Web search. The Web Search tool documentation lets Grok search and open web pages while answering. Developers can restrict it to up to five domains with allowed_domains, or exclude up to five with excluded_domains. Those settings belong to the developer calling the API, not to you as the site owner.
• X search. Grok can search posts, users and threads on X directly, as described above.
• Grok Bot. The agent can open any public page in its browser while doing a task for its user, such as comparing suppliers or filling in a form.

What xAI does not say is whether pages fetched by any of these are also used to train future Grok models. Operators such as OpenAI and Anthropic split training and search into separately named crawlers so that you can choose. xAI offers no such split, so allowing Grok means accepting whatever use xAI makes of the content.

What will change if xAI documents its crawler

Every major AI company that started without documentation has eventually published it, usually after pressure from publishers. If xAI does, expect three things to change, and plan to revisit your rules when it happens.

• The token names may change. A documented name might match one of the reported ones, or it might not. Keep your rules in one clearly labelled group so they are easy to update.
• Verification becomes possible. Published IP ranges or a reverse DNS pattern would let you tell a real xAI request from an impostor, which is not possible today.
• Training and search may split. If xAI separates the two, you could allow Grok to cite you while opting out of training, as you already can with OpenAI and Anthropic.

We check vendor documentation for every bot in the crawler directory each release. When xAI publishes a page, the GrokBot profile will move from Reported to Official and link to it, and this guide will change with it.

How to tell whether Grok is visiting you

For documented crawlers you can confirm a visit two ways: check the IP against the operator's published ranges, or run a reverse DNS check. Neither works for Grok, because xAI publishes no ranges and no hostname pattern. What you can do is look for the names.

grep -iE "GrokBot|xAI-SearchBot" /var/log/nginx/access.log | awk '{print $1}' | sort | uniq -c | sort -rn | head

This shows which IPs used the names and how often. A small, steady set of IPs is consistent with a real crawler. Hundreds of unrelated IPs, or requests hammering one page, look more like someone borrowing the name. Our guide to spotting spoofed AI crawlers explains the patterns, and tracking AI bots in server logs shows how to make this a routine report.

One more check is worth doing: make sure your firewall is not already blocking Grok without you knowing. The AI crawler checker tests your robots.txt against both xAI names, and the AI crawler diagnostic shows which layer answers a crawler on a given page: robots.txt, the server, or a firewall challenge. Look in your firewall's event log for requests carrying the two names, too.

Should you allow or block Grok?

Blocking Grok has no effect on your Google or Bing rankings. GrokBot and xAI-SearchBot are AI crawler names, not search engine indexers. The decision is only about whether Grok can read your pages.

If you are unsure where your content ends up after it is read, our guide to training bots vs search bots explains why the purpose of a crawler matters more than its name. And blocking AI training while allowing AI search shows how to split the decision for operators that document separate bots.

What robots.txt can never control: X posts

Grok has a source that no website setting reaches. xAI's X Search tool documentation describes a tool that lets Grok run "keyword search, semantic search, user search, and thread fetch on X". Posts about your brand on X can shape what Grok says about you whatever your robots.txt says.

That is a strong argument for allowing Grok's web crawlers if you care about how Grok describes you. If X conversation is going to inform Grok anyway, your own pages are the most accurate counterweight available. Blocking the crawler does not remove you from Grok. It only removes your version of the story.

Grok compared with the documented AI crawlers

To see why the lack of documentation matters, compare what you can do with Grok against what you can do with operators that publish a crawler page.

With OpenAI you can allow OAI-SearchBot so ChatGPT search can cite you, block GPTBot so your pages are not used for training, and check requests against the operator's published details. None of that is possible with Grok today. Every rule you write for xAI is a best guess at a name, with no way to check the traffic behind it.

That is not a reason to ignore Grok. It is a reason to keep your Grok rules simple, labelled and easy to update, and to lean on server-side controls for anything that really matters.

Three real-world examples

An online store that wants AI recommendations

A homeware shop wants to appear when people ask Grok for product suggestions. It allows both reported names, keeps product pages server-rendered so prices and stock are in the HTML, and checks that its bot-management tool is not challenging unfamiliar crawlers. It also removes a CAPTCHA from its product search, which had been stopping AI agents like Grok Bot that customers send to compare prices.

A research publisher protecting its archive

A subscription research site blocks both names in robots.txt and adds the Cloudflare rule above, because a robots.txt request alone is not enough for content it sells. It accepts that Grok will not quote its reports directly, and relies on its own posts on X to shape how Grok describes it.

An agency managing forty client sites

A marketing agency had copied User-agent: xAI into every client's robots.txt from an old guide. A batch check showed the rule matched nothing on all forty sites. It replaced the line with the two reported names, set each client's policy to match their goals, and added a note to its monthly audit to recheck xAI's documentation.

Your Grok crawler checklist

• □Search your robots.txt for GrokBot, xAI-SearchBot and any xAI or Grok lines. Replace unreported names with the two reported ones.
• □Decide allow or block on purpose, not by default, and write one group that names both tokens.
• □If you block, add a firewall or CDN rule for the same two names, because robots.txt is a request.
• □Check your firewall logs for challenges or 403s to xAI names, which override robots.txt entirely.
• □If AI agents matter to your business, test your key flows for CAPTCHAs and login walls that would stop Grok Bot.
• □Re-test with an AI bot access checker after every change, and check back here: if xAI publishes documentation, we will update the tokens and the badge.

For the wider picture of every AI crawler, see the AI and LLM bot directory, the xAI operator page, our explainer on Grok Bot vs GrokBot, and how to get cited by Grok. To check your own logs for the two names, use the Grok log checker.