robots.txt is a text file placed at the root of a website that tells web crawlers which pages to access and which to ignore. It follows the Robots Exclusion Protocol (RFC 9309).

Can I block AI bots like ChatGPT and Claude with robots.txt?

Yes. You can block AI training crawlers like GPTBot, ClaudeBot, and others using specific User-agent rules in robots.txt. Blocking AI bots has zero impact on your SEO rankings.

Does blocking bots in robots.txt affect SEO?

Blocking AI bots and scrapers has no SEO impact. However, blocking search engines like Googlebot or Bingbot will remove your site from search results. Our generator clearly marks which bots are safe to block.

How do I use the generated robots.txt?

Copy the generated content and save it as robots.txt in the root directory of your website. It must be accessible at yourdomain.com/robots.txt.

Robots.txt Generator

Create a custom robots.txt file to control which bots can crawl your website. Choose from presets or select individual bots from our database of 257+ crawlers.

Quick Presets

Start with a preset and customize from there. Click a preset to apply it.

Grok (xAI) presets

Applies to GrokBot and xAI-SearchBot, the two names public registries report. xAI has not documented either, so treat the rule as a request and add a firewall rule if you need enforcement. Neither preset affects Twitterbot (X link previews) or the Grok Bot agent, which robots.txt does not control.

0 bots blocked

Options

Sitemap URL (optional)

Crawl-delay (seconds) (optional)

Custom Disallow paths (one per line)

Add category comments Add generation timestamp Add Content-Signal line (search=yes, ai-train=no) what is this?

Generated robots.txt

User-agent: *
Allow: /

2 lines 24 bytes

Robots.txt generator presets compared

What each preset blocks
Preset	Blocks	Keeps	Best for
Allow AI Search, Block Training	Every bot whose purpose is training (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, CCBot and more)	Search engines, AI search indexes, user-triggered assistants	Most sites in 2026
Allow All	Nothing	Everything	Sites that want maximum AI reach
Block AI Bots	Every AI & LLM bot, including AI search	Search engines, SEO tools	Sites opting out of AI entirely
Block Scrapers	Data scrapers such as CCBot, Diffbot, Bytespider	AI bots, search engines	Reducing load
Block AI + Scrapers	Both groups	Search engines, SEO tools	Strict content protection
SEO Optimized	AI bots, scrapers, aggressive bots	Search engines	Classic SEO only
Block All	Everything	Nothing	Staging sites only

The new first preset reflects where the market went: Hostinger found GPTBot's coverage fell from 84% to 12% of sites while OAI-SearchBot rose to 55.67% (Hostinger, 20 January 2026).

Coverage by crawler: owners keep search bots and drop training bots.

Why the generator separates training from search

Because most AI crawling is training, and training sends no visitors. Cloudflare, 29 August 2025 measured training at 79% of AI crawling in July 2025, with search at 17% and user-triggered fetches at 3.2%.

Crawl purpose split, July 2024 vs July 2025.

The generator reads each bot's purpose from the same database the AI crawler checker uses, so the preset stays correct as bots are added.

Content-Signal: the optional content-use line

Tick "Add Content-Signal" in Options to prepend a line such as Content-Signal: search=yes, ai-train=no. Cloudflare introduced the signal and expanded it in July 2026 (Cloudflare, 1 July 2026); it states how content may be used after it is fetched.

Content-Signal: search=yes, ai-train=no

User-agent: GPTBot
Disallow: /

Deploy and verify your new robots.txt

Upload the file to the root of every host: example.com/robots.txt and blog.example.com/robots.txt are separate files.
Serve it with a 200 status and text/plain. A 5xx makes some crawlers stop entirely.
Wait up to 24 hours for crawlers to refresh their cache (RFC 9309).
Paste the output into the robots.txt validator before deploying, then run a free AI crawler check on the live site.
Check the CDN too. A Cloudflare AI Training block can stop Googlebot and Bingbot regardless of this file (Cloudflare, 1 July 2026); the AI crawler diagnostic will show it.

AI crawler checker: tests 257 bots against your robots.txt in one scan.
AI crawler diagnostic: fetches a page as each AI bot and names the layer that blocks it.
robots.txt validator: shows allowed, blocked and partial status per bot.
batch URL checker: compares up to 20 sites side by side, with CSV export.
llms.txt generator: writes the file AI assistants read for a site summary.
schema markup generator: produces JSON-LD for eight common types.
GEO audit: scores a page for generative engine optimization.

Frequently Asked Questions

What robots.txt should I use to allow AI search but block AI training?

Use the Allow AI Search, Block Training preset. It blocks bots whose purpose is training, such as GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and Meta-ExternalAgent, and keeps search engines, AI search indexes and user-triggered assistants allowed.

Should I add Content-Signal to robots.txt?

It is optional. Content-Signal states how content may be used after it is fetched, but it is not part of RFC 9309 and Google has said it does not read it. Keep the User-agent groups as the main control.

What is robots.txt and why do I need one?

robots.txt is a text file placed at the root of your website (e.g., yoursite.com/robots.txt) that tells web crawlers which pages they can and cannot access. It follows the Robots Exclusion Protocol (RFC 9309). While it's voluntary (bots can choose to ignore it), all major search engines and legitimate AI crawlers respect it. Having a properly configured robots.txt is essential for controlling how bots interact with your site.

Will blocking AI bots affect my SEO rankings?

No. Blocking AI training crawlers like GPTBot, ClaudeBot, PerplexityBot, and others has zero impact on your Google, Bing, or other search engine rankings. These AI bots are used for LLM training, not search indexing. However, blocking search engine bots (Googlebot, Bingbot, etc.) will remove your pages from search results. Our generator clearly marks which bots are safe to block.

How do I deploy the generated robots.txt?

Save the generated content as robots.txt and upload it to the root directory of your website. It must be accessible at https://yoursite.com/robots.txt. For WordPress, you can use a plugin like Yoast SEO or upload via FTP. For static sites, place it in your public folder. For Cloudflare Pages, add it to your public/ directory.

What's the difference between blocking AI bots and scrapers?

AI bots (GPTBot, ClaudeBot, etc.) collect content specifically for training language models. Blocking them prevents your content from being used in AI training datasets. Scrapers (CCBot, Bytespider, Diffbot, etc.) are general-purpose data collection bots that may aggregate content for various purposes. Both categories are safe to block without SEO impact.

Can I verify my robots.txt is working?

Yes! After deploying your robots.txt, use our AI Crawler Check tool to verify which bots are blocked and which are allowed. It scans your robots.txt against 257+ bots and gives you a detailed report. You can also use Google Search Console's robots.txt tester for Googlebot-specific validation.

Verify Your robots.txt

After deploying, check if your bots are properly blocked

Check AI Crawler Access