Free robots.txt builder

Robots.txt generator for the AI era

Build a robots.txt that keeps you in AI answers and out of AI training, or any policy you choose. Start from a preset, tick individual bots from our database of 265 crawlers, and download a file that is checked as you build it.

265crawlers to choose from
7one-click presets
0data sent to our servers

Choose a preset

Not sure? Start with Allow AI Search, Block Training. I warn you the moment a search engine gets blocked by mistake.

Quick Presets

Start with a preset and customize from there. Click a preset to apply it.

Grok (xAI) presets

Applies to GrokBot and xAI-SearchBot, the two xAI names site owners most often write rules for. xAI has not documented either, so treat the rule as a request and add a firewall rule if you need enforcement. Neither preset affects Twitterbot (X link previews) or the Grok Bot agent, which robots.txt does not control.

Which preset fits your site

What each preset blocks and when to use it
Preset	Blocks	Use it when	Watch out for
Allow AI Search, Block Training	Every bot whose purpose is training (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, CCBot and more)	You want to be cited by assistants but not used for training. Most sites land here.	Check that each operator's search crawler stays allowed.
Allow All	Nothing	You want maximum reach and have no licensing concerns.	Scrapers are allowed too.
Block AI Bots	Every AI and LLM bot, including AI search	You do not want any AI system reading your content.	This also removes you from AI answers.
Block Scrapers	Data scrapers such as CCBot, Diffbot and Bytespider	Harvesters are costing you bandwidth.	Aggressive scrapers often ignore robots.txt; add a firewall rule.
Block AI + Scrapers	Both groups above	You want strict content protection.	Same as Block AI Bots: no AI citations.
SEO Optimized	AI bots, scrapers and aggressive bots	You only care about classic search.	Assistants cannot read or cite you.
Block All	Everything	Staging or private sites.	Never ship it to a live site.

The first preset follows where site owners went. Hostinger found GPTBot's coverage fell from 84% to 12% of sites while OAI-SearchBot rose to 55.67% (Hostinger, 20 January 2026), and our robots.txt study of the Tranco top 10,000 (5,583 valid files) found 427 sites blocking GPTBot while leaving OAI-SearchBot open.

Coverage by crawler: owners keep search bots and drop training bots.

The options, one by one

Sitemap URL: Adds a Sitemap: line at the end of the file. Use the full URL, including https. Every major crawler reads it, whichever group it follows.
Crawl-delay: Adds Crawl-delay to each group, in seconds. Bingbot and some smaller crawlers honour it; Googlebot ignores it. Leave it empty unless a crawler is genuinely overloading your server.
Custom Disallow paths: One path per line, added to the default group, for example /admin/ or /cart/. Paths start with a slash and are case-sensitive.
Category comments: Adds a # comment above each group naming its category (AI training, AI search, scrapers). Crawlers ignore comments; they help the next person who edits the file.
Generation timestamp: Adds a dated comment at the top, so you can tell which version is live.
Content-Signal line: Adds Content-Signal: search=yes, ai-train=no, explained further down this page.

The counter under the output shows lines and bytes. Crawlers only have to read the first 500 KiB, which a generated file never comes close to.

Why the generator separates training from search

Because most AI crawling is training, and training sends no visitors. Cloudflare, 29 August 2025 measured training at 79% of AI crawling in July 2025, with search at 17% and user-triggered fetches at 3.2%. Blocking training costs you nothing in AI answers; blocking search removes you from them.

Crawl purpose split, July 2024 vs July 2025.

The generator reads each bot's purpose from the same database the free AI crawler check uses, so a preset stays correct as bots are added.

A worked example

A shop wants AI assistants to read product pages, wants training crawlers out, and has a private account area:

User-agent: *
Disallow: /account/
Disallow: /cart/

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
Disallow: /

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
Disallow: /account/
Disallow: /cart/

Sitemap: https://www.example.com/sitemap.xml

Every crawler not named follows the first group and stays out of the account and cart.
The three training crawlers share one group and are refused everything.
The two search crawlers have their own group, so they ignore the first one completely. That is why the account and cart lines are repeated for them.
The sitemap line helps every crawler find the product pages.

Content-Signal: the optional content-use line

Tick "Add Content-Signal" in Options to add a line such as Content-Signal: search=yes, ai-train=no. Cloudflare introduced the signal and expanded it in July 2026 (Cloudflare, 1 July 2026); it states how content may be used after it is fetched.

Publish and verify the file

Save it as plain text named exactly robots.txt, UTF-8 without a byte-order mark.
Upload it to the root of every host. example.com/robots.txt and blog.example.com/robots.txt are separate files. On WordPress use your SEO plugin's robots.txt editor; on static sites put it in the public folder.
Open it in a browser: it must load with HTTP 200 as plain text, not an HTML page. A 5xx makes some crawlers stop crawling the site entirely.
Paste it into the robots.txt validator to confirm each crawler gets the answer you meant.
Allow up to 24 hours for crawlers to refresh their copy (RFC 9309 allows that cache).
Check the CDN too. A Cloudflare AI Training block can stop crawlers regardless of this file; the AI crawler diagnostic shows that layer.

Where the file goes on common platforms

Publishing robots.txt by platform
Platform	Where to put it	Watch out for
WordPress	The robots.txt editor in your SEO plugin, or a physical file in the site root	A physical file overrides the virtual one WordPress generates
Shopify	The `robots.txt.liquid` theme template	Shopify ships a default; edit it rather than replacing it wholesale
Static sites and Cloudflare Pages	The public or build output folder	Check the deployed URL, not the source file
Nginx or Apache	The document root of each virtual host	Each subdomain needs its own file
Hosted builders (Wix, Squarespace)	The SEO settings panel, where the platform allows edits	Some plans only let you add lines, not remove defaults

Mistakes the generator helps you avoid

Blocking CSS or JavaScript folders. 483 top-site files do this, which can stop crawlers that render pages from seeing them properly.
Putting rules before any User-agent line. 25 top-site files do this, and most parsers ignore those rules.
Naming a crawler to add one rule and forgetting that it no longer reads the * group.
Setting a Crawl-delay for Googlebot, which Google does not support. 28 top-site files still do.
Blocking a search engine by accident. The generator shows an SEO Impact Warning the moment a search engine is ticked.

Robots.txt generator questions

What robots.txt should I use to allow AI search but block AI training?

The Allow AI Search, Block Training preset. It blocks bots whose purpose is training, such as GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and Meta-ExternalAgent, and keeps search engines, AI search crawlers and user-triggered assistants allowed.

Will blocking AI training crawlers affect my Google rankings?

No. Training crawlers such as GPTBot, ClaudeBot and Google-Extended have no part in Google or Bing ranking. Blocking Googlebot or Bingbot, on the other hand, removes your pages from search, which is why the generator warns you before it happens.

What is the difference between AI bots and scrapers in the presets?

AI bots are run by AI companies for training, search or assistants. Scrapers, such as CCBot, Bytespider and Diffbot, collect data in bulk for many uses. Blocking either has no effect on classic search rankings.

Should I add Content-Signal to robots.txt?

It is optional. Content-Signal states how content may be used after it is fetched, but it is not part of RFC 9309 and Google has said it does not read it. Keep the User-agent groups as the main control.

Does Disallow: /private block /Private/?

No. Paths are case-sensitive, while crawler names are not. Add a line for each spelling you serve.

Is robots.txt enough to keep a crawler out?

No. It is a request that well-behaved crawlers follow. To enforce a block, add a firewall or CDN rule for that user agent as well.

Where to go next

Batch URL CheckerCompare up to 20 sites, with CSV export.llms.txt GeneratorWrite a reading list for language models.