How to Create a robots.txt File for AI Bots: Step-by-Step (2026)

Your robots.txt file is the front door to your website for AI crawlers. Every time GPTBot, ClaudeBot, PerplexityBot, or any other AI bot visits your site, the first thing it checks is your robots.txt. If you do not have one, or if it is not configured correctly, you have no control over which AI bots access your content. The good news is that creating a robots.txt file is simple, and this guide will walk you through every step.

Whether you are a complete beginner who has never touched a robots.txt file, or an experienced webmaster who wants to add AI-specific rules, this tutorial covers everything. We will start with the basics of what a robots.txt file is, then move to writing rules for specific AI bots, provide copy-paste templates for common scenarios, and finish with testing and verification steps.

Before we begin, check your current setup. Use the AI crawl checker to scan your website and see which AI bots can currently access your content. This gives you a clear starting point.

What is robots.txt and Why Does It Matter for AI?

A robots.txt file is a simple text file that lives at the root of your website. When any web crawler visits your site, it first looks for this file at yoursite.com/robots.txt. The file contains rules that tell the crawler which parts of your site it can access and which parts it should skip.

The robots.txt standard (also called the Robots Exclusion Protocol) has been around since 1994. For decades, it was mainly used to manage search engine crawlers like Googlebot and Bingbot. But the explosion of AI crawlers in 2024 and 2025 changed the game completely. Now there are over 150 known AI crawlers, and your robots.txt file is your primary tool for controlling them.

Here is why robots.txt matters more than ever in the AI era:

The bottom line is that every website needs a well-configured robots.txt file. Without one, you are giving all AI crawlers unlimited access to your content by default.

Understanding robots.txt Syntax

Before you create your file, you need to understand the basic syntax. Robots.txt files use a very simple format with just a few directives. Here are the ones you will use most often:

Here is a simple example that blocks GPTBot but allows everything else:

Important rules about robots.txt syntax:

Step-by-Step: Creating Your robots.txt File

Follow these steps to create your robots.txt file from scratch. The entire process takes about 10 to 15 minutes.

Step 1: Decide your AI bot strategy

Before writing any code, decide how you want to handle AI crawlers. There are three main strategies:

Most website owners in 2026 choose Strategy 2 (selective access). This gives you the best balance between protecting your content and being visible in AI search. You can refine your strategy later as AI search evolves.

Step 2: Open a text editor

Create a new file called robots.txt in any plain text editor. You can use Notepad (Windows), TextEdit (Mac), VS Code, Sublime Text, or any code editor. Do not use Microsoft Word or Google Docs, because these add hidden formatting that will break the file.

If you are using a CMS like WordPress, Shopify, or Squarespace, you may not need to create the file manually. Many CMS platforms have built-in robots.txt editors or plugins that let you edit the file from your dashboard. Check your CMS documentation for specifics.

Step 3: Write the search engine rules first

Always start with rules for traditional search engine crawlers. This makes sure Google, Bing, and other search engines can access your site properly:

Step 4: Add your AI bot rules

Now add rules for the AI crawlers based on your chosen strategy. Here are the most important AI bots to include:

For a complete configuration with all 150+ bots, use the Robots.txt Generator tool. It lets you toggle each bot individually with a single click.

Step 5: Add the sitemap directive

At the end of your robots.txt file, add a Sitemap directive. This helps all crawlers find your pages more efficiently:

Replace yoursite.com with your actual domain name. If you have multiple sitemaps, you can add multiple Sitemap lines.

Step 6: Upload the file to your server

Save the file as robots.txt (all lowercase) and upload it to the root directory of your website. The file must be accessible at https://yoursite.com/robots.txt.

How you upload depends on your hosting setup:

Step 7: Test and verify

After uploading, verify that your file is working correctly. Open your browser and navigate to https://yoursite.com/robots.txt. You should see the plain text content of your file. Then use check which AI bots can crawl your site to confirm each bot is being handled correctly.

Ready-to-Use Templates

Here are three complete templates you can copy and paste. Choose the one that matches your strategy and customize it for your domain.

Template 1: Block all AI bots (maximum protection)

Template 2: Selective access (recommended)

Template 3: Maximum AI visibility

Common Mistakes to Avoid

Many website owners make these mistakes when creating their robots.txt file. Avoiding them will save you time and prevent issues:

Advanced robots.txt Techniques

Once you have the basics down, you can use these advanced techniques to fine-tune your AI crawler management:

Path-specific rules

Instead of blocking an AI bot from your entire site, you can block access to specific directories while allowing others:

This approach lets you get the benefits of AI visibility for your public content while protecting your premium or private pages.

Wildcard patterns

Use the * wildcard to match patterns in URLs. Use $ to match the end of a URL:

Combine with llms.txt

For maximum AI visibility, create an llms.txt file alongside your robots.txt. While robots.txt controls access, llms.txt provides AI models with structured information about your site, including what topics you cover, your key pages, and how you want to be cited. Together, these two files give you comprehensive control over how AI systems interact with your website.

Your robots.txt handles the "who can visit" question, while llms.txt handles the "what should AI know about us" question. Both are important for a complete AI optimization strategy.

Testing and Verification

After creating and uploading your robots.txt file, always test it. Here are the best ways to verify that everything is working:

It is a good practice to check your robots.txt file regularly, at least once a month. New AI crawlers appear frequently, and your strategy may need to evolve as the AI landscape changes.

Skip the Manual Work: Use the Robots.txt Generator

If writing robots.txt rules manually seems complicated, the Robots.txt Generator does all the work for you. It includes all 150+ known AI crawlers, lets you toggle each one individually, and generates a complete, properly formatted robots.txt file ready to download and upload to your server.

The generator also includes preset templates for common scenarios, so you can get started in seconds without knowing any robots.txt syntax. It is the fastest way to create a comprehensive robots.txt file that covers every known AI crawler.

Ongoing Monitoring and Maintenance

Creating your robots.txt file is not a one-time task. The AI crawler landscape changes rapidly. Here is how to keep your robots.txt file effective over time:

Key Takeaways