Your robots.txt file is the single most powerful lever you have over how crawlers, including AI bots, interact with your website. Get it right and you control your AI visibility. Get it wrong and you can disappear from AI answers overnight.
In this guide you will understand which directive actually keeps content out of AI. We will keep it practical, with clear steps, visual breakdowns, and specific actions you can take today. The first step in any AI visibility project is to run a free AI crawler check on your website so you know exactly where you stand against the 196 bots we track across 8 categories.
Key Takeaways
- noindex vs disallow: Which Stops AI Crawlers? is a practical, repeatable process, not a one-time fix.
- Most AI visibility problems trace back to access, not content.
- You can verify every change with the free AI crawler check and the robot checker.
- Document your approach so the whole team applies it consistently.
How to Change robots.txt Without Breaking Anything
robots.txt failures are rarely subtle in effect and almost always subtle in appearance, so the sequence is built around proving each change before the next one.
Read the file that is actually served
Request robots.txt from the live domain rather than opening the copy in your repository or your CMS settings screen. Those three frequently disagree, and only one of them is what crawlers see.
Identify which rule matches which agent
Group rules by user-agent block and work out which block a given crawler will obey, remembering that the most specific matching block wins and that a crawler obeys exactly one. This is where most misdiagnosis happens.
Change one thing, in one block
Broad rewrites make attribution impossible when something breaks. Edit a single directive, note what you expect to change, and generate the replacement with a robots.txt builder rather than by hand.
Test the rule against real paths before publishing
Run the edited file through a robots checker with the specific URLs you care about, including at least one you intend to block and one you intend to allow. A rule that passes syntax checking can still match nothing.
Verify from outside, then record what you changed
Fetch the live file again and re-run an AI crawler access checker. Then write down the change and the reason, because the next person to debug this file will otherwise be guessing at your intent.
The Part of noindex and Disallow That Trips People Up
noindex and Disallow are habitually treated as two strengths of the same control. They target different things: Disallow is about fetching, noindex is about listing, and for AI crawlers there is a third outcome neither addresses, which is whether your content is used to train a model.
That third outcome is why classical search advice transfers badly. A page can be excluded from an index and still be excellent training material, and a page can be indexed and never cited. Deciding what you actually object to has to precede choosing a directive, because the directives do not map onto the concern.
Four noindex and Disallow Rules Worth Knowing
Disallow is the only one of the two most AI crawlers act on
Training collectors are not building a search index, so an indexing directive is not necessarily meaningful to them. Preventing the fetch is the control with predictable effect. This inverts the usual preference for noindex as the more precise instrument, because precision about indexing is irrelevant to a crawler that is not indexing.
noindex requires the page to be fetched to work
The directive lives in the response, so the crawler must be permitted to request the page. Using both controls on one URL guarantees the noindex is never read. For AI crawlers this matters less than for search engines, but the mechanical trap is identical and catches people who apply search habits here.
Neither prevents content already collected from being used
A directive added today governs future fetches. Content already gathered into a training corpus is not withdrawn by a change to your file, and there is no directive that reaches backwards. This is the most commonly misunderstood limitation, and it makes the timing of the decision matter more than its wording.
Purpose is expressed by which bot you name, not by which directive you use
Since the format has no vocabulary for training versus citation, the only way to express that distinction is to name the crawlers individually. Blocking GPTBot while allowing OAI-SearchBot is a policy statement; there is no directive that says do not train on this. Getting the bot list right is therefore the substance of the work.
The One noindex and Disallow Error Worth Auditing For
Applying noindex in the belief that it prevents training use
noindex is familiar, feels like the careful choice, and is often reached for when the real concern is model training. It addresses search listing and says nothing about training, so the objectionable use continues while the page also loses search visibility. The outcome is the worst of both, arrived at by using the more sophisticated-looking tool.
The One noindex and Disallow Check That Settles It
The check that matters here: For each URL you have restricted, write down which of the three outcomes you were trying to prevent: fetching, listing, or training. Any case where the directive does not match the outcome is a mistake.
Where to Go From Here
noindex and Disallow is one piece of a larger picture. The full list of AI crawlers documents every crawler we track with its operator, purpose and safety rating, and the batch URL checker audits many sites in one pass if you manage a portfolio.
Syntax errors in robots.txt fail silently, so validate before you publish. Use the robots txt check, then confirm the result with an AI crawler access checker.
Your noindex and Disallow Action Checklist
Five concrete steps, specific to what this guide covered. Work through them in order, changing one thing at a time so you can tell which change produced the result.
- □Back up your current robots.txt before editing, so you can revert if a rule behaves unexpectedly.
- □Apply the pattern from this guide to a copy first, then diff it against the live file.
- □Run the robots.txt check to confirm the syntax resolves the way you intend.
- □Check that no Disallow rule shadows a path you meant to allow, which is the most common silent failure.
- □Re-crawl after the change, because engines cache robots.txt and the effect is not instant.
