In this guide you will learn how to allow citation bots while blocking training scrapers. We will keep it practical, with clear steps, visual breakdowns, and specific actions you can take today. The first step in any AI visibility project is to run an AI crawl checker on your website so you know exactly where you stand against the 265 bots we track across 8 categories.
Key Takeaways
- Block AI Training but Allow AI Search: The Hybrid Setup is a practical, repeatable process, not a one-time fix.
- Most AI visibility problems trace back to access, not content.
- You can verify every change with the quick AI crawlability check and the robots.txt validator.
- Document your approach so the whole team applies it consistently.
How to Change robots.txt Without Breaking Anything
Read the file that is actually served
For a hybrid setup that blocks training but allows search, open the live robots.txt in a browser instead of trusting the editor, because caches and platforms can serve a different version.
Identify which rule matches which agent
Walk each AI user agent through the groups in the file for a hybrid setup that blocks training but allows search and note which group wins, keeping the split between training tokens and answer-engine tokens in mind as you go.
Change one thing, in one block
For Block AI Training but Allow AI Search, start with robots.txt generator.
Test the rule against real paths before publishing
Paste real URLs from the training and search groups in your file into a tester and check each AI bot gets the result you intended before the file goes live.
Verify from outside, then record what you changed
Fetch the live file again and re-run an AI crawler checker. Then write down the change and the reason, because the next person to debug this file will otherwise be guessing at your intent.
Why the Training and Citation Split Is Harder Than It Looks
The position most publishers and businesses actually want is specific: do not use my content to train a model, but do cite me when answering a live question. robots.txt has no syntax for that distinction, so the entire policy has to be expressed as a list of which named crawlers you permit.
That makes the work factual rather than strategic. You need to know which agent belongs to which purpose, and the answer changes as vendors add crawlers. A policy that was correct when written becomes wrong through no action of yours, which is why this needs a review interval rather than a decision.
Where People Get the Training and Citation Split Wrong
Learn which agent does which job, per vendor
OpenAI operates separate crawlers for training and for answering live queries, and Google separates its search crawler from its AI training signal. Blocking the vendor by guessing at names catches the wrong one about as often as the right one. The list is short, it is documented, and reading it is the highest-value ten minutes in this whole exercise.
A named block must restate the wildcard protections you still want
Because a crawler follows only its most specific matching block, adding a named block for a training bot removes it from every rule under User-agent: *. If those wildcard rules protected paths you care about, they have to be repeated inside the new block. Skipping this is how a tightening change becomes a loosening one.
Verify from outside, because your firewall may disagree
A robots.txt that permits retrieval crawlers means nothing if a CDN bot-management rule blocks them at the edge. These are configured by different people and neither is visible from the other, so the only reliable check is an external fetch that reports what a crawler actually receives.
Expect the list to change and schedule the review
New crawlers appear, existing ones split into specialised agents, and vendors rename them. A policy is only as current as its last review, and nothing notifies you when it goes stale. A recurring calendar entry is a more effective control than any amount of care taken on the day.
The Training and Citation Split Mistake That Costs Most
Blocking a vendor wholesale to stop training
When the concern is training, the instinct is to block everything belonging to that vendor. This also blocks the retrieval agent that would have cited you with a link, so you give up the traffic you wanted while frequently failing to stop the training you objected to, because the training crawler was named differently. The loss appears as absence and nothing flags it.
The One Training and Citation Split Check That Settles It
The check that matters here: For each major vendor, name the crawler you are blocking and the crawler you are allowing. If you cannot name both, the policy is a guess rather than a decision.
Where to Go From Here
Training and Citation Split is one piece of a larger picture. The full list of AI crawlers documents every crawler we track with its operator, purpose and safety rating, and the multi-site crawler check audits many sites in one pass if you manage a portfolio.
Syntax errors in robots.txt fail silently, so validate before you publish. Use the robots.txt check, then confirm the result with an AI crawler checker.
Your Training and Citation Split Action Checklist
Applied to Block AI Training but Allow AI Search, it looks like this:
- □Back up your current robots.txt before editing, so you can revert if a rule behaves unexpectedly.
- □Read the file that is actually served.
- □Run the robots checker to confirm the syntax resolves the way you intend.
- □Identify which rule matches which agent.
- □Re-crawl after the change, because engines cache robots.txt and the effect is not instant.
