Different platforms give you very different levels of control over robots.txt. This guide shows exactly what you can and cannot change on your setup, and how to make AI crawlers welcome.
In this guide you will learn how to configure WordPress robots.txt for AI search. We will keep it practical, with clear steps, visual breakdowns, and specific actions you can take today. The first step in any AI visibility project is to run a free AI crawler check on your website so you know exactly where you stand against the 196 bots we track across 8 categories.
Key Takeaways
- robots.txt for WordPress: AI-Ready Configuration is a practical, repeatable process, not a one-time fix.
- Most AI visibility problems trace back to access, not content.
- You can verify every change with the free AI crawler check and the robots.txt validator.
- Document your approach so the whole team applies it consistently.
Working With a Platform That Owns the File
On a managed platform the question is not what the ideal file contains but where the file comes from, so establishing ownership precedes every other step.
Find out what is actually serving the file
Determine whether robots.txt comes from the platform default, a plugin setting, a theme template or a physical file. Editing the wrong one produces changes that appear to work and then revert, which is the single most common support case.
Establish a single owner and stop editing elsewhere
When two mechanisms both write the file, the result changes unpredictably and looks like a caching problem. Pick one, document it, and leave the others alone permanently.
Record the served baseline from the live domain
Run an AI crawler checker against the public URL, not the editor preview. On managed platforms these differ often enough that the preview should never be trusted.
Make the change through the owning mechanism
Generate the intended rules with a robots.txt builder, then apply them through whichever mechanism you established as the owner, and publish if the platform requires a separate publish step.
Verify at the custom domain, and again after any theme or platform update
Confirm with a robots checker at the domain your audience actually uses. Platform updates and theme changes silently discard customisations, so this check belongs in your release routine.
Where the Rules Really Live on WordPress
WordPress does not serve a robots.txt file by default. It generates one dynamically, and that virtual file continues to be served until a real file exists at the domain root. Most SEO plugins then take ownership of it from their own settings screen, which means the rules you believe are live are frequently not the rules being served.
The practical result is three possible sources of truth: the WordPress default, a plugin setting, and a physical file. They do not always agree, and the winner depends on which exist. Diagnosing a WordPress robots.txt problem starts with establishing which of the three you are actually looking at.
What to Check on WordPress, in Order
The virtual file wins until a real one exists
WordPress synthesises robots.txt on request, so editing a file that is not at the true web root changes nothing. Uploading a physical file overrides the virtual one, but only when it is genuinely reachable at the domain root, which is not always where the WordPress install lives. Subdirectory installs and reverse proxies both break this assumption routinely. Always confirm by requesting the file from the live domain rather than trusting the location you edited.
SEO plugins own the file and will overwrite you
Yoast, Rank Math and All in One SEO each expose a robots.txt editor and each write the file from their own stored setting. Editing on disk while a plugin also manages the file produces intermittent behaviour that looks like caching, because the file changes back whenever the plugin next writes. Pick one owner. If a plugin manages it, make every change there and never touch the file directly.
Legacy blanket rules predate AI crawlers
Robots.txt recommendations circulated widely years ago included broad Disallow rules aimed at duplicate content and crawl budget, written long before AI user agents existed. Those rules are still sitting in a great many WordPress installs, and they catch modern crawlers as collateral damage. A rule that made sense against a 2015 crawl problem can be the reason no AI engine can read your content today.
Block wp-admin, do not block wp-content
Disallowing wp-admin is sensible and standard. Disallowing wp-content is a common copied mistake that hides your uploads directory, which contains your images, and frequently your theme CSS and JavaScript as well. Blocking those degrades how any renderer sees the page and can make a well-built site look broken to a crawler. Keep the admin path blocked and leave the asset path open.
The Mistake That Costs WordPress Most
Editing the file while a plugin still owns it
The most common WordPress robots.txt support case is a change that appears to work, then reverts. The cause is almost always two owners: a physical file edited over FTP and a plugin that rewrites robots.txt from its own setting on a schedule or on save. The symptom looks like a caching problem, so people clear caches repeatedly and the change keeps disappearing. Establish the single owner first and the problem stops being mysterious.
How to Verify This Changed Anything for WordPress
The check that matters here: Request robots.txt from the live domain, not the editor, and compare it line by line with what your plugin setting says. If they differ, you have found the owner problem before it costs you anything.
Where to Go From Here
WordPress is one piece of a larger picture. The AI bot directory documents every crawler we track with its operator, purpose and safety rating, and the batch checker audits many sites in one pass if you manage a portfolio.
Platform defaults change with every theme and plugin update. Re-run the robot checker after any migration, and verify end to end with an AI crawler checker.
Your WordPress Action Checklist
Five concrete steps, specific to what this guide covered. Work through them in order, changing one thing at a time so you can tell which change produced the result.
- □Identify whether your platform generates robots.txt dynamically, which overrides any static file you upload.
- □Check for an SEO plugin writing its own rules, a frequent source of inherited blocks.
- □Apply the platform-specific pattern from this guide rather than a generic template.
- □Validate the published file at your live domain, not in the editor preview.
- □Re-check after the next theme or plugin update, since these commonly reset the file.
