Most AI visibility problems are not content problems. They are technical problems hiding in plain sight. This guide helps you find and fix them.

In this guide you will learn how to protect AI visibility during a migration. We will keep it practical, with clear steps, visual breakdowns, and specific actions you can take today. The first step in any AI visibility project is to run a free AI crawler check on your website so you know exactly where you stand against the 196 bots we track across 8 categories.

Key Takeaways

  • Migrate Your Site Without Losing AI Visibility is a practical, repeatable process, not a one-time fix.
  • Most AI visibility problems trace back to access, not content.
  • You can verify every change with the free AI crawler check and the robots txt check.
  • Document your approach so the whole team applies it consistently.
The four pillars of AI visibility Four pillars supporting AI visibility. Pillar 1 Access: AI crawlers can reach your pages. Pillar 2 Infrastructure: llms.txt, sitemap and HTTPS in place. Pillar 3 Structure: clear headings, FAQs and schema markup. Pillar 4 Authority: expertise signals and citations from trusted sources. AI VISIBILITY: read, trusted, and cited by AI engines 1 ACCESS Crawlers can reach your pages: robots.txt, WAF, no JS walls 2 INFRASTRUCTURE llms.txt, XML sitemap, HTTPS, clean canonical URLs 3 STRUCTURE Clear H2/H3 headings, FAQs, schema markup, quotable paragraphs 4 AUTHORITY E-E-A-T signals, author pages, mentions on trusted sources Work the pillars in order: authority means nothing if crawlers cannot access your pages in the first place.
The four pillars of AI visibility: access, infrastructure, structure, and authority.

A Diagnostic Order That Narrows Fast

Debugging crawler access is a process of elimination, and the sequence below is ordered by how much each check rules out rather than by how likely it is to be the cause.

5 Steps, in Order
1

Reproduce the failure as the bot experiences it

Request the URL with the crawler user-agent and no JavaScript execution, and record the status code and the returned body. A problem you cannot reproduce deterministically cannot be confirmed fixed.

2

Separate a status-code problem from a content problem

A 403 or 429 is an access decision made by your server or firewall. A 200 that returns an empty shell is a rendering problem. These have nothing in common and mixing them up wastes the most time.

3

Check the layers in front of your application

CDN rules, WAF policies, rate limits and bot-management features all block crawlers without touching robots.txt, and they are frequently configured by a different team. Run an AI bot access checker to see the external result rather than reasoning from your config.

4

Rule robots.txt in or out explicitly

Test the exact failing path against the live file with a robots.txt check. If the file permits it, robots.txt is eliminated and you can stop looking there, which is worth doing early precisely because it is cheap.

5

Fix one layer, retest, then move outward

Changing several layers at once produces a working system with an unknown cause, which will regress. Confirm each fix independently before proceeding.

The Part of Site Migration That Trips People Up

Migrations are planned around what users and search engines need, and AI crawler access is usually not on the checklist. The result is a launch that passes every test, ranks acceptably in search, and quietly stops appearing in AI answers, because a robots.txt default or a new security layer arrived with the new platform.

The recovery is also slower than in search. There is no console reporting that an AI crawler is being refused, so the loss is discovered through absence rather than through an alert, often a quarter later when someone asks why citations stopped.

Four Site Migration Rules Worth Knowing

Record the pre-migration state while you still have it

Capture the live robots.txt, a list of which crawlers can currently reach the site, and the URLs that matter most. After launch this baseline is the only way to tell what changed, and it cannot be reconstructed. Skipping this step is what turns a five minute diagnosis into a speculative investigation.

Assume the new platform ships restrictive defaults

Hosting platforms, CDNs and CMS installs all provide their own robots.txt and their own bot-management settings, and the defaults are frequently more restrictive than what you had. A staging Disallow: / carried into production is the most damaging version. Check the served file on the new platform before pointing the domain at it.

Map every old URL to a single destination, not a chain

Build the redirect map so each retired URL points directly at its final target with a 301. Chains formed during migration are the main way authority is lost, and they are much easier to prevent now than to unpick later when nobody remembers the intermediate steps.

Verify externally on launch day and again a week later

Fetch robots.txt from the live domain, request key pages with a crawler user agent from an external network, and confirm the raw HTML contains your content. Repeat after a week, because CDN configuration and platform defaults frequently settle into place after the initial cutover.

AI Visibility Score breakdown: 70 15 15 point model Horizontal bar chart of the 100-point AI Visibility Score. Bot Access is worth 70 points and covers 12 tier-one AI crawlers. AI Infrastructure is worth 15 points for llms.txt and llms-full.txt. Technical Readiness is worth 15 points for sitemap, schema markup and HTTPS. AI Visibility Score = 100 points Bot Access · 70 pts AI Infra · 15 Tech · 15 Bot Access (70) 12 tier-1 AI crawlers, equal weight Partial access earns 75% credit AI Infrastructure (15) llms.txt found: +10 llms-full.txt found: +5 Technical Readiness (15) XML sitemap: +6 · Schema: +6 HTTPS: +3 Most sites lose points in Bot Access: one blocked tier-1 crawler costs about 5.8 points.
The AI Visibility Score: 70 points for bot access, 15 for AI infrastructure files, 15 for technical readiness.

The Site Migration Mistake That Costs Most

Treating a successful launch as a successful migration

The site loads, the redirects resolve in a browser, search traffic holds, and the project closes. Nothing in that checklist looks at whether AI crawlers can still reach the content, so a new default or a new security layer goes undetected. The loss appears months later as an unexplained decline in citations, long after the change that caused it.

How to Confirm Site Migration Behaves the Way You Think

The check that matters here: On launch day, fetch the live robots.txt and request three key pages with a crawler user agent from outside your network. Compare against the baseline you captured beforehand; any difference is either intended or a bug.

Where to Go From Here

Site Migration is one piece of a larger picture. The AI crawler directory documents every crawler we track with its operator, purpose and safety rating, and the batch URL checker audits many sites in one pass if you manage a portfolio.

Most access failures are invisible from a browser, which is why they persist. Start with an AI bot access checker to see which layer is actually returning the block.

Your Site Migration Action Checklist

Five concrete steps, specific to what this guide covered. Work through them in order, changing one thing at a time so you can tell which change produced the result.

  • □Reproduce the failure with a request that sets the bot user-agent, not your browser default.
  • □Work through the layers in order: DNS, firewall, HTTP status, robots.txt, then rendering.
  • □Compare the raw HTML response against what you see on screen, because many bots never run JavaScript.
  • □Record which layer returned the block before changing anything, so the fix is targeted.
  • □Re-test the exact same request after the fix to confirm the status actually changed.