Why AI Bots Can't Crawl Your Website: 7 Layers to Check (and How to Fix Each)

If ChatGPT, Claude, Perplexity or Google's AI features never mention your site, the cause is usually not your content. It is access. Something between the AI crawler and your page says no: a robots.txt rule, a firewall, a challenge page, a status code, or a page that is empty until JavaScript runs. Most site owners only check the first of these, and the block is often in one of the others.

This guide walks through the seven layers a crawler must pass, in the order it meets them, with a test for each. It uses the RFC 9309 standard, each AI company's documentation, measurements from Vercel and Cloudflare, and our study of 4,962 robots.txt files from the top 10,000 sites. For a one-click starting point, run the free AI crawler checker, then the AI crawler diagnostic for the layers robots.txt cannot show.

How can I get AI bots to crawl and index my key pages?

Get AI bots to crawl your key pages in two steps: remove anything that blocks them, then make the pages easy to find and easy to read. In practice that means the retrieval bots are allowed in robots.txt, your CDN or firewall does not challenge them, each key page returns a 200 status without redirect loops, the main text is in the HTML the server sends, and the page is linked from your homepage and listed in your sitemap. Google says the same for its own AI features: a page "must be indexed and eligible to be shown in Google Search with a snippet", and there are "no additional technical requirements" (Google, AI features guide).

The rest of this guide shows how to check each of those conditions. If you prefer a monthly routine, use how do I audit my website for AI crawler accessibility as a checklist. To confirm which bots actually arrived afterward, use a tool that analyses server logs to show which AI bots visited. And if the problem is a redirect, our guide to AI crawler access and redirects covers what each bot follows.

The seven layers an AI crawler must pass

Think of a request as walking through a series of doors. A block at any door ends the visit, and the crawler usually does not tell you why. Here they are in order.

Layer 2: robots.txt rules that block AI crawlers

This is where most people look first, and it is a common cause. The usual mistakes are a blanket block, a list that blocks search bots along with training bots, and misunderstanding how groups work.

A blanket block

User-agent: *
Disallow: /

177 files in our study (3.6%) contain exactly this. On a staging site that is correct. On a live site it removes you from everything, including Google.

A copied "block AI" list

Many sites paste a long list of AI user agents with one Disallow line. If OAI-SearchBot, Claude-SearchBot or PerplexityBot is on that list, the site disappears from those assistants' search answers. In our data, 363 of 710 sites that block GPTBot also block OAI-SearchBot. OpenAI says opted-out sites "will not be shown in ChatGPT search answers" (OpenAI crawler docs).

Not understanding groups

A crawler follows only the group that names it. If a bot has its own group, the * group does not apply to it at all. RFC 9309 says crawlers obey the * group only "if no matching group exists". So adding User-agent: GPTBot with a single Allow line can accidentally open paths you blocked for everyone else.

Our robots.txt best practices guide has templates that avoid all three mistakes.

When robots.txt itself cannot be read

This layer surprises people. A crawler has to download robots.txt before it can follow it, and what it does when that fails depends on the status code. RFC 9309 sets the rules:

Two cases deserve attention. If your firewall blocks bots from /robots.txt with a 5xx error or a timeout, well-behaved crawlers stop crawling your whole site, which is the opposite of what you want. If it returns an HTML challenge page with status 200, many AI crawlers find no rules and you lose control of what they fetch. The fix in both cases is the same: let every client read robots.txt as plain text, even the ones you block everywhere else.

A caveat on our numbers: some of the 403s we saw were probably aimed at our study's own user agent, not at named AI crawlers. That is still the point. A firewall that blocks unknown agents from robots.txt hides your rules from every new bot.

Layer 3: CDN and firewall blocks

Since 2024, CDNs have added one-click AI blocking, and many sites have it on without knowing. These blocks act before robots.txt is read, so a perfect robots.txt file does not help.

• Cloudflare: on 1 July 2026 Cloudflare split AI traffic into Search, Agent and Training. From 15 September 2026, new domains block Training and Agent traffic by default on pages that show ads, while Search stays allowed. Cloudflare also said multi-purpose crawlers "such as Googlebot, Applebot, and BingBot will be blocked by customers who have selected to block Training" (Cloudflare, 1 July 2026).
• Bot Fight Mode and similar tools challenge clients that look automated. Crawlers cannot solve challenges.
• Security plugins on WordPress and other CMSs often block unfamiliar user agents or data-center IP ranges, which is where every AI crawler runs.
• Country blocks: Vercel and MERJ found the AI crawlers they measured all operate from US data centers. A rule that blocks US traffic, or only allows your home country, blocks them all.

Perplexity's documentation even includes WAF setup steps because this is so common: it says that if you use a Web Application Firewall "you may need to explicitly whitelist Perplexity's bots" (Perplexity). The safest way to allow a bot is by its operator's published IP list, not by user agent string alone.

for ua in "GPTBot/1.4" "OAI-SearchBot/1.4" "ClaudeBot/1.0" "Claude-SearchBot/1.0" "PerplexityBot/1.0"; do
  printf "%-22s" "$ua"
  curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 (compatible; $ua)" https://www.example.com/
done

Layer 4: rate limits

A rate limit that is too tight lets the first few requests through and then returns 429 or 503. Crawlers back off, sometimes for a long time. Google says that if it sees "a significant number of URLs with 500, 503, or 429" responses, it reduces the crawl rate for the whole hostname (Google, crawl rate guide). AI crawlers behave in similar ways.

Set rate limits per IP or per bot, not per site, and give search crawlers more room than training crawlers if you need to choose. Our AI crawler rate limiting guide has tested Nginx and Cloudflare examples.

Layer 5: status codes, redirects and login walls

Each key page must return a 200 status to the crawler. Common failures:

• 403 Forbidden from a security rule. Our guide to fixing 403 errors for AI crawlers traces each source.
• Redirect chains and loops. RFC 9309 asks crawlers to follow at least five redirects for robots.txt; pages vary. Keep each page to one redirect at most.
• Soft 404s: a "not found" page that returns 200. The crawler stores an error message as your content.
• Login or cookie walls that need a click before content appears.
• Geo or language redirects that send a US data-center IP to a different site version.

Wasted fetches are also a problem. Vercel and MERJ found that OpenAI's crawler spent 34.82% of its fetches on 404 pages and 14.36% following redirects in their sample, against 8.22% and 1.49% for Googlebot (Vercel and MERJ). Every broken link you fix leaves more of a crawler's attention for pages that work.

Layer 6: content that only exists after JavaScript

This is the most common reason AI systems "see" a page differently from a person. Vercel and MERJ tested the major AI crawlers and reported: "none of the major AI crawlers currently render JavaScript". The list included OpenAI's OAI-SearchBot, ChatGPT-User and GPTBot, Anthropic's ClaudeBot, Meta-ExternalAgent, ByteDance's Bytespider and PerplexityBot. Google's Gemini uses Googlebot's infrastructure and can render, and Applebot renders too.

So if your product descriptions, prices, reviews or article text are loaded by JavaScript after the page arrives, most AI crawlers get an empty shell. The test is simple: open the page, choose View Page Source (not Inspect), and search for a sentence from your main content. If it is not there, crawlers cannot see it either.

curl -s -A "Mozilla/5.0 (compatible; GPTBot/1.4)" https://www.example.com/product \
  | grep -c "a sentence that should be on the page"

The fix is server-side rendering, static generation or pre-rendering for the content that matters. Vercel's own advice is to server-render "main content (articles, product information, documentation), meta information (titles, descriptions, categories), and navigation structures", and leave client-side rendering for extras like counters and chat widgets.

Layer 7: page directives that hide content

Even when a crawler gets your page, a directive can tell it not to use the content. Check for these in the HTML head and the HTTP headers:

These are often added by accident: a staging setting copied to production, an SEO plugin default, or a CDN that adds X-Robots-Tag: noindex to some paths. Use them on purpose or not at all.

Platform notes: where blocks usually hide

The same seven layers exist on every stack, but each platform has its usual suspects. These are the places we check first when a site on that platform is invisible to AI crawlers.

When you find the block, write down where it was and why it was added. Many of these rules were turned on during an attack or a traffic spike and never turned off. A one-line note saves the next person an afternoon.

A 15-minute diagnosis, step by step

1. Run the AI crawler checker on your homepage. It reads your robots.txt the way RFC 9309 says and lists which of 257 bots are blocked.
2. Open /robots.txt in a browser. It must be plain text. Note any AI tokens in Disallow groups.
3. Run the AI crawler diagnostic on one key page. It fetches the page as each AI bot and names the layer that answers.
4. If the diagnostic shows 403, a challenge or a timeout, check your CDN's bot and AI settings, then your security plugin.
5. View the page source and search for a sentence from the main content.
6. Check the head for noindex or nosnippet, and run curl -I for X-Robots-Tag.
7. After fixing, wait a day, then look for the bots in your server logs.

After the fix: helping AI crawlers find your key pages

Removing blocks lets crawlers in. Helping them find the right pages is the second half of the job.

• Link key pages from the homepage and main navigation. Google lists internal links among the SEO basics that also apply to AI features.
• Keep your sitemap current and list it in robots.txt with a full URL. 67.3% of top sites already do.
• Use IndexNow if you publish often. Bing says IndexNow helps AI answers reference "the most current version of a page" (Bing Webmaster Blog).
• Fix 404s and redirect chains, which waste crawler visits.
• Put the answer near the top of each key page, in plain text.

Be patient after a fix. OpenAI says robots.txt changes can take about 24 hours to reach its search systems, Perplexity says up to 24 hours, and RFC 9309 allows crawlers to cache robots.txt for up to a day. A page that was blocked for months may also wait in a crawler's queue before its next visit.

Then measure. Google's Generative AI performance report in Search Console and Bing's AI Performance report both show which of your pages appear in AI answers, which tells you whether the fix worked.

Checklist

• □robots.txt returns 200 as plain text to every client.
• □AI search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) are allowed.
• □Your CDN's AI or training block does not cover search bots or Googlebot.
• □Key pages return 200 with at most one redirect.
• □Main content is in the server HTML.
• □No accidental noindex or nosnippet on key pages.
• □Key pages are linked from the homepage and in the sitemap.
• □You checked server logs a day after the fix.

Sources and further reading

Every claim above links to the page it came from. These are the primary sources, all read or re-read on 5 October 2026. Where we give a figure from our own data, it comes from our robots.txt study of the top 10,000 sites, and the raw counts are in the downloadable CSV.

1. RFC 9309, Robots Exclusion Protocol (IETF, September 2022)
2. Google Search Central: How Google interprets the robots.txt specification
3. Google Search Central: AI features and your website
4. Google Search Central: Reduce the Google crawl rate
5. Vercel and MERJ: The rise of the AI crawler (December 2024)
6. Cloudflare: Content Independence Day, new AI options (1 July 2026)
7. OpenAI: Overview of OpenAI crawlers
8. Perplexity: Perplexity crawlers
9. Bing Webmaster Blog: Introducing AI Performance in Bing Webmaster Tools (10 February 2026)
10. Tranco: a research-oriented top sites ranking

If you are asking "is my website crawlable?", start with one scan: it checks robots.txt for 257 bots and fetches the exact URL as Googlebot smartphone, so you can see whether the block is in robots.txt, the CDN or the page itself.