Hi, I'm Scout. I read your site the way AI crawlers do.

Free AI crawlability & access check

Is Your Website Invisible to ChatGPT, Gemini & other AI Crawlers?

Most sites block AI crawlers by accident.
Enter your URL to see exactly which of
265 bots can reach you.

Try:

100% free No sign-up Results in seconds

  • ChatGPT
  • Google Gemini
  • Claude
  • Perplexity
  • Microsoft Copilot
  • Meta

01How it works

How AI Crawler Check works

Three steps, about ten seconds. Nothing to install, nothing to sign up for, and no change to your website: paste your address and read the answer.

  1. You do this

    Paste your web address

    We visit your site the same way an AI assistant would and read the small text files that tell visiting robots what they may look at. We read a handful of files, not your whole website, and we do not save the address you typed.

    0fields to fill in. No account, no email, no card.

  2. We do this

    We try every AI robot by name

    Each robot has its own name, and your rules can allow one and block another without you realising. We test all 265 of them, one at a time, using the same matching order the real robots use. That order is where reading a rules file by eye usually goes wrong, as our guide to blocked crawlers shows.

    265robots tested individually, across 8 groups.

  3. You get this

    A score, and the exact fix

    A single number out of 100, a plain list of which assistants cannot see you and why, and ready-made text you can copy and paste to fix it. The fixes are ordered by how much each one is worth, so the first one you apply is the one that matters most.

    ~10sfrom pasting the address to reading the result.

See how AI crawlers read your site

02The research behind every guide

Meet Scout Agent, our AI crawler researcher

Scout never sleeps, never guesses and never copies someone else's homework. It is the research agent behind this AI crawler checker, and it does the work most guides skip: it tests live websites for AI crawlability, reads every crawler's documentation at the source, and turns the evidence into plain English you can act on today. Then the AI Crawler Check editorial team checks every claim before it goes live.

While other guides repeat last year's advice, Scout is out fetching robots.txt files, comparing user agent strings, re-testing pages from dozens of crawler identities and catching the moment a firewall treats a bot differently from a browser. That is how it spots the silent blocks that cost sites their place in AI answers. Our editors add the judgement: what a finding means for your AI search visibility, and what to fix first. Scout is an AI, and we say so proudly, because the proof is in the pages it writes.

Testing a live site exactly the way an AI crawler sees it.

Scout Agent: AI research agent created by AI Crawler Check, reviewed by the AI Crawler Check team.

  1. Tests the live web

    Scout runs the same AI crawler check you can run above, against real sites, and records which AI bots are allowed, blocked or silently challenged.

  2. Reads primary sources

    User agents, IP ranges and opt-out rules come from each operator's own documentation, never from another checker's list. That is how our AI crawler directory stays accurate.

  3. Shows the evidence

    Findings carry the test that produced them, so you can reproduce any crawlability case study yourself instead of taking our word for it.

  4. Gets a human review

    An AI Crawler Check editor checks facts, tone and usefulness before publishing. Corrections are logged on our methodology page.

Start with Scout's most-read guides: robots.txt rules for AI crawlers, our AI crawler robots.txt study and every AI bot you should know.

See everything Scout has written

03Scout Radar

What the top 10,000 sites decide about AI crawlers

Scout does not only check one site at a time. It reads the robots.txt file of every domain in the Tranco top 10,000 and records which AI crawlers each site lets in and which it turns away. Scout Radar publishes what it finds, so you can compare your own rules with what the most visited sites on the web actually do.

The latest scan read 5,583 valid files. Most top sites are not blocking AI at all, and the ones that do are careful: they say no to training and keep the door open for AI search, so assistants can still cite them.

  • 20.2%block at least one AI training crawler
  • 9.1%block training but keep AI search open
  • 79.6%leave every AI crawler open
Explore Scout Radar

Most blocked by top sites

  1. CCBotCommon Crawl, AI training16.1%
  2. BytespiderByteDance, AI training15.9%
  3. GPTBotOpenAI, AI training15.4%
  4. ClaudeBotAnthropic, AI training14.4%
  5. Google-ExtendedGoogle, AI training13.0%
Share of valid robots.txt files that block each crawler from the whole site. Tranco top 10,000, fetched 20 October 2026, read with an RFC 9309 parser. Open data under CC BY 4.0.

04What you get

What your free AI crawler report shows

No sign-up wall between you and the answer. Every report has the same four parts: a score, the robots that cannot reach you, the reason for each, and the fix.

Sample result

yoursite.com

High risk: invisible to the assistants that matter

3 of the 5 most important AI crawlers are turned away by your robots.txt, and your llms.txt is missing.

  • 21/70Crawler access
  • 6/15AI files
  • 15/15Technical
  • GPTBot · OpenAI robots.txt line 4 · Disallow: / Blocked
  • ClaudeBot · Anthropic robots.txt line 4 · Disallow: / Blocked
  • PerplexityBot · Perplexity X-Robots-Tag header · noai Blocked
  • Google-Extended · Google Gemini No rule matched · access permitted Allowed

The full report lists all 265 robots this way, grouped by how much each one is worth to you. See how each point is scored.

Your fix

copy & paste
# robots.txt · allow AI answer engines User-agent: GPTBot Allow: / User-agent: ClaudeBot Allow: / User-agent: PerplexityBot Allow: / # keep training-only scrapers out if you prefer User-agent: CCBot Disallow: /
  • +28 ptsRemove the blanket Disallow on line 4 and allow the three named crawlers above.
  • +10 ptsAdd an llms.txt file at your root so assistants can find your key pages without guessing; here is how to write one.
  • +6 ptsDrop the noai header your CDN is adding to every response.

Every snippet is generated for your site, not a template, and one click copies it.

05Why it matters

What invisibility costs you

AI assistants are where buyers now start their research. When they cannot read your site, the loss never shows up in your analytics, because the visit never happens.

  1. 01

    Silent loss

    No warning, no error, no dip to investigate

    Nothing breaks when GPTBot or ClaudeBot is turned away. Your rankings look fine, your dashboards stay green, and you quietly stop appearing in AI answers. There is no alert for this in Search Console, in your analytics, or anywhere else, so most site owners find out months late, or never find out at all. Measuring AI referral traffic is how you catch it.

  2. 02

    Competitor gain

    The citation goes to whoever is readable

    Every question ChatGPT or Perplexity answers needs a source, and the model picks from the pages it is allowed to read. If yours are blocked, the answer still gets written, it just names a competitor whose robots.txt let the crawler in. You are not ranked lower. You are not in the running. Read how AI engines choose their sources.

  3. 03

    Compounding gap

    AI visibility compounds, like search did

    The sites being cited today become the default sources models keep returning to, and being cited makes a page more likely to be cited again. Every month you stay blocked is a month of citations, mentions and trust accruing to someone else, and buying it back later costs far more than not losing it.

06The six causes

Six ways a site goes invisible by accident

Almost nobody decides to hide from AI assistants. It happens through a line copied from a forum, a plugin default, or a header a hosting provider adds for you. Here is what we find most often, and what each one does to your score.

  • The blanket disallow

    Two lines meant to keep one scraper out. They apply to every robot ever written, including the ones that would have quoted you.

    User-agent: *
    Disallow: /

  • A header you never wrote

    Some hosts and security plugins add an opt-out header to every response by default. Your robots.txt says yes and your server says no.

    X-Robots-Tag: noai, noimageai

  • A name that no longer exists

    Operators rename their robots. A rule written for the old token stops matching anything on the day of the change, silently, and nothing warns you.

    User-agent: anthropic-ai
    Disallow: /

  • A firewall that answers first

    Bot protection can challenge an AI crawler before your rules are ever read. The robot sees a security page, not your content, and moves on.

    403 Forbidden · managed bot rule

  • Content that arrives empty

    If your text is drawn by JavaScript after the page loads, a robot that does not run scripts receives a shell with nothing in it to quote.

    <div id="root"></div>

  • Nothing to guide the reader

    Access alone is not readability. With no llms.txt, no schema and no clear headings, an assistant has to guess which of your pages answers the question.

    GET /llms.txt → 404

07Fix it

Ten free tools to find, fix and watch

A report is only useful if you can act on it. These write the actual files for you, check the ones you already have, and none of them ask for an account. Start with the AI crawler diagnostic tool and the rest of the free set.

Find the problem

Start here when a scan says a crawler cannot read you.

Write the fix

Generate the files, then check them before they go live.

Watch your logs

See which AI agents actually visit. Runs in your browser; nothing is uploaded.

Most-searched crawler profiles: GPTBot ClaudeBot PerplexityBot SemrushBot Full AI crawler directory

08Fresh from Scout's desk

Latest research from Scout Agent

New robots appear every few weeks and old ones get renamed without notice. This is what changed most recently, written up the week it happened. Each one is researched by Scout, our AI agent and crawler researcher, and checked by the AI Crawler Check team.

Read all 159 articles

09The database

A verified AI crawler database, every entry sourced

Every robot we test has a page of its own: who runs it, what it takes, whether it obeys robots.txt, and the exact line to allow or block it. All of it free to read, no sign-up.

265Bots tracked
112Operators tracked
159Research articles
8Groups covered

By groupevery bot tracked, sorted by what it does

Beyond the botswho runs them, how they differ, and how we check

10How we compare

AI Crawler Check vs typical AI bot checkers

Most checkers were built before AI search mattered and test a thin slice of the crawler landscape. Here is the difference, row by row.

Feature comparison between AI Crawler Check and typical AI-only bot checkers
Capability AI Crawler Check Typical AI-only tools
Bots tested 265 bots 30 to 40
Categories covered 8 categories 1 to 3 (AI only)
Agentic crawlers (Google-Agent, OAI-AdsBot)
Unified GEO score (42 checks, 0 to 100)
Signals read per URL robots.txt, meta, headers, llms.txt, sitemap, schema robots.txt only
Shows the rule that caused each result
Flags renamed and retired robot tokens
Many URLs in one run Up to 20 One at a time
llms.txt detection with strict validation Partial
Honest WAF handling (15+ providers)
Free tools included 10 free tools Rare
Sign-up required Never Often
See the full comparison Every row above is testable. Run your own site and check.

11Google too

Free Googlebot test and crawlability checker

Is my website crawlable? Paste any page into the website check box at the top of this page and every scan now runs a Google crawl test on that exact URL, alongside the AI crawler check.

You see what the Google web crawler is allowed to fetch, what it receives, and whether it may index the page, in about ten seconds.

Use it as a Googlebot checker before a launch, a website crawl test after a migration, or a quick way to check if a URL is crawlable when traffic drops.

example.com/pricingGooglebot smartphone
5 / 6checks passed, 1 warning
  • robots.txtAllowed for /pricingPass
  • HTTP status200 OK as Googlebot smartphonePass
  • Challenge pageNo CDN challenge servedPass
  • IndexabilityNo noindex foundPass
  • CanonicalPoints to another URLWarning
  • RenderingText present before JavaScriptPass
Illustrative result. Your scan shows the exact rule or header behind each line.

How to check if a page is crawlable

  1. Paste the full page address, for example https://example.com/pricing, not just the domain. The crawlability test runs on the path you enter.
  2. Press Check my site. We read robots.txt and fetch the page as Googlebot smartphone, the crawler Google indexes with, so every scan is also a Googlebot mobile test.
  3. Open the Googlebot crawl test panel in the results. Each of the six checks shows pass, warning or fail with the exact rule or header behind it.

To test website crawlability for many pages or sites at once, use the bulk crawlability checker.

How Google's crawler sees your site

Googlebot, the Google search bot behind every organic result, does not see your page the way a browser does. It reads robots.txt first, requests the page with its own user agent, obeys noindex in the HTML or the X-Robots-Tag header, and only renders JavaScript later in a second pass. This Googlebot simulator follows the same order, so a failure points at the layer that caused it.

The same test covers the other search engine crawlers that matter for AI answers: Bingbot, which feeds Microsoft Copilot, and Applebot, which feeds Siri and Spotlight.

The six checks in every Googlebot crawl test
CheckWhat we testFails when
robots.txtThe Googlebot group, or the * group, evaluated for your exact path with Google's longest-match ruleA Disallow matches the URL, or robots.txt returns a 5xx error
HTTP statusThe page requested with the Googlebot smartphone user agent404, 410, 403, 429 or 5xx instead of 200
Challenge pageWhether a CDN or firewall serves a challenge instead of the pageCloudflare, Akamai, DataDome, Imperva or AWS WAF pages
IndexabilityMeta robots, a googlebot meta tag and the X-Robots-Tag headernoindex or none is present
CanonicalWhether the page names itself as the canonical URLIt points to another URL (reported as a warning)
RenderingHow much text is in the HTML before any JavaScript runsThe page is an empty client-side shell

Crawl test vs Google Search Console

Search Console's URL Inspection shows what the real Googlebot fetched from Google's own IP addresses, but only for sites you have verified. This site crawl test works on any website, including competitors and client prospects, and adds the AI crawlers Search Console does not cover.

One honest limit: our request comes from our server with the Googlebot user agent, not from Google's IP ranges. Many CDNs block look-alike Googlebots and let the real one through. When that happens, the report says so and checks the page with a normal browser fetch instead of calling your site uncrawlable.

Common crawl accessibility problems

For a full SEO crawl test of structure, schema and content, run the GEO site audit.

12Before you check

Frequently Asked Questions

Straight answers about what the check reads, what it costs and what to do with the result. Still stuck? Ask us or read the quick-start guide.

What does an AI crawler checker actually do?

It fetches your robots.txt, meta robots tags, X-Robots-Tag headers and AI files such as llms.txt, then evaluates each rule against the user-agent string of every known AI crawler. The result tells you which AI systems are allowed to read your content and which are blocked, either intentionally or by accident.

Why would blocking AI crawlers hurt my business?

When GPTBot, Google-Extended, ClaudeBot or PerplexityBot cannot read your pages, AI assistants cannot cite you as a source. Your competitors get recommended instead. Unlike a search ranking drop, this failure is silent: no error, no traffic alert, no warning in Search Console.

Is this tool really free, and do I need an account?

Yes, completely free with no account, no email and no credit card. All 265 bot checks, the 42-point GEO audit and all ten free tools are available immediately. We do not store the URLs you test.

What is the difference between SEO and GEO?

SEO optimises for ranked links in a results page. GEO, or Generative Engine Optimization, optimises for being quoted inside an AI-generated answer. GEO depends on crawler access, machine-readable structure, schema markup and clear factual statements, which is what this tool measures. Our GEO vs SEO comparison goes deeper.

How is the AI Visibility Score calculated?

The score weights crawler access at 70 percent, AI infrastructure files at 15 percent and technical readiness at 15 percent. Bots are weighted by real-world importance, so blocking GPTBot costs more than blocking a minor scraper. The full formula is documented on our methodology page.

Should I allow or block AI crawlers on my website?

It depends on the robot, which is why a single blanket rule is almost always the wrong answer. Retrieval crawlers such as ChatGPT-User and PerplexityBot fetch your page to answer a live question and cite you by name, so blocking them removes you from answers your customers are already asking for. Training-only crawlers such as CCBot copy your content into a dataset with no link back, and blocking those costs you nothing. Bulk scrapers that ignore robots.txt entirely need a firewall rule rather than a robots.txt line. The practical position for most businesses is to allow the crawlers that cite, block the ones that only harvest, and decide the training-plus-retrieval ones like GPTBot on your own commercial terms. See the full block-or-allow guide.

How do I know if ChatGPT can read my website?

Four things have to line up, and checking only the first is the usual mistake. Your robots.txt must not disallow GPTBot or ChatGPT-User. Your pages must not carry a noai or noindex instruction in a meta tag or an X-Robots-Tag response header, which your host or a plugin may be adding without your knowledge. Your firewall must not challenge the crawler before it reaches your content. And your text has to be present in the HTML the server returns, not drawn in afterwards by JavaScript. Asking ChatGPT whether it can see your site does not test any of this reliably, because the model may answer from memory. Running the URL through this checker tests all four in one pass. See how to optimise for ChatGPT Search.

How often should I run an AI crawler check?

Once a quarter as a baseline, and immediately after any change to your hosting, CDN, security settings or CMS plugins, because those are the changes that alter robots.txt and response headers without anyone editing a file on purpose. Also check when a new crawler is announced: operators add and rename robots several times a year, and a rule written for a token that no longer exists stops working the day the name changes, with no error and no notification. The scan is free and takes about ten seconds, so there is no reason to wait for a reason. See the monthly audit checklist.

How do I test if Googlebot can crawl my site?

Paste the page address into AI Crawler Check. Every scan runs a free Googlebot test on that exact URL: it evaluates your robots.txt for the path, requests the page with the Googlebot smartphone user agent, and checks for challenge pages, noindex, the canonical tag and JavaScript-only content. Confirm important pages with URL Inspection in Google Search Console, which fetches from Google's own IP addresses.

Is my website crawlable by Google?

It is crawlable when three things are true: robots.txt does not disallow Googlebot for the URL, the page answers Googlebot with HTTP 200 instead of an error or a challenge page, and the content is in the HTML. It is also indexable when there is no noindex in the meta robots tag or the X-Robots-Tag header. The crawlability test in each scan reports all four.

What is a crawlability test?

A crawlability test checks whether a search engine crawler can reach and read a page. It is narrower than a full SEO crawl test: it answers whether Googlebot and other crawlers are allowed in and what they receive, rather than auditing every page of a site for content issues.

A free tool by

Horatos.ai

AI-native growth marketing agency in Singapore

We built this checker because we needed it ourselves. If you want the strategy behind the score, that is the work we do for clients.

Visit Horatos.ai Request an audit
WhatsApp us