Using AI Crawler Check for client work

Most agency value here is in two things: finding the problem quickly across many sites, and explaining it in terms a client acts on. The tooling covers the first; this page covers both.

A repeatable audit workflow

1. 1

   Batch the portfolio first

   Put every client domain through the batch checker, up to 20 at a time, and export the CSV. This surfaces which sites have a real problem so you are not scanning one at a time.

2. 2

   Triage on Tier 1, not on score

   Sort by whether Tier 1 retrieval crawlers are blocked. A site scoring in the 70s with clean Tier 1 access is in better shape than one scoring in the 80s that blocks a retrieval crawler. The score is a summary; Tier 1 is the consequence.

3. 3

   Separate deliberate from accidental

   Some clients genuinely chose to block model training, and that decision should be respected rather than "fixed". What you are hunting for is the accidental blocks: usually a copied snippet that took out retrieval crawlers too.

4. 4

   Run the full report on the problem sites

   For each site that needs work, run the individual check and use the recommendations as the fix list. They arrive with copy-paste code, ordered by impact.

5. 5

   Export white-label and attach the evidence

   PDF and Excel exports carry your company name. Include the before state, because the value you are showing is the delta.

6. 6

   Re-run after the client deploys

   The scan reads the live site, so a re-run is your proof the change shipped. That second report is usually the most persuasive artefact in the engagement.

Explaining it to a client

The framing that lands is the distinction between training and retrieval, because it separates a values question from a mistake:

Two other points tend to matter to clients:

• This is not a rankings problem. A site can rank well and still be absent from AI answers. If they are judging you on rankings alone, this is invisible to both of you.
• A firewall can cause it. Bot-management settings block crawlers before robots.txt is consulted, so the fix sometimes belongs to whoever administers the CDN, not the marketing team.

Who builds this

AI Crawler Check is built by Horatos.ai, an AI-native growth marketing agency in Singapore. The tool exists because we needed it for client work and the available options only covered a fraction of the crawler landscape.