AI Crawler Check for Agencies: Client Audits at Scale

Most agency value here is in two things: finding the problem quickly across many sites, and explaining it in terms a client acts on. The tooling covers the first; this page covers both.

A repeatable audit workflow

Batch the portfolio first

Put every client domain through the batch checker, up to 20 at a time, and export the CSV. This surfaces which sites have a real problem so you are not scanning one at a time.
Triage on Tier 1, not on score

Sort by whether Tier 1 retrieval crawlers are blocked. A site scoring in the 70s with clean Tier 1 access is in better shape than one scoring in the 80s that blocks a retrieval crawler. The score is a summary; Tier 1 is the consequence.
Separate deliberate from accidental

Some clients genuinely chose to block model training, and that decision should be respected rather than "fixed". What you are hunting for is the accidental blocks: usually a copied snippet that took out retrieval crawlers too.
Run the full report on the problem sites

For each site that needs work, run the individual check and use the recommendations as the fix list. They arrive with copy-paste code, ordered by impact.
Export white-label and attach the evidence

PDF and Excel exports carry your company name. Include the before state, because the value you are showing is the delta.
Re-run after the client deploys

The scan reads the live site, so a re-run is your proof the change shipped. That second report is usually the most persuasive artefact in the engagement.

Explaining it to a client

The framing that lands is the distinction between training and retrieval, because it separates a values question from a mistake:

The sentence that usually works

Blocking AI from training on your content is a business decision, and a defensible one. Blocking AI from citing you is almost always an accident, and it costs you the exact traffic you were trying to win. They look nearly identical in a robots.txt file, which is why it happens so often.

Two other points tend to matter to clients:

This is not a rankings problem. A site can rank well and still be absent from AI answers. If they are judging you on rankings alone, this is invisible to both of you.
A firewall can cause it. Bot-management settings block crawlers before robots.txt is consulted, so the fix sometimes belongs to whoever administers the CDN, not the marketing team.

Who builds this

AI Crawler Check is run by Horatos.ai, an AI-native growth marketing agency in Singapore. The tool exists because we needed it for client work and the available options only covered a fraction of the crawler landscape.

Running AI crawler audits for clients

A one-page client summary

Headline. The AI Visibility Score and whether any Tier 1 crawler is blocked.
What it means. One sentence on whether AI assistants can read and cite the site.
Top three fixes. From the diagnostic, in order of impact.
Decision needed. Whether the client wants to allow AI training, which is a business choice, not a technical one.
Next check. The date of the next batch run.

Pricing the work

Audit: a fixed fee per site for the check, diagnostic and summary.
Implementation: robots.txt, firewall and structured data changes, priced by hours.
Monitoring: a monthly batch run and summary, priced per site.

The tools are free, so the value you sell is interpretation and implementation.

Measuring the value of the work

Clients ask what changed. Report three numbers: how many important AI crawlers could read the site before and after, the score before and after, and the date of each check. Add the robots.txt diff as an appendix. That is a short, honest report that any client can follow, and it avoids claims about citations or traffic that a crawler audit cannot prove on its own.

Scoping the first audit

List the client's hostnames, including www and any regional or app subdomains.
Pick one URL per template: home, a commercial page, an article, a category.
Ask who owns robots.txt, the CDN and the firewall, and get read access.
Agree the goal in one sentence: cited in AI answers, refuse training, or both.
Agree the recheck date before you start, so the follow-up is part of the job.

Keeping client work separate

Use a separate saved result or export for each client and each date, named with the client and the date. Never mix two clients in one batch: batch results are easier to read, and to share safely, when every URL in the run belongs to the same host.

Questions clients ask, and short answers

Will blocking AI training hurt my Google rankings?: Not if you block training tokens only. Google ranks with Googlebot; Google-Extended is a separate control.
How do I know the fix worked?: The dated recheck in your report shows each crawler before and after.
Do we need to do this again?: Yes, periodically. New crawlers appear and site changes can reintroduce blocks.
Can you guarantee AI citations?: No one can. Access makes citation possible; content and authority decide it.

A recheck calendar that works

Recheck cadence
Trigger	What to recheck
Monthly	Home page and top commercial pages for every client
After a site launch or migration	Every template, plus robots.txt and the sitemap
After a CDN or firewall change	Access for the Tier 1 AI crawlers
When a major operator announces a crawler	The client files that name that operator

What to put in the handover

At the end of an engagement, leave the client with three things: the final robots.txt with a comment above each group explaining why it exists, the dated before-and-after results, and a one-page note on who owns the file from now on. Files without an owner drift, and drift is the most common cause of the problems in fixing blocked crawlers.

Pricing the work honestly

The tools are free, so you are not reselling a report. What a client pays for is the judgement: which crawlers matter for their business, which fixes to make first and proof that the fixes worked. A useful structure is a fixed-price first audit, then a monthly or quarterly recheck. Quote the recheck as a separate line, because crawlers change: our database grew to 265 entries, and each new token can need a decision.

A client report outline

One-paragraph summary: score, grade and the single most important fix.
Access table: which AI crawlers can and cannot reach the key pages, with the reason for each refusal.
Recommended robots.txt, generated with the generator, with a line-by-line note.
GEO items: llms.txt, structured data and sitemap, each with an owner and a date.
Before and after: the original result and the recheck, dated.

Handling a client who wants to block everything

Some clients ask to block every AI crawler. Show them what that costs: crawlers that fetch pages for AI answers are how a site gets cited. Our study found 1,123 top-site files (20.1%) block at least one training crawler, but only 689 (12.3%) block an AI search crawler. Most large sites split the two. The block or allow guide is written to be shared with exactly this client.

Batch checks for portfolios

The batch checker takes up to 20 URLs at once. Use it for the home page and the top commercial pages of each client, not for every URL: the same robots.txt governs the whole host, so one check per template is usually enough.

Test yourself

Who should decide whether to allow AI training crawlers?

The client. It is a licensing and business decision; your job is to explain the trade-off.

Common questions

Can I put my own company name on the reports?

Yes. PDF and Excel exports support white-labelling, so you can hand a client a report carrying your branding.

How many client sites can I audit at once?

The batch checker takes up to 20 URLs per run and exports CSV. Larger portfolios go through in successive batches.

Is it really free for commercial client work?

Yes. The tools are free with no sign-up, including for paid client engagements.

What should I check first on a new client site?

Whether Tier 1 retrieval crawlers are allowed. That single row explains most cases of a site being absent from AI answers, and it is faster to check than anything else in the report.

A client deliberately blocks AI. Is that wrong?

No. Opting out of model training is a legitimate position, particularly for publishers and licensed content. The distinction worth raising is whether they also meant to opt out of being cited, because those are usually conflated by accident.

Ready to check your site?

265 crawlers, 42 GEO checks, no sign-up.

Run a free check