Batch URL checker for whole portfolios

Who runs a batch check, and what to look for

The migration row matters more in 2026: Cloudflare began blocking AI Training and Agent traffic by default on new ad-supported domains from 15 September 2026 (Cloudflare, 1 July 2026), so a freshly launched host can behave differently from the old one even with an identical robots.txt.

Building a useful list

• One host per line. Enter https://www.example.com and https://shop.example.com separately; each has its own robots.txt.
• Include the version people link to. If your site redirects from the bare domain to www, check the www address, because that is the file crawlers end up reading.
• Group comparable sites. A batch of five competitors in one market is easier to read than twenty unrelated sites.
• Start from a sample list to see the output before you paste your own. The four sample sets cover AI companies, news publishers, e-commerce and developer platforms, which show very different policies.
• Stay within 20. The counter under the box shows how many URLs you have entered; split a longer portfolio into several batches.

Reading the comparison table

Once two or more sites succeed, the page builds a side-by-side table. Sort it by any column.

Score and grade

The 0 to 100 AI Visibility Score, the same as a single check: 70 points for Tier 1 crawler access, 15 for llms.txt (10) and llms-full.txt (5), and 15 for sitemap, structured data and HTTPS.

Gap vs best

How many points each site trails the highest score in the batch. Useful for competitor sets.

Allowed, blocked, partial

How many of the 265 bots each verdict applies to on the homepage.

Tier 1 blocked

How many of the most influential crawlers are refused. This column usually matters more than the totals: one blocked Tier 1 crawler outweighs many low-tier ones.

Status cards

SEO Safe means every search engine is allowed. Warning and Critical mean a search engine or an important AI crawler is refused. Failed means the site could not be checked.

The CSV export, column by column

Export CSV mirrors the comparison table so a spreadsheet supports the same analysis. Copy Report gives a plain-text summary for an email or a client deck.

A site that fails gets its error in the Domain column, so the row still tells you what went wrong.

Reading a portfolio against market benchmarks

A batch run means most next to a benchmark. Across 5 million sites, Hostinger found OAI-SearchBot on 55.67% and GPTBot on only 12% (Hostinger, 20 January 2026). Among top news sites, 79% block a training bot and 71% also block a retrieval bot (BuzzStream via Search Engine Journal, January 2026). If most of your list blocks retrieval bots, you are with the publishers who gave up live AI citations.

Here is what a finished five-site comparison can look like (illustrative values):

A monthly portfolio routine

1. Keep one list of every site you look after, one URL per line.
2. Run the batch on the first working day of each month and export the CSV.
3. Compare the score and Tier 1 columns with last month. Any drop is worth a single check and, if needed, a run of the AI crawler diagnostic on that site.
4. Note infrastructure changes (new CDN rules, security presets, migrations) next to the month they happened; they explain most drops.
5. Share the export as a simple table. The column names above are all the explanation a client needs.

Patterns you will see, and what to do

Turning the table into a client report is straightforward: lead with the sites whose Tier 1 count changed, explain the cause in one line each, and attach the CSV for detail.

Before and after a CDN change: a worked run

1. The day before a firewall or CDN change, run the batch on every host it touches and export the CSV as your baseline.
2. Make the change on one host first.
3. Run the same batch again. Any row whose robots.txt status, Tier 1 blocked count or score moved points straight at the change.
4. If robots.txt now answers 403, the new rule refuses unknown clients; allow /robots.txt for everyone.
5. If the file is unchanged but Tier 1 blocked rose, the CDN is answering crawlers before the file matters. Run the diagnostic on that host to see which gate fails.
6. Roll the change out to the remaining hosts only once the batch matches the baseline.

Keep each month's CSV. Twelve files side by side show which sites drift, which is usually where an unmanaged plugin or a forgotten staging rule lives.

Limits worth knowing

• Each batch takes up to 20 URLs and checks them one after another, so no site is rate-limited and a full run takes a few minutes.
• Each site gets the homepage check. To test a deep page, enter that page's URL on its own line.
• Requests come from our servers, so a site that filters by IP can treat the real crawlers differently.
• www.example.com and blog.example.com are separate hosts with separate robots.txt files. List both if both matter.