Batch URL Checker
Check multiple websites at once for AI bot access, robots.txt configuration, and SEO readiness. Analyze up to 20 URLs simultaneously with our full 257-bot database.
Enter URLs (one per line, max 20)
Checking URLs...
0 / 0
0
Sites Checked
0
SEO Safe
0
Warnings
0
Critical
0
Failed
How to use the batch URL checker
- Paste up to 20 domains, one per line, or load a sample list (competitors, AI companies, or news publishers).
- Press Check All URLs. Each site gets the same 257-bot robots.txt analysis as a single AI crawler check, run one after another so no site is rate limited.
- Sort the comparison table by score, blocked bots or Tier 1 access, then export CSV or copy the report for a client deck.
- Open any weak result in the AI crawler diagnostic to find the CDN or HTTP layer a robots.txt scan cannot see.
Batch AI crawler check use cases
| User | List to paste | What to look for |
|---|---|---|
| SEO or GEO agency | Every client domain | Any client blocking OAI-SearchBot, Claude-SearchBot or PerplexityBot |
| Competitor research | Your site plus four rivals | Who lets AI search in while you block it |
| Publisher group | Each title and subdomain | Inconsistent training rules between titles; robots.txt is per host |
| Migration QA | Old and new hostnames | The new host lost the robots.txt groups |
| Monthly monitoring | Your key properties | Score drops after a CDN or plugin update |
The migration row matters more in 2026: Cloudflare began blocking AI Training and Agent traffic by default on new domains with ads from 15 September 2026 (Cloudflare, 1 July 2026), so a freshly launched host can behave differently from the old one even with an identical robots.txt.
Reading a portfolio against market benchmarks
A batch run is most useful next to a benchmark. Across 5 million sites, Hostinger found OAI-SearchBot on 55.67% and GPTBot on only 12% (Hostinger, 20 January 2026), and among top news sites 79% block a training bot while 71% also block a retrieval bot (BuzzStream via Search Engine Journal, January 2026). If most of your list blocks retrieval bots, you are with the publishers who gave up live AI citations.
Sample batch report: what a finished comparison looks like
| Site | Score | Tier 1 AI bots allowed | Main finding | Next step |
|---|---|---|---|---|
| brand.example | 86 | 9 of 10 | GPTBot blocked on purpose | None |
| shop.example | 41 | 3 of 10 | Wildcard Disallow on /products/ | Scope the Disallow |
| blog.example | 72 | 8 of 10 | No llms.txt | Use the llms.txt generator |
| news.example | 38 | 2 of 10 | OAI-SearchBot and PerplexityBot blocked | Unblock retrieval bots |
| docs.example | 0 | n/a | robots.txt returns 5xx | Fix the server; 5xx stops crawlers |
Other free tools for the same audit
- AI crawler checker: tests 257 bots against your robots.txt in one scan.
- AI crawler diagnostic: fetches a page as each AI bot and names the layer that blocks it.
- robots.txt validator: shows allowed, blocked and partial status per bot.
- robots.txt generator: builds a file with presets for search, agents and training.
- llms.txt generator: writes the file AI assistants read for a site summary.
- schema markup generator: produces JSON-LD for eight common types.
- GEO audit: scores a page for generative engine optimization.
Frequently Asked Questions
Can I bulk check AI crawler access for competitors?
Yes. The batch checker reads public robots.txt files, so any public domain works. Paste up to 20 domains, including competitors, and sort the comparison by score or blocked bots.
Why do two sites with the same robots.txt score differently in a diagnostic?
Because robots.txt is only the first layer. A CDN, WAF or hosting firewall can return 403 to AI bots regardless of robots.txt, and from 15 September 2026 new Cloudflare domains block AI training and agent traffic by default on ad-supported pages.
How many URLs can I check at once?
You can check up to 20 URLs simultaneously. Each URL is analyzed against our full database of 257+ bots, checking robots.txt rules and AI infrastructure files (llms.txt, llms-full.txt). Results include an AI Visibility Score from 0-100 based on Bot Access (70 pts), AI Infrastructure (15 pts) and Technical Readiness (15 pts).
What does the batch checker analyze for each URL?
For each URL we check: robots.txt (which of 257+ bots are blocked/allowed per RFC 9309), AI infrastructure files (llms.txt, llms-full.txt) with strict validation. Results are scored 0-100 across Bot Access (70 pts), AI Infrastructure (15 pts) and Technical Readiness (15 pts).
Can I export the batch results?
Yes! You can export all results as a CSV file for spreadsheet analysis, or copy a formatted text report to your clipboard. The CSV includes domain, score, grade, bots allowed/blocked/partial, robots.txt status, AI files status, and key recommendations.
What is the AI Visibility Score?
The AI Visibility Score (0-100) measures how well your site is configured for AI crawler access. It's calculated from 3 focused components: Bot Access (70 pts) based on Tier 1 AI bot accessibility in robots.txt, AI Infrastructure (15 pts) for llms.txt (10 pts) + llms-full.txt (5 pts), and Technical Readiness (15 pts) for sitemap, structured data and HTTPS. Higher scores indicate better AI discoverability.
Is the batch checker free?
Yes, the batch URL checker is completely free. There are no hidden fees, account requirements, or usage limits beyond the 20 URL maximum per batch. Each check runs the same comprehensive analysis as our single URL checker.