Free, no sign-up
Free AI crawler and GEO tools
Eleven tools to see how AI crawlers, assistants and agents treat your website, and to fix what they find: robots.txt, llms.txt, structured data, firewall gates and log analysis.
- 11free tools
- 265bots in the database
- 0accounts needed
All eleven tools
AI Crawler Check
The starting point. Score your whole site against every bot in the database in about a minute.
AI Crawler Diagnostic Tool
Find out why an AI crawler cannot read a page. Tests one URL as 18 crawlers across six gates and ranks the fixes.
AI Agent Readiness Check
See whether AI browsing agents such as Grok Bot can use a page, or are stopped by firewalls, CAPTCHAs or login walls.
Grok Log Checker
Paste access log lines to find GrokBot and xAI-SearchBot requests, grouped by IP. Runs in your browser.
AI Agent Log Checker
Paste access log lines to find personal AI agents such as Meta Muse, Grok Bot, ChatGPT and Claude user fetches. Runs in your browser.
GEO Site Audit
Run a full 42-point GEO/AEO audit. Scores your site out of 100 and returns a prioritised fix list for AI answer visibility.
Robots.txt Generator
Create a custom robots.txt with 265+ bot presets. One-click templates for AI bots, scrapers, and SEO tools.
Robots.txt Validator
Paste or fetch robots.txt to analyze which bots are blocked. SEO safety score and category breakdown.
Batch URL Checker
Check up to 20 URLs at once for AI bot access, robots.txt, and SEO readiness. CSV export included.
llms.txt Generator
Create a spec-compliant llms.txt file in 2 minutes. Templates for SaaS, blogs, e-commerce, and agencies.
Schema Generator
Generate valid JSON-LD structured data. 8 schema types including Organization, Article, FAQ, and Product.
Start from the question you have
Every tool answers one question well. Find yours in the left column, then open the matching card above.
| Your question | Tool | What you get |
|---|---|---|
| Can AI crawlers read my site at all? | AI crawler check (homepage) | A 0 to 100 AI Visibility Score and a verdict for each of 265 bots, in about a minute. |
| Why is one crawler failing on one page? | AI Crawler Diagnostic | Six gates per crawler, from robots.txt to JavaScript, with the fixes ranked. |
| How ready is this page to be cited? | GEO Site Audit | 42 checks across 5 categories and a prioritised fix list. |
| I need a correct robots.txt | Robots.txt Generator | A file built from presets, with a warning before you block a search engine. |
| Is my current robots.txt right? | Robots.txt Validator | Allowed, blocked or partial for all 265 bots, plus an SEO Safety Score. |
| I look after many sites | Batch URL Checker | Up to 20 sites side by side, with CSV export. |
| Can AI agents use my pages? | AI Agent Readiness Check | Eight checks for the challenges, CAPTCHAs and login walls that stop agents. |
| Which AI agents visited me? | AI Agent Log Checker | Personal agents and user-triggered fetchers found in pasted log lines. |
| Is this GrokBot traffic real? | Grok Log Checker | xAI names grouped by IP, so a borrowed name stands out. |
| I want an llms.txt file | llms.txt Generator | A file in the llmstxt.org format, from four starter templates. |
| I need structured data | Schema Generator | JSON-LD for 13 types, samples and a schema plan by industry. |
A sensible order for a full review
- Run the AI crawler check on your homepage and write down the AI Visibility Score.
- If a Tier 1 crawler is blocked or partial, run the diagnostic on that URL to find the gate that stops it.
- Fix robots.txt with the generator, then confirm the draft with the validator before you publish it.
- Run the GEO audit on your most important templates (homepage, a product or service page, an article).
- Add structured data where it describes the page honestly, and an llms.txt file if you want one.
- If you look after several sites, repeat step one for all of them at once with the batch checker, and repeat the whole review after any CDN, firewall or CMS change.
A first pass takes under an hour for a typical site. The tools are free and need no account.
A worked example: one site, five tools
- The check. A furniture shop scores 54. GPTBot and ClaudeBot are blocked on purpose, but OAI-SearchBot and PerplexityBot show as partial.
- The diagnostic. On a product URL, both search crawlers fail the robots.txt gate: a
Disallow: /*?line written to stop faceted URLs also blocks every product link that carries a tracking parameter. - The fix. The generator rebuilds the file with the training and search split and a narrower
Disallow: /*?sort=. The validator confirms both search crawlers are now allowed on product paths. - The audit. The GEO audit on the same product page flags missing Product structured data and a price that only appears after JavaScript runs.
- The markup. The schema generator's online store plan supplies Product and BreadcrumbList; the price moves into the server-rendered HTML.
A re-check a day later, once crawlers have fetched the new file, scores 81. Each step used a different tool because each answered a different question.
What the tools find most often
These are the problems that came up most in our robots.txt study of the Tranco top 10,000 (5,583 valid files), and the tool that catches each one.
| Problem | Top sites affected | Caught by |
|---|---|---|
| robots.txt answers HTTP 403, usually a firewall | 1,117 | Diagnostic (status gate) |
| An HTML page served instead of robots.txt rules | 501 | Validator in fetch mode |
| CSS or JavaScript folders disallowed | 483 | Validator |
User-agent: * with Disallow: / | 240 | Validator (SEO Safety Score) |
| Rules written before any User-agent line | 25 | Validator |
| A Crawl-delay aimed at Googlebot, which Google ignores | 28 | Validator |
Related: how AI engines choose their sources
The most common deliberate choice is the training and search split: 427 sites block GPTBot but keep OAI-SearchBot, so they stay in ChatGPT search without feeding training. The generator's first preset writes exactly that split. The full study, with the method, is in our robots.txt study, and the rules behind good files are in our robots.txt best practices.
What each tool reads, and where it runs
| Tool | What it requests | Where it runs |
|---|---|---|
| AI crawler check and Batch URL Checker | robots.txt, llms.txt files and the page HTML of each site | Our servers |
| AI Crawler Diagnostic | The page, once per crawler user agent and once as a browser | Our servers |
| GEO Site Audit | The page you enter and its robots.txt | Our servers |
| AI Agent Readiness Check | The page, twice: as a cloud browser and as a desktop browser | Our servers |
| Robots.txt Validator | robots.txt in fetch mode; nothing in paste mode | Our servers to fetch, your browser to analyse |
| Robots.txt, llms.txt and Schema generators | Nothing | Your browser only |
| AI Agent and Grok log checkers | Nothing; your log lines never leave the page | Your browser only |
Server-side checks request only public files that any visitor could open. They never log in, submit forms or follow links into private networks. How checked URLs are handled is set out in the privacy policy.
What these tools cannot do
- They never change your site. Every tool reads or generates; you publish the change.
- They test from our servers, not from each crawler's own network. A firewall that filters by IP can treat the real crawler differently, and the reports say so where it matters.
- They cannot see staging or intranet sites that are not reachable from the public internet.
- They report what crawlers may access, not whether an AI system will choose to cite you. Access is necessary, not sufficient.
The terms you will meet in the results
- Training crawler
- Collects pages to train AI models, for example GPTBot and ClaudeBot. Blocking it does not remove you from AI search.
- Retrieval or search crawler
- Fetches pages so AI answers can cite them, for example OAI-SearchBot, Claude-SearchBot and PerplexityBot.
- User-triggered fetcher
- Opens a page because a person asked an assistant to, for example ChatGPT-User. Operators say robots.txt may not apply.
- AI agent
- Uses a real browser to act for a person: compare, book, buy. Stopped by challenges and logins rather than robots.txt.
- Opt-out token
- A name you write in robots.txt that never fetches pages itself, for example Google-Extended or Applebot-Extended.
- Tier
- One of 6 priority levels in our database. Tier 1 crawlers are the ones whose absence you would notice in AI answers.
- AI Visibility Score
- Our 0 to 100 measure: 70 points for crawler access, 15 for AI files and 15 for technical readiness.
Questions about the tools
Do I need an account to use the tools?
No. Every tool on this page works without sign-up or an API key.
Which tool should I run first?
The AI crawler check on your homepage. It tells you whether there is an access problem at all; the other tools explain and fix what it finds.
Can I use the results in client reports?
Yes. The batch checker exports CSV and the GEO audit exports a PDF. Date each report, because sites and crawlers change.
How often should I re-check a site?
After every CDN, firewall, CMS or robots.txt change, and once a month otherwise. New crawlers launch regularly and bot-protection presets change without notice.