Changelog

Every one of the 257 bots now carries an evidence badge that says where its token came from. Official docs means the operator's own crawler page names the token; the profile links to that exact page and it was fetched and confirmed on the release date. Reported means no operator page exists but a credible public source names the token (the ai.robots.txt registry, Trakkr's AI Crawler Observatory, user-agents.net, or the operator's own non-crawler page); the profile says so in plain words, links the source, and tells you to treat the rule as best effort. A token in neither group is not listed, and the build fails if one slips in.

Why this release exists. Crawler lists across the industry, including earlier versions of this one, were built by accumulation: a token appeared in a server log or a community block list, it was added, and it was never checked. Some of those tokens are real crawlers whose operators simply never wrote a page about them; others are bare product names that were never user-agent strings at all. Deleting everything the vendor did not document would have thrown away real bots (Claude-Web still appears in logs; GrokBot and DeepSeekBot are real traffic). Keeping everything unlabelled would have kept presenting rumours as facts. The answer is to keep what a credible source can vouch for and label the difference on every card, every profile and every API response. Being right includes being clear about how sure we are.

Research update (September 27)

• 2025 TO 2026 MARKET DATA ON 24 PAGES: 13 guides, 4 tools, 6 bot profiles and the About page gained sourced sections from Cloudflare, Hostinger, BuzzStream, Google, Anthropic, Meta, Exa and Semrush, with comparison tables and charts.
• NEW TOOL FEATURES: an "Allow AI Search, Block Training" preset and an optional Content-Signal line in the robots.txt generator, five sample files in the validator, sample site lists in the batch checker, and worked examples in the schema generator.
• GEO AUDIT REBUILT: the GEO audit now runs 42 evidence-based checks instead of 58. Robots.txt is evaluated per crawler with RFC 9309 rules, a firewall probe compares AI crawler user agents with a browser, soft-404 pages no longer count as a valid sitemap or llms.txt, article-only checks are skipped on other page types, and points for having a particular plugin installed were removed. Blocking findings cap the score, every fix shows the points it adds, and the PDF and Excel reports were redesigned.
• CORRECTION: the Cohere guide no longer says cohere-ai collects training data. Cohere's crawler policy says it runs no training crawler today, and the ExaBot profile now points to Exa's documented ExaSearchBot.

The verification

• 177 OFFICIAL TOKENS, EACH WITH A PRIMARY SOURCE: OpenAI's bot page names GPTBot, ChatGPT-User, OAI-SearchBot and OAI-AdsBot. Anthropic's support article names ClaudeBot, Claude-User and Claude-SearchBot. Google's three crawler pages (common, special-case, user-triggered) account for every Google token. Yandex's official robot list contributed 38 tokens, Baidu's help page 7, Bing's crawler page 5. Meta, Apple, Amazon, Mistral, Perplexity, Moonshot, DuckDuckGo, Common Crawl, the Internet Archive, Kagi, LINER, SB Intuitions, Exa, Klaviyo, ByteDance's Toutiao Search team and the SEO tool vendors each publish a page that names their tokens, and every profile links to it.
• 71 REPORTED TOKENS, EACH WITH A NAMED PUBLIC SOURCE: Anthropic's legacy Claude-Web and anthropic-ai; xAI's GrokBot and xAI-SearchBot; DeepSeekBot, QwenBot, DoubaoBot, ERNIEBot, PanguBot; Cohere's two tokens; Google's Gemini-Deep-Research, Google-Gemini-CLI, GoogleAgent-Mariner and the superseded Google-NotebookLM; the AI coding agents Cursor, Devin, Trae and opencode; and the long tail of scrapers. Each profile shows an amber Reported badge, states that the operator has not documented the bot, and links the registry entry that names it.
• CHATGPT AGENT, EXPLAINED PROPERLY: OpenAI's help centre article on allowlisting the ChatGPT Work Cloud browser confirms that ChatGPT Agent is the bot name Akamai, HUMAN and Vercel use for that traffic, and that Cloudflare labels the same traffic ChatGPT Operator (detection ID 129220581). Both are now Official. The profiles explain that these are CDN bot-manager names, not robots.txt tokens: the traffic is verified by RFC 9421 Web Bot Auth signatures with public keys at chatgpt.com/.well-known, and the robots.txt token for on-demand ChatGPT fetches remains ChatGPT-User. This resolves a confusion that the first draft of this release made worse by calling the token fictional.
• EVERY PROFILE LINKS TO THE EXACT DOCUMENT: The "Official docs" link on each Official profile points at the specific page that names the token, not the operator's homepage. Reported profiles show "Official docs: none published" and a "Named by" link to the registry entry instead. The SoftwareApplication structured data carries the same URL.
• 38 STRINGS DELIBERATELY EXCLUDED, WITH REASONS: Dead Yahoo spellings (YahooSeeker, Yahoo-MMCrawler, Yahoo-Blogs, Yahoo-FeedSeeker), Teoma, Ai2Bot-Dolma (AI2's page names only AI2Bot), mis-spellings (LINER Bot, CloudVertexBot, SemrushBotSwa), bare company or product names (OpenAI, Operator, Grok, Firecrawl, Turnitin, Brandwatch) and tokens with no live public source at all. Their old URLs 301 to the category page. The list, with a reason for each, ships in the source data so the decision is auditable.
• A RETRACTION, IN WRITING: This release was drafted three times on the same day. The first draft was a 222-bot expansion built partly from a community list without checking sources. The second cut the database to 105 official-only tokens and, in doing so, wrongly removed eight documented bots (Slurp, Bytespider, ExaSearchBot, KlaviyoAIBot, SBIntuitionsBot, Arquivo-web-crawler, Cotoyogi, Webzio-Extended) and called ChatGPT Agent undocumented. Neither draft was deployed. This third version is what shipped, and both mistakes are recorded here because the whole value of this tool is that its claims can be trusted, and that includes claims about its own history.
• FILTER BY EVIDENCE: Every category page has All / Official docs / Reported only chips, the directory hub explains the two tiers at /directory#evidence, and /api/bots returns evidence and reportedSource for every token so integrators can apply their own threshold.
• STALE-TOKEN DETECTION, REWORDED: The robots.txt validator still flags legacy and never-a-token spellings (anthropic-ai, claude-web, google-notebooklm, notebooklm, chatgpt operator, chatgpt agent, operator, teoma), but the advice now reads "add the current token too" for legacy spellings that registries still carry, and "this is a CDN name, use your bot manager" for the ChatGPT Agent family.

One number, every page

• 257 EVERYWHERE, VERIFIED BY THE BUILD: The homepage hero, search placeholder, evidence strip, category tiles and comparison table; the directory badge; the robots.txt generator, validator, batch checker, llms.txt generator and schema generator; the compare pages; the footer; the WebApplication and FAQ structured data; and the meta descriptions that AI engines quote all read 257 from a single source file. The facts gate scanned 578 pages of source and rendered HTML and reports zero contradicting claims. The share card and every favicon were regenerated with a new cache-busting token.
• DIRECTORY REBUILT: 254 bot profile pages and 109 operator pages remain, each for a verified crawler. 120 profile pages and 51 operator pages were removed and 301 to their category or the operators index. The sitemap now lists 578 URLs, every one confirmed to return 200. The directory generator is category-aware, skips redirected slugs, and sets each page's documentation link from the vendor map.
• THE ROSTER ARTICLE IS GENERATED FROM THE DATABASE: 257 AI Bots You Should Know lists every token by category with its operator, purpose and safety rating, links each row to its profile, and explains which well-known tokens are deliberately absent and why. It is produced by a build script, so it cannot disagree with the checker. The 163, 196 and 222 slugs all redirect to it in one hop. The same pass fixed a templating defect that had left two Bot Profiles articles describing their subject as "operated by Unknown".
• BLOG SORTING AND READABILITY: "Latest published" and "Recently updated" now tie-break deterministically, and a regenerated article is dated as newly published rather than as an edit. Article body copy moved to 17px with a 1.75 line height, slightly wider word spacing and more room between paragraphs and headings; case study pages follow the same rules.
• SECURITY: Hono upgraded from 4.12 to 4.13.8, closing 30 published advisories including JSX escaping, serveStatic path handling, the query parser and CORS. The homepage probe re-runs the SSRF guard on the final URL after following redirects, so a public site cannot bounce the checker to a private address.

Every scan now includes a Googlebot crawl test

• THE EXACT URL, AS GOOGLEBOT SMARTPHONE: The homepage scan now runs a Googlebot crawlability test on the page you enter, not just the domain. It evaluates robots.txt for Googlebot, Googlebot-Image, Bingbot and Applebot on that path, then checks the HTTP status, challenge pages, noindex, the canonical and how much text is in the HTML before JavaScript.
• NO FALSE "NOT CRAWLABLE": Many CDNs block look-alike Googlebots and let the real one through. When our Googlebot request is refused but a normal browser gets the page, the report says "Probably crawlable", marks those checks Unverified and reads the page from the browser response. Our requests come from our server, not Google's IP ranges, and the report says so.

New tool: AI Crawler Diagnostic

• ONE PAGE, TWENTY CRAWLERS, SIX GATES: The free AI Crawler Diagnostic requests a single URL as 16 AI crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Meta-ExternalAgent, Amazonbot, Bytespider, CCBot, DuckAssistBot, MistralAI-User, cohere-ai and YouBot) plus Googlebot and Bingbot, and checks the Google-Extended and Applebot-Extended opt-out tokens. For each one it reports robots.txt, page directives, HTTP status, the redirect chain, firewall challenge pages and whether the content is readable without JavaScript, then ranks the fixes by how many crawlers each one unblocks.When the homepage scan says a bot is blocked, this says why, and which line to change.
• NEVER CALLS A GUESS A BLOCK: Every URL is also requested as a normal browser. If the site refuses the browser too, it is filtering our server, not the crawler, and the result is marked Unverifiable. If only the crawler is refused, the verdict is Firewall block: verify, because a real block and fake-bot protection (which lets the genuine crawler through from its own IP range) look identical from outside the operator's network. Neither is ever counted as reachable, and neither is reported as a confirmed block.An earlier draft reported sites that block datacenter traffic as blocking every AI crawler. It was caught in testing before release.
• ROBOTS.TXT READ THE WAY CRAWLERS READ IT: RFC 9309 group selection (a named group replaces the * group entirely), longest-match precedence with * and $, query strings included. A robots.txt that answers 5xx or 429 blocks the whole site, a 404 means no restrictions, and when a page redirects to another host that host's robots.txt is checked as well.
• DIRECTIVES BY CRAWLER PURPOSE: noindex blocks search engines and AI search crawlers. It is reported as a note, not a block, for user-triggered fetchers such as ChatGPT-User and for training crawlers, which follow robots.txt instead. Directives inside scripts and comments are ignored.
• FREE, NO SIGN-UP, AND SAFE TO POINT AT ANYTHING PUBLIC: Private, loopback, link-local and cloud-metadata addresses are refused on the first request and on every redirect hop, including IPv6 forms. Each run is capped at about 25 seconds and is rate limited per visitor, because every run sends about 20 requests to the site being tested.

Directory: search, filters and one count

• SEARCH AND FILTER EVERY CATEGORY: Each directory category page now has a search box, sorting (A to Z, latest added, operator, risk), operator, purpose, safety and evidence filters, one-click chips for the common providers, and filter URLs you can share. Every card stays in the page HTML, so crawlers and visitors without JavaScript still see the full list.
• 248 MEANS 248 ON EVERY PAGE: The footer's "All AI Crawlers" badge said 254 beside a site-wide claim of 257. Both were true: 257 user-agent tokens are tracked, on 254 profile pages, because three unversioned tokens (ChatGPT-User, MistralAI-User, Perplexity-User) share a profile with their versioned twin. Every total now states tokens, and category pages that differ say both, for example "97 user-agent tokens on 85 profiles". A new build check reads the rendered text of every page in the sitemap and fails on any bot count that disagrees.
• CLEANER PROFILES: Markdown asterisks that showed up as literal ** in bot profile copy now render as emphasis on all 254 profiles, and are stripped from the structured data.

Search visibility work

• CUT-OFF DESCRIPTIONS FINISHED: 110 articles had a meta description that stopped mid-word (one ended "Includes ste"), repeated in the social and structured-data copies. Each now ends on a complete sentence in all four places, and the build refuses a description that does not.
• ONE HEADLINE PER PAGE: The structured-data headline now matches the visible title on every page, so an AI answer and a search result quote the same string.
• QUESTIONS ANSWERED WHERE THEY ARE ASKED: 68 real search queries that the site was shown for but did not answer, such as "how do I monitor AI bot traffic to my website?", "robots.txt disallow all" and "what is PerplexityBot?", now have a heading that uses the exact wording and a direct answer underneath, across 33 pages.
• 34 NEW INTERNAL LINKS: Including four guides that no other page linked to, now reachable from the articles readers arrive on.
• SEVEN THIN ARTICLES MERGED: Seven posts that had stopped earning any search traffic now redirect to the stronger guide on the same topic, so readers land on one complete answer: Applebot vs Applebot-Extended, How to check if ChatGPT can read your site, Agentic AI browsing, robots.txt for WordPress, The complete robots.txt guide, GDPR and AI crawlers, and llms.txt examples. The blog now has 148 articles.
• 22 ARTICLES UPDATED, AND DATED: Every article whose content changed in this release shows "Updated September 23, 2026", carries the new date in its structured data and sitemap entry, and moves up the blog's "Recently updated" sort. Articles that only gained a related-post card keep their original date.
• CASE STUDY LINKS: The case study now links to the guides and bot profiles it references (16 links in English, 10 in Vietnamese) and ends with a short "Keep exploring" block. The downloadable PDF is unchanged.
• EASIER READING: Long-form text on articles, the case study, docs, profiles, the changelog and the legal pages is slightly larger with more line spacing.
• SITEMAP CHECKED BOTH WAYS: The sitemap lists 578 URLs. The build now confirms every listed URL returns 200 and names itself as canonical, and that no linked, indexable page is missing from it.

Case studies and a named diagnosis

• CASE STUDY, TWO LANGUAGES: The Invisible AI Crawler Block is a full technical investigation into a five-star hotel whose WordPress origin returns an infinite 301 loop to any request carrying the X-Forwarded-Proto header that Cloudflare and every other CDN adds. The site worked perfectly in a browser and was invisible to every AI crawler, and to this checker, at once. The English and Vietnamese editions and the downloadable PDFs are rendered from the same source file, so the page, the PDF and the post cannot drift apart. Paired with hreflang, TechArticle structured data, a feature image, the Share-to-AI toolbar and a dedicated header menu item.
• HONEST FETCH FAILURES: The checker now separates DNS failure, connection refusal, TLS errors, timeouts, redirect loops and hard HTTP blocks instead of collapsing them into one message. When it detects the redirect loop it names the trigger header, hands over the two curl commands that reproduce the difference, and says plainly that the site works in an ordinary browser. Retries stop the moment a loop or DNS failure is confirmed, so a diagnosis that used to time out after sixty seconds now returns in under three.
• EDGE EMULATION FOR LOCAL TESTING: The failure could not be reproduced from a developer machine because the header only exists behind a CDN. The checker's fetch layer can now emulate the edge headers locally, which is how the diagnosis was tested before it shipped.
• TWO MORE RETRACTIONS, IN WRITING: The investigation produced two confident wrong diagnoses before the right one. Both are documented in the case study, because diagnostic authority comes from being corrected by measurement, not from never being wrong.

Our biggest accuracy and honesty upgrade. We recalibrated the entire AI Visibility Score against what vendors actually document in 2026, added a live bot-intelligence layer that flags deprecated tokens and stealth crawling, and shipped four new free tools. Every article published at the time was re-verified for 2026 accuracy.

Why this release exists. Through 2025 most AI visibility tooling, including ours, quietly inherited assumptions that stopped being true. Scores rewarded publishing an llms.txt file as though it were a confirmed ranking input. Checkers kept treating anthropic-ai as a live crawler years after Anthropic replaced it. Opt-out preference tokens were counted as bots. The result was a familiar failure mode: a site scores well, the owner relaxes, and ChatGPT still cannot read a single page. v6.0 is a deliberate correction. Where the evidence is strong we score it heavily; where it is a reasonable hedge we say so and weight it lightly; and where nothing can be measured from the outside we tell you that rather than inventing a number.

Scoring, rebuilt from the evidence

• SCORE v6 (70/15/15): Bot Access now carries 70 of 100 points, AI Infrastructure an honest 15, and a brand-new Technical Readiness category the remaining 15 (sitemap 6, structured data 6, HTTPS 3). The reasoning: crawler access is binary and verifiable. If GPTBot is disallowed, your content cannot be cited, and no amount of schema markup compensates. So access dominates the score. The previous weighting let a site with excellent AI files but a blocked GPTBot outrank a plain site that was fully readable, which is exactly backwards. Every weight, and the argument behind it, is now documented on the methodology page.
• LLMS.TXT DEMOTED, DELIBERATELY: Publishing llms.txt used to be worth far more. It is now a small part of a 15-point category. No major AI vendor has confirmed llms.txt as a retrieval or ranking input. It remains a sensible, low-cost hedge and we still check it strictly, but inflating its weight produced falsely reassuring scores. We would rather lose the neat narrative than overstate the evidence.
• WEIGHTED BY REAL IMPACT: Blocking GPTBot, ClaudeBot, PerplexityBot or Googlebot costs materially more than blocking a minor scraper, and the score reflects that instead of counting every user-agent equally.

2026 bot intelligence

• DEPRECATED TOKEN DETECTION: We flag retired user-agents such as anthropic-ai and claude-web that are still sitting in robots.txt files. This is the most common silent failure we see. An operator renames its crawler, the old rule keeps matching nothing, and the site owner believes they are still governing a bot that no longer exists under that name. Worse, the intended policy is not being applied at all. We name the stale token and give you the current replacement.
• OPT-OUT TOKENS ARE NOT CRAWLERS: Google-Extended and Applebot-Extended are now correctly labelled as training-preference signals rather than fetchers. Neither ever requests a page. Disallowing Google-Extended opts you out of Gemini model training but does not remove you from AI Overviews, and blocking it will not reduce your crawl traffic by a single request. Tools that list it as a bot lead people to the wrong conclusion about what they just changed.
• FETCHERS THAT IGNORE ROBOTS.TXT: Some user-triggered fetchers do not consult robots.txt at all, and we now say which, instead of implying a rule will stop them. When a person pastes your URL into an assistant, that request is often treated as user-initiated rather than crawling. A robots.txt rule is the wrong tool there; edge or WAF rules are. Telling you a disallow line works when it does not is worse than saying nothing.
• STEALTH CRAWLING FINDINGS: Where independent research has documented undeclared or misrepresented crawling, most prominently the Perplexity findings, the report surfaces it so your policy accounts for traffic that may not honour your rules.
• AGENTIC BROWSERS, HONESTLY UNDETECTABLE: ChatGPT Atlas, Comet and Copilot Actions drive a real browser session on a user's behalf. They largely present as ordinary browser traffic, so no robots.txt audit, ours included, can confirm or deny their access. We list them as a known blind spot rather than scoring a category we cannot measure. Naming the limit is more useful than a confident guess.
• HONEST WAF HANDLING: If a firewall blocks our probe we report exactly that, across 15+ recognised providers, instead of recording a passing grade we did not verify.

Four new free tools

• LLMS.TXT GENERATOR: The llms.txt builder gives you live preview, a correctly formed Optional section, and starting templates for SaaS, blogs, e-commerce and agencies. Most hand-written llms.txt files we encounter are malformed in ways that make them useless to parse, so the generator enforces valid structure as you type.
• SCHEMA GENERATOR: The JSON-LD generator covers 8 types: Organization, LocalBusiness, Article, FAQPage, Product, WebSite, BreadcrumbList and HowTo. Structured data is one of the few levers that demonstrably helps machines read entities and relationships correctly.
• BOT COMPARISONS: The /compare hub ships 20 side-by-side matchups such as GPTBot vs ClaudeBot, each with user-agent strings, safety rating, SEO impact and copy-paste blocking rules, for the common case of allowing one operator while blocking another.
• SCORE BADGE: After any check you can embed a live SVG score badge in a site footer or README, with one-click HTML and Markdown embed codes.

Standards and content

• WEB BOT AUTH: We now detect the IETF Web Bot Auth directory at /.well-known/http-message-signatures-directory. This is the emerging answer to a real problem: user-agent strings are trivially spoofed, so allow-listing by name has never been trustworthy. Cryptographic signatures let a crawler prove which operator it belongs to. Adoption is early, which is precisely why it is worth knowing whether the operators reaching you support it.
• EVERY ARTICLE RE-VERIFIED: Every guide published at the time was reread against 2026 reality, not spot-checked. That means honest llms.txt framing, deprecated-token warnings, corrected user-triggered fetcher behaviour, the Perplexity compliance findings and recalibrated score references throughout. Advice that was accurate in 2024 is now actively misleading in places, and stale guidance from a tool that claims to measure accuracy is a contradiction we were not willing to ship.
• ONE BOT COUNT, ONE SOURCE OF TRUTH: The homepage checker, directory, validator, generator and batch checker all read the same bot database, now including agentic entries like Google-Agent (Project Mariner) and OAI-AdsBot. Today that database tracks 257 bots across 8 categories.

Verified numbers, documentation and tool comparisons

A tool whose pitch is honest measurement had been mis-stating its own figures: different pages claimed different bot totals, and the downloadable copy of the bot database had drifted to 154 entries while the real count was 257. Editing the text would have fixed nothing, because the cause was that any number could be typed by hand anywhere. The numbers now come from one source and a build gate refuses to ship the site when a claim disagrees with the database.

• One source of truth for every figure. Bot counts, page counts, category totals and the batch limit all come from one file, and the build fails if any page disagrees with it.
• The downloadable bot database is generated, not maintained by hand. It is rebuilt from the live database on every build, so it cannot fall behind again.
• New documentation section. Step-by-step guides for running a scan, reading your score, fixing a blocked crawler, and auditing client sites.
• New comparison section. How this tool compares with the alternatives, including the rows where the other tool wins and who should choose it instead.
• Better crawlability and accessibility. Corrected robots.txt rules, redirects for renamed pages, a real 404 page, AA link contrast and semantic landmarks throughout.
• Safer outbound links. External links no longer leak referrer data or expose the originating tab.

Mobile, rebuilt where it was actually broken

Most of this release was about being honest with numbers. This part is about being honest that the site was noticeably worse on a phone than on a desktop, which is where a majority of visitors read it. Every fix below was measured at real handset widths rather than judged by eye in a resized browser window, and desktop layout is deliberately untouched.

• THE MENU NOW OPENS AND CLOSES PROPERLY: The mobile navigation was rebuilt as a slide-in panel with every tool reachable in one tap. The old panel could leave the page unable to scroll after it closed, which reads to a visitor as a frozen site rather than a stuck menu. It also added invisible width to every page, so the whole site could be dragged sideways a little on every screen.
• THE MENU BUTTON IS VISIBLE ON EVERY PAGE: On the GEO audit tool the button was there and worked, but rendered as a completely blank square. The three lines of a menu button come from an icon font. That page was not loading the icon font at all, so all of its icons were invisible while the markup around them was correct. Nothing errored and nothing looked broken in the code, which is why it survived so long. The build now refuses to ship a page that mounts the navigation without the icons it needs.
• TAPPING A TOOL LANDS ON THE TOOL: Choosing a destination from the mobile menu used to land part-way into the section, with the heading scrolled up out of view.
• THE FIRST SCREEN IS THE FIRST SCREEN: On a phone the opening section now fills the screen exactly, instead of leaving a strip of the next section showing underneath. The first thing a visitor read below the main call to action was the start of an unrelated headline further down the page. It now tracks the live viewport, so it stays correct when the browser's address bar slides away.
• READABLE AT 320px: Text, tap targets and the comparison tables were checked at the narrowest phone widths still in use, not just at modern handset sizes.

Every page is now measurable, and privacy copy is tighter

• ANALYTICS COVERAGE ON EVERY PAGE: Two pages, the GEO audit tool and the not-found page, were loading no analytics at all. This is the quietest kind of fault. There is no error, nothing looks wrong, and the reporting does not warn you: it simply shows fewer pages, which is indistinguishable from those pages having no visitors. A release check now verifies that all 578 listed pages plus the unlisted tool and error pages carry analytics and the cookie banner, and separately confirms in a real browser that the tag stays silent until consent is given and genuinely starts once it is. Both error pages are included, because a not-found page is exactly the one you want in reporting: every visit to it is a broken link worth fixing.
• CONSENT STILL COMES FIRST: No analytics cookie is set and no tracking script we load runs until you accept. Rejecting also clears anything previously set, and the Global Privacy Control browser signal is honoured without needing a click. One exception is now written into the cookie policy rather than left unsaid: our hosting provider adds a cookieless visitor counter to every page after our code has run, so the banner cannot switch it off. It stores nothing on your device and cannot recognise you on a later visit or on any other site. It is disclosed in full, including how to block it.
• FEWER TECHNICAL DETAILS IN THE POLICIES: Our privacy and cookie pages still name every processor we use, which is what the disclosure is for, but no longer print internal account identifiers alongside them. Those identifiers gave a reader nothing useful and are not required by any disclosure rule. A release check now scans published pages for credentials, tokens and internal identifiers so none can reach a visitor by accident.

Upgrade note: because the weighting changed, a score from before v6.0 is not directly comparable with one after it. Sites that were blocking a major AI crawler while publishing tidy AI files will generally see a lower, more accurate number. That drop is the fix working, not a regression. Re-run your check to get a current baseline, and see the methodology page for the full formula.

• GEO AUDIT TOOL: Full GEO audit tool available at /geo-audit (now superseded by unified report in v4.6)
• SCHEMA FIX: Trimmed sameAs links in the author schema to one verified profile

• REBRAND: AI Crawler Check is now developed by Horatos.ai: an AI Search Optimization (AISEO) & Generative Engine Optimization (GEO) consultancy. All author links, legal pages, schema markup, and branding updated
• 163 BOTS DEPLOYED: All 8 new bots from v4.3 now live in production: OAI-AdsBot, MistralAI-Index, Google-NotebookLM, Google-Read-Aloud, Google-Site-Verification, FeedFetcher-Google, GoogleProducer, Amazon-Bedrock-AgentCore
• LEGAL PAGES: Privacy Policy, Terms of Use, and Cookie Policy updated to reflect Horatos.ai as operating entity

• 30 BLOG POSTS: Published 3 new E-E-A-T focused articles: E-E-A-T for AI Search, GEO vs SEO Complete Comparison, Why AI Bots Can’t Crawl Your Website
• 12 NEW IMAGES: 3 hero images + 9 inline diagrams covering trust pyramids, GEO/SEO frameworks, diagnostic flowcharts, and robots.txt fixes
• E-E-A-T AUTHORITY: Deep-dive into Experience, Expertise, Authoritativeness & Trustworthiness signals for AI search citation. Original frameworks and expert analysis
• INTERNAL LINKS: 80+ strategic internal links across 3 posts. Heavy tool CTAs, cross-referencing to crawler guides, robots.txt generator, schema markup, and audit checklist
• 163 BOTS: Added 9 new bots: OAI-AdsBot (ChatGPT Ads, Apr 2026), MistralAI-Index (Vibe search index), Google-NotebookLM, Google-Read-Aloud, Google-Site-Verification, FeedFetcher-Google, GoogleProducer, Amazon-Bedrock-AgentCore. Upgraded Amazonbot to Tier 2 (Nova AI training confirmed)
• WAF FIX: Fixed false-positive bot blocking when target sites use Cloudflare WAF. Multi-UA fallback strategy with WAF detection

• 27 BLOG POSTS: Published 3 new in-depth articles: Google-Agent & Project Mariner, WebMCP Guide, How to Create llms.txt
• 12 NEW IMAGES: 3 hero images + 9 inline diagrams covering agentic browsing, WebMCP architecture, and llms.txt structure
• TOPICS: Agentic Engine Optimization (AEO), WebMCP Tool Contracts, user-triggered-agents.json, Web Bot Auth, llms.txt templates for 4 website types

• NEW BOT: Google-Agent (Project Mariner): Google’s brand-new agentic AI crawler that navigates sites, fills forms, and takes actions on behalf of users. Monitor its activity to measure how important WebMCP is for your site
• 155 BOTS: Bot database expanded from 154 → 155. Google-Agent added to Google Bots category with full directory page, FAQs, and structured data. Each bot gained its own SEO-optimized directory page.
• 24 BLOG POSTS: Published 12 new articles (posts 13–24) covering robots.txt creation, Applebot-Extended, Meta-ExternalAgent, AI SEO Audit Checklist, Cohere AI Crawler, and more
• E-E-A-T FOCUS: All new posts emphasize Experience, Expertise, Authoritativeness, and Trustworthiness with author bylines, expert recommendations, and data-backed analysis
• ANCHOR TEXT: Strategic homepage anchor text distribution: “AI crawler checker free” (14x), “AI crawler checker online” (12x), “web crawler tool free” (6x), “AI crawlers” (4x)
• CLARITY: Microsoft Clarity analytics added to all pages: heatmaps, session recordings, and user behavior tracking across homepage, blog, tools, directory, and legal pages
• 48 NEW IMAGES: 12 hero images + 36 inline diagrams/infographics for blog posts 13–24. Custom illustrations for each article topic

• FOCUS: Removed homepage signals parsing (Schema, H1, Meta Description, Canonical, OG tags): tool now focuses solely on AI bot access & infrastructure
• SIMPLIFIED SCORE: New 2-component scoring: Bot Access (65pts) + AI Infrastructure (35pts) = 100pts. Removed Content (25) and Technical (10) components
• REMOVED: Browser Rendering API fallback, WAF/bot protection detection, homepage HTML parsing, extractSchemas(), extractMeta(), countTag(): eliminates all false-negative complaints from WAF-blocked sites
• NO MORE WAF ISSUES: Since homepage is no longer scanned, sites behind Cloudflare/LiteSpeed/Wordfence WAFs get accurate results with zero workarounds needed
• SMARTER RECS: Only actionable recommendations: unblock AI bots, create llms.txt/llms-full.txt, upgrade partial to full access. No more schema/meta/author suggestions
• CTA: “Want a Full AI Visibility Audit?” card replaces Homepage Signals section: directs users to paid GEO audit for deeper analysis
• FASTER: Removed homepage fetch + retry + Browser Rendering: checks complete ~3s faster on average

• BREAKTHROUGH: When WAF blocks homepage scan, tool now falls back to Cloudflare Browser Rendering API: headless Chromium renders the page like a real browser, bypassing CAPTCHAs and WAF challenges
• RESULT: WAF-protected sites (LiteSpeed, Cloudflare, Wordfence, Sucuri) now get full homepage analysis: Schema, H1, Meta Description, Canonical, OG tags all detected correctly
• 3-TIER FETCH: Homepage fetch now uses three strategies: (1) Chrome UA fetch → (2) Safari UA retry → (3) Headless Browser Rendering fallback
• UX: Green “Browser Rendering Used” badge shows when headless browser was needed to bypass WAF
• TYPED: Added Env type bindings for CF_API_TOKEN and CF_ACCOUNT_ID environment variables

• HONESTY: WAF-blocked signals now show ⚠ “Unable to verify” with yellow triangles instead of pretending they pass or fail: no more misleading results
• UX: New 2-column layout in WAF warning: “✓ Accurate Results” vs “⚠ Unable to Verify”: users instantly see what’s reliable
• NEW: Links to Google Rich Results Test & Schema.org Validator in WAF warning: guides users to verify schema manually
• SMART: Recommendations now skip “Add FAQ Schema” and “Add Author signal” when WAF blocks scan: avoids recommending things the site may already have
• UX: WAF-blocked signal rows show impact as “N/A” in yellow instead of misleading Critical/High labels
• CLEANUP: Removed temporary debug endpoint /api/debug-signals

• ENHANCED: WAF/Bot detection now covers 15+ providers: Cloudflare, LiteSpeed, Wordfence, Sucuri, Imperva, Akamai, StackPath, AWS WAF, hCaptcha, reCAPTCHA
• NEW: Auto-retry with Safari UA when initial Chrome UA triggers bot protection: bypasses simpler WAF rules
• IMPROVED: Bot-protected sites now score 19/25 content (was 13/25): assumes H1, meta desc, schema exist (standard for CMS sites behind WAF)
• UX: Homepage signals show yellow ⚠ warning icons for WAF-blocked items instead of misleading red ✗ marks
• IMPROVED: Homepage fetch now sends full browser headers (Accept, Sec-Fetch-*, Accept-Language): reduces WAF trigger rate

• FIXED: Schema detection now traverses @graph arrays (WordPress/RankMath/Yoast): previously only checked root-level @type
• FIXED: extractSchemas() rewritten as recursive collectSchemaTypes(): handles nested @graph, mainEntity, and arrays at any depth
• NEW: Bot Protection Detection: identifies WAF/CAPTCHA pages (LiteSpeed, Cloudflare, Wordfence) and flags botProtection: true in results
• IMPROVED: Bot-protected sites now receive partial content credit (13/25) instead of 0: no longer penalized for CAPTCHA pages
• PERF: Homepage HTML cached from STEP 0 probe: eliminated duplicate fetch, saving ~200ms per check
• IMPROVED: extractMeta() now handles multi-line attributes (WordPress meta tags with newlines between properties)

• Fixed False Positives: llms.txt and llms-full.txt detection now uses strict validation: redirect: manual prevents 301/302 homepage redirects from being counted as “Found”
• Content Validation: AI files must return 200 OK with non-HTML content. Responses containing HTML (custom 404 pages, redirected homepages) are correctly rejected
• Smart Redirect Follow: Follows one redirect only if the target URL ends with the same filename (e.g., shopify.com/llms.txt → www.shopify.com/llms.txt). Homepage redirects are rejected
• Removed ai.txt: /ai.txt has been removed from all checks: no actual standard exists for this file. Only llms.txt and llms-full.txt (official llmstxt.org standard) are checked
• Score Rebalanced: Bot Access (40), AI Infrastructure (25), Content (25), Technical (10). Removed 5pts from ai.txt, redistributed to Content quality signals
• Social Preview: All pages now have OG image + Twitter Card meta tags for rich social previews when sharing on Facebook, LinkedIn, X/Twitter, Slack

• 154 Bots: Added 3 new versioned AI bots: ChatGPT-User/2.0, MistralAI-User-1.0, Perplexity-User-1.0: closing a gap in our AI bot coverage
• Deprecated Bot Labels: Claude-Web and anthropic-ai are now marked as Deprecated per Anthropic’s official docs (replaced by Claude-User, Claude-SearchBot, ClaudeBot)
• ClaudeBot Relabeled: Changed from “Claude AI (Legacy)” to “Claude AI Training” to accurately reflect its official purpose per Anthropic docs
• Versioned Bot Matching: New parser logic handles versioned user-agents (e.g., ChatGPT-User/2.0 inherits ChatGPT-User rules from robots.txt). Same for -1.0 suffixes
• Cross-Verified: AI bot list verified against official sources: OpenAI Bots docs, Anthropic Crawler docs, CrawlerCheck.com directory, Search Engine Journal’s 2025 list

• Partial = Accessible: Bots with partial access (e.g., /admin blocked but / allowed) now count as “accessible” in overview. Only fully BLOCKED bots are counted as denied. This prevents misleading “0% allowed” for sites that only block admin paths
• Category Overview Rework: Shows “X full · Y partial” breakdown instead of just “X/Y allowed”. Percentage now reflects real accessibility
• Bot Card Labels: Status now shows “FULLY ALLOWED”, “PARTIALLY ALLOWED”, or “BLOCKED” for clarity
• Report Naming: PDF/Excel exports renamed from “AI Visibility Report” to “AI Crawler Check Report”: accurate to the tool’s function
• Score Adjustment: PARTIAL bots now receive 75% score credit (was 50%). Reflects that partial access still means content is crawlable

• Double Verification: robots.txt is now fetched twice and results compared. If content changed between fetches (CDN cache, etc.), the freshest version is used
• Enforced Backend Processing: Minimum 8-second server-side processing time ensures all checks genuinely complete. No shortcuts, no cached guesses
• Frontend Sync: Animation steps now match real backend processing. 10+ second minimum display with step-by-step progress visible to the user
• Error UX Improved: Invalid domains show immediately clear error messages. No more fake results for non-existent websites
• Version Tag: Reports now include processingTime to prove thorough analysis

• Critical Fix: Added mandatory domain reachability check (STEP 0). Invalid or non-existent domains (e.g., example.com.sgsg) now return a clear error instead of fake results
• DNS Validation: The tool now verifies the website actually responds before analyzing. DNS failures, connection timeouts, and Cloudflare 5xx errors are properly detected
• robots.txt Validation: Content is now validated to ensure it’s actually a robots.txt file (not a custom error page returning 200). Per RFC 9309: 404 = all allowed, 403 = all blocked
• Minimum Display Time: Results always show for at least 6 seconds with progressive step animation to confirm thorough analysis
• Error UX: Clear, actionable error messages for unreachable domains, DNS failures, and timeout scenarios

• Critical Fix: Completely removed HTTP simulation. robots.txt is now the SOLE source of truth for bot access status. This eliminates ALL false positives caused by WAF/CDN/IP-based blocking (Cloudflare, Akamai, etc.)
• Why: HTTP 403 from datacenter IPs does NOT mean the site owner blocks a bot: it's often generic bot-protection. Only robots.txt reflects the site owner's explicit intent
• Partial Explained: Added tooltip explaining what "Partial" means on every bot card
• Bot Purpose Labels: Every bot card shows its purpose: Training, Indexing, User Request, Both, Scraping, Monitoring, Social
• Check Timing: Increased analysis animation to 14+ seconds for user trust
• Directory UI Fix: Fixed broken layout in Web Crawler & Bot Directory pages

• Critical Fix: Resolved false-positive blocking reports caused by WAF/CDN detection (e.g., Cloudflare-protected sites returning HTTP 403 for all datacenter IPs)
• Accuracy: robots.txt is now the canonical source of truth. HTTP simulation only upgrades severity, never overrides a robots.txt ALLOWED status
• Bot Purpose Labels: Every bot now shows its purpose: Training, Indexing, User Request, Both (Training + User Request), Scraping, Monitoring, Social
• Partial Explained: PARTIAL status now includes an inline explanation showing which paths are blocked vs allowed
• New Logo: Custom SVG logo for AI Crawler Check replaces old BH branding in header
• Full Navigation: Directory, Tools, Generator, Validator, Batch Checker links on every page
• Check Animation: Progressive step-by-step loading with realistic timing for better user trust
• Changelog: This page: track all updates and version history

• Robots.txt Generator: Create custom robots.txt with 257 bot presets, 6 one-click presets, per-bot toggles
• Robots.txt Validator: Paste or fetch robots.txt, analyze against every bot in the database (257 today), SEO Safety Score
• Batch URL Checker: Check up to 20 URLs at once with CSV export and formatted reports
• Bot Directory: A page for every bot, category and operator (today: 254 bot profiles, 8 category pages, 109 operator pages)
• SEO: JSON-LD structured data, FAQ rich snippets, XML sitemap covering every page (578 URLs today)

• Expanded to 151 bots across 8 categories (AI bots, search engines, Google bots, SEO tools, social bots, scrapers, cloud services, other agents)
• PDF export with branded report
• Meta-robots and X-Robots-Tag analysis
• AI Visibility Score with 4-component breakdown

• First release with basic robots.txt checking
• Core AI bot database and analysis engine