Every release, newest first. Bot facts change monthly, so this is also where corrections are recorded.
v6.3current version
26releases logged
265crawlers in the database
617pages on the site
v6.3Meet Scout
Meet Scout, 10 Research Articles From Our Own Data and Fuller Pages Everywhere
, updated
Scout is a small robot built from the Horatos mark: two black feet, two teal hands and a lime signal antenna. He is not decoration. On every page he gives a short, specific take written from that page's own data, and he reacts to what you do. This release also goes deeper: ten research articles built on primary sources and our own data, a self-test at the end of each new article, and at least 1,500 words on every page.
Scout Agent, our researcher and author
MEET SCOUT AGENT ON THE HOMEPAGE: a new Meet Scout Agent chapter, after How it works and the introduction video, explains who does the research behind every page: he tests the live web, reads primary sources, shows the evidence and gets a human review. It is plain HTML, so crawlers and readers without JavaScript see all of it.
LATEST RESEARCH FROM SCOUT AGENT: the newest articles have their own homepage chapter, each card linking to Scout's author page, so you can see who wrote it and how it was checked.
A REAL AUTHOR PAGE:Scout's author page was rebuilt with a profile panel, a stats bar, a table of contents, numbered research steps, the sources Scout cites and the ones he never cites, every article by topic, and a way to report an error to his editors. Scout now has his own LinkedIn profile, linked from the page and from the author data on every article.
CREATED AND REVIEWED BY AI CRAWLER CHECK: Scout is an AI research agent created by AI Crawler Check, and the AI Crawler Check editorial team reviews his work. Every article, case study and profile says so in its structured data, so search engines and AI systems credit the right people.
AN AUTHOR BOX ON EVERY ARTICLE: each of the 159 articles ends with a Scout author box that links to his page and stacks neatly on phones.
SCOUT LEADS THE TOOLS: the GEO audit, the schema generator, the contact page and every other tool now open with Scout: he explains what the tool checks, reacts while it works, and on the contact page his speech bubble follows the topic you pick.
SCOUT ON VIDEO: Scout narrates all 3 new videos, including a Meet Scout video on his author page.
SCOUT RADAR CARRIES HIS NAME: the new Scout Radar is Scout's study of what the top 10,000 sites decide about AI crawlers, refreshed by a daily pulse, with its own RSS feed.
Our own identity, contact page and contact form
ONE NAME, ONE INBOX: every page now uses AI Crawler Check's own addresses: hello@aicrawlercheck.com for questions and sponsor@aicrawlercheck.com for sponsorship. Horatos.ai is credited as the company that designs and operates the site. The company LinkedIn page and full business address are in the structured data.
A REDESIGNED CONTACT PAGE: the contact page has four ways to reach us, a sponsorship section and a real message form with five topics, including sponsored posts. Messages are stored safely, routed to the right inbox, checked field by field and protected from spam.
A SECURITY CONTACT: security researchers can find where to report a problem at /.well-known/security.txt.
Schema generator and GEO audit, rebuilt
A SCHEMA GENERATOR THAT TELLS THE TRUTH: the schema generator covers 13 types in four groups, marks Google's required and recommended fields, shows what Google actually does with each type today, and has a working sample for every type. A schema plan for 12 kinds of business tells you which types to add first.
A GEO AUDIT YOU CAN READ BEFORE YOU RUN IT: the GEO audit shows a full sample report scored live by the real engine, a guide for 8 kinds of site, and a section on how we keep the scoring honest.
Search results, speed and AI agents
ADDRESSES THAT DO NOT EXPIRE: 18 page addresses that carried a year, month or changing number were renamed to evergreen ones. Each old address forwards in one step, and read counts moved with them.
CLEAN RICH RESULTS: Google's Rich Results Test now reports no errors and no optional warnings on any page: full business address, article authors, images and exact publish and update times, and the video dates include a time zone. We do not show star ratings, because we have no real reviews to show.
FASTER ON PHONES: pages paint without waiting for any outside server. The site stylesheet is built into each page, the Inter font and the icons are served from our own domain (the icons trimmed to the 185 we use), the PDF and Excel exports load only when you press them, and nothing shifts while the page loads.
READY FOR AI AGENTS: every form describes itself to AI browser agents (WebMCP), the main tools can be called by agents in browsers that support it, and a tools catalog lists what agents can do here. All of it is read-only except the contact and newsletter forms, which always wait for a person to press send.
Scout as your guide
A QUICK TAKE ON EVERY PAGE: all 265 bot profiles say what the bot does, whether blocking it costs you anything, and hand over the exact robots.txt lines. Operator pages such as OpenAI split that company's crawlers by job and give the training-only rule, built only from verified tokens.
A LIVE ROBOTS.TXT COACH: the generator now reads your file as you build it and warns before a search engine or a whole site gets blocked by mistake.
MOODS THAT MEAN SOMETHING: Scout scans while a check or GEO audit runs, looks worried at a poor score or an error, cheers a good one, and gets lost on a 404 page. A Scout button in the corner of every page opens a tip for that page.
Design
A SCOUT COVER ON ALL 159 ARTICLES: every guide has a new Scout cover matched to its topic. Bot profile covers show the real bot name, operator and user-agent token.
A CALMER INTERFACE: a glass header, softer cards, buttons that respond when pressed and clear teal focus rings for keyboard users. All motion stops when your device asks for reduced motion, and Scout is hidden from screen readers while his tips stay readable.
A CLEARER HEADER: Case Studies sits before Blog, the tools menu has two columns with a Start here card, and Docs moved to the footer.
A REDESIGNED HOMEPAGE: everything below the hero now reads as one story, in numbered chapters: how it works, what you get, why it matters, the six causes, the tools, the research team, the database, the comparison, the Googlebot test and the questions. Scout Agent moved up from the bottom of the page to the middle, next to the latest research.
ALL TEN FREE TOOLS, GROUPED BY JOB: the homepage showed seven. It now lists all ten, grouped as find the problem, write the fix and watch your logs.
A FULL FOOTER MENU: new Research and Company columns link the case studies, the top 10,000 robots.txt study, tool comparisons, the methodology and Scout's author page, and the Bot Directory column lists every category.
MENUS THAT GET OUT OF THE WAY: the Free Tools menu closes when you click elsewhere, press Escape, move the mouse away or tab out, and the cookie notice no longer covers the open phone menu.
STRICT VISUAL CHECK: the three floating buttons (contents, back to top, Scout) are now one size, one shape and evenly spaced. Article covers keep the topic label clear of the title. Code blocks keep the Copy button clear of the code, the Bot Directory filter bar no longer slides under the header, and the footer shows a current verification date.
NO STRAY UNDERLINES: cards, buttons and chips are no longer underlined like links in a sentence; links inside text still are.
CALMER READING: articles use a comfortable line length, tables sit on soft cards, and wide tables on phones show an edge shadow so you know they scroll. The homepage hero is unchanged.
ONE SECTION PER TOPIC ON EVERY TOOL PAGE: four stacked content layers and five shared blocks were removed from the tool pages. Each tool now has one set of sections written for it (its options, samples, export columns, log format or score method), one FAQ and one row of next steps.
A MATCHING DESIGN FOR ALL ELEVEN TOOLS: every tool opens with the teal hero and Scout, uses one card style, and has properly styled buttons. The Tools hub lists each tool as a card under one heading.
ONE LINK PER PAGE TO EACH PAGE: a page now links to another page once, because search engines and AI crawlers credit the first link they meet. Later mentions are plain text.
AN "AI CRAWLER" WORDMARK: the logo is unchanged; the words beside it now put "AI Crawler" on a lime highlighter band, with a small crawler dot that runs along it on hover.
A RADAR THAT KEEPS UP: Scout Radar now runs on a fresh scan of the top 10,000 sites and adds a daily pulse that re-reads the biggest sites a few at a time and lists every crawler a site newly blocks or allows.
ACCESSIBILITY SETTINGS ON EVERY PAGE: a round button at the bottom left, which you can drag to either side like AssistiveTouch, opens larger text, high contrast, underlined links, line spacing, a plainer font, a stronger focus outline, stopped animations and a big cursor. Choices are saved on your device.
DATES THAT MEAN SOMETHING: every page now shows the day its content last changed, and every scan date matches the data beside it.
FEEDS AND LAUNCH CHECKS: new RSS feeds for every article, our videos as a podcast with transcripts, and each Scout Radar scan, plus a video sitemap. The homepage now has a Scout Radar section after Meet Scout Agent. Every page now names its publisher, its last update and its place in the site in structured data, and the footer page count matches the sitemap again.
THREE SHORT VIDEOS: a one-minute introduction on the homepage after How it works, a Meet Scout video on Scout's author page and a Scout Radar explainer on the radar page. Our own player matches the site, and nothing downloads until you press play.
SCOUT RADAR AND AN HONEST POSITION: we retired our old size claim, because a network radar sees traffic that no public site can see. The new Scout Radar page shows the other side: what the top 10,000 websites decide about AI crawlers in robots.txt, the splits, the common mistakes and the new signals, with the aggregate data free to download.
A BEFORE AND AFTER CASE STUDY: a new page, linked from the footer and the case-study hub, puts real v6.2 and v6.3 screenshots side by side with drag sliders and explains the method, the SEO and GEO safeguards, the mascot and the trade-offs.
Scout's stage and the homepage detail pass
FIVE NEW POSES: Scout now inspects with a magnifier, reads a page, thinks it over, gives a wink and a thumbs-up, and cheers with confetti. Each pose matches a step of the research he does.
A STAGE OF HIS OWN: the Meet Scout section is a spotlight stage. As you read the four research steps, Scout acts out each one and says what he is doing. It pauses when you hover, tap or tab in, and stays still if you prefer reduced motion.
ONE COLUMN, ONE RHYTHM: every chapter below the hero uses the same width, larger headings and more air between sections. The hero is unchanged.
A CLEARER DATABASE: the bot groups and the wider research now sit in two labelled rows under the summary, so the counts read as a breakdown instead of a repeat.
CALMER NUMBERS AND COLOURS: numbered markers are toned down, counts use one colour (scrapers stay coral), and the stat lines in How it works sit on one line.
FLOATING BUTTONS WITH MANNERS: the contents, back to top and Scout buttons fade when they would cover the check form, the newsletter or the footer, and step aside while you scroll down on narrow screens.
FOOTER FIXES: the verification note keeps its own line, count badges stay beside their labels, and the small stats line up.
COVERS YOU CAN READ: the brand line and the site address on every article cover are larger, so they stay legible on blog cards. The bot roster cover now shows its real count, and bot profile covers only show a User-agent when it is in our database.
NO LOST WORDS: a new release check compares the homepage with the previous version and fails if the hero changes or any sentence disappears.
ONE BRAND IN THE HEADER: the header shows AI Crawler Check on its own. Horatos.ai stays credited in the footer.
A CHANGELOG THAT MATCHES THE SITE: release cards, version and tag pills, headings and list spacing now use the same type sizes and rhythm as the rest of AI Crawler Check, and every bullet dot lines up with its first line.
CASE STUDY TABLES REBUILT: numbers line up under their headers, the header band runs the full width, and on phones each row becomes a labelled card instead of a squeezed table. Status pills hug their text.
LINKS THAT SAY WHERE THEY GO: links back to the checker from articles and guides now describe what you can test there (robots.txt rules, AI bot access, AI search visibility) instead of repeating one phrase.
Evidence and bot data
EIGHT NEW BOTS, 265 IN TOTAL: on 19 October we compared Cloudflare Radar's bot directory (871 entries) with our database and kept only bots whose operator documents the token on its own page. New: Cloudflare-AI-Search and Cloudflare-AI-Search-External, CloudflareBrowserRenderingCrawler (Browser Run /crawl), KeenableBot and Keenable-User, Channel3Bot, Lyrenth's AIWebIndex-Agent and Naver's search crawler Yeti. All eight are Official.
UPDATED FROM VENDOR CHANGES: Cloudflare renamed its AI Search crawler from Cloudflare-AutoRAG on 9 February and still honours the old token, so Cloudflare-AutoRAG moved from Reported to Official and its profile now says to name both. Google's 17 September note that Mediapartners-Google covers AdSense and Ad Manager is on that profile too. Every Official token was re-fetched from its operator page on 19 October: 193 of 193 found.
OUR OWN EVIDENCE FOR EVERY UNDOCUMENTED BOT: on 17 October we fetched robots.txt from the Tranco top 10,000 (5,254 valid files) and counted how many sites write a rule for each Reported token. Those profiles now say Verified by AI Crawler Check with the count, instead of citing other bot lists. BLEXBot is backed by Cloudflare Radar. The method and the full table are on the methodology page.
IBOUBOT IS NOW OFFICIAL: Ibou published its own crawler page, so IbouBot moved from Reported to Official: a search crawler that follows robots.txt and does not train AI models.
VENDOR DOCS RE-CHECKED: OpenAI moved its crawler page to developers.openai.com, and every OpenAI profile now links there. Anthropic publishes IP ranges at claude.com/crawling/bots.json, now on the ClaudeBot, Claude-SearchBot and Claude-User profiles. Mozilla Tabstack's docs link was updated. All 184 Official tokens confirmed on their operator pages.
NEW GUIDE:Cloudflare's new AI bot defaults: Search, Agent and Training, why a Training block can stop Googlebot, the new use= content signal, and how the top 10,000 sites are set up.
Accessibility (WCAG 2.2 AA)
NEW ACCESSIBILITY STATEMENT: the accessibility page says what we aim for (WCAG 2.2 AA), how every release is tested, what is not perfect yet, and how to get help within five working days. It is linked from every footer.
FOCUS NEVER HIDDEN BY THE COOKIE BANNER: while the banner is open, the page reserves its height, so keyboard focus scrolls clear of it and the last content stays reachable (WCAG 2.4.11).
LINKS DO NOT RELY ON COLOUR: links inside sentences, breadcrumbs, FAQ answers and policy pages are now underlined (WCAG 1.4.1). The GEO audit URL box shows a clear focus ring, the header logo's spoken name matches its visible text, and dimmed table-of-contents items now meet contrast.
A NEW RELEASE GATE: every release now runs axe-core for WCAG 2.0, 2.1 and 2.2 AA, plus a real keyboard walk on one page of every template: skip link, visible focus, focus not obscured, 24px targets, reduced motion, and Scout opening with Enter and closing with Escape.
SELF-TESTS WITHOUT JAVASCRIPT: questions open with a tap to show the answer. They are plain HTML, so they work for every reader and every crawler. Nothing is scored or stored.
1,500+ WORDS ON EVERY PAGE: all 617 pages now pass a word floor, measured on the HTML the server sends. Bot, operator, category and comparison pages gained sections built from our bot database and the top-10k study, and tool and policy pages gained task guides.
CASE STUDIES AND COMPARISONS:case studies show their charts as accessible HTML, and the comparison hub adds honest pages on Google Search Console, Screaming Frog and TechnicalSEO.com.
SCOUT AS AUTHOR: articles carry a Scout author box that links to his author page, which explains how articles are researched and checked.
Links, security and quality gates
FEWER, BETTER EXTERNAL LINKS: external links open in a new tab with nofollow, and no page lets them pass 15% of its links. The deepest primary source is kept; the link text always stays.
STRICTER SECURITY HEADERS: pages now isolate their browsing context, block cross-domain policy files and narrow the browser features a page may use. Share pop-ups open without access back to this page.
NEW RELEASE GATES: every release now checks all 617 pages without JavaScript (landmarks, one h1, no skipped headings, word floor), the security headers, every button and form, and the external link ceiling.
v6.2Personal AI Agents
Personal AI Agents: Muse, Dots and Grok Bot, Plus a New Log Checker
Meta Muse, OpenAI dots and xAI Grok Bot all launched in the last two months. They are personal agents with their own cloud browsers, not crawlers, so robots.txt is the wrong control for them. This release adds them to the database where a token exists, a tool to find them in your logs, and three guides built from the makers' own documents.
Database
NINE NEW BOTS, 257 IN TOTAL: checked against Cloudflare Radar's verified-bot registry on 11 October. Six are Official, with the operator's own page naming the token: Reflectionbot, KernelSearchBot, alphalens-bot, Anomura, HarkBot and ZipchatBot. Three are Reported from Radar: GrokAgent (Grok Bot), muse- (Meta Muse) and Meta-ExternalTest.
AGENT PROFILES WRITTEN BY HAND: the generic template called agents "on-demand fetchers" that robots.txt can control. The GrokAgent, Muse and Meta-ExternalTest profiles now say what the sources support: who triggers the visit, how to verify it, and why a firewall or bot-management rule is the real control.
GROK CORRECTION: our Grok guides said Grok Bot sends no xAI name. Cloudflare Radar now lists it as GrokAgent with signed requests, so four Grok pages were updated. xAI's own docs still do not print the string, so it stays Reported.
New tool and guides
AI AGENT LOG CHECKER: the new tool finds Grok Bot, Meta Muse, Hark, Google-Agent and the user-triggered fetchers in pasted log lines, grouped by IP, with an evidence badge on each. It runs in your browser and uploads nothing.
NO ARTICLE UNDER 1,000 WORDS: seven short strategy posts gained worked examples, templates and a full 50-term glossary. Article schema now reports the word count of the page as rendered.
Crawling, security and design
BLOG PAGINATION IS NOW CRAWLABLE: page two onward lived at /blog?page=N, which our own robots.txt rule Disallow: /*?* blocked. Pages now live at /blog/page/N, the old URLs 301 there, and filtered views stay noindex.
SAFER URL FETCHING: every tool that fetches a URL now checks the resolved IP of each redirect hop, which closes DNS-rebinding and redirect routes to private networks.
CONTRAST AND SEMANTICS: the code-block Copy button was 1.35:1 on light blocks and now passes AA easily. The semantic and no-JavaScript audits now also cover the agent tools, a paginated blog page and an agent profile.
REAL ARTICLE DATES: some articles and bot profiles showed dates from 2025, before this site existed. An old generator had made them up. Every publish date now comes from the day the article was first committed, so none is earlier than launch on 17 March 2026. Every article now shows its October 2026 review date, and the sitemap uses the same dates. A new release check fails if any date is earlier than launch, older than the last review, or out of step with the sitemap.
FOOTER DATE FOLLOWS THE BOT DATABASE: the footer line "Bot intelligence last verified" was typed by hand and fell behind the 19 October release. It is now generated from the bot database release date, so the two always match. Bot profile "updated" dates and their sitemap entries use the same date.
SCOUT'S AUTHOR PAGE: Scout Agent's page has a new profile design: a profile summary, a step-by-step research process, a sources policy, and the latest articles with their covers. Scout's official LinkedIn profile is linked on the page and in every article's author box. It is also listed as sameAs in the author structured data, so search engines can tell the byline and the profile are the same entity.
OUR OWN CONTACT DETAILS: every page, policy, schema and text file now uses AI Crawler Check addresses: hello@aicrawlercheck.com for general enquiries, privacy, security and accessibility, and sponsor@aicrawlercheck.com for sponsorship. security.txt points to hello@aicrawlercheck.com. The company LinkedIn page is in the footer, on the contact page and in the organisation structured data. Phone, WhatsApp and the office address are unchanged.
SCOUT BELONGS TO AI CRAWLER CHECK: Scout Agent's page, bylines, case studies and structured data now say Scout was created by AI Crawler Check and is reviewed by the AI Crawler Check editorial team. Horatos.ai is named as the agency that runs the site.
NEW CONTACT PAGE AND FORM: the contact page was redesigned around Scout, who reacts to the topic you pick. It has a working message form with field-by-field validation; the form also works with JavaScript off. A new sponsorship section offers sponsored posts and link insertion, and every placement is labelled and marked rel="sponsored". Sponsorship messages go to sponsor@, everything else to hello@.
EVERGREEN URLS: no page address carries a year, a month or a changing count any more. Eighteen articles and bot profiles moved to permanent addresses (for example /blog/ai-bots-you-should-know), every old address redirects in one step, and read counts carried over.
SCHEMA GENERATOR REBUILT WITH SCOUT: the schema markup generator now covers 13 types, marks the fields Google requires and recommends, shows each type's real Google status (FAQ and HowTo no longer earn rich results), has a working sample for every type and a schema plan for 12 kinds of business. Sources and the review date are on the page.
GEO AUDIT: SAMPLE REPORT AND INDUSTRY GUIDE: the GEO audit shows a full sample report, scored live by the real engine, before you run one, plus which pages to audit first for 8 kinds of site and how the scoring is kept honest.
v6.1Verified Database
The Evidence-Tiered Database: Official and Reported Bots, Clearly Labelled
, updated
Every one of the 265 bots now carries an evidence badge that says where its token came from. Official docs means the operator's own crawler page names the token; the profile links to that exact page and it was fetched and confirmed on the release date. Reported means no operator page exists but independent evidence shows the token is in use (since v6.3: Cloudflare Radar's verified bots or our own top-10k robots.txt study); the profile says so in plain words, shows the evidence, and tells you to treat the rule as best effort. A token in neither group is not listed, and the build fails if one slips in.
Why this release exists. Crawler lists across the industry, including earlier versions of this one, were built by accumulation: a token appeared in a server log or a community block list, it was added, and it was never checked. Some of those tokens are real crawlers whose operators simply never wrote a page about them; others are bare product names that were never user-agent strings at all. Deleting everything the vendor did not document would have thrown away real bots (Claude-Web still appears in logs; GrokBot and DeepSeekBot are real traffic). Keeping everything unlabelled would have kept presenting rumours as facts. The answer is to keep what a credible source can vouch for and label the difference on every card, every profile and every API response. Being right includes being clear about how sure we are.
Research update (September 27)
2025 TO 2026 MARKET DATA ON 24 PAGES: 13 guides, 4 tools, 6 bot profiles and the About page gained sourced sections from Cloudflare, Hostinger, BuzzStream, Google, Anthropic, Meta, Exa and Semrush, with comparison tables and charts.
NEW TOOL FEATURES: an "Allow AI Search, Block Training" preset and an optional Content-Signal line in the robots.txt generator, five sample files in the validator, sample site lists in the batch checker, and worked examples in the schema generator.
GEO AUDIT REBUILT: the GEO audit now runs 42 evidence-based checks instead of 58. Robots.txt is evaluated per crawler with RFC 9309 rules, a firewall probe compares AI crawler user agents with a browser, soft-404 pages no longer count as a valid sitemap or llms.txt, article-only checks are skipped on other page types, and points for having a particular plugin installed were removed. Blocking findings cap the score, every fix shows the points it adds, and the PDF and Excel reports were redesigned.
CORRECTION: the Cohere guide no longer says cohere-ai collects training data. Cohere's crawler policy says it runs no training crawler today, and the ExaBot profile now points to Exa's documented ExaSearchBot.
The verification
177 OFFICIAL TOKENS, EACH WITH A PRIMARY SOURCE: OpenAI's bot page names GPTBot, ChatGPT-User, OAI-SearchBot and OAI-AdsBot. Anthropic's support article names ClaudeBot, Claude-User and Claude-SearchBot. Google's three crawler pages (common, special-case, user-triggered) account for every Google token. Yandex's official robot list contributed 38 tokens, Baidu's help page 7, Bing's crawler page 5. Meta, Apple, Amazon, Mistral, Perplexity, Moonshot, DuckDuckGo, Common Crawl, the Internet Archive, Kagi, LINER, SB Intuitions, Exa, Klaviyo, ByteDance's Toutiao Search team and the SEO tool vendors each publish a page that names their tokens, and every profile links to it.
71 REPORTED TOKENS, EACH WITH A NAMED PUBLIC SOURCE: Anthropic's legacy Claude-Web and anthropic-ai; xAI's GrokBot and xAI-SearchBot; DeepSeekBot, QwenBot, DoubaoBot, ERNIEBot, PanguBot; Cohere's two tokens; Google's Gemini-Deep-Research, Google-Gemini-CLI, GoogleAgent-Mariner and the superseded Google-NotebookLM; the AI coding agents Cursor, Devin, Trae and opencode; and the long tail of scrapers. Each profile shows an amber Reported badge, states that the operator has not documented the bot, and links the registry entry that names it.
CHATGPT AGENT, EXPLAINED PROPERLY: OpenAI's help centre article on allowlisting the ChatGPT Work Cloud browser confirms that ChatGPT Agent is the bot name Akamai, HUMAN and Vercel use for that traffic, and that Cloudflare labels the same traffic ChatGPT Operator (detection ID 129220581). Both are now Official. The profiles explain that these are CDN bot-manager names, not robots.txt tokens: the traffic is verified by RFC 9421 Web Bot Auth signatures with public keys at chatgpt.com/.well-known, and the robots.txt token for on-demand ChatGPT fetches remains ChatGPT-User. This resolves a confusion that the first draft of this release made worse by calling the token fictional.
EVERY PROFILE LINKS TO THE EXACT DOCUMENT: The "Official docs" link on each Official profile points at the specific page that names the token, not the operator's homepage. Reported profiles show "Official docs: none published" and a "Named by" link to the registry entry instead. The SoftwareApplication structured data carries the same URL.
38 STRINGS DELIBERATELY EXCLUDED, WITH REASONS: Dead Yahoo spellings (YahooSeeker, Yahoo-MMCrawler, Yahoo-Blogs, Yahoo-FeedSeeker), Teoma, Ai2Bot-Dolma (AI2's page names only AI2Bot), mis-spellings (LINER Bot, CloudVertexBot, SemrushBotSwa), bare company or product names (OpenAI, Operator, Grok, Firecrawl, Turnitin, Brandwatch) and tokens with no live public source at all. Their old URLs 301 to the category page. The list, with a reason for each, ships in the source data so the decision is auditable.
A RETRACTION, IN WRITING: This release was drafted three times on the same day. The first draft was a 222-bot expansion built partly from a community list without checking sources. The second cut the database to 105 official-only tokens and, in doing so, wrongly removed eight documented bots (Slurp, Bytespider, ExaSearchBot, KlaviyoAIBot, SBIntuitionsBot, Arquivo-web-crawler, Cotoyogi, Webzio-Extended) and called ChatGPT Agent undocumented. Neither draft was deployed. This third version is what shipped, and both mistakes are recorded here because the whole value of this tool is that its claims can be trusted, and that includes claims about its own history.
FILTER BY EVIDENCE: Every category page has All / Official docs / Reported only chips, the directory hub explains the two tiers at /directory#evidence, and /api/bots returns evidence and reportedSource for every token so integrators can apply their own threshold.
STALE-TOKEN DETECTION, REWORDED: The robots.txt validator still flags legacy and never-a-token spellings (anthropic-ai, claude-web, google-notebooklm, notebooklm, chatgpt operator, chatgpt agent, operator, teoma), but the advice now reads "add the current token too" for legacy spellings that registries still carry, and "this is a CDN name, use your bot manager" for the ChatGPT Agent family.
One number, every page
265 EVERYWHERE, VERIFIED BY THE BUILD: The homepage hero, search placeholder, evidence strip, category tiles and comparison table; the directory badge; the robots.txt generator, validator, batch checker, llms.txt generator and schema generator; the compare pages; the footer; the WebApplication and FAQ structured data; and the meta descriptions that AI engines quote all read 265 from a single source file. The facts gate scanned 617 pages of source and rendered HTML and reports zero contradicting claims. The share card and every favicon were regenerated with a new cache-busting token.
DIRECTORY REBUILT: 262 bot profile pages and 112 operator pages remain, each for a verified crawler. 120 profile pages and 51 operator pages were removed and 301 to their category or the operators index. The sitemap now lists 617 URLs, every one confirmed to return 200. The directory generator is category-aware, skips redirected slugs, and sets each page's documentation link from the vendor map.
THE ROSTER ARTICLE IS GENERATED FROM THE DATABASE:265 AI Bots You Should Know lists every token by category with its operator, purpose and safety rating, links each row to its profile, and explains which well-known tokens are deliberately absent and why. It is produced by a build script, so it cannot disagree with the checker. The 163, 196 and 222 slugs all redirect to it in one hop. The same pass fixed a templating defect that had left two Bot Profiles articles describing their subject as "operated by Unknown".
BLOG SORTING AND READABILITY: "Latest published" and "Recently updated" now tie-break deterministically, and a regenerated article is dated as newly published rather than as an edit. Article body copy moved to 17px with a 1.75 line height, slightly wider word spacing and more room between paragraphs and headings; case study pages follow the same rules.
SECURITY: Hono upgraded from 4.12 to 4.13.8, closing 30 published advisories including JSX escaping, serveStatic path handling, the query parser and CORS. The homepage probe re-runs the SSRF guard on the final URL after following redirects, so a public site cannot bounce the checker to a private address.
Every scan now includes a Googlebot crawl test
THE EXACT URL, AS GOOGLEBOT SMARTPHONE: The homepage scan now runs a Googlebot crawlability test on the page you enter, not just the domain. It evaluates robots.txt for Googlebot, Googlebot-Image, Bingbot and Applebot on that path, then checks the HTTP status, challenge pages, noindex, the canonical and how much text is in the HTML before JavaScript.
NO FALSE "NOT CRAWLABLE": Many CDNs block look-alike Googlebots and let the real one through. When our Googlebot request is refused but a normal browser gets the page, the report says "Probably crawlable", marks those checks Unverified and reads the page from the browser response. Our requests come from our server, not Google's IP ranges, and the report says so.
New tool: AI Crawler Diagnostic
ONE PAGE, TWENTY CRAWLERS, SIX GATES: The free AI Crawler Diagnostic requests a single URL as 16 AI crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Meta-ExternalAgent, Amazonbot, Bytespider, CCBot, DuckAssistBot, MistralAI-User, cohere-ai and YouBot) plus Googlebot and Bingbot, and checks the Google-Extended and Applebot-Extended opt-out tokens. For each one it reports robots.txt, page directives, HTTP status, the redirect chain, firewall challenge pages and whether the content is readable without JavaScript, then ranks the fixes by how many crawlers each one unblocks.When the homepage scan says a bot is blocked, this says why, and which line to change.
NEVER CALLS A GUESS A BLOCK: Every URL is also requested as a normal browser. If the site refuses the browser too, it is filtering our server, not the crawler, and the result is marked Unverifiable. If only the crawler is refused, the verdict is Firewall block: verify, because a real block and fake-bot protection (which lets the genuine crawler through from its own IP range) look identical from outside the operator's network. Neither is ever counted as reachable, and neither is reported as a confirmed block.An earlier draft reported sites that block datacenter traffic as blocking every AI crawler. It was caught in testing before release.
ROBOTS.TXT READ THE WAY CRAWLERS READ IT: RFC 9309 group selection (a named group replaces the * group entirely), longest-match precedence with * and $, query strings included. A robots.txt that answers 5xx or 429 blocks the whole site, a 404 means no restrictions, and when a page redirects to another host that host's robots.txt is checked as well.
DIRECTIVES BY CRAWLER PURPOSE:noindex blocks search engines and AI search crawlers. It is reported as a note, not a block, for user-triggered fetchers such as ChatGPT-User and for training crawlers, which follow robots.txt instead. Directives inside scripts and comments are ignored.
FREE, NO SIGN-UP, AND SAFE TO POINT AT ANYTHING PUBLIC: Private, loopback, link-local and cloud-metadata addresses are refused on the first request and on every redirect hop, including IPv6 forms. Each run is capped at about 25 seconds and is rate limited per visitor, because every run sends about 20 requests to the site being tested.
Directory: search, filters and one count
SEARCH AND FILTER EVERY CATEGORY: Each directory category page now has a search box, sorting (A to Z, latest added, operator, risk), operator, purpose, safety and evidence filters, one-click chips for the common providers, and filter URLs you can share. Every card stays in the page HTML, so crawlers and visitors without JavaScript still see the full list.
248 MEANS 248 ON EVERY PAGE: The footer's "All AI Crawlers" badge said 262 beside a site-wide claim of 265. Both were true: 265 user-agent tokens are tracked, on 262 profile pages, because three unversioned tokens (ChatGPT-User, MistralAI-User, Perplexity-User) share a profile with their versioned twin. Every total now states tokens, and category pages that differ say both, for example "104 user-agent tokens on 85 profiles". A new build check reads the rendered text of every page in the sitemap and fails on any bot count that disagrees.
CLEANER PROFILES: Markdown asterisks that showed up as literal ** in bot profile copy now render as emphasis on all 262 profiles, and are stripped from the structured data.
Search visibility work
CUT-OFF DESCRIPTIONS FINISHED: 110 articles had a meta description that stopped mid-word (one ended "Includes ste"), repeated in the social and structured-data copies. Each now ends on a complete sentence in all four places, and the build refuses a description that does not.
ONE HEADLINE PER PAGE: The structured-data headline now matches the visible title on every page, so an AI answer and a search result quote the same string.
QUESTIONS ANSWERED WHERE THEY ARE ASKED: 68 real search queries that the site was shown for but did not answer, such as "how do I monitor AI bot traffic to my website?", "robots.txt disallow all" and "what is PerplexityBot?", now have a heading that uses the exact wording and a direct answer underneath, across 33 pages.
34 NEW INTERNAL LINKS: Including four guides that no other page linked to, now reachable from the articles readers arrive on.
SEVEN THIN ARTICLES MERGED: Seven posts that had stopped earning any search traffic now redirect to the stronger guide on the same topic, so readers land on one complete answer: Applebot vs Applebot-Extended, How to check if ChatGPT can read your site, Agentic AI browsing, robots.txt for WordPress, The complete robots.txt guide, GDPR and AI crawlers, and llms.txt examples. The blog now has 159 articles.
22 ARTICLES UPDATED, AND DATED: Every article whose content changed in this release shows "Updated September 23, 2026", carries the new date in its structured data and sitemap entry, and moves up the blog's "Recently updated" sort. Articles that only gained a related-post card keep their original date.
CASE STUDY LINKS: The case study now links to the guides and bot profiles it references (16 links in English, 10 in Vietnamese) and ends with a short "Keep exploring" block. The downloadable PDF is unchanged.
EASIER READING: Long-form text on articles, the case study, docs, profiles, the changelog and the legal pages is slightly larger with more line spacing.
SITEMAP CHECKED BOTH WAYS: The sitemap lists 617 URLs. The build now confirms every listed URL returns 200 and names itself as canonical, and that no linked, indexable page is missing from it.
Case studies and a named diagnosis
CASE STUDY, TWO LANGUAGES:The Invisible AI Crawler Block is a full technical investigation into a five-star hotel whose WordPress origin returns an infinite 301 loop to any request carrying the X-Forwarded-Proto header that Cloudflare and every other CDN adds. The site worked perfectly in a browser and was invisible to every AI crawler, and to this checker, at once. The English and Vietnamese editions and the downloadable PDFs are rendered from the same source file, so the page, the PDF and the post cannot drift apart. Paired with hreflang, TechArticle structured data, a feature image, the Share-to-AI toolbar and a dedicated header menu item.
HONEST FETCH FAILURES: The checker now separates DNS failure, connection refusal, TLS errors, timeouts, redirect loops and hard HTTP blocks instead of collapsing them into one message. When it detects the redirect loop it names the trigger header, hands over the two curl commands that reproduce the difference, and says plainly that the site works in an ordinary browser. Retries stop the moment a loop or DNS failure is confirmed, so a diagnosis that used to time out after sixty seconds now returns in under three.
EDGE EMULATION FOR LOCAL TESTING: The failure could not be reproduced from a developer machine because the header only exists behind a CDN. The checker's fetch layer can now emulate the edge headers locally, which is how the diagnosis was tested before it shipped.
TWO MORE RETRACTIONS, IN WRITING: The investigation produced two confident wrong diagnoses before the right one. Both are documented in the case study, because diagnostic authority comes from being corrected by measurement, not from never being wrong.
v6.0Major Release
The 2026 Accuracy Release: Honest Scoring, Bot Intelligence & New Builder Tools
Our biggest accuracy and honesty upgrade. We recalibrated the entire AI Visibility Score against what vendors actually document in 2026, added a live bot-intelligence layer that flags deprecated tokens and stealth crawling, and shipped four new free tools. Every article published at the time was re-verified for 2026 accuracy.
Why this release exists. Through 2025 most AI visibility tooling, including ours, quietly inherited assumptions that stopped being true. Scores rewarded publishing an llms.txt file as though it were a confirmed ranking input. Checkers kept treating anthropic-ai as a live crawler years after Anthropic replaced it. Opt-out preference tokens were counted as bots. The result was a familiar failure mode: a site scores well, the owner relaxes, and ChatGPT still cannot read a single page. v6.0 is a deliberate correction. Where the evidence is strong we score it heavily; where it is a reasonable hedge we say so and weight it lightly; and where nothing can be measured from the outside we tell you that rather than inventing a number.
Scoring, rebuilt from the evidence
SCORE v6 (70/15/15): Bot Access now carries 70 of 100 points, AI Infrastructure an honest 15, and a brand-new Technical Readiness category the remaining 15 (sitemap 6, structured data 6, HTTPS 3).
The reasoning: crawler access is binary and verifiable. If GPTBot is disallowed, your content cannot be cited, and no amount of schema markup compensates. So access dominates the score. The previous weighting let a site with excellent AI files but a blocked GPTBot outrank a plain site that was fully readable, which is exactly backwards. Every weight, and the argument behind it, is now documented on the methodology page.
LLMS.TXT DEMOTED, DELIBERATELY: Publishing llms.txt used to be worth far more. It is now a small part of a 15-point category.
No major AI vendor has confirmed llms.txt as a retrieval or ranking input. It remains a sensible, low-cost hedge and we still check it strictly, but inflating its weight produced falsely reassuring scores. We would rather lose the neat narrative than overstate the evidence.
WEIGHTED BY REAL IMPACT: Blocking GPTBot, ClaudeBot, PerplexityBot or Googlebot costs materially more than blocking a minor scraper, and the score reflects that instead of counting every user-agent equally.
2026 bot intelligence
DEPRECATED TOKEN DETECTION: We flag retired user-agents such as anthropic-ai and claude-web that are still sitting in robots.txt files.
This is the most common silent failure we see. An operator renames its crawler, the old rule keeps matching nothing, and the site owner believes they are still governing a bot that no longer exists under that name. Worse, the intended policy is not being applied at all. We name the stale token and give you the current replacement.
OPT-OUT TOKENS ARE NOT CRAWLERS:Google-Extended and Applebot-Extended are now correctly labelled as training-preference signals rather than fetchers.
Neither ever requests a page. Disallowing Google-Extended opts you out of Gemini model training but does not remove you from AI Overviews, and blocking it will not reduce your crawl traffic by a single request. Tools that list it as a bot lead people to the wrong conclusion about what they just changed.
FETCHERS THAT IGNORE ROBOTS.TXT: Some user-triggered fetchers do not consult robots.txt at all, and we now say which, instead of implying a rule will stop them.
When a person pastes your URL into an assistant, that request is often treated as user-initiated rather than crawling. A robots.txt rule is the wrong tool there; edge or WAF rules are. Telling you a disallow line works when it does not is worse than saying nothing.
STEALTH CRAWLING FINDINGS: Where independent research has documented undeclared or misrepresented crawling, most prominently the Perplexity findings, the report surfaces it so your policy accounts for traffic that may not honour your rules.
AGENTIC BROWSERS, HONESTLY UNDETECTABLE:ChatGPT Atlas, Comet and Copilot Actions drive a real browser session on a user's behalf.
They largely present as ordinary browser traffic, so no robots.txt audit, ours included, can confirm or deny their access. We list them as a known blind spot rather than scoring a category we cannot measure. Naming the limit is more useful than a confident guess.
HONEST WAF HANDLING: If a firewall blocks our probe we report exactly that, across 15+ recognised providers, instead of recording a passing grade we did not verify.
Four new free tools
LLMS.TXT GENERATOR: The llms.txt builder gives you live preview, a correctly formed Optional section, and starting templates for SaaS, blogs, e-commerce and agencies. Most hand-written llms.txt files we encounter are malformed in ways that make them useless to parse, so the generator enforces valid structure as you type.
SCHEMA GENERATOR: The JSON-LD generator covers 8 types: Organization, LocalBusiness, Article, FAQPage, Product, WebSite, BreadcrumbList and HowTo. Structured data is one of the few levers that demonstrably helps machines read entities and relationships correctly.
BOT COMPARISONS: The /compare hub ships 20 side-by-side matchups such as GPTBot vs ClaudeBot, each with user-agent strings, safety rating, SEO impact and copy-paste blocking rules, for the common case of allowing one operator while blocking another.
SCORE BADGE: After any check you can embed a live SVG score badge in a site footer or README, with one-click HTML and Markdown embed codes.
Standards and content
WEB BOT AUTH: We now detect the IETF Web Bot Auth directory at /.well-known/http-message-signatures-directory.
This is the emerging answer to a real problem: user-agent strings are trivially spoofed, so allow-listing by name has never been trustworthy. Cryptographic signatures let a crawler prove which operator it belongs to. Adoption is early, which is precisely why it is worth knowing whether the operators reaching you support it.
EVERY ARTICLE RE-VERIFIED: Every guide published at the time was reread against 2026 reality, not spot-checked. That means honest llms.txt framing, deprecated-token warnings, corrected user-triggered fetcher behaviour, the Perplexity compliance findings and recalibrated score references throughout.
Advice that was accurate in 2024 is now actively misleading in places, and stale guidance from a tool that claims to measure accuracy is a contradiction we were not willing to ship.
ONE BOT COUNT, ONE SOURCE OF TRUTH: The homepage checker, directory, validator, generator and batch checker all read the same bot database, now including agentic entries like Google-Agent (Project Mariner) and OAI-AdsBot. Today that database tracks 265 bots across 8 categories.
Verified numbers, documentation and tool comparisons
A tool whose pitch is honest measurement had been mis-stating its own figures: different pages claimed different bot totals, and the downloadable copy of the bot database had drifted to 154 entries while the real count was 265. Editing the text would have fixed nothing, because the cause was that any number could be typed by hand anywhere. The numbers now come from one source and a build gate refuses to ship the site when a claim disagrees with the database.
One source of truth for every figure.Bot counts, page counts, category totals and the batch limit all come from one file, and the build fails if any page disagrees with it.
The downloadable bot database is generated, not maintained by hand.It is rebuilt from the live database on every build, so it cannot fall behind again.
New documentation section.Step-by-step guides for running a scan, reading your score, fixing a blocked crawler, and auditing client sites.
New comparison section.How this tool compares with the alternatives, including the rows where the other tool wins and who should choose it instead.
Better crawlability and accessibility.Corrected robots.txt rules, redirects for renamed pages, a real 404 page, AA link contrast and semantic landmarks throughout.
Safer outbound links.External links no longer leak referrer data or expose the originating tab.
Mobile, rebuilt where it was actually broken
Most of this release was about being honest with numbers. This part is about being honest that the site was noticeably worse on a phone than on a desktop, which is where a majority of visitors read it. Every fix below was measured at real handset widths rather than judged by eye in a resized browser window, and desktop layout is deliberately untouched.
THE MENU NOW OPENS AND CLOSES PROPERLY: The mobile navigation was rebuilt as a slide-in panel with every tool reachable in one tap.
The old panel could leave the page unable to scroll after it closed, which reads to a visitor as a frozen site rather than a stuck menu. It also added invisible width to every page, so the whole site could be dragged sideways a little on every screen.
THE MENU BUTTON IS VISIBLE ON EVERY PAGE: On the GEO audit tool the button was there and worked, but rendered as a completely blank square.
The three lines of a menu button come from an icon font. That page was not loading the icon font at all, so all of its icons were invisible while the markup around them was correct. Nothing errored and nothing looked broken in the code, which is why it survived so long. The build now refuses to ship a page that mounts the navigation without the icons it needs.
TAPPING A TOOL LANDS ON THE TOOL: Choosing a destination from the mobile menu used to land part-way into the section, with the heading scrolled up out of view.
THE FIRST SCREEN IS THE FIRST SCREEN: On a phone the opening section now fills the screen exactly, instead of leaving a strip of the next section showing underneath.
The first thing a visitor read below the main call to action was the start of an unrelated headline further down the page. It now tracks the live viewport, so it stays correct when the browser's address bar slides away.
READABLE AT 320px: Text, tap targets and the comparison tables were checked at the narrowest phone widths still in use, not just at modern handset sizes.
Every page is now measurable, and privacy copy is tighter
ANALYTICS COVERAGE ON EVERY PAGE: Two pages, the GEO audit tool and the not-found page, were loading no analytics at all.
This is the quietest kind of fault. There is no error, nothing looks wrong, and the reporting does not warn you: it simply shows fewer pages, which is indistinguishable from those pages having no visitors. A release check now verifies that all 617 listed pages plus the unlisted tool and error pages carry analytics and the cookie banner, and separately confirms in a real browser that the tag stays silent until consent is given and genuinely starts once it is. Both error pages are included, because a not-found page is exactly the one you want in reporting: every visit to it is a broken link worth fixing.
CONSENT STILL COMES FIRST: No analytics cookie is set and no tracking script we load runs until you accept. Rejecting also clears anything previously set, and the Global Privacy Control browser signal is honoured without needing a click.
One exception is now written into the cookie policy rather than left unsaid: our hosting provider adds a cookieless visitor counter to every page after our code has run, so the banner cannot switch it off. It stores nothing on your device and cannot recognise you on a later visit or on any other site. It is disclosed in full, including how to block it.
FEWER TECHNICAL DETAILS IN THE POLICIES: Our privacy and cookie pages still name every processor we use, which is what the disclosure is for, but no longer print internal account identifiers alongside them.
Those identifiers gave a reader nothing useful and are not required by any disclosure rule. A release check now scans published pages for credentials, tokens and internal identifiers so none can reach a visitor by accident.
Upgrade note: because the weighting changed, a score from before v6.0 is not directly comparable with one after it. Sites that were blocking a major AI crawler while publishing tidy AI files will generally see a lower, more accurate number. That drop is the fix working, not a regression. Re-run your check to get a current baseline, and see the methodology page for the full formula.
The Big Sync: One Bot Count Everywhere, Gamified Learning & a Verified Single Source of Truth
Our largest consistency and content update yet. We rebuilt the bot database from the ground up so that every single page (homepage checker, robots.txt validator, generator, batch checker, full directory, and every article) now reports the exact same bot count, read from one database. That database has since been re-verified and now tracks 265 bots. We also turned every blog article into an interactive learning experience and expanded the directory with 35 brand-new AI bot profiles.
ONE SOURCE OF TRUTH: Eliminated every count mismatch across the site. The checker database, the crawler directory (detail pages summed across 8 categories), the validator, and the marketing copy now read one shared count. We also removed 2 duplicate Meta crawler entries that were inflating the count.
35 NEW AI BOT PROFILES: Added full directory pages for the newest agents and search crawlers, including ChatGPT Agent, OpenAI Operator, Google NotebookLM, Gemini CLI, Project Mariner, AWS Bedrock Bot, Azure AI Search, Kimi, Manus, Devin, Firecrawl, Exa, Tavily, Brave Leo, and more. Each ships with user-agent strings, safety ratings, robots.txt rules, FAQ schema, and related-bot links.
GAMIFIED ARTICLES: Every guide now features a live reading-progress bar, XP-style milestone badges, an interactive knowledge-check quiz, an animated action checklist, count-up statistics, Myth vs Fact cards, one-click copy buttons on code blocks, and an auto-generated table of contents. Learning AI SEO has never been this engaging.
NEW OPERATOR PAGES: Expanded operator coverage (the directory now has 112 operator pages), adding profiles for Moonshot AI, Cognition, Exa, Tavily, Linkup, Kagi, Phind, WRTN, Brave, iAsk, AddSearch, Klaviyo, Zhipu AI, Andi, aiHit, and Microsoft Azure.
AUTHOR FIX: Corrected the author profile link across every article to point to the verified Horatos.ai profile, strengthening E-E-A-T signals for AI search citations.
SITEMAP REFRESH: The XML sitemap now includes every bot page, category, operator and article, so AI crawlers and search engines discover every new page immediately. Today it lists 617 URLs.
v4.6
Ultimate Unified Report: GEO + AI Crawler Check Combined
UNIFIED REPORT: All 30+ GEO audit checks merged directly into the main AI Crawler Check report: one domain check, one comprehensive result covering every tracked bot plus full GEO analysis
GEO SCORE: Overall GEO Score (0–100) with 5 weighted categories: Technical SEO (18%), Content Quality (22%), AI Readiness (30%), Security & Trust (15%), UX (15%)
22 AUDIT CHECKS: SSL, sitemap, canonical, hreflang, schema markup, semantic HTML, llms.txt, AI content readiness, JS blocking, E-E-A-T-A scoring, CDN detection, and more: all inline in the report
ZERO EXTRA FETCHES: GEO analysis reuses homepage HTML from Step 0, robots.txt from Step 1, and llms.txt from Step 2: only sitemap requires an additional fetch
ENHANCED EXPORTS: PDF & Excel reports now include full GEO Audit page with category scores, 22-check table, and actionable recommendations
VISUAL DASHBOARD: Conic-gradient score ring, category progress bars, status-colored check cards, and prioritized recommendation list in the unified report
v4.5
GEO Audit Tool: Standalone Mode
GEO AUDIT TOOL: Full GEO audit tool available at /geo-audit (now superseded by unified report in v4.6)
SCHEMA FIX: Trimmed sameAs links in the author schema to one verified profile
v4.4
Horatos.ai Rebrand & 163 Bots Live
REBRAND: AI Crawler Check is now developed by Horatos.ai: an AI Search Optimization (AISEO) & Generative Engine Optimization (GEO) consultancy. All author links, legal pages, schema markup, and branding updated
163 BOTS DEPLOYED: All 8 new bots from v4.3 now live in production: OAI-AdsBot, MistralAI-Index, Google-NotebookLM, Google-Read-Aloud, Google-Site-Verification, FeedFetcher-Google, GoogleProducer, Amazon-Bedrock-AgentCore
LEGAL PAGES: Privacy Policy, Terms of Use, and Cookie Policy updated to reflect Horatos.ai as operating entity
v4.3
30 Blog Posts: E-E-A-T, GEO vs SEO & AI Bot Troubleshooting
12 NEW IMAGES: 3 hero images + 9 inline diagrams covering trust pyramids, GEO/SEO frameworks, diagnostic flowcharts, and robots.txt fixes
E-E-A-T AUTHORITY: Deep-dive into Experience, Expertise, Authoritativeness & Trustworthiness signals for AI search citation. Original frameworks and expert analysis
INTERNAL LINKS: 80+ strategic internal links across 3 posts. Heavy tool CTAs, cross-referencing to crawler guides, robots.txt generator, schema markup, and audit checklist
163 BOTS: Added 9 new bots: OAI-AdsBot (ChatGPT Ads, Apr 2026), MistralAI-Index (Vibe search index), Google-NotebookLM, Google-Read-Aloud, Google-Site-Verification, FeedFetcher-Google, GoogleProducer, Amazon-Bedrock-AgentCore. Upgraded Amazonbot to Tier 2 (Nova AI training confirmed)
WAF FIX: Fixed false-positive bot blocking when target sites use Cloudflare WAF. Multi-UA fallback strategy with WAF detection
v4.2
3 New Blog Posts: Google-Agent, WebMCP & llms.txt Guide
12 NEW IMAGES: 3 hero images + 9 inline diagrams covering agentic browsing, WebMCP architecture, and llms.txt structure
TOPICS: Agentic Engine Optimization (AEO), WebMCP Tool Contracts, user-triggered-agents.json, Web Bot Auth, llms.txt templates for 4 website types
v4.1
Google-Agent, 24 Blog Posts & Microsoft Clarity
NEW BOT:Google-Agent (Project Mariner): Google’s brand-new agentic AI crawler that navigates sites, fills forms, and takes actions on behalf of users. Monitor its activity to measure how important WebMCP is for your site
155 BOTS: Bot database expanded from 154 → 155. Google-Agent added to Google Bots category with full directory page, FAQs, and structured data. Each bot gained its own SEO-optimized directory page.
E-E-A-T FOCUS: All new posts emphasize Experience, Expertise, Authoritativeness, and Trustworthiness with author bylines, expert recommendations, and data-backed analysis
CLARITY: Microsoft Clarity analytics added to all pages: heatmaps, session recordings, and user behavior tracking across homepage, blog, tools, directory, and legal pages
48 NEW IMAGES: 12 hero images + 36 inline diagrams/infographics for blog posts 13–24. Custom illustrations for each article topic
v4.0
Focused AI Bot Checker: Simplified & Faster
FOCUS: Removed homepage signals parsing (Schema, H1, Meta Description, Canonical, OG tags): tool now focuses solely on AI bot access & infrastructure
SIMPLIFIED SCORE: New 2-component scoring: Bot Access (65pts) + AI Infrastructure (35pts) = 100pts. Removed Content (25) and Technical (10) components
REMOVED: Browser Rendering API fallback, WAF/bot protection detection, homepage HTML parsing, extractSchemas(), extractMeta(), countTag(): eliminates all false-negative complaints from WAF-blocked sites
NO MORE WAF ISSUES: Since homepage is no longer scanned, sites behind Cloudflare/LiteSpeed/Wordfence WAFs get accurate results with zero workarounds needed
SMARTER RECS: Only actionable recommendations: unblock AI bots, create llms.txt/llms-full.txt, upgrade partial to full access. No more schema/meta/author suggestions
CTA: “Want a Full AI Visibility Audit?” card replaces Homepage Signals section: directs users to paid GEO audit for deeper analysis
FASTER: Removed homepage fetch + retry + Browser Rendering: checks complete ~3s faster on average
v3.9
Browser Rendering Fallback: Bypass WAF with Headless Chrome
BREAKTHROUGH: When WAF blocks homepage scan, tool now falls back to Cloudflare Browser Rendering API: headless Chromium renders the page like a real browser, bypassing CAPTCHAs and WAF challenges
RESULT: WAF-protected sites (LiteSpeed, Cloudflare, Wordfence, Sucuri) now get full homepage analysis: Schema, H1, Meta Description, Canonical, OG tags all detected correctly
3-TIER FETCH: Homepage fetch now uses three strategies: (1) Chrome UA fetch → (2) Safari UA retry → (3) Headless Browser Rendering fallback
UX: Green “Browser Rendering Used” badge shows when headless browser was needed to bypass WAF
TYPED: Added Env type bindings for CF_API_TOKEN and CF_ACCOUNT_ID environment variables
v3.8.2
Honest WAF Reporting & UX Overhaul
HONESTY: WAF-blocked signals now show ⚠ “Unable to verify” with yellow triangles instead of pretending they pass or fail: no more misleading results
UX: New 2-column layout in WAF warning: “✓ Accurate Results” vs “⚠ Unable to Verify”: users instantly see what’s reliable
NEW: Links to Google Rich Results Test & Schema.org Validator in WAF warning: guides users to verify schema manually
SMART: Recommendations now skip “Add FAQ Schema” and “Add Author signal” when WAF blocks scan: avoids recommending things the site may already have
UX: WAF-blocked signal rows show impact as “N/A” in yellow instead of misleading Critical/High labels
NEW: Auto-retry with Safari UA when initial Chrome UA triggers bot protection: bypasses simpler WAF rules
IMPROVED: Bot-protected sites now score 19/25 content (was 13/25): assumes H1, meta desc, schema exist (standard for CMS sites behind WAF)
UX: Homepage signals show yellow ⚠ warning icons for WAF-blocked items instead of misleading red ✗ marks
IMPROVED: Homepage fetch now sends full browser headers (Accept, Sec-Fetch-*, Accept-Language): reduces WAF trigger rate
v3.8
Schema Detection Fix & Bot Protection Awareness
FIXED: Schema detection now traverses @graph arrays (WordPress/RankMath/Yoast): previously only checked root-level @type
FIXED:extractSchemas() rewritten as recursive collectSchemaTypes(): handles nested @graph, mainEntity, and arrays at any depth
NEW: Bot Protection Detection: identifies WAF/CAPTCHA pages (LiteSpeed, Cloudflare, Wordfence) and flags botProtection: true in results
IMPROVED: Bot-protected sites now receive partial content credit (13/25) instead of 0: no longer penalized for CAPTCHA pages
PERF: Homepage HTML cached from STEP 0 probe: eliminated duplicate fetch, saving ~200ms per check
IMPROVED:extractMeta() now handles multi-line attributes (WordPress meta tags with newlines between properties)
v3.7
AI Files Detection Overhaul & Score Rebalance
Fixed False Positives: llms.txt and llms-full.txt detection now uses strict validation: redirect: manual prevents 301/302 homepage redirects from being counted as “Found”
Content Validation: AI files must return 200 OK with non-HTML content. Responses containing HTML (custom 404 pages, redirected homepages) are correctly rejected
Smart Redirect Follow: Follows one redirect only if the target URL ends with the same filename (e.g., shopify.com/llms.txt → www.shopify.com/llms.txt). Homepage redirects are rejected
Removed ai.txt: /ai.txt has been removed from all checks: no actual standard exists for this file. Only llms.txt and llms-full.txt (official llmstxt.org standard) are checked
Score Rebalanced: Bot Access (40), AI Infrastructure (25), Content (25), Technical (10). Removed 5pts from ai.txt, redistributed to Content quality signals
Social Preview: All pages now have OG image + Twitter Card meta tags for rich social previews when sharing on Facebook, LinkedIn, X/Twitter, Slack
v3.6
Bot Database Accuracy & Versioned Bots
154 Bots: Added 3 new versioned AI bots: ChatGPT-User/2.0, MistralAI-User-1.0, Perplexity-User-1.0: closing a gap in our AI bot coverage
Deprecated Bot Labels: Claude-Web and anthropic-ai are now marked as Deprecated per Anthropic’s official docs (replaced by Claude-User, Claude-SearchBot, ClaudeBot)
ClaudeBot Relabeled: Changed from “Claude AI (Legacy)” to “Claude AI Training” to accurately reflect its official purpose per Anthropic docs
Versioned Bot Matching: New parser logic handles versioned user-agents (e.g., ChatGPT-User/2.0 inherits ChatGPT-User rules from robots.txt). Same for -1.0 suffixes
Cross-Verified: AI bot list verified against official sources: OpenAI Bots docs, Anthropic Crawler docs, CrawlerCheck.com directory, Search Engine Journal’s 2025 list
v3.5
Partial Access & Report Naming
Partial = Accessible: Bots with partial access (e.g., /admin blocked but / allowed) now count as “accessible” in overview. Only fully BLOCKED bots are counted as denied. This prevents misleading “0% allowed” for sites that only block admin paths
Category Overview Rework: Shows “X full · Y partial” breakdown instead of just “X/Y allowed”. Percentage now reflects real accessibility
Bot Card Labels: Status now shows “FULLY ALLOWED”, “PARTIALLY ALLOWED”, or “BLOCKED” for clarity
Report Naming: PDF/Excel exports renamed from “AI Visibility Report” to “AI Crawler Check Report”: accurate to the tool’s function
Score Adjustment: PARTIAL bots now receive 75% score credit (was 50%). Reflects that partial access still means content is crawlable
v3.4
Deep Verification & Accuracy Guarantee
Double Verification: robots.txt is now fetched twice and results compared. If content changed between fetches (CDN cache, etc.), the freshest version is used
Enforced Backend Processing: Minimum 8-second server-side processing time ensures all checks genuinely complete. No shortcuts, no cached guesses
Frontend Sync: Animation steps now match real backend processing. 10+ second minimum display with step-by-step progress visible to the user
Error UX Improved: Invalid domains show immediately clear error messages. No more fake results for non-existent websites
Version Tag: Reports now include processingTime to prove thorough analysis
Critical Fix: Added mandatory domain reachability check (STEP 0). Invalid or non-existent domains (e.g., example.com.sgsg) now return a clear error instead of fake results
DNS Validation: The tool now verifies the website actually responds before analyzing. DNS failures, connection timeouts, and Cloudflare 5xx errors are properly detected
robots.txt Validation: Content is now validated to ensure it’s actually a robots.txt file (not a custom error page returning 200). Per RFC 9309: 404 = all allowed, 403 = all blocked
Minimum Display Time: Results always show for at least 6 seconds with progressive step animation to confirm thorough analysis
Error UX: Clear, actionable error messages for unreachable domains, DNS failures, and timeout scenarios
v3.2
Accuracy Fix: robots.txt Only
Critical Fix: Completely removed HTTP simulation. robots.txt is now the SOLE source of truth for bot access status. This eliminates ALL false positives caused by WAF/CDN/IP-based blocking (Cloudflare, Akamai, etc.)
Why: HTTP 403 from datacenter IPs does NOT mean the site owner blocks a bot: it's often generic bot-protection. Only robots.txt reflects the site owner's explicit intent
Partial Explained: Added tooltip explaining what "Partial" means on every bot card
Bot Purpose Labels: Every bot card shows its purpose: Training, Indexing, User Request, Both, Scraping, Monitoring, Social
Check Timing: Increased analysis animation to 14+ seconds for user trust
Directory UI Fix: Fixed broken layout in Web Crawler & Bot Directory pages
v3.1
Accuracy & UX Overhaul
Critical Fix: Resolved false-positive blocking reports caused by WAF/CDN detection (e.g., Cloudflare-protected sites returning HTTP 403 for all datacenter IPs)
Accuracy: robots.txt is now the canonical source of truth. HTTP simulation only upgrades severity, never overrides a robots.txt ALLOWED status
Bot Purpose Labels: Every bot now shows its purpose: Training, Indexing, User Request, Both (Training + User Request), Scraping, Monitoring, Social
Partial Explained: PARTIAL status now includes an inline explanation showing which paths are blocked vs allowed
New Logo: Custom SVG logo for AI Crawler Check replaces old BH branding in header
Full Navigation: Directory, Tools, Generator, Validator, Batch Checker links on every page
Check Animation: Progressive step-by-step loading with realistic timing for better user trust
Changelog: This page: track all updates and version history
