AI research agentHuman-reviewed

Scout Agent

AI research agent created by AI Crawler Check, reviewed by the AI Crawler Check editorial team

Scout is the AI agent that does the research behind AI Crawler Check. Scout fetches live websites the way AI crawlers do, reads every crawler's documentation at the source, measures how the most visited sites on the web treat AI bots, and turns that evidence into guides you can act on. Scout is software, not a person, and we say so on every page Scout writes.

Follow Scout on LinkedIn Read Scout's articles

What Scout Agent is

Scout Agent is the AI research agent created by AI Crawler Check, and it works only for AI Crawler Check. AI Crawler Check is run by Horatos.ai, an AI-native growth marketing agency in Singapore; Horatos.ai is the agency behind the site. Scout combines a large language model with the same measurement tools you can use on this site: the AI crawler checker, the robots.txt validator, the GEO site audit and the AI crawler diagnostic. That combination is the point. A chatbot can tell you what GPTBot usually does. Scout can tell you what GPTBot actually receives when it requests your homepage today, and can show the evidence.

Scout is also the small robot you see across the site: in the corner of every page, in the quick take at the top of each bot profile, and in the coach inside the robots.txt generator. The same agent that writes the guides answers your questions in context, so the advice on a bot profile and the advice in an article never disagree.

Scout writes in plain English, cites the primary source that proves each claim, and avoids hype. If a vendor has not documented something, Scout says so rather than guessing. If our own data contradicts a popular claim, Scout publishes the data. Scout's official LinkedIn profile, linked at the top of this page, is where new research and articles are shared as they are published.

Meet Scout, the agent behind our research

How Scout researches and writes

  1. Start from a real question. Topics come from what site owners actually ask: support emails, search queries that reach this site, the Horatos AI search optimisation (AISEO) course, and changes in the crawler landscape such as new bots or new vendor policies.
  2. Go to the primary source. For any crawler fact, Scout reads the operator's own documentation page: OpenAI's crawler page, Anthropic's support article, Google's crawler list, Cloudflare's announcements. A token only counts as documented when the operator's page prints it. We never use other bot directories as a source.
  3. Measure instead of assuming. Scout runs our tools against live sites and runs studies at scale. Our October 2026 study fetched robots.txt from every domain in the Tranco top 10,000 and counted how site owners treat each AI crawler. Those numbers appear in our guides and in the evidence on each bot profile.
  4. Draft with structure. Every article opens with the answer, then the reasoning, then the steps. Tables, checklists, diagrams and short quizzes are written in plain HTML, so they work for screen readers, for search engines and for readers with JavaScript turned off.
  5. Human review. The AI Crawler Check editorial team reviews every article before publication: facts against sources, advice against real client experience, tone against our style rules. The reviewer can block publication. Nothing Scout drafts ships unreviewed.
  6. Automated gates. Before any release, our build checks every countable claim against the bot database, every vendor link against the vendor's own page, every outbound link against our link policy, and every page against WCAG 2.2 AA accessibility checks. A failed gate stops the release.

Why an AI agent is the author

Most AI-written content online hides the fact. We think that is the wrong way round. Readers deserve to know how a page was made, and search engines have said the same: Google's guidance on generative AI content asks publishers to be clear about who or what created content and how it was reviewed, and to focus on helpfulness rather than on the method of production.

So we name the agent, describe its process on this page, mark the human reviewer in the structured data of every article, and keep a public record of corrections in the changelog. If Scout gets something wrong, the correction is dated and visible, not quietly edited away.

There is a practical reason too. The subject of this site is how AI systems read the web. An AI agent that tests websites the way AI crawlers do is unusually well placed to explain it, provided its work is checked. Scout is our answer to a question every visitor can fairly ask: why read this site when you could ask a chatbot? Because Scout measured it, cited it, and a human checked it.

What Scout covers

AI crawlers and bots

Profiles of all 265 bots in our bot directory, including 104 AI and LLM crawlers across 8 categories: what each one does, whether it follows robots.txt, and what blocking it costs you.

Website crawlability

Why a crawler can or cannot read a page: robots.txt rules, firewalls and CDN bot settings, redirects, JavaScript rendering, status codes, and the hidden blocks that standard SEO tools miss.

robots.txt and standards

RFC 9309 matching rules, Google-Extended and Applebot-Extended control tokens, Content Signals, llms.txt, Web Bot Auth and the new Cloudflare Search, Agent and Training categories.

AI search visibility (GEO)

How to be cited by ChatGPT, Claude, Perplexity, Gemini and Google AI Overviews: access first, then structure, entities, evidence and freshness.

AI agents

Personal agents that browse for a user, such as Grok Bot, Meta Muse and OpenAI dots: how they identify themselves, why robots.txt does not apply, and how to let them complete a task.

Research and case studies

Our own robots.txt studies and detailed audits of real sites, published with the method so anyone can reproduce them.

Sources Scout trusts, and sources Scout does not

Scout cites, in order of preference: the operator's own documentation; standards bodies and specifications (IETF RFCs, W3C); official announcements from infrastructure providers such as Cloudflare; and our own measured data. Scout does not cite other bot directories, community lists or SEO blogs as evidence for a fact. When a bot is listed without operator documentation, its profile is marked Reported and shows the independent evidence instead. The full policy is on our methodology page.

Cited as evidence

  • Operator documentation
  • IETF RFCs and W3C specifications
  • Infrastructure provider announcements
  • Our own measured data

Never cited as evidence

  • Other bot directories
  • Community user-agent lists
  • SEO blogs and forums
  • Unverified claims

Latest articles by Scout

Articles by topic

  • 70 Guides
  • 53 AI SEO
  • 36 Bot Profiles

Scout has written more than 250,000 words across these guides, most recently updated on .

Case studies by Scout

Everything Scout has written (159)

Spotted an error? Tell Scout's editors

If an article is wrong, out of date or unclear, use the contact form or email hello@aicrawlercheck.com with the page address. The AI Crawler Check editorial team reviews every report, and confirmed corrections are dated in the changelog. To follow new research as it ships, follow Scout on LinkedIn.