Bot analytics platform

AI Crawler Check vs Known Agents (formerly Dark Visitors)

These tools answer different questions. Known Agents tells you which agents did visit, by logging live traffic through a script you install. AI Crawler Check tells you which agents can visit, by reading your rules and testing them against 265 bots without installing anything. Most serious sites end up wanting both.

Competitor facts verified 2026-08-08 Source: knownagents.com/pricing

Short answer

Use AI Crawler Check to audit whether your rules allow or block the right AI crawlers, with no install and no sign-up. Use Known Agents when you need ongoing analytics of agent traffic that actually arrived. They are complementary, not substitutes: one is a diagnostic, the other is a monitor.

Choose Known Agents when

You want a running log of which AI agents actually hit your site, over time.
You need to measure human referral traffic arriving from ChatGPT, Claude, Gemini and Perplexity.
You are on WordPress and want a plugin that both reports and enforces.
You want a robots.txt that updates itself as new bots are discovered, and you accept a hosted dependency in exchange.
You need CSV export or an agent identification API for your own code, and can use a paid tier.

Choose AI Crawler Check when

You want an answer in about a minute with nothing installed, no script and no account.
You need to know whether your rules are correct, not just what traffic arrived. A bot that never visited still shows as blocked if your robots.txt blocks it.
You are auditing a site you do not control, such as a prospect, a client before onboarding, or a competitor.
You want the reasoning shown: which rule matched, why a bot is blocked, and what to change. See the scoring methodology.
You need the wider picture in one report: robots.txt, llms.txt, HTTPS, schema and the rest of the 42 GEO checks.

Side by side

AI Crawler Check compared with Known Agents
What you are comparing	AI Crawler Check	Known Agents
Comparable. Core question answered	Which AI crawlers can reach this site, and is that what you intended	Which AI agents did reach this site, and how often
Advantage AI Crawler Check. Install required	None. Enter a URL and read the report	Yes. A script, plugin or API integration on the site you are measuring
Advantage AI Crawler Check. Account required	No sign-up for the core scan	Yes, a project must be created
Advantage AI Crawler Check. Works on sites you do not own	Yes. This is the normal case for agency and competitor research	No. You must be able to install on the site
Comparable. Bot coverage	265 bots profiled across 8 categories and 6 priority tiers, with 112 operators documented	A large, actively maintained agent database, with new agents added as they are found
Advantage Known Agents. Historical traffic data	None. Each scan is a point-in-time snapshot of your configuration	Yes, this is the product. Trends over time, per agent
Advantage Known Agents. AI chat referral tracking	Not offered	Yes, referrals attributed by AI platform
Comparable. Firewall and WAF blocking detection	Detects when a WAF blocks the robots.txt fetch itself and says so rather than guessing	Detects agents that ignore robots.txt, and can enforce via plugin or WAF rules on higher tiers
Advantage AI Crawler Check. Wider GEO audit	42 checks across 5 categories, scored 0 to 100	Out of scope. Focused on agent traffic
Advantage AI Crawler Check. Bulk checking	Up to 20 URLs per run in the batch checker	Projects are per site, with plan limits on how many
Advantage AI Crawler Check. Free tier limits	No event cap. The scan, score, robots analysis and bot directory are free	Free plan covers 1 project and 100K events per month, with a hard cap
Advantage Known Agents. Paid plans	None for the core tools	Paid tiers add projects, events, CSV export, identification API and team members
Advantage AI Crawler Check. Ongoing dependency	None. Nothing keeps running after you close the tab	A live integration, which is the price of continuous data

Dot colour shows which tool has the edge on that row: AI Crawler Check, Known Agents, comparable.

The distinction that actually matters

Most people comparing these two tools are really asking one question without realising it: do I need a diagnostic or a monitor?

A monitor tells you what happened. Known Agents installs on your site, watches requests, and builds a record of which agents arrived and what they did. That record is genuinely valuable and AI Crawler Check does not produce it.

A diagnostic tells you whether your configuration is correct. That is a different thing, and the difference has a sharp edge to it.

Why traffic logs cannot answer the access question Suppose your robots.txt blocks ClaudeBot. Anthropic's crawler reads the rule and stays away. Your traffic log now shows zero ClaudeBot visits. That zero is indistinguishable from "Claude has never heard of us" and from "Claude crawls us happily but rarely". Absence of traffic does not tell you the cause of the absence. Reading the rule does.

This is why the two tools are complementary rather than competing. The log tells you about the bots that came. The audit tells you about the bots that were turned away at the door, which are precisely the ones that generate no data.

The install question is not a detail

Known Agents requires code on the site being measured. That is unavoidable for its purpose and not a flaw. But it does set a hard boundary: you can only measure sites you control.

A large share of real AI visibility work happens outside that boundary:

Prospecting. An agency wants to show a prospect that their site blocks GPTBot. There is no access to install anything, and asking for access before the pitch inverts the sales process.
Due diligence. You are acquiring a site, or onboarding a client, and want to know what you are inheriting before you touch it.
Competitive research. You want to know whether the competitor who keeps getting cited in ChatGPT answers has done anything you have not.
Pre-launch and staging. There is no traffic yet, so there is nothing for a monitor to log. The configuration still needs to be right on day one.

In all four cases a URL-based audit is the only tool that works, because it reads what is publicly published rather than what is privately logged.

On bot counts, and why we will not claim to win that row

It would be easy to put our 265 next to somebody else's number and declare victory. We are not going to, because the numbers do not measure the same thing.

A blocklist counts user-agent strings. We count bots we have profiled: assigned to one of 8 categories, given a priority tier, attributed to one of 112 operators, and documented on its own page. Those are different units. A list of 300 strings and a set of 265 profiled bots are not comparable quantities, and pretending otherwise would be the same sleight of hand we criticise elsewhere.

What we will say instead Coverage is roughly comparable and both databases are actively maintained. The meaningful difference is not how many strings each tool knows, it is what each one does with them. We attach a category, a tier, an operator and a recommendation to every bot, so a result tells you whether the block was a mistake or a deliberate choice you should keep.

The Dark Visitors rename

If you are searching for Dark Visitors, you are looking for this product. It was rebranded as Known Agents, reflecting a broader scope than crawler blocking alone: agent analytics, AI chat referral tracking and agentic commerce.

We mention it because a comparison that still called the product Dark Visitors would tell you, accurately, that nobody had checked it recently.

Using both together

The honest recommendation for a site that takes AI visibility seriously:

Audit first. Run the LLM crawler checker to find out what your rules currently permit. Fix anything blocked by accident, using the guide to fixing blocked crawlers.
Then monitor. With correct rules in place, a traffic monitor becomes informative. Installed over a broken configuration, it mostly measures your own mistakes.
Re-audit after changes. A redesign, a migration or a new WAF rule can silently reintroduce a block. A monitor shows the traffic drop weeks later; an audit shows the cause the same day.

Order matters. Fix the door, then count who walks through it.

A worked example: the silent ClaudeBot block

Here is the scenario that shows why a diagnostic and a monitor answer different questions. A retailer adds a broad rule to robots.txt to stop aggressive scrapers. The rule names several user-agents, and by accident it also covers ClaudeBot and Claude-SearchBot.

Week one, in a traffic monitor: Anthropic's crawlers read the new rule and stop requesting pages. The monitor shows Claude traffic falling to zero. Nothing flags it as an error, because a crawler politely staying away looks exactly like a crawler that lost interest.
Week three, in the business: product pages stop appearing in Claude's answers. Nobody connects the two, because the change was made for a different reason by a different person.
One minute, in an audit: an outside-in check lists both Anthropic crawlers as Blocked, names the group that applied and shows the line that decided it. The cause and the fix are on one screen.

The monitor was not wrong. It accurately reported that ClaudeBot stopped visiting. It simply cannot tell you that the reason was your own configuration, because the evidence of a refused visit is the absence of a visit. That is the core reason we recommend running both: the monitor for what happened, the audit for what is allowed to happen.

Questions to ask before you pick either tool

Do I control the site? A monitor needs an install. An audit needs only a URL, so it also works for prospects, competitors and clients before access is granted.
Do I need history or a snapshot? Trends over weeks come from logs. A point-in-time answer after a change comes from an audit.
Who needs to read the result? Engineers like raw request data. Clients and managers usually want a plain list of which assistants can and cannot read the site.
What changed recently? After any robots.txt, CDN or firewall change, run an audit first. It answers the urgent question in a minute.

Frequently asked questions

Is AI Crawler Check a Dark Visitors alternative?

Partly. Dark Visitors is now Known Agents. If what you want is a fast audit of whether AI crawlers can access your site, with no install, then yes, AI Crawler Check does that job and it is free. If what you want is a continuous log of agent traffic and AI referral analytics, then no, we do not offer that and Known Agents does. The two tools answer different questions.

Do I need to install anything to use AI Crawler Check?

No. You enter a URL and get a report in about a minute. There is no script, no plugin, no DNS change and no account. That is what makes it usable on sites you do not control, such as a prospect's or a competitor's. See the quick start guide.

Why does traffic monitoring not tell me if a bot is blocked?

Because a blocked crawler generates no traffic, and neither does a crawler that simply never visited. In a log both look like zero. Reading your robots.txt rules directly distinguishes the two, which is what an audit does.

Which one should an agency use?

Both, at different stages. An audit works before you have access, so it suits prospecting, pitching and onboarding. A monitor works after access, so it suits ongoing retainer reporting. The agency workflow guide covers the audit side, including the 20-URL batch checker for auditing a whole client list at once.

Is AI Crawler Check really free, or is there a paid tier later?

The core scan, the AI Visibility Score, the robots.txt analysis, the 42-check GEO audit and the 265-bot bot directory are free with no sign-up and no event cap. It is built and maintained by Horatos.ai, an AI-native growth marketing agency in Singapore, as a public tool.

See where your site actually stands

Free check against 265 AI crawlers. No sign-up, nothing to install.