AI Crawler Check Blog

Find the right article

• Bot guides explain one crawler or agent: what it does, its token and whether to allow it.
• How-to guides walk through a task, such as blocking training crawlers or fixing a 403.
• Strategy covers decisions: training versus search, licensing, and AI visibility for different kinds of site.
• Research publishes our own data, including our robots.txt study of the top 10,000 sites.
• News covers launches and policy changes from AI companies and CDNs.

How we write

• Facts about a bot come from the operator's own documentation where it exists, and are labelled when they come from elsewhere.
• Numbers about the web come from studies we ran ourselves, with the date and method stated.
• Articles are researched and drafted by Scout Agent, the AI research agent created by AI Crawler Check, and reviewed by the AI Crawler Check editorial team before publishing.
• When a fact changes, we update the article and its date rather than adding a footnote.

Five articles worth reading first

1. How to block AI crawlers with robots.txt
2. Training bots vs search bots
3. Who blocks AI crawlers? A study of the top 10,000 sites
4. Fixing 403 errors for AI crawlers
5. Why your website needs an AI visibility score

How articles are sourced

Articles link to the primary source behind their main claims: the company's own crawler documentation, the standards body's draft or our own published data. Counts from our robots.txt scans state the date of the fetch and how many files were parsed, and the method is on the methodology page. The newest articles also end with a short self-test that works without JavaScript.

Getting new articles

New articles appear at the top of this page, newest first, and every article lists its publish and update dates. Each one is also linked from the bot profiles, operators and tools it covers, so you will usually meet the newest guidance wherever you are on the site. If a fact in any article has changed, tell us through the contact page and we will update it.

A note on dates and accuracy

Every article shows when it was published and when it was last updated. Crawler details change quickly: operators rename tokens, publish new IP lists and change how they treat robots.txt. When that happens we update the articles that mention the crawler and move their update date, so a recent date means the facts were rechecked, not that a comma moved. If you find something out of date, the contact page explains how to tell us, and corrections are recorded in the changelog. For live facts about a single crawler, its profile in the bot directory is always the most current place to look, because it is rechecked before every release.

Reading an article well

1. Read the key takeaways at the top to decide whether the article answers your question.
2. Use the "On this page" list to jump to the section you need.
3. Check the last updated date; crawler details change.
4. Follow the primary-source links for anything you will act on.
5. Run the check on your own site from the tip at the end.

How often new articles appear

New articles are added most weeks, and existing ones are updated when an operator changes its documentation or our scans show a shift. The update date on each card reflects the last real change to that article, not a cosmetic edit. To see what changed recently across the whole site, use the "Recently updated" sort above or read the changelog.

How to suggest a topic

Most articles start from a question a reader sent. If you cannot find an answer here, send us the question and the context: what you were trying to do, which crawler or engine was involved, and what you have already tried. Questions that come up more than once become articles.

Where to start if you are new

1. robots.txt best practices for AI SEO, for the rules that matter.
2. Training bots versus search bots, for the one distinction behind most decisions.
3. Why AI bots cannot crawl your site, for the causes beyond robots.txt.
4. GEO for beginners, for what happens after access.

What the blog covers

159 articles, grouped into the topics shown above. Bot profiles explain one crawler each. Guides walk through a task. GEO strategy covers how AI engines choose and cite sources. The news roundups track what changed each month, and the study articles report our own robots.txt scans. Use the topic buttons to filter and the sort options to see the newest or most read.

Test yourself