Bingbot and Copilot: How Microsoft AI Uses Your Site
Everything you need to know about Bingbot (Copilot): what it does, how it affects your AI visibility, and exactly how to allow or block it in robots.txt.
AI Crawler Check Blog
148 guides on AI crawlers, robots.txt and AI search visibility, written and maintained by Horatos.ai.
Latest articles
FacebookBot and Meta-ExternalAgent: Meta AI Crawlers
Everything you need to know about Meta-ExternalAgent: what it does, how it affects your AI visibility, and exactly how to allow or block it in robots.txt.
DuckAssistBot: DuckDuckGo's AI Answer Crawler
Everything you need to know about DuckAssistBot: what it does, how it affects your AI visibility, and exactly how to allow or block it in robots.txt.
Kagi Crawler: Privacy-First Search and AI
Everything you need to know about Kagibot: what it does, how it affects your AI visibility, and exactly how to allow or block it in robots.txt.
AhrefsBot vs SemrushBot: SEO Crawlers and Crawl Budget
Everything you need to know about AhrefsBot / SemrushBot: what it does, how it affects your AI visibility, and exactly how to allow or block it in robots.txt.
Allow vs Disallow in robots.txt: Rules, Order, and Gotchas
A practical, visual guide to understand Allow/Disallow precedence and avoid mistakes. Includes step-by-step instructions, common mistakes to avoid, and free tools to verify your results.
robots.txt Wildcards and Pattern Matching Explained
A practical, visual guide to use * and $ patterns correctly in robots.txt. Includes step-by-step instructions, common mistakes to avoid, and free tools to verify your results.
12 robots.txt Mistakes That Accidentally Block AI Bots
A practical, visual guide to find and fix robots.txt errors hurting AI visibility. Includes step-by-step instructions, common mistakes to avoid, and free tools to verify your results.
How to Test Your robots.txt (Free Validator Walkthrough)
A practical, visual guide to validate robots.txt and confirm bot access. Includes step-by-step instructions, common mistakes to avoid, and free tools to verify your results.
robots.txt for Shopify: What You Can and Cannot Change
A practical, visual guide to edit Shopify robots.txt.liquid for AI access. Includes step-by-step instructions, common mistakes to avoid, and free tools to verify your results.
robots.txt on Wix and Squarespace: Workarounds for AI
A practical, visual guide to manage AI access on Wix and Squarespace. Includes step-by-step instructions, common mistakes to avoid, and free tools to verify your results.
robots.txt for Webflow and Framer Sites
A practical, visual guide to set AI crawler rules on Webflow and Framer. Includes step-by-step instructions, common mistakes to avoid, and free tools to verify your results.
Page 3 of 13, showing 25 to 36 of 148 articles
Stay Updated on AI SEO
Check your website's AI visibility and discover optimization opportunities
Free AI Crawler Check