Meta Muse and Your Website: What the AI Agent Does and How to Handle It
Meta's Muse agent opens websites to book and buy for its users. What the muse- user agent means, how to find it in your logs, and how to decide whether to welcome it.
AI Crawler Check Blog
148 guides on AI crawlers, robots.txt and AI search visibility, written and maintained by Horatos.ai.
Latest articles
OpenAI Dots and Your Website: How to Spot, Allow or Block Them
Dots browse from their own cloud computer and sign every request instead of sending a name. How to see them, allowlist them in your CDN, or keep them out.
Personal AI Agents and Your Website: Muse, Dots, Grok Bot and More (2026)
Six personal AI agents now open websites for their users. How each one identifies itself, where they get stuck, and a single plan that works for all of them.
Who Blocks AI Crawlers? A Study of the Top 10,000 Sites' robots.txt (2026)
Original data: which AI crawlers the top 10,000 sites block in robots.txt, how training and search blocks differ, and the file errors we found.
Can You Block Grok Bot? Why robots.txt Does Not Apply to AI Agents
robots.txt does not apply to AI browsing agents. What actually stops Grok Bot, how to block it by IP if you must, and why most sites should not.
Grok Bot vs GrokBot: The AI Agent and the Crawler Name Explained
One is an AI agent with its own cloud browser, the other a crawler name in your logs. How Grok Bot and GrokBot reach your website, and which controls work on each.
10 Grok Bot Workflows for SEO and GEO Teams (With Prompts)
Ten practical Grok Bot workflows for SEO and GEO teams, each with a starter prompt, built only on features xAI documents and kept safely behind your approval.
How to Get Cited by Grok: What xAI Documents and What It Does Not
What xAI actually documents about how Grok searches and cites, what it does not, and seven evidence-labelled steps to make your pages easier to read and cite.
How to Make Your Website Work for AI Agents Like Grok Bot
AI agents now browse, compare and fill in forms for their users. Seven steps to make sure they can finish the job on your site, and a free check to test it.
257 AI Bots You Should Know (and What They Do)
Every one of the 257 bots AI Crawler Check tests against, grouped into 8 categories with operator, purpose and safety rating, plus the 164 release 6.1 additions and the Official versus Reported evidence badge on every token.
GeminiBot: How Google Gemini Crawls and Uses Your Content
Everything you need to know about Gemini (Google AI): what it does, how it affects your AI visibility, and exactly how to allow or block it in robots.txt.
Grok Crawler User Agent: The Official Answer and robots.txt Rules
xAI has published no crawler documentation. This guide gives the two reported Grok user agents, the robots.txt rules to use, and explains why Grok Bot is a different thing.
Page 1 of 13, showing 1 to 12 of 148 articles
Stay Updated on AI SEO
Check your website's AI visibility and discover optimization opportunities
Free AI Crawler Check