160 bots across 8 categories — complete user-agent database with safety ratings, robots.txt rules, and blocking guides.
AI training crawlers collecting content for LLM datasets. Blocking has zero SEO impact.
Search engine crawlers that index your content for organic search visibility. Critical to allow.
Commercial SEO platform crawlers (Ahrefs, Semrush etc.). Blocking hides your data from competitors.
Social media crawlers fetching Open Graph metadata for link previews.
Google's specialized crawlers for Images, Video, News and other services.
Data aggregation and research crawlers. No SEO impact from blocking.
Cloud provider crawlers used for security scanning, CDN warm-up or threat intelligence.
Miscellaneous crawlers not fitting other categories.