AI & LLM Bots
88 crawlersAI training crawlers collecting content for LLM datasets. Blocking has zero SEO impact.
ChatGPT Agent (Cloud browser) ChatGPT Operator (Cloudflare name) ChatGPT-User/2.0 GPTBot OAI-SearchBot
Web Crawler & Bot Directory
248 bots across 8 categories, with 245 written profiles (some unversioned tokens share a profile with their versioned twin) covering user-agent strings, safety ratings, robots.txt rules and blocking guides.
Most visits start with the AI bots directory (GPTBot, ClaudeBot, PerplexityBot and the rest) or the SEO bots such as SemrushBot and AhrefsBot.
Search Engines
62 crawlersSearch engine crawlers that index your content for organic search visibility. Critical to allow.
Googlebot Amzn-SearchBot Bing Ads Crawler Bing Page Snapshot Bing Video Preview
SEO Tools
19 crawlersCommercial SEO platform crawlers (Ahrefs, Semrush etc.). Blocking hides your data from competitors.
Lighthouse Audit Ahrefs Site Audit AhrefsBot AwarioBot AwarioRssBot
Social Media Bots
9 crawlersSocial media crawlers fetching Open Graph metadata for link previews.
Facebook Crawler facebookexternalhit Meta Ads Crawler LinkedInBot Pinterestbot
Google Bots
24 crawlersGoogle's specialized crawlers for Images, Video, News and other services.
APIs-Google Chrome Web Store Fetcher FeedFetcher-Google Gemini Notebook Gemini-Deep-Research
Data Scrapers
39 crawlersData aggregation and research crawlers. No SEO impact from blocking.
Amazon Kendra Apify Crawler Barkrowler BLEXBot Scraper Bright Data Crawler
Cloud Services
3 crawlersCloud provider crawlers used for security scanning, CDN warm-up or threat intelligence.
Google-CloudVertexBot Amazonbot AliyunSecBot
Other Agents
4 crawlersMiscellaneous crawlers not fitting other categories.
archive.org_bot Arquivo.pt Web Archive ia_archiver TurnitinBot
How we decide what goes in this directory
A crawler list is only useful if you can tell what is confirmed from what is rumoured. Every profile here carries one of two badges, and the badge tells you exactly where the token came from.
Official docs 174 profiles
The operator's own documentation names the token. The profile links to that exact page, not the company homepage, and it was fetched and re-checked on September 22, 2026. GPTBot, ClaudeBot, Googlebot, Bytespider and ChatGPT Agent are in this group.
Reported 71 profiles
No operator page names the token, but a credible public source does (the ai.robots.txt registry, Trakkr's observatory, user-agents.net, or the operator's own non-crawler page). The profile says so in plain words and links the source. Claude-Web, GrokBot, DeepSeekBot and QwenBot are in this group. Treat these rules as best effort.
Tokens with neither an operator page nor a credible public source are not listed at all, and bare product names (Grok, OpenAI, Operator) are not tokens. Each category page has a filter to show only one group.