Kernel Search
SafeUser-agent: KernelSearchBot
Category: AI & LLM Bots
SEO impact if blocked: 2/10
1 crawler operated by Kernel
Kernel operates 1 crawler in our directory, filed under AI & LLM Bots. Its crawl behaviour is rated safe.
Blocking Kernel carries an SEO impact score of 2/10. Blocking impact is rated: Low - No SEO ranking impact.
Our recommendation for Kernel: Generally safe to allow; provides legitimate crawling value.
| Operator | Kernel |
|---|---|
| Crawler in directory | 1 |
| Categories | AI & LLM Bots |
| SEO impact if blocked | 2/10 |
| Crawl behaviour | 1 rated safe |
| Official crawler docs | Kernel crawler documentation |
Kernel runs 1 crawler in our database. Grouped by the job each one does:
KernelSearchBotThe highest-priority token is Tier 2. That puts Kernel among the operators whose crawlers have a direct effect on whether AI answers or search results can include your pages.
None of Kernel's tokens were among the names tracked in our robots.txt study of the Tranco top 10,000, which usually means few large sites write rules for them yet. Your own logs are the best evidence of how often they visit.
A robots.txt file cannot say "Kernel"; it can only name Kernel's tokens, one group at a time. With a single token, one group is the whole policy.
# One token controls everything Kernel fetches through it
User-agent: KernelSearchBot
Disallow: /private/
Allow: /
User-agent: *.* group, so copy any Disallow lines you still want it to obey.User-agent: Kernel block all of Kernel's crawlers?No. robots.txt matches crawler tokens, not company names. Use the token KernelSearchBot.
No. Each token is read separately. Blocking one leaves the rest following whatever group applies to them.
A one-paragraph brief for every Kernel token in our database, covering what it does, how it treats robots.txt and how often large sites write rules for it:
KernelSearchBot, Tier 2): a retrieval crawler that feeds AI search answers that builds a search or answer index. Rated safe. Kernel Search's operator does not clearly document how it treats robots.txt. Write the Kernel Search rule anyway, then confirm in your logs that its requests stop. Not tracked in our robots.txt study.The errors we see most often with Kernel's crawlers in the files and firewalls we test:
User-agent: KernelSearchBot group exists, Kernel's crawlers ignores every rule under User-agent: *.All three hide inside a file that looks correct, so check Kernel's crawlers from outside after every change.
Everything below comes from the AI Crawler Check bot database and our two most recent robots.txt scans of the Tranco top 10,000, not from Kernel's marketing. Kernel has one token in our database, so the whole picture fits in one place.
KernelSearchBot in our recordsWe first recorded KernelSearchBot on October 11, 2026. Kernel documents it on kernel.sh, and we recheck that page before each release. No file in our top-10,000 scans names KernelSearchBot, so any rule you write for it puts you ahead of almost every large site.
Among the 104 AI and LLM crawlers in our database, KernelSearchBot sits 54th by priority (Tier 2), level with AddSearchBot, AI2Bot, AI2Bot-DeepResearchEval, aiHitBot. 81 of the 104 share its safe rating. Other operators with a crawler that builds a search or answer index: AddSearchBot (AddSearch, Tier 2), AIWebIndex (Lyrenth, Tier 2), atlassian-bot (Atlassian, Tier 2), AzureAI-SearchBot (Microsoft, Tier 2), Cloudflare-AI-Search (Cloudflare, Tier 2). A policy for KernelSearchBot is usually a policy for these too.
KernelSearchBotTeams that review crawlers once a quarter keep one short entry per token, so anyone can see why a rule exists and when it was last checked. Filled in from our data, the KernelSearchBot entry reads:
User-agent: KernelSearchBot
Allow: /
Paste the line into the validator with one of your own URLs before it goes live. Revisit the KernelSearchBot entry whenever Kernel announces a change, because a Tier 2 crawler affects visibility directly.
KernelSearchBot on your own siteBecause Kernel runs only KernelSearchBot in our records, one check covers the whole operator. Search your access logs for the string KernelSearchBot over the last 30 days, note how many requests it made and which paths it asked for, then compare that with what your robots.txt allows. Match the requests against what Kernel publishes about the crawler, since that page is the only authority on its behaviour. A Tier 2 crawler rated safe is worth confirming every month, and our log checker does the counting for you.
From our robots.txt study of the Tranco top 10,000 (5,583 valid files, fetched October 20, 2026):
None of the tokens for Kernel's crawlers is tracked by name in our study. For scale, 20.1% of top-site files block at least one AI training crawler and 4.3% refuse every unnamed crawler with User-agent: * and Disallow: /, which would catch Kernel's crawlers too.
KernelSearchBot, add Disallow lines for the paths to keep private and Allow lines for exceptions. The longest matching rule wins.User-agent group per decision and repeat any shared rules inside it.This rule covers the single crawler Kernel operates. Add it to the robots.txt file at the root of your domain.
These Kernel rules ask; they do not enforce. If a Kernel token ignores them, block it at the server or CDN. Paths are case-sensitive, and each rule matches the token, not the full user-agent string.
Text size
Saved on this device only. Drag the round button to move it. Accessibility statement