Kernel

1 crawler operated by Kernel

Who Kernel is, for site owners

Kernel operates 1 crawler in our directory, filed under AI & LLM Bots. Its crawl behaviour is rated safe.

Blocking Kernel carries an SEO impact score of 2/10. Blocking impact is rated: Low - No SEO ranking impact.

Our recommendation for Kernel: Generally safe to allow; provides legitimate crawling value.

Kernel at a glance

OperatorKernel
Crawler in directory1
CategoriesAI & LLM Bots
SEO impact if blocked2/10
Crawl behaviour1 rated safe
Official crawler docsKernel crawler documentation

Crawler operated by Kernel

What Kernel's crawlers are for

Kernel runs 1 crawler in our database. Grouped by the job each one does:

  • a retrieval crawler that feeds AI search answers: KernelSearchBot

The highest-priority token is Tier 2. That puts Kernel among the operators whose crawlers have a direct effect on whether AI answers or search results can include your pages.

How site owners treat Kernel

None of Kernel's tokens were among the names tracked in our robots.txt study of the Tranco top 10,000, which usually means few large sites write rules for them yet. Your own logs are the best evidence of how often they visit.

Writing one policy for Kernel

A robots.txt file cannot say "Kernel"; it can only name Kernel's tokens, one group at a time. With a single token, one group is the whole policy.

# One token controls everything Kernel fetches through it
User-agent: KernelSearchBot
Disallow: /private/
Allow: /
  1. Decide per job, not per company. Training, search and user-triggered fetching have different trade-offs, even from the same operator.
  2. Name every token you mean. A token you leave out follows User-agent: *.
  3. Repeat shared rules. A named group ignores the * group, so copy any Disallow lines you still want it to obey.
  4. Verify from outside. Run the website crawlability test after any change to see the result for each Kernel token, including firewall refusals.
Test yourself: does User-agent: Kernel block all of Kernel's crawlers?

No. robots.txt matches crawler tokens, not company names. Use the token KernelSearchBot.

Test yourself: if you block one Kernel token, are the others blocked too?

No. Each token is read separately. Blocking one leaves the rest following whatever group applies to them.

Each Kernel crawler, explained

A one-paragraph brief for every Kernel token in our database, covering what it does, how it treats robots.txt and how often large sites write rules for it:

  • Kernel Search (KernelSearchBot, Tier 2): a retrieval crawler that feeds AI search answers that builds a search or answer index. Rated safe. Kernel Search's operator does not clearly document how it treats robots.txt. Write the Kernel Search rule anyway, then confirm in your logs that its requests stop. Not tracked in our robots.txt study.

Common mistakes with Kernel's crawlers

The errors we see most often with Kernel's crawlers in the files and firewalls we test:

  1. Blocking Kernel's crawlers while trying to stop AI training. Retrieval crawlers fetch pages so answers can cite them; blocking it can remove you from those answers without affecting training.
  2. Letting a firewall or bot-fight mode refuse Kernel's crawlers while robots.txt allows it. The file looks right, but the crawler still gets a 403 or a challenge page.
  3. Forgetting that once a User-agent: KernelSearchBot group exists, Kernel's crawlers ignores every rule under User-agent: *.

All three hide inside a file that looks correct, so check Kernel's crawlers from outside after every change.

What our own data shows about Kernel

Everything below comes from the AI Crawler Check bot database and our two most recent robots.txt scans of the Tranco top 10,000, not from Kernel's marketing. Kernel has one token in our database, so the whole picture fits in one place.

KernelSearchBot in our records

We first recorded KernelSearchBot on October 11, 2026. Kernel documents it on kernel.sh, and we recheck that page before each release. No file in our top-10,000 scans names KernelSearchBot, so any rule you write for it puts you ahead of almost every large site.

Among the 104 AI and LLM crawlers in our database, KernelSearchBot sits 54th by priority (Tier 2), level with AddSearchBot, AI2Bot, AI2Bot-DeepResearchEval, aiHitBot. 81 of the 104 share its safe rating. Other operators with a crawler that builds a search or answer index: AddSearchBot (AddSearch, Tier 2), AIWebIndex (Lyrenth, Tier 2), atlassian-bot (Atlassian, Tier 2), AzureAI-SearchBot (Microsoft, Tier 2), Cloudflare-AI-Search (Cloudflare, Tier 2). A policy for KernelSearchBot is usually a policy for these too.

A decision record for KernelSearchBot

Teams that review crawlers once a quarter keep one short entry per token, so anyone can see why a rule exists and when it was last checked. Filled in from our data, the KernelSearchBot entry reads:

  • Operator and job: Kernel; it builds a search or answer index.
  • Priority and behaviour: Tier 2, rated safe, blocking impact low.
  • Suggested starting point: Allow, and watch it in your logs.
  • Enforcement: KernelSearchBot's operator does not clearly document how it treats robots.txt. Write the KernelSearchBot rule anyway, then confirm in your logs that its requests stop.
User-agent: KernelSearchBot
Allow: /

Paste the line into the validator with one of your own URLs before it goes live. Revisit the KernelSearchBot entry whenever Kernel announces a change, because a Tier 2 crawler affects visibility directly.

Checking KernelSearchBot on your own site

Because Kernel runs only KernelSearchBot in our records, one check covers the whole operator. Search your access logs for the string KernelSearchBot over the last 30 days, note how many requests it made and which paths it asked for, then compare that with what your robots.txt allows. Match the requests against what Kernel publishes about the crawler, since that page is the only authority on its behaviour. A Tier 2 crawler rated safe is worth confirming every month, and our log checker does the counting for you.

How top sites treat Kernel's crawlers

From our robots.txt study of the Tranco top 10,000 (5,583 valid files, fetched October 20, 2026):

None of the tokens for Kernel's crawlers is tracked by name in our study. For scale, 20.1% of top-site files block at least one AI training crawler and 4.3% refuse every unnamed crawler with User-agent: * and Disallow: /, which would catch Kernel's crawlers too.

More questions about Kernel

Will blocking Kernel affect my Google rankings?
Only if one of its tokens is a Google search crawler. None of Kernel's tokens are Googlebot, so classic Google rankings are unaffected.
How quickly does a robots.txt change take effect for Kernel?
Crawlers cache robots.txt, commonly for up to a day. Expect a change to apply within about 24 hours, and confirm it in your logs.
Can I allow Kernel on some sections and block it on others?
Yes. Inside the group for KernelSearchBot, add Disallow lines for the paths to keep private and Allow lines for exceptions. The longest matching rule wins.
Does blocking Kernel delete content it already collected?
No. robots.txt only controls future visits. For content already collected, check whether Kernel offers a removal or opt-out process.
Is a firewall block better than robots.txt for Kernel?
A firewall enforces the decision, while robots.txt asks politely. Many operators recommend robots.txt because blocking their IPs can stop them reading your rules at all. Use robots.txt first, and add a firewall rule only if requests continue.

A five-minute Kernel review

  1. Confirm the token. Compare the user-agent in your logs with the tokens listed on this page. A close spelling is not the same crawler.
  2. Decide the outcome per crawler. Use the briefs above to choose allow, limit or block for each token.
  3. Write the group. Add or edit one User-agent group per decision and repeat any shared rules inside it.
  4. Publish and wait. Allow about a day for crawlers to fetch the new file.
  5. Confirm the result. Check your logs, then run the AI crawler checker to see the outcome from outside, including any firewall layer.

How to block or allow Kernel in robots.txt

This rule covers the single crawler Kernel operates. Add it to the robots.txt file at the root of your domain.

Block Kernel completely
User-agent: KernelSearchBot Disallow: /
Allow Kernel explicitly
User-agent: KernelSearchBot Allow: /

These Kernel rules ask; they do not enforce. If a Kernel token ignores them, block it at the server or CDN. Paths are case-sensitive, and each rule matches the token, not the full user-agent string.

Frequently asked questions about Kernel

What crawler does Kernel run?
Kernel operates 1 crawler in our directory: Kernel Search (user-agent KernelSearchBot).
Should I block Kernel?
Generally safe to allow; provides legitimate crawling value. The SEO impact score for blocking is 2/10, where 10 means blocking costs you the most visibility.
How do I block Kernel in robots.txt?
Add the following to robots.txt at your domain root: User-agent: KernelSearchBot then Disallow: /.
Does blocking Kernel hurt my search rankings?
Blocking impact is rated: Low - No SEO ranking impact. Google's own ranking crawler is Googlebot, which this operator does not run, so blocking Kernel does not by itself remove you from Google's index. What it can change is whether AI assistants and analytics tools can read your pages.
WhatsApp us