LINER AI

LINER AI in context

LINER AI is a training crawler operated by LINER. In plain terms, it collects pages both for training and for search or answers. Our database files it under AI and LLM crawlers and ranks it Tier 2, important crawlers from established AI and search operators. 74 of the 104 AI and LLM crawlers in our database share this tier.

Its safety rating is safe and its blocking impact is low. So the decision about LINER AI rests on what you want from LINER, not on its behaviour. How we rate safety and impact.

How site owners treat LINER AI

Our robots.txt study of the Tranco top 10,000 did not track LinerBot as a named token, so we do not publish an adoption figure for it. If you run a large site, your own logs are the best evidence of how often it visits.

Does it follow robots.txt?

LINER AI's operator does not clearly document how it treats robots.txt. Write the LINER AI rule anyway, then confirm in your logs that its requests stop.

LINER crawlers and close alternatives

LINER runs only this one crawler in our database, so a single User-agent: LinerBot group controls everything LINER fetches through it.

Other AI and LLM crawlers that do the same job (collects pages both for training and for search or answers) include PerplexityBot from Perplexity AI, Andibot from Andi, cohere-ai from Cohere, DeepSeekBot from DeepSeek, DoubaoBot from ByteDance, GrokBot from xAI. Deciding about LINER AI is usually a decision about the whole group, so check how your file treats these too.

We first recorded LinerBot on September 22, 2026. LINER documents it on docs.getliner.com, and we recheck that page before each release.

Which rule LINER AI actually follows

LINER AI reads exactly one group in robots.txt, the most specific one that names LinerBot, and never combines rules across groups. This short file shows the trap:

User-agent: *
Disallow: /private/

User-agent: LinerBot
Disallow: /drafts/

1. Find the group. LINER AI looks for a group naming LinerBot. One exists, so it ignores User-agent: * completely.
2. Apply only that group. It is refused /drafts/ and nothing else.
3. The surprise. /private/ is open to LINER AI, because that rule lives in the * group it no longer reads. To keep it out of both, repeat Disallow: /private/ inside its own group.

Should you allow LINER AI? Decide by situation

Check LINER AI on your own site

1. Open /robots.txt on your domain and search for LinerBot. If it is not named, it follows User-agent: *.
2. List the Disallow lines in the group LINER AI follows and test your key URLs against them; the longest match wins, and Allow wins a tie.
3. Look beyond the file. A firewall or bot manager can refuse LINER AI even when robots.txt allows it, and that never shows in the file.
4. Run our robots.txt AI crawler test to see how LINER AI and every other crawler we track are treated in one pass, including firewall refusals.