Safe Official docs AI & LLM Bots

AI2Bot

Q: What is the official user-agent string for AI2Bot?

The official user-agent string for AI2Bot is: AI2Bot . This is the exact string you must use in robots.txt, Nginx, Apache, or Cloudflare firewall rules to target this bot. User-agent matching in robots.txt is case-insensitive, but the string must be spelled correctly. You can verify that a request genuinely comes from AI2Bot by performing a reverse-DNS lookup on the source IP - legitimate bots resolve back to their operator's domain.

Q: Is AI2Bot safe?

Yes. AI2Bot is a documented bot: Allen Institute for AI publishes a page that names it and explains what it does.

Q: How do I block AI2Bot in robots.txt?

Add the following lines to your /robots.txt file: User-agent: AI2Bot Disallow: / This instructs AI2Bot not to crawl any path on your site. The Disallow: / directive covers the entire domain including subfolders. To only block specific sections, replace / with the path (e.g., Disallow: /blog/ ). Note: robots.txt is publicly readable - any bot or human can inspect it at yourdomain.com/robots.txt .

Q: How do I verify if AI2Bot is crawling my site?

Search your web server access logs for the string AI2Bot (case-insensitive grep: grep -i "AI2Bot" /var/log/nginx/access.log ). You can also check Google Search Console → Coverage → Crawl Stats for Googlebot variants. For AI2Bot specifically, filter by user-agent in your log analysis tool (GoAccess, AWStats, etc.).

Operated by Allen Institute for AI

Scout's quick take

AI2Bot collects pages for Allen Institute for AI's AI training. Blocking it keeps your content out of future training runs and does not change your Google or Bing rankings.

User-agent: AI2Bot
Disallow: /

Run an AI crawlability test Build a full robots.txt

Quick Facts

User-Agent:: AI2Bot
Category:: AI & LLM Bots
Operator:: Allen Institute for AI
Safety:: Safe
Blocking Impact:: Low - No SEO ranking impact
SEO Impact Score:: 2/10
Official docs:: allenai.org

Confirmed by the operator

Allen Institute for AI names AI2Bot on its own documentation page, linked above. The token, purpose and robots.txt behaviour on this profile come from that page, re-checked on October 19, 2026.

What is AI2Bot?

AI2Bot is the Allen Institute for AI's crawler. AI2 says it collects public web data for open AI research and datasets.

Content may be used in open research datasets and models. It identifies itself with the token AI2Bot. Allen Institute for AI documents it at allenai.org.

What happens if you block AI2Bot?

No effect on Google or Bing.

Generally safe to allow; provides legitimate crawling value.

Want to see whether AI2Bot actually visits you? Read how do I monitor AI bot traffic to my website?

How to block AI2Bot with robots.txt

<code>User-agent: AI2Bot</code> - Matching is case-insensitive. Robots.txt is fetched from the root of each subdomain separately.

Block completely (robots.txt)

User-agent: AI2Bot
Disallow: /

Allow all (robots.txt)

User-agent: AI2Bot
Allow: /

Block private only (robots.txt)

User-agent: AI2Bot
Disallow: /private/
Disallow: /api/
Disallow: /admin/
Allow: /

Nginx server block

# Nginx: Hard-block AI2Bot
if ($http_user_agent ~* "AI2Bot") {
    return 403 "Bot blocked";
}

Apache .htaccess

# Apache: Hard-block AI2Bot
SetEnvIfNoCase User-Agent "AI2Bot" bad_bot
Order Allow,Deny
Allow from all
Deny from env=bad_bot

Meta robots tag

X-Robots-Tag header

X-Robots-Tag: noindex, nofollow

Is AI2Bot safe to allow?

Yes. AI2Bot is a documented bot: Allen Institute for AI publishes a page that names it and explains what it does.

Verify by reverse-DNS lookup: legitimate AI2Bot requests resolve to allen-institute-for-ai's domain.

What does AI2Bot do?

From Allen Institute for AI's own documentation.

AI2Bot is the Allen Institute for AI's crawler.
Content may be used to train AI models
No effect on Google or Bing.

AI2Bot in context

AI2Bot is a training crawler operated by Allen Institute for AI. In plain terms, it collects pages to train AI models. Our database files it under AI and LLM crawlers and ranks it Tier 2, important crawlers from established AI and search operators. 74 of the 104 AI and LLM crawlers in our database share this tier.

Its safety rating is safe and its blocking impact is low. So the decision about AI2Bot rests on what you want from Allen Institute for AI, not on its behaviour. How we rate safety and impact.

How site owners treat AI2Bot

Our robots.txt study of the Tranco top 10,000 did not track AI2Bot as a named token, so we do not publish an adoption figure for it. If you run a large site, your own logs are the best evidence of how often it visits.

Does it follow robots.txt?

AI2Bot's operator does not clearly document how it treats robots.txt. Write the AI2Bot rule anyway, then confirm in your logs that its requests stop.

Allen Institute for AI crawlers and close alternatives

AI2Bot is one of 2 Allen Institute for AI crawlers in our database. A User-agent: AI2Bot group controls AI2Bot only; the other Allen Institute for AI tokens below need their own lines.

AI2Bot: this crawler (Tier 2)
AI2Bot-DeepResearchEval: Ai2 Deep Research, fetches a page when a person asks an assistant or app to open it (Tier 2)

To give AI2Bot the same answer as its siblings, stack User-agent: AI2Bot with their lines above one set of rules; to treat AI2Bot differently, give it its own group.

Other AI and LLM crawlers that do the same job (collects pages to train AI models) include ClaudeBot from Anthropic, Google-Extended from Google, GPTBot from OpenAI, Meta-ExternalAgent from Meta, aiHitBot from aiHit, anthropic-ai from Anthropic. Deciding about AI2Bot is usually a decision about the whole group, so check how your file treats these too.

We first recorded AI2Bot on March 17, 2026. Allen Institute for AI documents it on allenai.org, and we recheck that page before each release.

Which rule AI2Bot actually follows

AI2Bot reads exactly one group in robots.txt, the most specific one that names AI2Bot, and never combines rules across groups. This short file shows the trap:

User-agent: *
Disallow: /private/

User-agent: AI2Bot
Disallow: /drafts/

Find the group. AI2Bot looks for a group naming AI2Bot. One exists, so it ignores User-agent: * completely.
Apply only that group. It is refused /drafts/ and nothing else.
The surprise. /private/ is open to AI2Bot, because that rule lives in the * group it no longer reads. To keep it out of both, repeat Disallow: /private/ inside its own group.

Test yourself: does a rule under User-agent: * apply to AI2Bot if the file also has a User-agent: AI2Bot group?

No. Once a group names it, AI2Bot reads only that group. Put every rule you want it to follow inside its own group.

Test yourself: will blocking AI2Bot remove your pages from AI answers?

No. AI2Bot collects training data. Blocking it changes what future models learn, not which pages AI answers cite today.

Test yourself: is User-agent: ai2bot the same as User-agent: AI2Bot?

Yes. RFC 9309 makes user-agent matching case-insensitive. Paths are a different story: Disallow: /Blog/ does not block /blog/.

Should you allow AI2Bot? Decide by situation

Recommended setting for AI2Bot by type of site
If you are...	Recommended setting
You publish content and want AI answers to cite it	Your call. AI2Bot is a training crawler, so blocking it does not remove you from AI answers; retrieval crawlers do that job.
You license content and want to stop AI training	Block it by name with `User-agent: AI2Bot` and `Disallow: /`.
You run an online shop	Allow it unless your logs show it is costly.
Your server is under load from bots	Rate-limit before blocking. It is rated safe in our database.

Check AI2Bot on your own site

Open /robots.txt on your domain and search for AI2Bot. If it is not named, it follows User-agent: *.
List the Disallow lines in the group AI2Bot follows and test your key URLs against them; the longest match wins, and Allow wins a tie.
Look beyond the file. A firewall or bot manager can refuse AI2Bot even when robots.txt allows it, and that never shows in the file.
Run our blocked AI crawler check to see how AI2Bot and every other crawler we track are treated in one pass, including firewall refusals.

Common mistakes with AI2Bot

The errors we see most often with AI2Bot in the files and firewalls we test:

Assuming that blocking AI2Bot removes you from AI answers. It is a training crawler; AI answers are fed by retrieval and user-triggered crawlers with different tokens.
Blocking AI2Bot with a broad User-agent: * rule that also catches search engines. Name AI2Bot in its own group instead.
Expecting a AI2Bot block to remove content already collected. robots.txt only affects future visits.

All three hide inside a file that looks correct, so check AI2Bot from outside after every change.

Spotting AI2Bot in your logs

robots.txt says what AI2Bot may do; your access logs show what it actually did.

Count requests

On a server with standard Nginx or Apache logs, this counts matching requests per day (adjust the log path for your host):

grep -iE "AI2Bot" /var/log/nginx/access.log \
  | awk '{print $4}' | cut -d: -f1 | sort | uniq -c

And this shows which paths it requested most, which is the quickest way to see whether your Disallow lines are being respected:

grep -iE "AI2Bot" /var/log/nginx/access.log \
  | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

Check the requests are genuine

Anyone can send the AI2Bot user-agent, so confirm the source before acting on a match:

AI2Bot: we hold no published IP list, so use reverse DNS where the operator documents a hostname and treat an unverifiable AI2Bot request with caution.

Frequently Asked Questions

What is the official user-agent string for AI2Bot?

The official user-agent string for AI2Bot is: AI2Bot. This is the exact string you must use in robots.txt, Nginx, Apache, or Cloudflare firewall rules to target this bot. User-agent matching in robots.txt is case-insensitive, but the string must be spelled correctly. You can verify that a request genuinely comes from AI2Bot by performing a reverse-DNS lookup on the source IP - legitimate bots resolve back to their operator's domain.

Is AI2Bot safe?

Yes. AI2Bot is a documented bot: Allen Institute for AI publishes a page that names it and explains what it does.

Will blocking AI2Bot hurt my SEO?

No effect on Google or Bing.

How do I block AI2Bot in robots.txt?

Add the following lines to your /robots.txt file:

User-agent: AI2Bot
Disallow: /

This instructs AI2Bot not to crawl any path on your site. The Disallow: / directive covers the entire domain including subfolders. To only block specific sections, replace / with the path (e.g., Disallow: /blog/). Note: robots.txt is publicly readable - any bot or human can inspect it at yourdomain.com/robots.txt.

Does AI2Bot respect robots.txt?

Yes. AI2 documents robots.txt control on its crawler page.

How do I verify if AI2Bot is crawling my site?

Search your web server access logs for the string AI2Bot (case-insensitive grep: grep -i "AI2Bot" /var/log/nginx/access.log). You can also check Google Search Console → Coverage → Crawl Stats for Googlebot variants. For AI2Bot specifically, filter by user-agent in your log analysis tool (GoAccess, AWStats, etc.).

What is the crawl frequency of AI2Bot?

AI2Bot crawls at a moderate rate. If you notice excessive traffic in your logs, you can add a Crawl-delay directive:

User-agent: AI2Bot
Crawl-delay: 10

(10 second delay between requests).

Can I block AI2Bot from specific pages only?

Yes. Instead of a global Disallow: / you can restrict AI2Bot to specific paths:

User-agent: AI2Bot
Disallow: /private/
Disallow: /staging/
Allow: /

This allows AI2Bot everywhere except the listed paths. Path matching in robots.txt uses prefix matching - Disallow: /private/ blocks /private/page.html but NOT /public/private/.

Does blocking AI2Bot prevent AI training on my content?

A robots.txt Disallow for AI2Bot is the opt-out Allen Institute for AI documents. Robots.txt is voluntary, so for enforcement add a firewall or CDN rule. Content may be used in open research datasets and models.

Is there an alternative to robots.txt to opt out of AI2Bot?

The opt-out Allen Institute for AI documents is robots.txt. A firewall, CDN or server rule that returns 403 for AI2Bot enforces it. Per-bot meta tags and llms.txt are not opt-out mechanisms any major operator documents.