GPTBot vs OAI-SearchBot: Which AI Crawler Should You Allow or Block?

Mistakes when writing rules for GPTBot and OAI-SearchBot together

GPTBot is named in 18.1% of the top-site robots.txt files in our study and blocked in 15.4%; OAI-SearchBot is named in 10.3% of the top-site robots.txt files in our study and blocked in 7.8%. Large sites block GPTBot more often, which usually reflects a decision about training rather than about OAI-SearchBot.

1. They are different kinds of agent (a training crawler versus a retrieval crawler that feeds AI search answers), so the usual error is giving them the same rule out of habit. Blocking both to stop training only needs the GPTBot line; the other one changes something else entirely.
2. Both come from OpenAI. Operators change token names and add new agents, so a rule that covers GPTBot and OAI-SearchBot today can miss the next OpenAI agent; recheck after each announcement.
3. Putting User-agent: GPTBot and User-agent: OAI-SearchBot in separate groups and then editing only one. Two groups drift apart over time; keep a comment above each that says why it differs from the other.

Each crawler's own profile lists the mistakes that apply to it alone; this list is only about combining the two.

Spotting GPTBot and OAI-SearchBot in your logs

robots.txt says what GPTBot and OAI-SearchBot may do; your access logs show what they actually did.

Count requests

On a server with standard Nginx or Apache logs, this counts matching requests per day (adjust the log path for your host):

grep -iE "GPTBot|OAI-SearchBot" /var/log/nginx/access.log \
  | awk '{print $4}' | cut -d: -f1 | sort | uniq -c

And this shows which paths it requested most, which is the quickest way to see whether your Disallow lines are being respected:

grep -iE "GPTBot|OAI-SearchBot" /var/log/nginx/access.log \
  | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

Check the requests are genuine

Anyone can send these user-agents, so confirm the source before acting on a match:

• GPTBot: compare each address with the ranges at openai.com/gptbot.json. A request claiming to be GPTBot from outside them is not genuine. Check it separately from OAI-SearchBot.
• OAI-SearchBot: compare each address with the ranges at openai.com/searchbot.json. A request claiming to be OAI-SearchBot from outside them is not genuine. Check it separately from GPTBot.

Related: writing content AI engines can cite

How GPTBot and OAI-SearchBot moved in our latest scans

Between our scans of October 17, 2026 and October 20, 2026, files naming GPTBot went from 941 to 1,008 (+67) and files blocking it from 789 to 859 (+70), so more sites wrote a rule for GPTBot and more of those rules block it. For comparison, 2026 and October 20, 2026, files naming OAI-SearchBot went from 549 to 574 (+25) and files blocking it from 416 to 434 (+18), so more sites wrote a rule for OAI-SearchBot and more of those rules block it.

Over the 3 days between those scans, blocking of GPTBot moved +70 files and blocking of OAI-SearchBot moved +18. Owners are acting faster on GPTBot, which is the one to recheck first if you have not touched your file this month.

We first recorded GPTBot on March 17, 2026. OpenAI documents it on developers.openai.com, and we recheck that page before each release. We first recorded OAI-SearchBot on March 17, 2026. OpenAI documents it on developers.openai.com, and we recheck that page before each release.

Among the 104 AI and LLM crawlers in our database, GPTBot sits ninth by priority (Tier 1), level with ChatGPT Agent, ChatGPT Operator, ChatGPT-User, ChatGPT-User/2.0. 81 of the 104 share its safe rating. On the OAI-SearchBot side: Among the 104 AI and LLM crawlers in our database, OAI-SearchBot sits 13th by priority (Tier 1), level with ChatGPT Agent, ChatGPT Operator, ChatGPT-User, ChatGPT-User/2.0. 81 of the 104 share its safe rating.

How top sites treat GPTBot and OAI-SearchBot

From our robots.txt study of the Tranco top 10,000 (5,583 valid files, fetched October 20, 2026):

• GPTBot, the most blocked of the GPTBot and OAI-SearchBot tokens: blocked in 15.4% of files (663 with its own group, 196 through User-agent: *), more often by the top 1,000 (20.4%) than the long tail (14.5%).
• OAI-SearchBot, second of the GPTBot and OAI-SearchBot tokens: blocked in 7.8% of files (234 with its own group, 200 through User-agent: *), more often by the top 1,000 (12.2%) than the long tail (6.6%).

For scale, 20.1% of all files block at least one AI training crawler. GPTBot is one of the tokens driving that figure.

More questions about GPTBot and OAI-SearchBot

Can I allow one and block the other?

Yes. Give each its own User-agent group with its own rules. Each crawler reads only the group that names it.

Which of the two visits more often?

It depends on your site, your audience and how often your content changes. Your access logs are the reliable answer; the commands above count both.

How long until a change applies?

Crawlers commonly cache robots.txt for up to a day, so allow about 24 hours and confirm in your logs.

Do I need to update my rules when either operator launches a new crawler?

Often, yes. A new token is not covered by rules written for the old ones. Check this comparison and both profiles periodically, or rerun the AI bot crawlability test after any announcement.