Apple

Common mistakes with Apple's crawlers

The errors we see most often with Apple's crawlers in the files and firewalls we test:

1. Blocking Apple's crawlers to save bandwidth and then wondering why pages drop out of that search engine. Use crawl-rate settings or caching first.
2. Writing a User-agent: Applebot group to add one rule for Apple's crawlers and accidentally dropping every rule it used to follow under User-agent: *.
3. Disallowing CSS and JavaScript folders, which can stop Apple's crawlers rendering your pages properly.

All three hide inside a file that looks correct, so check Apple's crawlers from outside after every change.

What our own data shows about Apple

Everything below comes from the AI Crawler Check bot database and our two most recent robots.txt scans of the Tranco top 10,000, not from Apple's marketing. It covers 2 Apple tokens in turn.

Applebot in our records

We first recorded Applebot on March 17, 2026. Apple documents it on support.apple.com, and we recheck that page before each release. Between our scans of October 17, 2026 and October 20, 2026, files naming Applebot went from 239 to 250 (+11) and files blocking it from 265 to 283 (+18), so more sites wrote a rule for Applebot and more of those rules block it.

Among the 63 search engines in our database, Applebot sits fourth by priority (Tier 3), level with AdIdxBot, Amzn-SearchBot, baidu, Baiduspider. 62 of the 63 share its safe rating. Other operators with a crawler that builds a search or answer index: AddSearchBot (AddSearch, Tier 2), AIWebIndex (Lyrenth, Tier 2), atlassian-bot (Atlassian, Tier 2), AzureAI-SearchBot (Microsoft, Tier 2), Cloudflare-AI-Search (Cloudflare, Tier 2). A policy for Applebot is usually a policy for these too.

A decision record for Applebot

Teams that review crawlers once a quarter keep one short entry per token, so anyone can see why a rule exists and when it was last checked. Filled in from our data, the Applebot entry reads:

• Operator and job: Apple; it builds a search or answer index.
• Priority and behaviour: Tier 3, rated safe, blocking impact critical.
• Suggested starting point: Allow by default; revisit if it costs you bandwidth.
• Enforcement: Applebot's operator says it follows robots.txt, so a correctly written Applebot group is enough to control it.

User-agent: Applebot
Allow: /

Paste the line into the validator with one of your own URLs before it goes live. Revisit the Applebot entry once a quarter, or when Apple publishes a new token next to it.

Applebot-Extended in our records

We first recorded Applebot-Extended on March 17, 2026. Apple documents it on support.apple.com, and we recheck that page before each release. Between our scans of October 17, 2026 and October 20, 2026, files naming Applebot-Extended went from 556 to 600 (+44) and files blocking it from 643 to 694 (+51), so more sites wrote a rule for Applebot-Extended and more of those rules block it.

Among the 63 search engines in our database, Applebot-Extended sits first by priority (Tier 2), alone in its tier. 1 of the 63 share its caution rating. Other operators with a crawler that collects pages to train AI models: ClaudeBot (Anthropic, Tier 1), Google-Extended (Google, Tier 1), GPTBot (OpenAI, Tier 1), Meta-ExternalAgent (Meta, Tier 1), AI2Bot (Allen Institute for AI, Tier 2). A policy for Applebot-Extended is usually a policy for these too.

A decision record for Applebot-Extended

Teams that review crawlers once a quarter keep one short entry per token, so anyone can see why a rule exists and when it was last checked. Filled in from our data, the Applebot-Extended entry reads:

• Operator and job: Apple; it collects pages to train AI models.
• Priority and behaviour: Tier 2, rated caution, blocking impact critical.
• Suggested starting point: Disallow only if you want to refuse that use of your content.
• Enforcement: Applebot-Extended is a control token, not a crawler, so it never appears in your logs. You set Applebot-Extended in robots.txt and the operator reads that setting through its other crawlers.

User-agent: Applebot-Extended
Disallow: /

Paste the line into the validator with one of your own URLs before it goes live. Revisit the Applebot-Extended entry sooner than most: a caution rating means Apple traffic deserves a monthly look at your logs.

How top sites treat Apple's crawlers

From our robots.txt study of the Tranco top 10,000 (5,583 valid files, fetched October 20, 2026):

• Applebot-Extended, the most blocked of the Apple's crawlers tokens: blocked in 12.4% of files (478 with its own group, 216 through User-agent: *), more often by the top 1,000 (17.6%) than the long tail (11.3%).
• Applebot, second of the Apple's crawlers tokens: blocked in 5.1% of files (90 with its own group, 193 through User-agent: *), more often by the top 1,000 (6.4%) than the long tail (4.7%).

For scale, 20.1% of all files block at least one AI training crawler. Applebot-Extended is one of the tokens driving that figure.

More questions about Apple

Will blocking Apple affect my Google rankings?

Only if one of its tokens is a Google search crawler. None of Apple's tokens are Googlebot, so classic Google rankings are unaffected.

How quickly does a robots.txt change take effect for Apple?

Crawlers cache robots.txt, commonly for up to a day. Expect a change to apply within about 24 hours, and confirm it in your logs.

Can I allow Apple on some sections and block it on others?

Yes. Inside the group for Applebot, add Disallow lines for the paths to keep private and Allow lines for exceptions. The longest matching rule wins.

Does blocking Apple delete content it already collected?

No. robots.txt only controls future visits. For content already collected, check whether Apple offers a removal or opt-out process.

Is a firewall block better than robots.txt for Apple?

A firewall enforces the decision, while robots.txt asks politely. Many operators recommend robots.txt because blocking their IPs can stop them reading your rules at all. Use robots.txt first, and add a firewall rule only if requests continue.