Wikipedia Leaves the Door Open to AI Crawlers

Every other site in this series uses robots.txt to keep AI out. Wikipedia does the opposite. Its file is 712 lines long, the longest we tested, and yet it blocks 0 of 97 AI crawlers. We ran our live AI crawler test on en.wikipedia.org to show what an open policy looks like, why Wikipedia chose it despite the cost, and how it manages heavy crawling without closing the door.

• 73/100AI Visibility Score (Good)
• 0/97AI crawlers blocked
• 0/14Tier 1 bots blocked
• 1/257All bots blocked
• 55/100GEO audit score

The short version

en.wikipedia.org allows 256 of the 257 bots we tested. Every AI crawler, search crawler and user-triggered agent can read article pages, from GPTBot and ClaudeBot to PerplexityBot and Google-Extended. The only bot from our list that the file blocks completely is MJ12bot, and the file explains why: it was "observed spamming large amounts" of URLs and "ignoring 429 ratelimit responses". Wikipedia blocks behaviour, not purpose.

Its AI Visibility Score is 73/100 (Good), the highest of the five sites. The remaining points are lost on things like a missing homepage sitemap and llms.txt, not on access.

A file written for crawl load, not for AI policy

Most of Wikipedia's 712 lines are not about AI at all. They are about server load. Three kinds of rules make up the file:

• Named blocks for misbehaving tools. Site copiers and offline downloaders (HTTrack, WebCopier, WebZIP, wget) and bots that ignored rate limits get Disallow: /, each with a comment explaining the reason.
• A crawl delay for a heavy SEO crawler. SemrushBot gets Crawl-delay: 5 with a note that this keeps it near 20 requests a second across all Wikimedia sites.
• A wildcard group that protects expensive pages. User-agent: * blocks /w/, /api/, /trap/ and the Special: pages in dozens of languages. Those are dynamically generated and uncached, so they cost far more to serve than an article. Article pages under /wiki/ stay open.

The comment above the wildcard group sums up the philosophy: "Friendly, low-speed bots are welcome viewing article pages, but not dynamically-generated pages please." There is no AI section, because Wikipedia's content is already freely licensed under Creative Commons. Anyone may reuse it with attribution. Blocking AI in robots.txt would contradict the licence.

Crawler by crawler

Every row is allowed via the wildcard group. The "path rules" in each entry are the expensive-page blocks described above, which apply to every crawler equally.

The price of staying open

Openness is not free. In April 2025 the Wikimedia Foundation reported that bandwidth for multimedia downloads had grown 50% since January 2024, and that the growth came mainly from automated scrapers rather than people. It found that at least 65% of its most resource-intensive traffic came from bots, even though bots made up about 35% of pageviews. Bots tend to request rare, uncached pages that people seldom visit, which is why they cost so much more.

In October 2025 the Foundation added that human pageviews had fallen about 8% year on year, partly because people now get answers from AI summaries and search features built on Wikipedia's content without visiting the site. That is the same pressure the New York Times responded to by closing the door. Wikipedia responded differently.

Related: writing content AI engines can cite

Open by default, paid for scale

Rather than block AI crawlers, Wikimedia sells a better way in. Wikimedia Enterprise offers companies high-volume, reliable access to the content through an API, without scraping. In January 2026 the Foundation announced Amazon, Meta, Microsoft, Mistral AI and Perplexity as Enterprise partners, joining Google and others. The content stays free for everyone. Companies that need it at industrial scale pay for an efficient feed instead of hammering the public servers.

Compare this with Reddit, which also monetises AI access: Reddit closes its public file and sells the only door, while Wikipedia leaves the public door open and sells a faster one. Both work because both have content AI companies need. The lesson for most sites is that you do not have to choose between "blocked" and "unmanaged". You can be open to everyone and still protect the pages that are expensive to serve.

What our Googlebot and GEO tests saw

• Passrobots.txt allows GooglebotNo rule in the User-agent: * group matches /
• FailPage answers GooglebotHTTP 403 to the Googlebot smartphone user agent. A firewall, CDN or rate limit is refusing it.
• PassNo challenge or block pageThe response is the real page, not a challenge
• Not readPage is indexable (no noindex)Page not read
• Not readCanonical points to this URLPage not read
• Not readContent is in the HTML before JavaScriptPage not read

Tool verdict: Not crawlable: the page returns HTTP 403 to Googlebot.

Our homepage request carrying Googlebot's user agent was refused with HTTP 403. As with nytimes.com and reddit.com, that is because the request came from our server, not from Google's IP ranges. Wikimedia's infrastructure rejects unverified bot traffic that claims to be a major crawler, which fits its 2025 move to rate-limit and block abusive scrapers. Real Googlebot is unaffected.

The GEO audit scores 55. Access is strong at 88, while content and structured-data scores are low because the audited URL is the language portal, not an article, and our request was refused at the edge. Wikipedia articles themselves are among the most cited sources in AI answers precisely because they are open, structured, sourced and constantly updated.

What any site can copy from Wikipedia

1. Block bad behaviour, not whole categories. A bot that ignores 429 responses deserves a block. A bot that reads politely and cites you does not.
2. Protect expensive paths. Search results, filters, faceted navigation and admin areas are costly to serve and worthless to index. Disallow them for every crawler.
3. Explain your rules in comments. Wikipedia's file reads like documentation. Crawler operators can see what you want and why.
4. Use Crawl-delay where it is respected. Some crawlers, including Bing and several SEO tools, honour it. Google does not, so manage Googlebot's rate in Search Console instead.
5. Offer a better door if you have scale. A feed, API or sitemap that serves content cheaply is better than a fight with scrapers.

Sources: robots.txt and homepage fetched by AI Crawler Check on 17 October 2026. Crawler impact: Wikimedia Diff, 1 April 2025. Pageview trends: Wikimedia Diff, 17 October 2025. Enterprise partners: Wikimedia Enterprise, 15 January 2026.