Reddit Blocks Every Crawler. So Why Is It in Google?

Reddit's robots.txt is 9 lines long and says one thing: User-agent: * followed by Disallow: /. Read literally, that blocks every crawler on the internet, Google included. Yet Reddit threads rank in Google every day and turn up in Google's AI answers. We ran our live AI crawler test on reddit.com to explain the gap, because it is the clearest example on the web of a site where robots.txt is a policy statement and the real access control lives somewhere else.

• 3/100AI Visibility Score (Critical)
• 97/97AI crawlers blocked
• 14/14Tier 1 bots blocked
• 257/257All bots blocked
• 25/100GEO audit score

The short version

Our checker reads the file exactly as a well-behaved crawler would, and the verdict is total: 257 of 257 bots blocked, every category at 100%, an AI Visibility Score of 3/100. On paper reddit.com is invisible to Google, Bing, ChatGPT, Claude, Perplexity and everyone else.

In practice, Reddit is one of the most visible sites in search. The explanation is commercial. In February 2024 Reddit signed an AI content licensing agreement with Google, reported at about $60 million a year, and in May 2024 it announced a data partnership with OpenAI. Partners get Reddit content through licensed access rather than by crawling under robots.txt rules. When Reddit published the blanket file in mid-2024, it said it would keep rate-limiting and blocking unknown bots that do not have an agreement. The public file is a "no" to everyone. Contracts are the "yes".

The whole file

This is the complete content we fetched, minus nothing:

There are no named groups and no exceptions, and the file declares no sitemap. Every bot, whatever its purpose, gets the same answer. That simplicity is deliberate. A list of named bots invites the question "why them and not us". A single wildcard rule says the decision is not made in this file at all.

What a rule-following crawler sees

Every row says blocked and every source says User-agent: *, including Googlebot. For any other site this would be a disaster. It is exactly the defect we warn about in the robots.txt mistakes that block AI: two lines copied from a forum that remove a site from every search engine at once.

The edge agrees with the file

Robots.txt is voluntary. To see whether Reddit enforces it, our tool fetched the homepage as Googlebot from our own server.

• Failrobots.txt allows GooglebotDisallow: / (line 8, the User-agent: * group)
• FailPage answers GooglebotHTTP 403 to the Googlebot smartphone user agent. A firewall, CDN or rate limit is refusing it.
• PassNo challenge or block pageThe response is the real page, not a challenge
• Not readPage is indexable (no noindex)Page not read
• Not readCanonical points to this URLPage not read
• Not readContent is in the HTML before JavaScriptPage not read

Tool verdict: Not crawlable: robots.txt blocks Googlebot from this URL.

The request was refused with HTTP 403. When we fetched robots.txt itself with an ordinary browser user agent, we did not get the file. We got a page titled "Blocked" that said our request had been blocked "due to a network policy". Reddit is not relying on crawlers to obey the file. Unrecognised automated traffic is stopped at the network edge before it reaches a page. Reddit's own files (llms.txt and llms-full.txt) also returned 403, not 404, which is the same wall again.

So how does Google index Reddit? Google can tell its real crawler apart from our imitation: real Googlebot requests come from Google's published IP ranges and pass a reverse DNS check, and Reddit's network can recognise them. Independent testing published in 2024, after the file changed, found that Reddit served different robots.txt responses to different requesters, a pattern the testers described as more of a public stance than a practical restriction. We could only test what an outside crawler receives, and that is a firm no.

The GEO audit, and why it bottoms out

The audit caps at 25 because, as it reports, "robots.txt blocks both search engines and AI search bots from this URL." With robots.txt and the edge both refusing, no content, schema or headings could be read. That is the correct score for what an unlicensed crawler can see.

Why Reddit can do this and you cannot

Reddit's content is mostly written by users, it is in very high demand as AI training data, and the company had a small number of large buyers ready to pay for it. Put those together and blocking everyone is a sales strategy: scarcity in the public file creates value in the private licence. Reddit's 2024 filings described its data licensing agreements as worth more than $200 million in total.

Almost no other site has that position. If your robots.txt says User-agent: * / Disallow: /, you do not have a licensing strategy. You have a site that Google, Bing and every AI assistant has been told to ignore, and nobody is paying you for the privilege. We see this exact file on staging sites pushed to production, on WordPress installs with "discourage search engines" ticked, and on sites where a developer copied Reddit's file believing it was best practice.

Five lessons from Reddit's file

1. Robots.txt is not access control. It tells polite crawlers what you want. Enforcement happens at your CDN, firewall or rate limiter. Reddit has both, and they agree.
2. A blanket Disallow is only safe if you have another way in for the crawlers you need. Reddit's way in is contracts and verified access. Without that, you are simply deindexed.
3. Policy belongs in comments too. Reddit's file links its Public Content Policy and a research programme, so anyone who reads the file knows where permission comes from.
4. Test from outside. What your team sees in a browser is not what a crawler sees. Reddit's file is only readable to some requesters, and our browser fetch hit a block page.
5. Do not copy famous files. A big brand's robots.txt reflects its own contracts. Start from your goals, then build a file for them.

Sources: robots.txt, homepage and llms.txt requests made by AI Crawler Check on 17 October 2026. Google agreement: Reuters, 22 February 2024. Robots.txt change and bot policy: TechCrunch, 25 June 2024. Response differences: MERJ, July 2024.