AI Crawler and AI Search News: October 2026 Roundup

October 2026 is a month of consolidation rather than surprises. Google's two big AI controls for site owners, the generative AI performance report and the Search generative AI control, have now been live for every site since 31 August. The September spam update finished on 8 October. The IETF published new drafts for AI usage preferences. And our own scan of the 10,000 most visited sites shows how slowly robots.txt files keep up with all of it.

This roundup covers what changed, what it means for AI visibility, and where to read more. Every item links to a primary source or to our own data, and each one ends with the action we suggest.

Google: AI features, now fully measurable and switchable

Since 31 August 2026, two Search Console features are available to every site worldwide. The generative AI performance report shows impressions in AI Overviews and AI Mode by page, country, device and date. It shows no clicks and no queries. The Search generative AI control, under Settings, lets a property include or exclude its content from AI Overviews, AI Mode and Discover's AI features, or inherit the choice from a parent property.

Two points cause the most confusion. The control does not affect training; Google-Extended in robots.txt does that. And Google says the control is not used as a ranking signal for other parts of Search. Our guides to reading the generative AI report and the three Google AI switches cover both in detail.

Action: check the control on your highest-level property, then record last month's AI impressions as a baseline.

Google: September 2026 spam update complete

The Google Search Status Dashboard lists the September 2026 spam update as released on 24 September at 09:15 Pacific, applying "globally and to all languages", and complete as of 8 October. That is about two weeks, in line with Google's estimate.

Spam updates move AI visibility as well as classic rankings, because AI Overviews draw on the same index. If your AI impressions changed between those dates, compare the timing with the update before you blame anything else. If they changed at another time, check crawl access first; a firewall or CDN change is a more common cause than people expect.

Action: annotate 24 September to 8 October in your reporting, and compare AI impressions before and after.

Search Console: multimodal search data

On 24 September, Google's multimodal reporting post announced a new web multimodal search type filter in the performance reports, including the generative AI report. It covers searches made with images: Google Lens, Circle to Search on Android, image uploads and Chrome's "Search this image".

For shops, travel, food, fashion, home and anything people photograph, this is worth a look. A product that is often searched by photo may get AI impressions you never saw in text queries. Good product images, clear alt text and accurate structured data all help here.

Action: switch the search type to multimodal in both reports and note which pages appear.

Google: learning paths and a policy change in the EEA

On 6 October, Google published two curated YouTube playlists, Search fundamentals for beginners and Technical SEO, announced in Google's learning paths post. They are a useful free resource for training junior team members or clients.

Separately, Google's site reputation update on 28 August explained that from 30 August, manual actions under the site reputation abuse policy no longer affect results for users searching inside the European Economic Area. Instead, the affected section may be separated so that, over time, it ranks independently. Outside the EEA, the manual action applies as before. If you work with sites that host third-party content on a strong domain, note the difference between markets.

Crawler infrastructure: IP lists and new paths

Verifying a crawler by IP is only as good as the list you check against, and those lists move. Google's IP range move said its crawler IP files are moving from /search/apis/ipranges/ to /crawling/ipranges/, with the old path kept for now. We updated our own directory links to the new location this month.

Related: how AI engines choose their sources

If your firewall allows or blocks crawlers by IP, refresh these lists on a schedule. A stale allow list can block a real crawler as easily as a stale block list can let a fake one in. Our guide to verifying AI bots with reverse DNS explains the method, and why AI fetchers may ignore robots.txt covers the user-triggered fetchers, which several companies say may not follow robots.txt.

Action: if you use IP rules, confirm they pull from the current URLs.

Standards: AIPREF moves forward

The IETF AIPREF working group's attachment draft, draft-ietf-aipref-attach-05, defines a Content-Usage line for robots.txt and an HTTP header with the same name. Values such as train-ai=n state what may happen to content after it is fetched. Two rules stand out: usage preferences only apply to pages a crawler may fetch, and when statements conflict the strictest one wins. Cloudflare's Content Signals offer a similar idea with an extra ai-input category.

Read our Content-Usage explainer and Content-Signal vs Content-Usage for the details and a robots.txt that uses both.

Action: keep crawler-token rules as your main control; add usage lines if you want your position on record.

Publishers: payment and licensing experiments

Google is testing payments to about 100 invited publishers when their content contributes significantly to AI answers in AI Overviews, AI Mode and the Gemini app, according to reporting by The Information cited by Ars Technica. Reported payments are small for most participants. Separately, Google's News AI pilot update says more than 400 publications now take part in its News AI licensing programme. Our guide to Google's AI contribution pilot covers what is known and what it means for everyone else.

Our data this month

We fetched robots.txt from the Tranco top 10,000 on 17 October 2026 and parsed 5,254 valid files. A few numbers from that scan that matter for AI visibility:

• 4,195 files leave every AI crawler we track open. Most of the web has still not made an explicit AI decision.
• 457 files block at least one training crawler while leaving every AI search crawler we track open: the "cite me, do not train on me" position.
• 592 files block both training and at least one AI search crawler, which removes them from some AI answers.
• 529 files block Google-Extended and keep Googlebot allowed.
• 629 files name at least one stale AI token; see the stale token audit.

We also published a method for measuring your own crawl-to-referral ratio, so you can test the headline numbers against your own logs.

Also published this month

We published eight in-depth guides alongside this roundup.

• our Content-Usage explainer: the IETF's Content-Usage line and header, with worked examples.
• Content-Signal vs Content-Usage: Cloudflare's format against the IETF's, and a file that uses both.
• reading the generative AI report: what Google's AI impressions mean and a monthly routine.
• the three Google AI switches: the Search generative AI control, Google-Extended and noindex.
• why AI fetchers may ignore robots.txt: what Perplexity, OpenAI, Google and Anthropic promise about user fetchers.
• measuring your own crawl-to-referral ratio: a step-by-step method using your logs and analytics.
• the stale token audit: eight outdated names and what replaced them.
• Google's AI contribution pilot: what is known about Google paying publishers for AI answers.

The data behind every count in these guides comes from one scan of the Tranco top 10,000 on 17 October 2026, parsed with an RFC 9309 parser. Our methodology page explains how we collect and check it.

What these changes add up to

Put the month's news together and a clear pattern appears. Control over AI use is moving from one blunt switch to several precise ones, each tied to a specific product or use.

That precision is good news, but it creates a new risk: settings spread across Search Console, robots.txt, CDN dashboards and HTTP headers, often owned by different people. The most common failure we see is not a wrong decision but a decision recorded in one place and contradicted in another. A one-page AI access policy, kept with your robots.txt, solves most of it. Our robots.txt generator gives you the robots.txt part; the rest is a short document listing each switch, its setting and who owns it.

A note on how we verify news

AI search news spreads fast and is often wrong in the details. Before we include an item, we read the primary source: the company's own documentation, the standards body's draft, or the Search Status Dashboard. Where only reporting exists, as with Google's payment pilot, we name the outlet and the figures it reported, and we say that the company has not confirmed them. If you spot an error in this roundup, tell us through the contact page and we will correct it with a dated note.

What to watch in November

• AIPREF drafts. The vocabulary still lacks working group consensus. Watch for changes to category names, and for an ai-input-like category.
• Google's covered features. Google says the list of AI features covered by the report and the control will change over time. Check the help pages monthly.
• IP list changes. The old Google path will eventually redirect; make sure your automation follows redirects or uses the new path.
• New crawlers. New AI tokens keep appearing. Our new AI bots watchlist and crawler directory track them.
• Publisher payment programmes. Any widening of Google's AI contribution pilot would be significant news for publishers.

For agencies: a client update you can send

Here is a short, accurate update that works for most clients this month. "Google now lets every site measure and control its appearance in AI Overviews and AI Mode through Search Console. We have checked your setting and recorded your AI impressions as a baseline. Google's September spam update finished on 8 October; any movement in that window is likely related. We have also reviewed your robots.txt against current AI crawler names and fixed outdated entries. No action is needed from you." Adjust the last two sentences to what you actually did. Our case studies show the kinds of problems these reviews usually find.

Your October checklist

• □Run AI Crawler Check on your homepage and one key landing page.
• □Check the Search generative AI control on every Search Console property you manage.
• □Export last month's generative AI report, including the multimodal filter.
• □Mark 24 September to 8 October in your reporting as the spam update window.
• □Search your robots.txt for stale tokens and add their current replacements.
• □If you verify crawlers by IP, confirm your lists use the current URLs.
• □Decide whether to add a Content-Usage or Content-Signal line.
• □Schedule the same review for November, using our monthly AI crawler audit.