AI search optimization, from eligibility to control

What changed in 2026

1. 3 June 2026

   Search Console begins showing a generative AI performance report and testing an AI opt-out, the same day a UK regulator requires one.

2. 31 August 2026

   Google confirms the site-level "Search generative AI" setting has rolled out to all websites worldwide.

3. March 2027

   The UK regulator's deadline for Google to add page-level controls.

It works on the whole site at once, and Google states that choosing to exclude a site has no effect on how it ranks in regular results. The report shows impressions by page, country, device and date, without clicks or queries. Snippet-level controls are compared on adult AEO.

Three kinds of AI crawler

The major AI companies now separate their bots by job: one collects training data, one builds the index their search features cite, and one fetches a page when a user asks the assistant to read it. Each needs its own decision, and engines differ in which sources they lean on, as our note on how AI engines cite adult brands shows.

The user-requested column behaves differently. OpenAI's documentation says robots.txt rules may not apply to ChatGPT-User, because a person started the visit; Anthropic's documentation says all three of its bots follow robots.txt; Perplexity treats its user fetches as user-initiated too. If something must never be read by any assistant, robots.txt is the wrong tool: it has to sit behind a login. Bot names and rules change, so we recheck this table against each company's documentation every quarter.

Checking that the rules actually work

1. Read your server logs for the bots above, and confirm which ones visit and which pages they request.
2. Check that a visitor claiming to be a bot really is one: OpenAI publishes the IP ranges its crawlers use, and Google documents how to verify Googlebot.
3. Look at your CDN and firewall. Bot protection settings can block AI crawlers before robots.txt is ever read, and some providers now offer AI crawler blocking as a simple switch.
4. Ask each assistant to summarize one of your public pages and see whether it can; the full list of checks is in our AI search readiness checklist.

In our audits, a forgotten firewall rule is a more common reason for being invisible to AI search than anything in robots.txt.

When crawlers arrive faster than your server likes

AI crawlers can request pages in bursts, and on a small store or membership site that can slow things down for real visitors. Blocking them outright solves the load problem by creating a visibility problem. There are gentler tools. Anthropic documents that its bots respect the non-standard Crawl-delay rule in robots.txt; Google ignores that rule and adjusts its own pace when a server slows or returns errors. Caching public pages, serving them from a CDN and returning a proper 429 or 503 status under strain, rather than an error page with a 200 status, all help crawlers back off without forgetting you.

We look at crawler load in the logs before recommending any block, and when a block is right we make it as narrow as possible: one bot, one section, for as long as needed. If you want us to look at your own logs, request a readiness audit.

Questions