change(robots-for-ai-crawlers): cover the shared AI training preference

src/content/changelog/2026-11-02-shared-ai-training-preference.md

@@ -0,0 +1,8 @@
+---
+title: "AI crawlers: a shared robots.txt training preference is on the way"
+date: "2026-11-02"
+type: changed
+relatedSlugs: [robots-for-ai-crawlers, content-signals]
+---
+
+The [robots.txt for AI crawlers page](/spec/agent-readiness/robots-for-ai-crawlers/#a-shared-training-preference) now covers the `__usage_ai-training` token, an Internet-Draft from 30 October 2026 that Microsoft and Google plan to honour in early 2027. The [Content Signals page](/spec/agent-readiness/content-signals/) now makes clear that those commitments are to this token, not to the `Content-Signal` directive.

src/content/spec/agent-readiness/content-signals.md

@@ -7,7 +7,7 @@ status: optional
 order: 42
 appliesTo: [all]
 relatedSlugs: [robots-for-ai-crawlers, robots-txt, tdmrep, agent-readiness-overview, web-bot-auth]
-updated: "2026-09-26T00:00:00.000Z"
+updated: "2026-11-02T00:00:00.000Z"
 sources:
   - title: "IETF AI Preferences WG (aipref) — drafts"
     url: "https://datatracker.ietf.org/wg/aipref/documents/"
@@ -57,11 +57,13 @@ The part worth knowing is that **the IETF vocabulary does not use the names you
 
 That is not a reason to avoid it, but it is a concrete reason to expect the syntax to move. Treat Content Signals as recommended-to-experiment-with, not as a finalised standard. The directive will be ignored by every crawler that does not yet parse it — which today is most of them.
 
+**The provider commitments announced in October 2026 are not for this directive.** On 30 October, [Google](https://developers.google.com/search/blog/2026/10/ai-training-control) and [Microsoft](https://blogs.bing.com/search/2026/10/your-content-your-choice-clearer-preferences-for-training-and-grounding/) announced support for a different mechanism: a `User-agent: __usage_ai-training` group, which expresses a training preference using only existing `robots.txt` syntax. It covers training alone and says nothing about search or AI input, which `Content-Signal` does. See [the shared training preference](/spec/agent-readiness/robots-for-ai-crawlers/#a-shared-training-preference) for how it works.
+
 ## Why it matters
 
 - **A declarative opt-in / opt-out at the right layer.** Existing `User-agent: GPTBot / Disallow:` directives are coarse — they block a specific bot from fetching anything. Content Signals separates *what* the bot may do from *who* the bot is.
 - **Validators already check for it.** [isitagentready.com](https://isitagentready.com/) explicitly looks for `Content-Signal:` lines in `robots.txt`. Sites that want a clean agent-readiness scorecard should add them.
-- **Some major model providers have signalled future support.** Cloudflare, Google, and others have published positions on the IETF drafts.
+- **It expresses more than training.** A single line separates search, AI input and AI training, which no shared mechanism with announced provider support does yet.
 
 ## How to implement
 
@@ -99,6 +101,7 @@ Content-Signal: search=yes, ai-input=yes, ai-train=yes
 - Inventing values beyond `yes` / `no`. The vocabulary is small on purpose.
 - Writing the IETF draft's labels (`train-ai`, `ai-use`) in a `Content-Signal` directive. Nothing reads those; the deployed directive uses `ai-train` and `ai-input`.
 - Treating it as a substitute for `Disallow:`. It is complementary.
+- Assuming Google's or Microsoft's announced training-preference support means they read `ai-train=no`. Those announcements cover the `__usage_ai-training` token, so a site that wants to reach them needs that group.
 
 ## Verification
 

src/content/spec/agent-readiness/robots-for-ai-crawlers.md

@@ -7,7 +7,7 @@ status: recommended
 order: 40
 appliesTo: [all]
 relatedSlugs: [robots-txt, content-signals, tdmrep, agent-readiness-overview, llms-txt, web-bot-auth]
-updated: "2026-10-29T00:00:00.000Z"
+updated: "2026-11-02T00:00:00.000Z"
 sources:
   - title: "OpenAI — GPTBot"
     url: "https://platform.openai.com/docs/bots"
@@ -21,6 +21,9 @@ sources:
   - title: "Apple — Applebot-Extended"
     url: "https://support.apple.com/en-us/119829"
     publisher: "Apple"
+  - title: "Machine-Readable Training Preference Signal (Internet-Draft)"
+    url: "https://datatracker.ietf.org/doc/draft-kelley-aipref-training-preference-signal/"
+    publisher: "IETF"
 ---
 
 ## What it is
@@ -96,12 +99,31 @@ Rules of thumb:
 - Combine with [/llms.txt](/spec/agent-readiness/llms-txt/) when you want to allow access but steer agents to specific pages.
 - For paywalled or member content, keep auth in place — robots.txt is not a security boundary.
 
+### A shared training preference
+
+An [Internet-Draft published on 30 October 2026](https://datatracker.ietf.org/doc/draft-kelley-aipref-training-preference-signal/) defines one name for stating a training preference to every provider that honours it, instead of listing each vendor:
+
+```txt
+User-agent: __usage_ai-training
+Disallow: /
+Allow: /press-kit/
+```
+
+**This looks like a crawler rule, but it is not one.** No bot needs to send that name, and the group does not itself stop anything being fetched. `Disallow` states a preference against generative AI training on those paths, and `Allow` states a preference in favour. To limit crawling, keep rules for each crawler's own user-agent. A `User-agent: *` group does not count, so a blanket `Disallow: /` under `*` expresses no preference through this mechanism, and a file without the group expresses none at all.
+
+Nothing has been announced as live yet. [Microsoft](https://blogs.bing.com/search/2026/10/your-content-your-choice-clearer-preferences-for-training-and-grounding/) will honour it from 1 February 2027 and [Google](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers) intends to in early 2027. [OpenAI](https://openai.com/index/helping-websites-express-ai-training-preferences/) aims to in the coming months, and [Anthropic](https://mailarchive.ietf.org/arch/msg/ai-control/mddjatOxSR0h6GmLRBWlc839c7w/) has started implementing it. The draft is an individual submission, not yet a working-group document, so the details may change.
+
+Keep your per-vendor rules, because the shared group sits alongside them and does not replace them. A vendor's own token can take precedence: [Google](https://developers.google.com/search/blog/2026/10/ai-training-control) applies the shared opt-out only where `Google-Extended` is not explicitly specified.
+
+This site does not publish the group: it has no training opt-out to express, and it already allows these crawlers by name.
+
 ## Common mistakes
 
 - Blocking `*` and forgetting that this also blocks Google.
 - Blocking only `GPTBot` and assuming no OpenAI surface can see your site — `OAI-SearchBot` and `ChatGPT-User` are separate. The same split applies to Anthropic: `ClaudeBot`, `Claude-SearchBot` and `Claude-User` are governed independently.
 - Leaving stale user-agents from 2023 lists. Names change; the vendor docs are the source of truth.
 - Treating disallow as deletion. Content already in training sets stays there.
+- Using `User-agent: __usage_ai-training` to keep a crawler out. It states a preference about training, not a crawling rule.
 
 ## Verification