From 6ab57a4a6a83908d5acca00510bc7d5d6b488617 Mon Sep 17 00:00:00 2001 From: Joost de Valk Date: Mon, 2 Nov 2026 08:10:21 +0100 Subject: [PATCH] change(robots-for-ai-crawlers): cover the shared AI training preference Google and Microsoft announced support on 30 October 2026 for a __usage_ai-training robots.txt group, defined in an individual Internet-Draft. Explain what it does and does not do, and correct the Content Signals page, which implied those providers back its directive. Co-Authored-By: Claude Opus 5.5 --- ...026-11-02-shared-ai-training-preference.md | 8 +++++++ .../spec/agent-readiness/content-signals.md | 7 ++++-- .../agent-readiness/robots-for-ai-crawlers.md | 24 ++++++++++++++++++- 3 files changed, 36 insertions(+), 3 deletions(-) create mode 100644 src/content/changelog/2026-11-02-shared-ai-training-preference.md diff --git a/src/content/changelog/2026-11-02-shared-ai-training-preference.md b/src/content/changelog/2026-11-02-shared-ai-training-preference.md new file mode 100644 index 00000000..2d753dc2 --- /dev/null +++ b/src/content/changelog/2026-11-02-shared-ai-training-preference.md @@ -0,0 +1,8 @@ +--- +title: "AI crawlers: a shared robots.txt training preference is on the way" +date: "2026-11-02" +type: changed +relatedSlugs: [robots-for-ai-crawlers, content-signals] +--- + +The [robots.txt for AI crawlers page](/spec/agent-readiness/robots-for-ai-crawlers/#a-shared-training-preference) now covers the `__usage_ai-training` token, an Internet-Draft from 30 October 2026 that Microsoft and Google plan to honour in early 2027. The [Content Signals page](/spec/agent-readiness/content-signals/) now makes clear that those commitments are to this token, not to the `Content-Signal` directive. diff --git a/src/content/spec/agent-readiness/content-signals.md b/src/content/spec/agent-readiness/content-signals.md index 56ff2a55..d332f1b4 100644 --- a/src/content/spec/agent-readiness/content-signals.md +++ b/src/content/spec/agent-readiness/content-signals.md @@ -7,7 +7,7 @@ status: optional order: 42 appliesTo: [all] relatedSlugs: [robots-for-ai-crawlers, robots-txt, tdmrep, agent-readiness-overview, web-bot-auth] -updated: "2026-09-26T00:00:00.000Z" +updated: "2026-11-02T00:00:00.000Z" sources: - title: "IETF AI Preferences WG (aipref) — drafts" url: "https://datatracker.ietf.org/wg/aipref/documents/" @@ -57,11 +57,13 @@ The part worth knowing is that **the IETF vocabulary does not use the names you That is not a reason to avoid it, but it is a concrete reason to expect the syntax to move. Treat Content Signals as recommended-to-experiment-with, not as a finalised standard. The directive will be ignored by every crawler that does not yet parse it — which today is most of them. +**The provider commitments announced in October 2026 are not for this directive.** On 30 October, [Google](https://developers.google.com/search/blog/2026/10/ai-training-control) and [Microsoft](https://blogs.bing.com/search/2026/10/your-content-your-choice-clearer-preferences-for-training-and-grounding/) announced support for a different mechanism: a `User-agent: __usage_ai-training` group, which expresses a training preference using only existing `robots.txt` syntax. It covers training alone and says nothing about search or AI input, which `Content-Signal` does. See [the shared training preference](/spec/agent-readiness/robots-for-ai-crawlers/#a-shared-training-preference) for how it works. + ## Why it matters - **A declarative opt-in / opt-out at the right layer.** Existing `User-agent: GPTBot / Disallow:` directives are coarse — they block a specific bot from fetching anything. Content Signals separates *what* the bot may do from *who* the bot is. - **Validators already check for it.** [isitagentready.com](https://isitagentready.com/) explicitly looks for `Content-Signal:` lines in `robots.txt`. Sites that want a clean agent-readiness scorecard should add them. -- **Some major model providers have signalled future support.** Cloudflare, Google, and others have published positions on the IETF drafts. +- **It expresses more than training.** A single line separates search, AI input and AI training, which no shared mechanism with announced provider support does yet. ## How to implement @@ -99,6 +101,7 @@ Content-Signal: search=yes, ai-input=yes, ai-train=yes - Inventing values beyond `yes` / `no`. The vocabulary is small on purpose. - Writing the IETF draft's labels (`train-ai`, `ai-use`) in a `Content-Signal` directive. Nothing reads those; the deployed directive uses `ai-train` and `ai-input`. - Treating it as a substitute for `Disallow:`. It is complementary. +- Assuming Google's or Microsoft's announced training-preference support means they read `ai-train=no`. Those announcements cover the `__usage_ai-training` token, so a site that wants to reach them needs that group. ## Verification diff --git a/src/content/spec/agent-readiness/robots-for-ai-crawlers.md b/src/content/spec/agent-readiness/robots-for-ai-crawlers.md index cbb41902..6da9ef80 100644 --- a/src/content/spec/agent-readiness/robots-for-ai-crawlers.md +++ b/src/content/spec/agent-readiness/robots-for-ai-crawlers.md @@ -7,7 +7,7 @@ status: recommended order: 40 appliesTo: [all] relatedSlugs: [robots-txt, content-signals, tdmrep, agent-readiness-overview, llms-txt, web-bot-auth] -updated: "2026-10-29T00:00:00.000Z" +updated: "2026-11-02T00:00:00.000Z" sources: - title: "OpenAI — GPTBot" url: "https://platform.openai.com/docs/bots" @@ -21,6 +21,9 @@ sources: - title: "Apple — Applebot-Extended" url: "https://support.apple.com/en-us/119829" publisher: "Apple" + - title: "Machine-Readable Training Preference Signal (Internet-Draft)" + url: "https://datatracker.ietf.org/doc/draft-kelley-aipref-training-preference-signal/" + publisher: "IETF" --- ## What it is @@ -96,12 +99,31 @@ Rules of thumb: - Combine with [/llms.txt](/spec/agent-readiness/llms-txt/) when you want to allow access but steer agents to specific pages. - For paywalled or member content, keep auth in place — robots.txt is not a security boundary. +### A shared training preference + +An [Internet-Draft published on 30 October 2026](https://datatracker.ietf.org/doc/draft-kelley-aipref-training-preference-signal/) defines one name for stating a training preference to every provider that honours it, instead of listing each vendor: + +```txt +User-agent: __usage_ai-training +Disallow: / +Allow: /press-kit/ +``` + +**This looks like a crawler rule, but it is not one.** No bot needs to send that name, and the group does not itself stop anything being fetched. `Disallow` states a preference against generative AI training on those paths, and `Allow` states a preference in favour. To limit crawling, keep rules for each crawler's own user-agent. A `User-agent: *` group does not count, so a blanket `Disallow: /` under `*` expresses no preference through this mechanism, and a file without the group expresses none at all. + +Nothing has been announced as live yet. [Microsoft](https://blogs.bing.com/search/2026/10/your-content-your-choice-clearer-preferences-for-training-and-grounding/) will honour it from 1 February 2027 and [Google](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers) intends to in early 2027. [OpenAI](https://openai.com/index/helping-websites-express-ai-training-preferences/) aims to in the coming months, and [Anthropic](https://mailarchive.ietf.org/arch/msg/ai-control/mddjatOxSR0h6GmLRBWlc839c7w/) has started implementing it. The draft is an individual submission, not yet a working-group document, so the details may change. + +Keep your per-vendor rules, because the shared group sits alongside them and does not replace them. A vendor's own token can take precedence: [Google](https://developers.google.com/search/blog/2026/10/ai-training-control) applies the shared opt-out only where `Google-Extended` is not explicitly specified. + +This site does not publish the group: it has no training opt-out to express, and it already allows these crawlers by name. + ## Common mistakes - Blocking `*` and forgetting that this also blocks Google. - Blocking only `GPTBot` and assuming no OpenAI surface can see your site — `OAI-SearchBot` and `ChatGPT-User` are separate. The same split applies to Anthropic: `ClaudeBot`, `Claude-SearchBot` and `Claude-User` are governed independently. - Leaving stale user-agents from 2023 lists. Names change; the vendor docs are the source of truth. - Treating disallow as deletion. Content already in training sets stays there. +- Using `User-agent: __usage_ai-training` to keep a crawler out. It states a preference about training, not a crawling rule. ## Verification