AI Training Opt-Out: One robots.txt Line, Three Different Outcomes
TL;DR: On 30 October 2026, Google, Microsoft and OpenAI said they will honour a shared robots.txt token, __usage_ai-training, that lets a website state one AI training preference for several providers at once. The line is the same everywhere, but the effect is not. At Microsoft it covers training only. At Google, an opt-out without an explicit Google-Extended rule is read as a Google-Extended disallow, which also takes content out of grounding in Gemini Apps. Companies that care about being cited in AI answers should decide on training and AI visibility separately, before the first provider starts reading the token on 1 February 2027.
Key facts
- The signal: a "policy token" written on the User-Agent line of a normal robots.txt group. It is defined in an individual IETF Internet-Draft by Kevin Kelley of Anthropic, which states that it "is not endorsed by the IETF" and expires on 3 May 2027 (IETF Datatracker).
- Wildcards do not count: under the draft, a
User-Agent: *group must not be read as matching the token. A site that blocks all bots with a wildcard has expressed no AI training preference through this mechanism. - Google: plans to support the control in 2027 and will treat a shared-token opt-out as a Google-Extended disallow unless Google-Extended is specified separately (Google Search Central).
- Microsoft: starts honouring the new training and grounding preferences as it crawls from 1 February 2027, and stops reading NoArchive and NoCache as AI preferences after 1 March 2027 (Microsoft Bing Blog).
- OpenAI: says it will honour the common signal for "foundational generative AI training" and aims to implement it "in the coming months", while its existing crawler controls stay in place (OpenAI).
What the new signal actually says
Until now, a company that did not want its website used for AI training had to list each provider's crawler or control token separately in robots.txt, and keep that list current as new names appeared. The shared token replaces that list with one group:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/
According to the draft, this example expresses a preference against AI training for the whole site, except the press kit folder. Two details matter for anyone editing the file.
First, the token is a statement about use, not about access. The draft says plainly that it "is not a directive for crawling behavior". Rules for named crawlers keep doing what they do today. Blocking a crawler and opting out of training remain two different actions.
Second, "training" has a narrow meaning. The draft borrows the IETF AI Preferences working group definition: using content to modify the learned parameters of a generative AI model. Using a page at answer time to ground a response does not change a model's parameters, so it is not training under this definition. That gap is where the providers diverge.
Three providers, three readings of the same line
Microsoft separates the two uses explicitly. AI Training is the shared robots.txt preference, with an extra page-level override through a meta tag or HTTP header. AI Grounding is a separate, page-level-only preference with three values: Allow, Disallow, and a new DisallowExtended that keeps content in Microsoft's own experiences such as Copilot and Microsoft 365 but out of grounding services sold to third parties. Microsoft's post gives the example of disallowing training while allowing grounding, so content stays eligible to inform generated answers without being used to train models. It also warns that restricting grounding "will significantly reduce" visibility in AI-generated experiences, and that neither choice "guarantees inclusion, traffic, or payment."
Google maps the shared token onto its existing control. If a site opts out with the token and does not mention Google-Extended, Google plans to treat it as if Google-Extended were disallowed. According to Google's crawler documentation, Google-Extended covers both training of future Gemini models and grounding in Gemini Apps and Grounding with Google Search on Vertex AI. It does not affect inclusion in Google Search, and it is not a ranking signal. Google's announcement adds that sites which specifically allow Google-Extended "are eligible to appear in responses and receive clicks via grounding in Gemini Apps."
OpenAI has committed to honour the signal without publishing implementation details yet. Its wording, "foundational generative AI training", leaves open how narrowly the opt-out will be read, and OpenAI says a further update will follow.
The practical result: the same three lines in robots.txt can mean "no training, still cited" at Microsoft and "no training, no Gemini grounding" at Google, unless the file also says something specific about Google-Extended.
Training is not grounding: the GEO trade-off
For companies investing in generative engine optimisation (GEO), the distinction is the core of the decision. Training shapes what future models know in general. Grounding decides which pages are retrieved and cited, with links, in the answer a buyer reads today. A manufacturer, a bank or an e-commerce brand that wants its product and service pages quoted in AI answers has a different interest from a publisher that sells its archive.
A blanket opt-out written by a well-meaning web team could quietly reduce visibility in Gemini answers once Google starts reading the token. The reverse risk also exists. Microsoft says that once it detects new AI Training or AI Grounding preferences on a site, it will disregard any remaining NoArchive and NoCache tags. A site that relied on NoCache to stay out of Copilot answers and then adds only the shared training token could lose its grounding restriction unless it also publishes page-level grounding preferences.
Neither outcome is a disaster. Both are avoidable with an explicit decision instead of a copied snippet.
Why this matters under EU rules
Google ties its plan to the EU's General-Purpose AI Code of Practice, saying the code recognises robots.txt as a primary method for web publishers to express machine-readable preferences. The code's copyright chapter offers model providers a way to meet the AI Act's obligation to have a policy for complying with EU copyright law. For European companies, that makes robots.txt more than a technical file: it is where a rights reservation is expected to be readable by machines.
Two caveats keep this in proportion. The token is a voluntary signal with no formal IETF standing, and OpenAI itself describes its participation as a "voluntary effort". And a preference attaches to content when it is retrieved, so it says nothing about pages collected before the line was added.
Five decisions to make before February 2027
- Decide training and AI visibility as two separate questions. Write down the answer to each: may public pages train generative models, and should they be cited in AI answers? Different sections of the site can have different answers.
- Audit the current robots.txt and meta tags. List every AI-related rule already in place: GPTBot, Google-Extended, NoArchive, NoCache and any vendor list added by a CDN. Check whether a wildcard rule is assumed to cover AI training. Under the draft, it does not.
- If opting out of training but staying in Gemini answers matters, say so explicitly. Google only applies its mapping when Google-Extended is not specified, so the file needs its own Google-Extended group that reflects the chosen position rather than relying on the shared token alone.
- Plan the Microsoft transition if NoArchive or NoCache are in use. Temporary grounding preferences open in Bing Webmaster Tools on 15 December 2026, and the old tags stop counting as AI preferences after 1 March 2027.
- Keep provider-specific rules for now. Google and OpenAI both say their existing controls continue to work alongside the shared token. Removing them before every provider has implemented the token creates a gap.
Who should own the decision
This is not only a web team task. The choice touches marketing (AI visibility), legal (copyright and licensing) and IT (how robots.txt and headers are deployed). Without a named owner, the file usually reflects whoever edited it last. The decision belongs in the same place as other AI policy choices: a short written position, one owner, and a review date when Google and OpenAI publish their implementation details. Encorp.ai's AI governance work helps companies set exactly this kind of policy, from content-use preferences to internal AI rules.
Next step: put "AI training and AI visibility preferences" on the agenda of the next marketing, legal and IT meeting, and agree a written position before 1 February 2027. If an outside view would help, contact Encorp.ai for a short review of the current robots.txt and AI policy.
