AI providers back shared training preferences, but migration is required

What happened

OpenAI, Google and Microsoft say they support a shared robots.txt signal for expressing whether a website permits AI training. The proposed __usage_ai-training policy token is intended to reduce the need for separate, provider-specific instructions. Anthropic’s standards lead said implementation has begun, though the supplied material does not establish that rollout is complete across its services.

The change is voluntary. It is not an access-control mechanism, a blanket scraping ban, a copyright ruling or an enacted regulation. The underlying Internet-Draft has no IETF endorsement or formal standing, so businesses should treat this as a coordinated proposal rather than an adopted web standard. OpenAI expects work to continue in the coming months, while Google points to 2027.

Microsoft has set the clearest operational timetable. It will begin honoring the new preferences as pages are crawled on February 1, 2027. From March 1, it will stop treating NoArchive and NoCache as preferences about AI training or grounding. Publishers using those directives must also use temporary preferences in Bing Webmaster Tools during the grounding migration. See the Microsoft announcement and OpenAI’s explanation.

Why it matters

For publishers, retailers and other businesses with valuable public content, a common signal could make AI-content governance less tedious. Fewer duplicated rules should mean less maintenance when crawler policies change. That is useful—but only if the signal is implemented consistently and the business knows what it is trying to control.

Training and grounding are different uses. A preference about model training may not govern whether a search or assistant can use a page to generate an answer. Google’s mapping to Google-Extended can also affect Gemini grounding unless company-specific settings override it. Microsoft provides separate controls for training and grounding, so a single switch should not be assumed to cover both.

• Audit current robots.txt, crawler and webmaster settings.
• Record which rules address training, search indexing, caching or AI-answer grounding.
• Plan the Bing migration before March 1, 2027.
• Keep provider-specific controls until implementation and behavior are confirmed.

The practical business question is not whether to opt out by default. It is whether the value of AI discovery, referral traffic and content licensing—none of which is guaranteed here—outweighs the cost and risk of making content available for particular uses. Google’s guidance and the proposed specification provide useful implementation context.