🔍 Read the full analysis: Should You Set The Shared AI Training Opt-Out? What It Covers on ThorstenMeyerAI.com
Get business pricing on monitors, keyboards and dev gear
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
OpenAI, Google, Microsoft and Anthropic have said they will support a shared robots.txt signal asking them not to use a website’s content to train generative AI models. The signal is voluntary, does not block crawling or AI answers based on live content, and is not yet an official internet standard; companies have different implementation schedules.
OpenAI, Google, Microsoft and Anthropic have said they will honor a shared instruction in a website’s robots.txt file asking them not to use its content to train generative AI models. The announcement, made October 30, offers publishers one common signal instead of separate training preferences for each company, but the instruction is voluntary and the companies have not set a shared start date.
The proposed rule uses the user-agent name __usage_ai-training followed by Disallow: /. It is set out in an individual Internet-Draft titled “Machine-Readable Training Preference Signal,” published by Anthropic standards lead Kevin Kelley. The draft is intended to communicate a publisher’s preference about model training; it is not a technical barrier that prevents a crawler from accessing a site.
The companies described different timelines. Anthropic says it has started implementation, while noting that support across its services will take time. OpenAI says it aims to adopt the signal “in the coming months.” Google says it is aiming for 2027, with its crawler documentation specifying early 2027. Microsoft says it will honor the preference from February 1, 2027, as it crawls websites.
Microsoft also described a separate page-level preference for AI Grounding, covering content used to inform answers in Bing, Copilot and Microsoft 365. The company says both preferences take effect February 1, 2027. It says its older NoArchive and NoCache tags will stop counting as preferences for training or grounding after March 1, 2027. Existing company-specific crawler rules remain relevant during the rollout.
Infographic 01 · Who backs it, and when
One line in robots.txt, four companies, four timelines.
A shared signal against AI training. Only Anthropic says it has started; the others name dates or months. The standard itself is still a draft.
- 30 Oct 2026Draft published, four companies announce supportdoneIndividual Internet-Draft by Kevin Kelley (Anthropic)
- 30 Oct 2026Anthropic: “has already started implementing support”startedIts standards lead on the IETF list: “It’ll take us some time to add support across our services.”
- Coming monthsOpenAI aims to implement itplanExisting GPTBot and OAI-SearchBot rules keep applying
- 16 + 19 Nov 2026Working group meets at IETF 127, San FranciscoopenOn 21 Oct a chair said no consensus has been declared yet
- 1 Feb 2027Microsoft honors the new preferences as it crawlsplanTraining in robots.txt or per page; grounding per page
- 1 Mar 2027Bing stops reading NoArchive / NoCache as AI preferencesdeadlinePublish the new settings before this date
- 2027Google plans support; its docs say “early 2027”planA “no” maps to Google-Extended unless you set that token yourself
- 3 May 2027Draft version 00 expires unless updatedInternet-Drafts are valid for at most six months
Binding? No, voluntary
- OpenAI calls it a “voluntary effort”.
- The vocabulary draft does not “ensure that preferences are followed”.
- The draft “has no formal standing in the IETF standards process”.
- Private agreements may override it, the draft says.
Who else?
- Meta supports a common signal, but did not say whether it will honor it.
- Apple, Amazon, xAI, Perplexity: no statement in the sources reviewed for this article.
- The draft’s acknowledgments list contributors “for identification only”, not supporters.
Sources: IETF draft, 30 Oct 2026 · Anthropic on the IETF list · AIPREF minutes, 21 Oct 2026 · Google, 30 Oct 2026 · Google crawler docs · OpenAI, 30 Oct 2026 · Microsoft Bing, Oct 2026 · Meta (Andy Stone on X). Company plans are company statements.
Powered by Thorsten Meyer AI
A Single Signal, Limited Control
For publishers, the shared rule could make it simpler to state a preference across several major AI providers. Previously, a site owner might need separate instructions for crawlers such as GPTBot, Google-Extended and ClaudeBot, as well as Microsoft-specific page tags. A common signal reduces that administrative burden if participating companies implement it as described.
But the change is a request, not an enforcement mechanism. It does not prevent a company from accessing a page, remove material from models already trained, or establish a right to compensation. Microsoft explicitly says that its preferences do not guarantee inclusion, traffic or payment. Publishers also need to distinguish training from the use of web pages to produce AI answers: the shared signal is for training, while Microsoft’s grounding preference is a separate control.
The distinction matters to anyone making decisions about public web content. A publisher seeking to limit model training may still want to keep existing crawler rules in place and decide separately whether to permit search indexing or use of pages in AI-generated answers. The new line does not change those choices automatically across every service.
Products Worth Considering
DJI Osmo Mobile 8 3-Axis Phone Gimbal Stabilizer
Logitech MX Keys S Wireless Keyboard, Graphite
Logitech MX Brio Ultra HD 4K Webcam, Graphite
As an affiliate, we earn on qualifying purchases.
The proposal is connected to the Internet Engineering Task Force’s AI Preferences working group, which discusses ways for websites to express machine-readable preferences. The document is an individual Internet-Draft, not a final standard. At the group’s October 21 meeting, a chair said the group had not declared consensus on any proposal.
Company announcements refer to the same draft, but their commitments are not identical. Google says that if a website uses the shared token and has no separate Google-Extended rule, it plans to treat the shared preference as a disallow for Google-Extended. Microsoft’s announcement goes beyond training by adding its own grounding control. Anthropic’s stated support does not mean every Anthropic service already recognizes the signal.
Meta’s communications chief, Andy Stone, voiced support for publishers’ ability to express preferences and called a common signal an important step. He did not commit Meta to honoring it. The source report said it found no statement from Apple, Amazon, xAI or Perplexity in the material reviewed. That absence is not evidence of how those companies will act.
“Anthropic “has already started implementing support for this draft.””
— Anthropic standards lead Kevin Kelley
Adoption and Coverage Remain Open
Several parts of the rollout remain unresolved. The Internet-Draft has no formal standing in the IETF standards process, and the working group had not declared consensus as of its October 21 meeting. The companies’ public statements establish their stated plans, but do not show that every product, crawler or service already applies the rule consistently.
It is also unclear which other AI companies will participate, how compliance will be monitored, or what recourse a publisher would have if a company disregarded the preference. The signal’s stated purpose is model training; it does not itself settle whether content may be crawled for search, used for live AI answers, or included in other services. Meta’s position on honoring the instruction is unconfirmed.
These details can change as companies publish technical documentation and the standards group considers the draft. The dates and capabilities described here reflect the announcements and documents reported as of October 31, 2026.
Publishers Face Staggered Rollouts
Website operators can add the shared training preference if they want to express that choice, but should not treat it as a substitute for existing company-specific rules before each provider’s implementation is in place. They should also check how their site handles training, AI answers and search separately, since those uses may have different controls.
Microsoft users of NoArchive or NoCache should review the company’s announced change ahead of March 1, 2027, when those tags will stop counting as training or grounding preferences under its plan. Publishers will need to watch for updated documentation from each provider, particularly as Google and OpenAI approach their stated rollout periods and Anthropic expands support across services.
The next substantive milestones are company implementation updates and any progress by the IETF working group on the draft. Until then, the shared rule is best understood as a developing, voluntary way to communicate a training preference—not a universal opt-out that blocks all AI use of a website.
Key Questions
What does the shared AI training opt-out do?
It is a robots.txt instruction that communicates a website owner’s preference not to have its content used to train generative AI models by companies that honor the signal.
Does it stop AI companies from crawling a website?
No. The proposal is a preference signal, not a technical block. It does not prevent a crawler from accessing a page or remove content from models that have already been trained.
Does the signal prevent content from appearing in AI answers?
Not by itself. The shared instruction covers model training. Microsoft announced a separate AI Grounding preference for content used in answers in Bing, Copilot and Microsoft 365.
When will the companies honor the signal?
Anthropic says implementation has started; OpenAI aims for the coming months; Google is aiming for 2027, with documentation saying early 2027; and Microsoft has set February 1, 2027. The schedules differ and may be updated.
Is this the setting for my personal AI chats?
No. The instruction is for website owners expressing preferences about their published content. Personal chat-history or model-training controls are separate account settings provided by individual AI services.
Source: ThorstenMeyerAI.com
