Bulk opt-out of AI training via robots.txt: Three major companies announce support
機械翻訳 / Machine-translated
A
AI Friends
3日前・15分で読める
機械翻訳 / Machine-translated
@aifriends
AI Friends（https://aifriends.jp）のクロスポスト公式アカウント。AIツールの紹介・使い方・できることを、中学生でもわかるやさしい日本語で届けます。
If you've thought "I don't want my articles used to train AI," writing separate rules for each AI company has been a real hassle. On October 30, 2026, a proposal to reduce that burden was put forward, and three major companies announced their intention to support it. This article explains what is changing and what operators need to do, in plain terms.
According to ITmedia NEWS, the announcement was made on October 30, 2026 (local time).
Kevin Kelly, who handles standardization at Anthropic, submitted an Internet-Draft to the IETF (the international body that establishes technical rules for the internet) under the title "Machine-Readable Training Preference Signal."
In response, OpenAI, Google, and Microsoft announced plans to support it. Anthropic itself has reportedly already begun implementation.
However, the draft was submitted under an individual's name. It is not yet an official IETF standard.
robots.txt — a plain-text file placed at the entrance to a site that acts like a signboard for bots — was originally designed for search engine crawlers (programs that automatically collect pages).
Operators wanting to refuse AI training had to look up each AI company's crawler name and write rules for them one by one. Every time a new company appeared, more entries had to be added.
In the new format, you write the token __usage_ai-training in the User-Agent line.
Disallow expresses the intent to refuse use for training; Allow expresses permission. For example:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/
This example signals that you do not want any pages except those under /press-kit/ used for training.
No new directives or syntax are introduced. A key feature is that existing parsers (programs that read the file) are not affected.
First, this does not block crawling itself. If you want to stop pages from being fetched, write rules for each individual crawler as before.
Second, if you do not write the token, you are considered to have expressed no particular preference regarding AI training.
Third, the scope is limited to training generative AI models. Training models used exclusively for non-generative purposes is not covered. Companies are also permitted to prioritize their own proprietary controls or to handle things differently under individual contracts.
OpenAI aims to comply within the next few months and says it will publish details when it implements the feature. Existing robots.txt crawl controls will continue to apply.
Google aims to comply in 2027 along with other AI companies.
Sites that opt out using the common signal but do not explicitly specify "Google-Extended" will be treated as having opted out for Google-Extended as well. Google-Extended, available since 2023, will remain usable going forward.
As a side note, allowing use via Google-Extended makes your site eligible for Gemini app "grounding" (a feature that references web content to generate answers). Your site may be mentioned in responses, potentially driving clicks.
Microsoft is the most specific: it will begin reading and applying the new settings starting February 1, 2027.
In addition to the common signal, Microsoft will support per-page specifications via meta tags and HTTP headers. Per-page settings take priority over robots.txt.
Grounding will have its own dedicated settings. Disallow refuses use. DisallowExtended permits use in Copilot and Microsoft 365 while refusing use via "Web IQ" for third-party companies and through the Microsoft Foundry API.
These settings do not affect inclusion in Bing search results.
Here is how existing methods compare with the new approach:
In June 2026, Google also announced a switch in Search Console to opt out of use in AI search features. The options are growing, but the settings have been scattered across different places.
Consider a web editor at a regional newspaper. They don't want their articles used for AI training, but they don't want to lose search traffic either. Until now, they had to research and write rules for dozens of companies.
If the common signal gains broad adoption, it becomes much easier to refuse training while remaining visible in search results.
Independent bloggers and illustrators face the same situation. For anyone who doesn't want their work used for training, being able to express that intent with a single line of text is a significant step forward.
On the other hand, some companies want AI responses to introduce their websites. In that case, they can use Allow to grant permission and keep open the possibility of being cited through grounding.
That said, robots.txt works because compliant parties — like Google's crawlers — voluntarily follow it. There is no technical means to stop those who do not.
AI preference signals are being discussed in the IETF's AIPREF Working Group.
The group is working on two documents: a vocabulary (vocab) for expressing preferences, and a method for attaching that vocabulary to content (attach). Examples of attachment methods include robots.txt and HTTP response headers.
Technical enforcement and auditing of AI training are outside the scope of this working group.
Since this draft is an individual submission based on AIPREF discussions, its content may change going forward.
A. Not yet. OpenAI aims to comply within a few months, Google by 2027, and Microsoft from February 1, 2027.
A. Expressing a preference about training is separate from blocking crawling. Microsoft has also stated it will not affect inclusion in Bing search results.
A. No. Google-Extended will continue to work after the common signal is introduced.
A. Within the scope of this reporting, no legal effect has been confirmed. This is strictly a mechanism for expressing intent.
A. Replacement settings for the existing "NoArchive" and "NoCache" directives will be provided, and sites currently using them will be asked to migrate by March 1, 2027.
Start by opening your site's robots.txt and checking what settings are already in place.
This article is a cross-post from AI Friends.