From 2027, you can refuse AI training from several providers with 1 group in robots.txt: User-Agent: __usage_ai-training and then Disallow: /. Google, Microsoft Bing, OpenAI, and Anthropic announced support on 30 October 2026, but no provider reads this token as of 31 October 2026.
This token does not change rankings in Google Search or Bing. The risk is in a different place. Google reads this token as a Disallow for Google-Extended, and Google-Extended also controls citations in Gemini Apps.
What is the __usage_ai-training token?
The __usage_ai-training token is a special user-agent name in robots.txt that carries a content-use preference, not a crawl instruction. Kevin Kelley of Anthropic wrote the rules in the IETF draft draft-kelley-aipref-training-preference-signal-00, dated 30 October 2026.
Disallowunder the token means that you refuse AI training for that path.Allowmeans that you agree. If no group names this token, your site states no preference.- A
User-Agent: *group does not apply to this token. Write the token name explicitly. - This token does not limit crawling. To limit crawling, keep per-crawler groups such as GPTBot.
This draft is still an individual draft. The IETF page states that the IETF has not endorsed it and that it has no formal standing. The text and syntax can change before 2027.
Who supports it, and when does reading start?
Four AI providers announced support on 30 October 2026, each with a different date.
| Provider | Date | Note from the primary source |
|---|---|---|
| 2027. The crawler documentation states early 2027. | The token with no Google-Extended group reads as a Disallow for Google-Extended. Source: Google Search Central, 30 October 2026. | |
| Microsoft Bing | From 1 February 2027 | Bing separates training and grounding preferences. Source: Bing Blogs, 30 October 2026. |
| OpenAI | In the coming months | GPTBot and OAI-SearchBot controls stay in effect. Source: OpenAI announcement. |
| Anthropic | In the coming months | Work has started, and it adds to the current crawler controls. Source: IETF ai-control list, 30 October 2026. |
Google also states that this proposal agrees with the EU Code of Practice for general-purpose AI model providers under the EU AI Act. That code accepts robots.txt as a primary method to state machine-readable preferences.
What changes for Google Search, Gemini, and Copilot?
This token does not change ranking or inclusion in Google Search and Bing, but it can stop citations in Gemini Apps.
The Google common crawlers documentation, updated on 30 October 2026, states that Google-Extended controls 2 uses. The first use is Gemini model training. The second use is grounding in Gemini Apps and in Grounding with Google Search on Vertex AI. The same page states that Google-Extended does not change inclusion in Google Search and is not a ranking signal.
Thus, a training opt-out through the shared token also removes your site from Gemini Apps grounding. Google states that sites that specifically allow Google-Extended can appear in responses and get clicks through Gemini Apps grounding. At Google, you cannot yet refuse training and keep Gemini citations.
Bing gives more detailed choices. The ai-training and ai-grounding preferences are separate. The DisallowExtended value for grounding keeps Microsoft products such as Copilot and Bing, but it refuses third-party grounding services. Bing states that Disallow or DisallowExtended for grounding will decrease brand visibility in AI experiences.
The IETF draft and the Google announcement cover training only. Neither one mentions AI OverviewsAIOThe AI answer summary above search results. It answers part of the question before anybody presses a link.Open the glossary or AI Mode in Google Search. Do not use this token to control those 2 features.
How do you write the rule in robots.txt?
Write the token group apart from your crawler groups, and then select 1 of the 3 scenarios below.
Scenario 1: refuse AI training at all providers, and stay indexed in search engines.
User-Agent: __usage_ai-training
Disallow: /
User-Agent: *
Allow: /Scenario 2: refuse training at other providers, but keep Google-Extended. The Google announcement states that the token reads as a Disallow only when no Google-Extended group exists. With an explicit Google-Extended group, Google uses that group, so your content stays in Gemini training and Gemini Apps grounding.
User-Agent: __usage_ai-training
Disallow: /
User-Agent: Google-Extended
Allow: /Scenario 3: refuse training for 1 folder only. Example: paid course material.
User-Agent: __usage_ai-training
Disallow: /courses/For Bing at page level, use the meta tag <meta name="Bingbot" content="ai-training:disallow"> or the header X-Robots-Tag: Bingbot: ai-training:disallow. A page preference overrides robots.txt.
Which sites must add this token?
Select a scenario from the revenue source of your site, not from fear of AI.
| Site type | Main revenue | Recommendation |
|---|---|---|
| Online store, local service, small business | Orders and chats from search | Do not use Scenario 1. Gemini and Copilot citations bring buyers. If you still want an opt-out, use Scenario 2. |
| Media and blogs with original content | Display ads and subscriptions | Consider Scenario 1. First count visits from Gemini, because Scenario 1 stops Gemini Apps grounding. |
| Courses, research, and paid content | Paid access | Use Scenario 3 for paid folders. For Bing, add ai-grounding:disallow on paid pages. |
Robots.txt only states a preference. The IETF draft states that a private agreement between a site owner and an AI provider can override this preference. For content that must stay closed, keep a login.
What is the effect on cost, measurement, and team work?
The token costs almost nothing to add, but 1 error in robots.txt can stop traffic without a visible sign.
- Cost: 1 change to the robots.txt file. No licence or new tool is necessary. The largest risk is a typing error in the
User-Agent: *group that blocks Googlebot. - Measurement: record a baseline before the change. Get visits from gemini.google.com, copilot.microsoft.com, and chatgpt.com in GA4GA4The Google analytics tool that records visits and actions on your site as events.What is Google Analytics 4 (GA4)?, plus the AI Performance report in Bing Webmaster Tools. Compare 28 days before and after the token is live.
- Team work: give 1 person ownership of robots.txt. WordPress SEOSEOThe work of making your page easy to find and easy to choose in search results, without paying for ads.Open the glossary plugins and Shopify templates can overwrite this file. Check the result in the Google Search Console robots.txt report after each release.
- Bing deadline: after 1 March 2027, Bing stops reading NoArchive and NoCache as training and grounding preferences. If your site uses them, add the new preferences before that date.
Our SEO and AI visibility audit examines robots.txt rules and AI citations in 1 report. For ongoing work, see our monthly SEO service.
What must you do this week?
Do not change robots.txt in a hurry, because no provider reads this token as of 31 October 2026.
- Open the robots.txt of your site. Record the current GPTBot, Google-Extended, ClaudeBot, NoArchive, and NoCache rules.
- Select 1 scenario from the site type table above.
- Record a baseline of visits from Gemini, Copilot, and ChatGPT.
- If your site uses NoArchive or NoCache, schedule the Bing change before 1 March 2027.
- Check the Google, OpenAI, and Anthropic announcements each month. The draft syntax can change.
To see if AI tools cite your brand now, read our AEO and GEO brand visibility report.
Frequently asked questions
Does the __usage_ai-training token decrease Google rankings?
No. Google maps this token to Google-Extended, and the Google documentation states that Google-Extended does not change inclusion in Google Search and is not a ranking signal.
Must I add this token now?
No. As of 31 October 2026, no provider reads it. If you already selected a scenario, you can add it now, because providers read it when their support goes live.
Do GPTBot and Google-Extended still work?
Yes. Google, OpenAI, and Anthropic state that per-provider controls continue with the shared token. At Google, an explicit Google-Extended group has priority over the shared token.
Does this token stop AI tools from reading my page when a user asks a question?
No. This token states a training preference only, not a crawl instruction. For AI answers in Bing, use the ai-grounding preference. For Gemini Apps, the rule follows Google-Extended.
What about sites that use NoArchive or NoCache?
Bing stops reading NoArchive and NoCache as training and grounding preferences after 1 March 2027. Add the new ai-training and ai-grounding preferences, and then record the temporary preference in Bing Webmaster Tools.
