BLOG
On October 30, 2026, Google announced plans to support a shared preference for AI training in robots.txt, and Microsoft Bing described a related transition with separate choices for training and grounding. These are not a universal switch already active everywhere. Site owners should first decide what they want for AI model training, use of pages as sources for answers, and ordinary search visibility. The three outcomes have different controls and implications.
What the shared training signal proposes
The industry proposal introduces a shared policy token in robots.txt so a publisher can express a training preference across participating AI providers. Google’s announcement says that, when a site opts out with this token and has no explicit Google-Extended instruction, Google plans to treat the preference as if Google-Extended were disallowed. Existing provider-specific controls continue alongside it. This is a planned interpretation, not a claim that every provider will apply identical behavior on the day a site changes its file.
Keep training, grounding, and search distinct
Training uses content to modify a model’s learned parameters. Grounding uses a source to support a response. Search discovery concerns crawling, indexing, and showing pages in results. A local consultant may want public guides to be found and cited while declining training use. A different site may make another choice. State the desired outcome in those terms before editing any crawler rule. The IETF Internet-Draft describes the proposed signal; it is a draft, not a finalized standard.
| Decision | What to review |
|---|---|
| AI training | Shared and provider-specific preference signals |
| Grounded answers | Provider controls and citation eligibility |
| Search visibility | Crawl and indexing rules independently |
| Existing restrictions | Old directives before any migration |
Google’s plan and Google-Extended
Google said it plans to support the shared control in 2027, without giving an exact activation day in the announcement. Its existing Google-Extended documentation describes a provider-specific control for certain AI uses, distinct from the normal Googlebot crawl used for Search. Do not block Googlebot for search simply because your policy concerns model training. Google also says a specific Google-Extended allowance can affect eligibility for grounding in Gemini Apps. Review the latest provider documentation before making a decision for a live site.
Bing’s transition has dates
Microsoft says the new preferences begin on February 1, 2027 as pages are recrawled. Its announcement also sets transition steps for publishers using legacy NoArchive or NoCache instructions, including a temporary grounding choice in Bing Webmaster Tools from December 15, 2026 and a March 1, 2027 decision point. New training and grounding preferences are independent. According to Bing, when it detects the newer settings it will disregard remaining legacy NoArchive or NoCache for those purposes. This makes a partial edit risky: confirm the full policy you want before switching syntax.
Inventory the current site settings
- Save a copy of the current robots.txt and relevant HTTP or HTML directives.
- List which plugin, theme, server rule, or CDN publishes each setting.
- Record the owner-approved decision for training, grounding, and search separately.
- Check existing Google-Extended and Bing legacy controls.
- Set a review date for the providers’ rollout and the draft specification.
- After an authorized change, fetch the actual public response and retain a rollback copy.
Two tools can generate conflicting rules. A WordPress SEO plugin might output one instruction while a CDN or custom file outputs another. Read the live response served to crawlers rather than relying only on a settings panel. The date you edit a file is not necessarily the date a provider recrawls and applies it.
Do not promise traffic or citations
A preference about AI training does not improve content quality, ensure indexing, generate citations, or guarantee leads. It is a publisher policy choice whose behavior depends on each participant’s implementation. Keep ordinary SEO work, accessibility, useful information, and clear contact paths on their own track. For a small business, the practical task today is to document its preference and current rules, then schedule a measured migration once the relevant support is confirmed.
What remains uncertain
Prepare a migration record
For a site maintained by an agency, record who authorized the policy and why. Include a copy of the old directives, the proposed new directives, the intended outcome for each provider, the rollout date, and the public response observed after deployment. This is especially useful when a plugin updates robots.txt dynamically: a later plugin change can alter the served content without anyone editing a physical file. Recheck the live response after major updates. If a site uses a CDN, make sure the edge response agrees with the origin response and that the rule has the intended scope across subdomains.
The shared signal is still an Internet-Draft, and Google’s support is future-facing in the October announcement. Bing has published more specific transition dates but implementation still depends on crawling and the site’s actual directives. Review official updates before altering a production site. This article describes the announced controls as of November 2, 2026 and does not claim a rollout already completed.
Do not copy a sample robots.txt line without checking its user-agent group and path context. A preference token in the wrong group may express a different policy from the one the owner approved. Technical syntax and provider support are still evolving, so preserve the decision first and implement it against the current official documentation when the relevant service supports it.
A useful internal review has three columns: desired policy, live rule, and observed provider behavior. The first is a business decision, the second can be checked on your server, and the third may lag until the provider recrawls or publishes a new implementation. Keeping those columns separate prevents a team from declaring a platform compliant merely because a line exists in robots.txt. It also makes later changes auditable if a provider clarifies how training or grounding controls work.
