One Robots.txt Line Could Soon Speak for Every AI Company

You already decided whether to block AI. Google just changed what that decision does.

Somewhere in your site's files sits a robots.txt, and odds are decent that at some point in the last two years, you, a developer, or a WordPress plugin added a line or two about AI. Maybe it was a blog post that told you to disallow GPTBot. Maybe it was a security plugin that shipped an "AI blocking" toggle you flipped once and never thought about again. Either way, that file has been sitting there quietly doing exactly what it was told, and until recently, each line in it spoke to exactly one company.

That is the part that just changed. On October 30, 2026, Google published a post saying it plans to support a new, shared signal in robots.txt, one line that multiple AI companies agree to read the same way, instead of each company requiring its own separate rule¹. The underlying mechanism comes from a draft filed the same day with the Internet Engineering Task Force, the body that writes the technical standards the internet runs on, titled the Machine Readable Training Preference Signal².

Nothing about this is a crisis. The draft is early, the rollout target is 2027, and nothing in your current file stops working today. But it is worth fifteen minutes, because the whole point of a shared signal is that one line now has more reach than it used to. A mistake in a single user agent group used to affect one crawler. The design intent of this new token is that a mistake, or an unread default, could someday speak for several companies at once. That is worth understanding before it matters, not after.

Why one file now has three different ways to say the same thing

Robots.txt is a plain text file, specified by an actual internet standard, RFC 9309, and for most of its history it did one job: tell a given crawler, identified by its own name, which paths it may fetch. A line reading User-agent: Googlebot followed by Disallow: /admin/ speaks only to Googlebot. A different crawler reads its own group, or falls back to the wildcard group if it has none. Simple, and for twenty five years, that was the entire vocabulary.

AI changed what people wanted to say with that file, and the vocabulary has been playing catch up ever since, in three separate layers that now all sit in the same document on most agency sites without most owners realizing there are three.

The first layer is the one everyone already knows: per crawler rules. A group named for one exact user agent token, with an Allow or Disallow for that crawler only. This is how a site tells OpenAI's GPTBot, used to train the company's generative AI foundation models⁵, apart from OpenAI's entirely separate OAI-SearchBot, which exists to surface a site in ChatGPT's search features, and from ChatGPT-User, which handles a live, user-initiated fetch inside a chat and is not automated crawling at all⁵. Three different jobs, three different tokens, and disallowing one says nothing about the other two. The same split exists on Anthropic's side with ClaudeBot, Claude-SearchBot, and Claude-User, and on Perplexity's side with PerplexityBot and Perplexity-User.

The second layer is the Content-Signal line, a robots.txt extension Cloudflare published on September 24, 2025, built as the Content Signals Policy⁶. Instead of naming a crawler, it names a use, in one line: Content-Signal: search=yes, ai-train=no for example. Cloudflare defines three fields exactly: search covers building a search index and providing search results, and explicitly does not include AI-generated search summaries; ai-input covers feeding content into an AI model for a real-time answer, the retrieval-augmented-generation and grounding use case; and ai-train covers training or fine-tuning a model⁶. Any crawler that chooses to read this line gets all three preferences in one place, rather than needing a separate rule per company. Cloudflare's own managed robots.txt feature is active on over 3.8 million domains, and the company sets new customers by default to search=yes, ai-train=no, leaving ai-input unset entirely because, in Cloudflare's own words, it does not want to guess a site owner's preference on that one⁶.

The third layer is the one that is new as of this proposal: a dedicated token meant to be read identically by every participating AI company, covered in full in the next section.

None of these three layers replaces either of the others. A site can, and increasingly will, carry all three at once: per crawler rules for fine control, a Content-Signal line for a three-way preference in one spot, and the new shared training token once it exists. The practical risk is not that any one of them is dangerous on its own. It is that an owner sets one, assumes it covers everything, and never checks whether a different layer is quietly saying something else.

What Google and the IETF actually proposed on October 30

Here is the mechanism in exact terms, because the vague version of this story is more confusing than the precise one. The draft, authored by Kevin Kelley at Anthropic and filed with the IETF's AI Preferences working group, defines a policy token: __usage_ai-training, following the same product token syntax every robots.txt user agent name already uses². It goes on a User-Agent line, the same way a crawler's name does, and the example the draft itself gives is this:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

A few details matter more than they first look like they should. The draft states consumers "MUST match policy tokens exactly, apart from case, as for any product token," and that wildcard groups, a plain User-Agent: *, "MUST NOT be interpreted as matching the policy token"². In other words, a generic catch-all allow rule does not accidentally opt a site into or out of anything here, the token has to be addressed directly. The draft is also explicit that this is "not a directive for crawling behavior"², meaning it does not control whether a bot may fetch a page at all, only what the fetched content may be used for. Crawling itself is still governed by the ordinary per-crawler rules in layer one.

On status: this is an individual Internet Draft, version 00, published October 30, 2026, and set to expire May 3, 2027 under the IETF's standard six-month draft lifecycle unless it is renewed or formally adopted²³. The IETF's own datatracker lists it as an individual submission, not a working group document, meaning the IETF has not endorsed it and it currently carries no formal standing in the standards process³. Discussion is happening in public, on the AI Preferences working group's mailing list, which is the normal, deliberately slow way an internet standard gets built. Google's own post states its aim to support the resulting signal alongside other AI providers in 2027, and says it will post updates as that work progresses¹.

What Google-Extended already does, and what it has never touched

Because Google is the company that made this announcement, and because Google already has the oldest AI training control of the three layers, it is worth being precise about what that existing control covers before adding a new one on top. Google's documentation names two uses Google-Extended governs: training future generations of Gemini models, in both Gemini Apps and the Vertex AI API, and grounding, which is supplying Google Search index content to a model at the moment someone asks it a question, again in both Gemini Apps and Grounding with Google Search on Vertex AI⁴.

On Search specifically, Google's own documentation is direct: "Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search"⁴. That sentence is doing real work. It means an agency that disallowed Google-Extended at some point, perhaps along with a blanket "block all AI" pass across its robots.txt, did not change its Search rankings even slightly by doing so. It changed two narrower things: whether that content can train a future Gemini model, and whether Google's own grounding feature can pull it into a live Gemini answer at the moment someone asks.

Google's October 30 announcement adds one more wrinkle worth knowing. If a site sets the new shared training token to opt out, and never separately addresses Google-Extended by name, Google has said it will treat that as equivalent to having disallowed Google-Extended too¹. Company specific controls keep working alongside the shared one. A site can set a broad preference with the new token and still carve out an exception for one provider with that provider's own rule, the same layering the per crawler groups in layer one already allow today.

Google counts eleven distinct crawler and control tokens on its own documentation page as of this writing: Googlebot, Googlebot-Image, Googlebot-Video, Googlebot-News, Storebot-Google, Google-InspectionTool, GoogleOther, GoogleOther-Image, GoogleOther-Video, Google-CloudVertexBot, and Google-Extended⁴. That is one company. Add OpenAI's three, Anthropic's several, Perplexity's two, and the rest of the named AI crawlers, and the realistic count of tokens an agency site might need an opinion about runs well past twenty, which is exactly the complexity the new shared signal is trying to collapse back down to one line for the training question specifically.

What the growing pile of robots.txt rules already costs an agency that never checks

It is tempting to treat all of this as too technical to matter for a two or three person agency. The honest cost is not a dollar figure, because this is a configuration problem, not a billing problem. The cost is a silent mismatch between what an owner thinks their site is doing and what it is actually doing, and the mismatch gets wider every time a new layer gets added to the same file.

Consider the scale of just one default. Cloudflare's managed robots.txt feature, active on over 3.8 million domains as of the policy's September 2025 launch, sets ai-train=no as the out of the box default for every one of them⁶. Some meaningful share of those domain owners chose that deliberately. A meaningful share almost certainly never looked at the setting at all, because it shipped as a default inside a security feature they turned on for an unrelated reason, DDoS protection or bot management, and never revisited. Multiply that pattern across every agency that installed a WordPress security plugin's "block AI bots" toggle, every site on a hosting platform that quietly ships its own AI-blocking default, and every robots.txt copied from a blog post written for a different kind of business entirely, and the honest picture is a large share of small business websites carrying an AI preference nobody at the business actually chose.

That matters specifically for an insurance agency because of what the preference actually governs. Training is the uncontroversial one to opt out of, your blog posts and plan pages improving a model you get no say in, if that is genuinely a concern for your agency. The problem is when the same blunt instrument that blocked training also blocked a citation-relevant crawler by mistake, because the person who added the rule, or the plugin that shipped the default, did not separate the tokens the way the companies that built them do. A site can end up invisible to the exact moment a prospect asks ChatGPT or Perplexity who handles Medicare enrollment in their county, not because a competitor out-marketed it, but because a setting nobody at the agency discussed quietly removed it from consideration.

Put a rough shape on it without inventing a number that was not fetched this session. Say a mid-sized agency's site answers, across a typical month, somewhere between a handful and a few dozen AI-driven questions about who handles Medicare or ACA enrollment in its county, a plausible range given how much of that research has moved off a traditional search box and into a chat window. If one crawler among the several that could have surfaced that answer is blocked by accident, the agency does not lose a fraction of those chances. It loses all of them, for the whole month, for that one assistant, because an AI crawler either sees the page or it does not, there is no partial credit. That is the actual shape of the risk: not a gradual decline, a binary switch flipped by a line nobody remembers writing.

Here is the self-audit we built as a companion to this post, written for exactly this blind spot, and it goes deeper into the per crawler layer specifically, with a sample file and a five minute check⁷: Can ChatGPT and Claude Even See Your Website? What follows here is the shorter version, focused on the two newer layers this post is actually about.

Read your own robots.txt in five minutes

You do not need a developer for this. Open yoursite.com/robots.txt in any browser. It is a plain text file, and every line in it is readable English once you know what the three layers above are looking for.

If what you find matches what you actually want, you are done, and that is a genuinely common outcome, not a trick answer. If you find a Disallow rule for a crawler you do not remember adding, or a Content-Signal line set to values nobody at your agency chose, that is worth fifteen more minutes figuring out where it came from, a plugin, a host default, a template copied from another site, before deciding whether to change it.

Three places those unexplained lines tend to come from, in rough order of how often we see them on an agency site we are auditing for the first time. A WordPress SEO or security plugin that shipped an "AI bot blocking" feature as a selling point, turned on by default or toggled once during setup and never revisited. A hosting platform or CDN that ships its own managed robots.txt, the way Cloudflare's does for over 3.8 million domains, with its own default baked in before the site owner ever opens the file⁶. And a template or starter theme, copied from a different kind of business entirely, carrying rules that made sense for whoever wrote them originally and mean nothing in particular for an insurance agency. None of the three is malicious. All three are the same root problem: a default chosen by someone who was never thinking about your agency specifically, sitting in a file almost nobody ever reads.

Should an insurance agency actually opt out of AI training?

It is a fair question with a real answer, not an obvious one either direction, and it is worth saying plainly where the honest tradeoff sits rather than pretending there is only one right choice.

The case for leaving training open: an agency chasing visibility in AI answers is, by definition, trying to be the thing a model recognizes and names. A large share of AI training still comes from the open web, and a site that disallows training broadly is choosing to not be part of that corpus going forward, on the same logic that caused the agency to want AEO in the first place. If the whole strategy is "be findable and citable," training is upstream of that goal, not opposed to it, for most agencies most of the time.

The case for opting out is real too, and it deserves to be said without hedging: some site owners simply do not want a company building a commercial model off content they wrote, full stop, as a matter of principle rather than strategy, and that is a legitimate business decision a site owner is entitled to make regardless of what it does to AI visibility. If that is your position, the Content-Signal line and the per crawler rules both support it cleanly today, and the new shared token will add a third, simpler way to say the same thing once it exists.

There is a version of this specific to insurance worth naming directly. An agency's own county level research, a plan comparison it wrote up, or a local market note built from its own carrier relationships is proprietary work in a way a generic "five tips for choosing insurance" post is not. Wanting that narrower category of original work excluded from training, while still wanting the same page fully visible to the crawlers that drive AI citation and grounding, is a coherent, defensible position, and it is exactly the kind of split the three-layer system above is built to support: a training preference set one way, a citation and search preference set another, on the same page, without the two ever needing to match.

Our own default, stated plainly: we want to be read, indexed, and cited, so every site we build ships open across all three layers by default, Allow on every named search and AI crawler, Content-Signal: search=yes, ai-input=yes, ai-train=yes, verified live on our own robots.txt this session⁷. That is our call for our own content, not a claim that it is the only defensible one for yours.

What we ship instead

This section stays narrow, describing only what is actually live, because that is the whole point of writing about compliance and configuration honestly. Every Digital Foundation build ships a robots.txt that names search engines and AI crawlers by their exact token rather than relying on a bare wildcard allow, since a wildcard can be silently overridden by a CDN or security plugin's own defaults in a way a named rule cannot. It carries a Content-Signal line stating all three preferences in one place from day one, and it is built on Astro, which renders pages to finished HTML at build time rather than shipping a JavaScript runtime the browser, or a crawler, has to execute before the content exists⁸.

The part that matters most for this specific topic: it updates. Sitemap, llms.txt, and robots.txt are generated by the site itself, not maintained by hand, and they update with the full absolute URL the moment a new page or post publishes. When the shared training signal moves from an individual draft to something crawlers actually read in practice, that is one change we make once, across every site we operate, rather than an email every client has to act on individually or forget to.

All three tiers are month to month with a 14-day free look period, verified live on the Digital Foundation page this session⁹. If what you actually need is narrower, just a correct, future-proofed crawler and citation setup on a site you are keeping as is, that is exactly the kind of scoped check the free Audit runs.

What changes once the file says what you actually mean

Strip the standards talk away and the outcome is small and concrete. Your robots.txt stops being an inherited artifact nobody at the agency understands and starts being a short, deliberate statement of two separate preferences: what may crawl and index your site, and what may train a model on it. Those two questions get answered on purpose instead of by whatever a plugin or a host defaulted to, and the gap between what you think your site allows and what it actually allows closes.

There is a second benefit that compounds quietly. A site built to welcome the citation-relevant crawlers by name, with the training question answered separately and on purpose, is positioned correctly for every future layer this standard adds, not just the one that exists today. The companies building these controls are explicit that they plan to keep crawler specific rules working alongside any shared signal¹², so a site that gets the fundamentals right now does not have to redo this work when the shared token moves from draft to something crawlers actually check.

What changes if you run more than one agency site

Everything above holds for one site run by one owner who can spend fifteen minutes reading a text file. It compounds for an FMO or IMO running several branded sites, or for downline agents each running their own, because the same inherited default, a plugin's AI-blocking toggle, a template copied from one site to the next, is not a problem that is five times smaller across five sites. It is five times larger, quietly repeated, usually without any one person responsible for noticing it across all five.

A shared training signal arriving sometime in 2027 raises the stakes on exactly this pattern, because a mistake in one line will eventually speak for more AI companies at once than it does today. Standardizing a correct robots.txt, Content-Signal line, and crawler welcome list across every site a multi-office operation runs, while each office keeps its own brand and its own content, is exactly the kind of work worth a real conversation rather than five separate, inconsistent fixes.

Questions agents ask