On October 30, four companies that rarely agree on anything agreed on one line of robots.txt.
Google, OpenAI, Microsoft and Anthropic each said they will honour a shared signal that tells AI developers not to train on your site. It replaces the need to maintain a separate rule for every crawler. If you run a website, it’s a real improvement, and it takes about two minutes to deploy.
It’s also the least important control you’ll set this year. The signal covers training, and training is no longer where most of the value in web content is captured.
1 token: __usage_ai-training, a robots.txt user-agent group defined in an IETF draft by Anthropic’s Kevin Kelley.
4 providers: Google, OpenAI, Microsoft and Anthropic have committed to honouring it. Anthropic started on October 30. Google and Microsoft target early 2027.
25,000:1 to 100,000:1: the range of Anthropic’s crawl-to-referral ratio through most of 2025, by Cloudflare’s measurement. That’s pages crawled per visitor sent back.
3: the uses the IETF draft vocabulary separates: AI Training, AI Use and Search. The new signal covers only the first.
What actually shipped
The mechanism is deliberately boring. The Internet-Draft adds no new directives. It reuses the ordinary robots.txt structure from RFC 9309 and gives it a reserved name that no real crawler uses:
User-Agent: __usage_ai-trainingDisallow: /Allow: /press-kit/That says “don’t train on anything except my press kit.” A User-Agent: * group doesn’t count, and a file with no group for the token expresses no preference at all. The draft also states that the token is not a crawling directive. Crawlers can still fetch your pages, so this records what you want, not who gets in.
Each company handles precedence slightly differently, and the differences matter. Anthropic’s support page says a rule that names ClaudeBot overrides the shared signal. Google’s Search Central post says opting out through the token without naming Google-Extended will be treated as disallowing Google-Extended. Microsoft is going further and splitting its controls in two: an AI Training preference, set in robots.txt or per page, and a separate AI Grounding preference set per page. Both replace the old NoArchive and NoCache tags. Bing says they take effect on February 1, 2027, both default to Allow, and neither changes whether you appear in Bing search.
The vocabulary underneath is honest about its limits. The IETF draft by Mozilla’s Martin Thomson and Open Future’s Paul Keller says that a recipient “has a choice whether to follow those preferences” and that “preferences are not a security mechanism.” The European Commission has given those preferences some legal weight. Signatories of the EU’s General-Purpose AI Code of Practice commit to honour robots.txt and other machine-readable rights reservations under Article 4 of the copyright directive. Outside that framework, the signal works only because the companies say they’ll respect it.
Training was the one-time payment
Think about what training actually takes. A model reads your page once, absorbs it into its weights, and never needs to come back. Opting out stops future uses of that kind. It does nothing about models already trained, and it doesn’t touch what happens when someone asks a question.
Value has moved to that second moment. When an assistant answers a question by fetching your page, quoting your price or summarising your review, your content is being used live, every time someone asks. The IETF vocabulary calls this “AI Use.” Microsoft calls it grounding. It’s the use the shared signal doesn’t cover.
The economics of that moment are poor for publishers. Cloudflare’s 2025 Radar review found that Anthropic’s crawlers fetched somewhere between 25,000 and 100,000 pages for every visitor Claude sent back, and OpenAI’s ratio peaked at around 3,700:1. Cloudflare itself notes that the Claude app doesn’t send a Referer header, so Anthropic’s figure is probably overstated. Even so, the trend points one way: AI systems read far more than they send back.
Agents make this harder still. OpenAI’s crawler documentation lists ChatGPT-User, which fetches pages when a person asks ChatGPT to, and states plainly that “because these actions are initiated by a user, robots.txt rules may not apply.” Anthropic’s Claude-User works the same way. That’s a reasonable position: a browser doesn’t check robots.txt either. But it means the fastest-growing kind of AI traffic sits outside the file everyone has just agreed on.
What about content that’s already been used?
The signal only works going forward, and that’s by design. The draft says a preference applies according to the robots.txt in effect when the page was retrieved. Anthropic’s support page frames blocking ClaudeBot the same way: it excludes future material from training datasets. Adding a Disallow today doesn’t reach models that have already been trained on your pages, or datasets already collected.
The reason is technical, not just policy. A trained model has no stored copy of your page to delete. Anthropic’s privacy documentation says models “do not store text like a database” and can’t retrieve their original training data. The fix researchers are working on is “machine unlearning”: removing a specific piece of data’s influence without retraining from scratch. It’s still an experimental field. A 2024 paper on unlearning in pre-trained models describes its methods as over 100,000 times cheaper than retraining. That figure shows how expensive the only exact method, retraining from scratch, would be.
You do have three partial options:
- Find out whether you’re in a model. Under the EU AI Act, providers of general-purpose models must publish a training-data summary using the Commission’s template. For scraped data, it names their crawlers and the collection period, and lists the top 10% of domains collected (top 5%, or 1,000 domains, for smaller providers). Models already on the market before August 2025 have until August 2, 2027 to publish theirs.
- Use privacy law for personal data. If what you need removed is personal data, GDPR rights to object and to request erasure apply. Anthropic, for one, accepts these requests for training data, while warning that it’s limited in what it can identify within a trained model.
- Negotiate, rather than retract. For everything else, past use is settled through commercial agreements, not robots.txt. Both IETF drafts say private agreements can override machine-readable preferences.
In practice, you can’t take back what’s already been used. What you can control is future models, and how your content is used each time someone asks a question.
From “no” to a price
The tools for charging, rather than just refusing, are arriving at the same time.
Cloudflare launched Pay Per Crawl in 2025. It returns HTTP 402 Payment Required to crawlers that haven’t agreed to a price. On September 30 it added a beta called Pay Per Use, built on the argument that “AI products fetch far more than they use.” Buyers report each actual use, such as an excerpt shown in a search answer, and pay per use rather than per fetch. For APIs and data feeds, where every request is a use, its Monetization Gateway charges agents per call using the open x402 protocol.
RSL 1.0, published in December 2025 and supported by Cloudflare, Akamai and Creative Commons, provides the licensing language. It separates ai-train from ai-input and ai-index, and it lets you require payment per crawl, per training use or per AI-generated output.
None of this works unless you know who’s asking, and user-agent strings are trivial to fake. Web Bot Auth, now in an IETF working group, has agents sign each request with a published key. OpenAI says it signs its Operator agent’s requests this way, and Cloudflare says Visa and Mastercard will host key directories for agents they register to make purchases. Identity is the basis for the rest: you can’t price or permit what you can’t recognise.
What this means for your company
Most enterprises aren’t publishers, but almost all of them publish something AI wants: documentation, product catalogues, pricing, reviews, research, support answers.
- Deploy the shared signal, but decide per path. A blanket
Disallow: /is easy and usually wrong. You probably want your docs and product pages in every model, since that’s how agents learn to recommend you. Your proprietary research, premium content and customer-generated material may be different. The draft’sAllow/Disallowstructure is built for exactly that split. - Keep vendor-specific rules. Named crawler rules override the shared token at Anthropic and Google. Audit your existing GPTBot, ClaudeBot and Google-Extended lines so they don’t contradict the new one by accident.
- Treat grounding as a separate decision from training. If you use NoArchive or NoCache on Bing, you have until March 1, 2027 to move to Microsoft’s new grounding preference. Being excluded from grounding may mean being missing from the answer entirely, so decide that deliberately.
- Measure before you charge. Find out which bots fetch what and how much traffic each sends back. Your CDN’s logs or a crawl-to-referral dashboard will show this. Then price according to the value of your content, not the number of times it’s fetched.
- Start building agent identity now. If you expose APIs or data, plan to accept signed agent requests and to meter them. That’s where per-request revenue will come from.
The shared signal settles an old question: whether your content can be trained on. That was mostly decided years ago, by the models that already read the web. The question that matters now is what an agent pays to use your content each time someone asks it something, and robots.txt doesn’t answer that.
Sources: IETF — Machine-Readable Training Preference Signal (draft-kelley-aipref-training-preference-signal-00), IETF — A Vocabulary For Expressing AI Usage Preferences (draft-ietf-aipref-vocab-08), OpenAI — Making it easier for websites to express AI training preferences, OpenAI — Overview of OpenAI crawlers, Google Search Central — Supporting a shared AI training control for websites, Google — Common crawlers (Google-Extended), Microsoft Bing — New AI preferences, Anthropic — Does Anthropic crawl data from the web, and how can site owners block the crawler?, Anthropic — Privacy rights requests relating to Anthropic’s training data, European Commission — Explanatory Notice and Template for the Public Summary of Training Content for general-purpose AI models, Yao et al. — Machine Unlearning of Pre-trained Large Language Models (arXiv:2402.15159), European Commission — Consultation on protocols for reserving rights from text and data mining under the AI Act and the GPAI Code of Practice, Cloudflare — The 2025 Cloudflare Radar Year in Review, Cloudflare — Introducing pay per crawl, Cloudflare — Pay Per Use: when AI uses your work, you should get paid, Cloudflare — The age of agents: cryptographically recognizing agent traffic, RSL — Really Simple Licensing 1.0 specification, IETF — HTTP Message Signatures for automated traffic (Web Bot Auth)
Opinions are my own.