The biggest names in AI are backing a new way to opt out of scraping
TL;DR
OpenAI, Google, Microsoft, and Anthropic say they will adopt a new standard that publishers can use to opt out of being scraped by AI, as reported earlier by Search Engine Roundtable. With this control, publishers can apply their AI training preferences across multiple AI companies, instead of setting them individually. The shared "policy token" signal is proposed by Kevin Kelley, a technical lead for standards at Anthropic, as part of the Internet Engineering Task Force (IETF).
Nauti's Take
For small teams, this is mainly a control and maintenance question: first verify whether the AI providers you care about actually honor the signal and how they distinguish training, search, and archiving. Until the IETF proposal is settled and widely implemented, treat robots.
txt as an additional instruction rather than reliable protection against every form of data use.
Sources