OpenAI, Google, Microsoft and Anthropic unite behind a shared AI scraping opt-out standard

OpenAI, Google, Microsoft and Anthropic are backing a shared robots.txt-based standard that would let publishers signal, in one place, that they do not want their work used for AI scraping or model training. The move, reported by The Verge, could make it easier for sites to set AI training preferences across multiple crawlers instead of repeating the same request company by company.

What the new standard changes

The proposed system centers on a new “policy token” that would sit inside robots.txt, the long-used file websites already use to communicate crawler access rules. The token is being developed under Internet Engineering Task Force work, according to The Verge, and was proposed by Anthropic’s Kevin Kelley.

If adopted broadly, the token would let a publisher express a preference once and have participating crawlers read that instruction in the same format. That is a notable shift from the current setup, which requires publishers to signal AI training preferences separately to each company.

According to the reporting, companies backing the standard say crawlers must stop accessing a site when they encounter the shared token. Microsoft also said it will add finer controls that distinguish between AI training opt-outs and AI grounding opt-outs, while noting that Bing search results will still show sites regardless of those preferences.

Why it matters

The change matters because publishers have spent the past few years pushing back on AI companies’ use of web content for model training. A shared opt-out token does not end that conflict, but it could reduce the friction around saying no. Instead of negotiating separately with every major AI provider, publishers could rely on a common signal recognized by multiple crawlers.

That has two major implications. First, it gives websites a more practical tool for controlling access to their material. Second, it may help normalize the idea that crawler access can be managed through a shared technical standard, which could make future disputes about scraping, licensing and compensation more structured, but not necessarily smaller.

The standard also sits alongside another direction the industry has been exploring: giving publishers the ability to charge crawlers. The Really Simple Licensing, or RSL, standard aims to do that, according to the fact sheet. In other words, the debate is no longer just about blocking access. It is increasingly about whether AI companies should have to ask, pay, or stop.

How does the robots.txt token work?

The shared signal is designed to live inside robots.txt, a file that has long served as a site-level instruction sheet for web crawlers. The new token would extend that familiar mechanism to AI-specific uses, giving publishers a simple place to state their preferences about training and related access.

In practical terms, the promise is consistency. Right now, a publisher that wants to discourage AI use of its content may have to deal with multiple separate systems and policies. The new token aims to make that simpler by creating one shared instruction that participating companies can recognize.

How the companies are lining up

The Verge reported that Anthropic already supports the signal. Google and Microsoft plan to support it next year, while OpenAI says implementation is due in the coming months. Those different timelines matter: a standard can only shape behavior once enough major crawlers actually honor it.

Microsoft’s position is also more nuanced than a simple opt-out. The company said it will introduce finer controls so publishers can distinguish between AI training and AI grounding preferences. At the same time, Microsoft said Bing search results will still include sites regardless of those preferences, an important reminder that search indexing and AI data use are not identical.

What is publisher pushback really about?

Publisher complaints have intensified as AI crawlers increasingly scrape content for model training. The issue is not only traffic loss, though that matters. It is also about control over valuable text, images and other material that publishers argue is being harvested without clear permission or compensation.

This is why the shared standard is significant even if it does not fully solve the underlying dispute. It gives publishers a cleaner technical objection, but it does not automatically force every AI company to obey. The power of the system will depend on adoption, enforcement and whether publishers trust that a token in robots.txt will actually be respected.

Cloudflare has already been blocking AI crawlers by default, according to the fact sheet, which shows that some infrastructure providers are moving beyond voluntary requests. The new cross-company standard, by contrast, appears aimed at broad interoperability rather than hard blocking. That distinction could matter as publishers choose between a common signal and a stronger gatekeeper.

Background: how we got here

robots.txt is not new. For years, websites have used it to tell search engines and other crawlers what parts of a site they can access. What is new is the pressure AI has put on that old system. As model makers have raced to build larger, more capable systems, publishers have become more vocal about how their material is collected and used.

The reporting from The Verge places this latest move in the middle of a wider industry response. AI companies have faced growing publisher pushback over crawlers used to collect training data. In response, publishers have pursued different tools: technical standards, blocking systems and licensing approaches. The proposed policy token is one more attempt to make those preferences machine-readable and widely recognized.

That history helps explain why the announcement matters beyond a single file format. It is not simply a coding update. It is part of an effort to define the rules of access on the web at a moment when AI systems depend heavily on online content and publishers are looking for leverage.

Where the standard comes from

The proposed token is linked to work at the Internet Engineering Task Force, the body that helps develop internet standards. Kevin Kelley of Anthropic is named in the fact sheet as the person who proposed the “policy token.” That matters because standards are often only as useful as the degree to which major firms and institutions adopt them.

By placing the token inside robots.txt, the proposal borrows from an existing convention rather than inventing a separate system from scratch. That lowers the barrier to entry, at least in theory. Site operators already understand the file, and crawlers already know how to read it.

What happens next?

The most immediate question is whether publishers will use the token widely. A standard can be elegant on paper and still fail if too few sites adopt it. Publishers will likely watch closely to see whether the major AI crawlers actually honor the signal in practice, rather than only in policy statements.

Another key question is implementation speed. OpenAI says it will support the signal in the coming months, while Google and Microsoft are aiming for next year. Those timelines leave a gap in which behavior could remain uneven, especially if different crawlers or different services interpret the standard differently.

There is also the broader issue of enforcement. The fact sheet does not suggest a formal penalty system, so the standard’s effectiveness will depend on trust, pressure and industry alignment. If that weakens, publishers may continue to rely on separate licensing deals, blocking tools or other defenses.

What to watch

• Whether major publishers begin adding the shared token to robots.txt.
• Whether OpenAI, Google and Microsoft roll out support on the timelines described.
• Whether other AI crawlers outside this group accept the standard.
• Whether the RSL licensing approach gains traction alongside opt-out tools.
• Whether Cloudflare-style blocking expands as an alternative to voluntary signals.

By the numbers

The story is less about hard numeric targets than about timing and scope. Even so, the rollout schedule helps frame the stakes for publishers deciding whether to adopt the new token now or wait for broader support.

How the industry may change

If the standard sticks, it could reshape the way publishers and AI companies negotiate access. A shared token would not eliminate disputes, but it would create a common starting point. That could make future discussions more predictable, especially for publishers that want a single technical control rather than a patchwork of rules.

At the same time, the move may also make it easier for AI firms to frame access as orderly and consensual, even as deeper questions about compensation remain unresolved. In that sense, the standard could stabilize the process without settling the politics.

“websites will still appear in Bing’s search results regardless of their preferences”

That statement from Microsoft captures the tension at the heart of the proposal. Search visibility and AI training are being separated into different buckets, but publishers are likely to keep asking where the line really runs and who gets to draw it.

Bottom line

OpenAI, Google, Microsoft and Anthropic are moving toward a shared robots.txt-based opt-out standard that could give publishers one place to signal they do not want their content scraped for AI training. The shift may simplify compliance, but it also shows how far the industry still is from a settled agreement on how web content should be used in the AI era.