Cloudflare Draws a Hard Line on AI Crawlers, Forcing a New Payment Era for Publishers

Cloudflare is making one of its boldest moves yet in the fight over who gets paid when artificial intelligence systems consume online content. The internet infrastructure company has announced a policy change that will, by default, block so-called mixed-use crawlers from ad-supported pages starting September 15, 2026 — a step designed to separate ordinary search indexing from the scraping used to train AI models and power agentic products.

The shift is more than a technical setting. It is a direct challenge to the current economics of the web, where publishers have long allowed search engines to index their work in exchange for traffic, but increasingly face AI systems that extract value from their content without sending readers back. Cloudflare’s new rules aim to make that imbalance harder to sustain.

Under the company’s updated defaults, crawlers that combine search, agent behavior and training use will be denied access to sites that display ads unless a publisher explicitly changes the setting. The policy will apply automatically to new Cloudflare customers, new websites created by existing customers, and all free-tier sites already on the platform.

For AI companies, the change could raise the cost and complexity of gathering web data. For publishers, it could become a lever to negotiate compensation, enforce access rules and preserve bandwidth. Cloudflare is framing the update as part of a broader attempt to build what it calls a more sustainable internet economy in the age of machine traffic.

What Cloudflare changed and why it matters

The company’s announcement targets a blurry category of web bots that no longer fit neatly into the old search-engine model. Traditional crawlers were easy to understand: they indexed pages so users could discover them through search. Newer AI crawlers do more. They may crawl for indexing, collect data for model training, and fuel interactive AI assistants or agents that answer questions directly instead of referring users to the source.

Cloudflare’s concern is that many sites have not clearly consented to that broader use. A publisher may be comfortable being indexed by a search engine, but not with that same content being folded into a model or used to generate synthetic answers that reduce referrals and advertising revenue.

The company says the default change is meant to give site owners better control while still allowing legitimate discovery. In practice, the policy is intended to force a clearer line between bots that index the web and bots that extract content to support AI products.

A change aimed at the ad-supported web

The new default specifically targets pages that host advertising. That matters because ad-supported publishing remains one of the web’s core business models, and those sites are especially vulnerable when AI systems reuse their content without compensation or click-throughs.

Cloudflare argues that the existing arrangement increasingly disadvantages publishers. Search engines can still send traffic, but AI products can also absorb the substance of the page and surface the answer themselves. That means the content creator bears the costs of production while the platform captures much of the downstream value.

The company is trying to alter that equation by making access a choice rather than a default entitlement.

The Google factor: search, AI and the murky middle

Cloudflare’s announcement also takes aim at the tension created by the largest search platforms, particularly Google. The company says the biggest search engine has access to roughly twice as much information as other AI companies because website owners often have to remain open to search indexing in order to stay visible online, even if they would prefer to block AI use.

That point touches a longstanding frustration among publishers: opting out of search can mean disappearing from the open web, so the practical choice is often to accept whatever access the dominant platform requires.

Google has previously rejected broad claims that its indexing and AI systems are inseparable. The company has said that publishers can use Google Extended to opt out of content being used for training or AI-related products such as Gemini apps and Vertex API services, without affecting inclusion in Google Search. At the same time, Googlebot still crawls content for search results and for AI-powered search features like AI Overviews and AI Mode.

That distinction is important, but it has not ended the debate. Cloudflare’s move suggests that infrastructure providers believe the industry still needs stricter defaults, not just opt-out tools buried in settings panels.

Matthew Prince, Cloudflare’s co-founder and chief executive, said the company is acting faster because bots now make up the majority of internet traffic, and that the web needs new rules to support a sustainable ecosystem.

Prince’s framing is notable because it treats the bot problem as a structural internet issue, not merely an AI industry problem. If machine traffic now outweighs human traffic, then the economics of crawling, indexing and access may need a complete reset.

From Pay Per Crawl to Pay Per Use

Cloudflare is not starting from zero. In recent years, the company has introduced several tools to help publishers regulate how bots interact with their sites. One of the most ambitious was a marketplace model known as Pay Per Crawl, which gave websites a way to charge AI bots for scraping content.

Now that concept is expanding into something broader: Pay Per Use.

The difference is important. Pay Per Crawl focuses on the act of fetching content. Pay Per Use is designed to compensate publishers when their material actually creates value inside an AI product, not merely when a bot downloads it. That could mean payment when a publisher’s article appears in AI search results, surfaces in a premium response, or otherwise contributes to a monetized AI experience.

In theory, this moves closer to a licensing model. Instead of treating the web as a free reservoir for AI systems, it establishes a mechanism for economic participation when content powers a commercial service.

Cloudflare says the new framework could also cut waste. The company’s analysis suggests that more than half of AI crawler traffic is spent repeatedly requesting pages that have not changed. If that figure holds, then a large share of crawl activity is not just costly to publishers, but inefficient for AI companies as well.

How the new default works

Cloudflare’s policy is not a universal ban. It is a default setting. That means publishers can still decide to allow access if they want to, but the company is shifting the baseline toward restriction rather than openness for mixed-use bots.

The distinction matters because defaults shape behavior. Many website owners never revisit bot rules after setup, especially smaller publishers without technical staff. By changing the default, Cloudflare is effectively changing the starting point for millions of sites that rely on its infrastructure.

The policy will automatically apply in several cases:

• new Cloudflare customers
• new websites created by existing customers
• all existing free-tier customers

That broad rollout suggests Cloudflare wants the new rule to become normal rather than exceptional. For larger customers, the change will likely prompt direct conversations about bot permissions, licensing and revenue-sharing arrangements.

Which crawlers are affected

The emphasis is on mixed-use crawlers — bots that do not serve a single, clean purpose. Cloudflare wants to separate three functions that have become entangled in the AI era:

1. traditional search indexing
2. AI agent access
3. model training data collection

In the old web model, indexing was the main game. In the new model, a single crawler may feed several products at once. That overlap is exactly what Cloudflare is trying to unwind.

For site owners, the practical effect is likely to be more granular permissions. A publisher may allow search indexing while denying training access, or permit one AI partner to use content under a commercial agreement while blocking other bots entirely.

Why publishers have been pushing back

Publishers have spent the last two years arguing that generative AI platforms are changing the value chain of the internet. When a chatbot answers a query using a publisher’s reporting, the user may never click through to the original article. That removes page views, weakens ad performance and reduces subscription conversion opportunities.

At the same time, the publisher still bears the cost of reporting, editing, hosting and compliance. The more AI systems absorb that content, the more publishers worry that they are being transformed from traffic sources into invisible training fuel.

Cloudflare’s policy is likely to resonate with publishers for that reason. It does not solve every problem, but it gives them another mechanism to assert ownership over access and to demand payment where AI systems create commercial value from their work.

There is also a bandwidth argument. If bots are repeatedly retrieving the same content without a business relationship, publishers are subsidizing the AI industry’s infrastructure usage. Cloudflare says its data show that much of that traffic is redundant, with bots revisiting unchanged pages over and over.

Why AI companies may be forced to adapt

The policy could affect several parts of the AI stack.

Training data pipelines

Model developers that rely on large-scale web scraping may find some sources more difficult to access. That could encourage more licensing deals, more first-party data collection, or a shift toward open datasets and content partnerships.

Agentic products

AI agents often require live web access to browse, summarize and act on current information. If more sites block mixed-use crawlers by default, agent providers may need to prove intent more clearly and negotiate separate access terms.

Search products with AI layers

Search engines that have increasingly embedded AI features may be pushed to clarify which bot is doing what. That could accelerate the breakup of the one-crawler-does-everything approach.

For AI firms, the broader message is clear: access to the web may no longer be assumed, especially for commercial products that monetize content indirectly.

Cloudflare’s own position in the AI ecosystem

Cloudflare’s move is notable because the company is not anti-AI. It has also built products that help customers deploy AI systems of their own. That makes this policy less of an ideological stance and more of an infrastructure decision about who should control access to content and under what terms.

That dual role gives Cloudflare a strong vantage point. It sits between publishers and AI companies, managing traffic, permissions and delivery at scale. As a result, it can turn policy into infrastructure rather than just commentary.

In the current debate over AI and publishing, that matters. The company is not simply arguing for fair compensation; it is designing a mechanism that could make compensation enforceable.

What happens on September 15, 2026

That date is the first major checkpoint. From then on, the new default blocking of mixed-use crawlers will begin across the affected Cloudflare-managed sites. For publishers, that means the burden shifts toward deciding which bots to allow, which partners to trust and whether to opt into monetization models tied to AI usage.

For AI companies, the deadline creates urgency. If they want to keep broad access to publisher content, they may need to separate crawlers, disclose intent more clearly or negotiate commercial agreements before the new rules take hold.

It is not yet clear how many publishers will keep the default block in place, or how many will open access in exchange for compensation. But Cloudflare’s move is likely to influence the wider market by making restrictions easier and monetization more normalized.

Timeline of the Cloudflare policy shift

The bigger fight over the future of the web

Cloudflare’s announcement is part of a much larger argument about whether the open web can remain open in its traditional form while still supporting AI at scale. Search engines built the first economic contract of the internet: index our pages, send us traffic, and everyone benefits. AI has complicated that bargain.

Now, publishers want a second contract — one in which access to content is limited, negotiated or paid for when it is used to train models and drive commercial AI experiences. Infrastructure companies like Cloudflare appear increasingly willing to enforce that contract on behalf of their customers.

That does not mean the entire web will suddenly shut off from AI systems. But it does mean the default assumption is changing. The era of unrestricted crawling is giving way to a more explicit permission economy, where publishers are likely to ask three questions before allowing access: who are you, what are you using this for, and how do we get paid if our content helps you make money?

As Prince suggested, the scale of machine traffic is forcing the internet toward a new operating model. Whether that model becomes sustainable will depend on how many publishers keep the gates closed, how many AI companies are willing to pay, and how aggressively infrastructure providers like Cloudflare enforce the new rules.

What is clear is that the balance of power is shifting. The web’s raw material is becoming harder to take for free.

Cloudflare says its tools and partnerships are intended to give publishers more visibility, more commercial leverage and better control over how their work is used by AI systems.

What to watch next

• Whether major publishers adopt the new default block and keep it in place.
• How quickly AI companies split their crawlers into separate search, training and agent functions.
• Whether licensing and revenue-sharing models become standard for high-value content.
• How Google and other major search platforms respond to growing pressure for clearer permissions.
• Whether Cloudflare’s approach becomes an industry norm for bot access control.

For now, Cloudflare has done what few infrastructure players have been willing to do: it has used its position in the middle of the web to redraw the rules. The result could be a tougher internet for AI crawlers — and a more valuable one for the publishers that still feed it.

superintelligencenews@gmail.com