Cloudflare Sets A September 15 Crawler Deadline

Cloudflare plans to change its default crawler rules on September 15, 2026, targeting bots that combine traditional search crawling with AI agent activity or model-training access.

Under the policy, mixed-use crawlers will be blocked by default from pages that host ads unless the site owner chooses another setting.

Cloudflare said the change will apply to new customers, new sites created by existing customers and all existing free customers.

The rule gives publishers a sharper control point over AI access to web pages.

It also forces AI companies to make clearer distinctions between crawlers that support search discovery and crawlers that support training or agent services.

Cloudflare framed the change around publisher control rather than a blanket block on AI.

The company said website owners often want search discovery and some AI visibility, but they also want protection against content being used without payment.

Pay Per Use Expands The Publisher Payment Model

Cloudflare is also expanding its publisher-payment model from Pay Per Crawl to Pay Per Use.

The company said the newer model would let publishers charge AI companies when content creates value, not only when a crawler fetches it.

Cloudflare said Ceramic.ai and You.com are its first partners for the approach.

When a publisher opts in, Cloudflare said payment can occur when content appears in Ceramic's AI search results or when You.com accesses premium content.

The company also cited efficiency as part of the policy.

Cloudflare said its data suggested that more than 50% of AI crawler traffic is spent fetching pages that have not changed, adding bandwidth and compute costs for publishers and model providers.

Google Boundary Remains A Search Dispute

Cloudflare's announcement also challenges how large search operators separate discovery from AI use.

The company said the world's largest search engine has access to about 2x more information than other AI companies because publishers find it difficult to stay discoverable without also being used for AI.

Google has previously pointed to Google Extended, a crawler control that lets site owners opt out of training and AI product uses such as Gemini Apps and Vertex API without affecting inclusion in Google Search.

Googlebot still crawls for Search, including AI Overviews and AI Mode.

Cloudflare co-founder and CEO Matthew Prince said most internet traffic is now non-human and said clearer bot intent could support a more sustainable web ecosystem.

Cloudflare did not disclose pricing for Pay Per Use, publisher revenue terms, or which other AI companies will adopt the model.