Cloudflare is about to change the default settings for how a big chunk of the internet gets crawled, and it targets a very specific kind of bot: mixed-use crawlers that do two jobs at once. Starting September 15, 2026, new Cloudflare customers and new sites for existing customers will, by default, allow search crawling but block training and “agents” from pages with ads. Cloudflare is also applying the same change to free tier customers that have not changed their settings.

In plain English: if your site makes money from ads, Cloudflare wants you to decide whether those crawlers can harvest your content for AI training and agent use. Cloudflare’s stated rationale is that content driving revenue “cannot be crawled without explicit permission of those content owners.”

This is not happening in a vacuum. The web’s crawling ecosystem has been around for decades, with search engines sending automated requests to index pages. Googlebot is the flagship example: Google has used crawlers for search inclusion for years. Over the past few years, many crawlers started visiting sites with a second mission, harvesting content to train AI models. That shift triggered countermeasures from publishers who feel they are not being fairly compensated for the content used to feed AI systems.

The hard part is that some of the bots doing both jobs are also the ones publishers fear blocking. The source points out that Googlebot combines crawling for search indexing with content harvesting for AI training. That means publishers have often tolerated the bot because they worry blocking it could mean disappearing from Google Search results. Microsoft’s Bingbot faces a similar “two-in-one” problem, and Apple has also used Applebot for both indexing and AI-related data gathering.

Cloudflare is stepping into that tension by trying to give site owners a clearer, declarative control gate. The company specifically says it wants to separate search from “agent use and training,” and it frames the shift as an ecosystem maintenance issue: “Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge,” Matthew Prince, Cloudflare co-founder and CEO, said in a statement. Cloudflare also argues that its new tools and partnerships will increase “visibility and commercial opportunities” while rewarding AI companies with bots that have “clear and transparent intent.”

So what happens to the major crawler operators mentioned in the source? Apple, Google, and Microsoft’s Bing crawlers could fall afoul of Cloudflare’s default behavior. But the source also notes that each of the tech giants offers an AI opt-out that may allow them to escape sanctions. In the Apple and Google model, publishers can use opt-outs via robots.txt directives, specifically Applebot-Extended and Google-Extended. Bing, meanwhile, supports a robots meta tag attribute, content="noarchive", that also blocks data harvesting. Other crawler operators often ignore voluntary robots.txt, which is one reason Cloudflare is aiming to make the control less optional and more enforceable through default behavior.

This is where decision-makers should focus: it is not just about whether bots can access a page, it is about what you can afford to lose in discovery. If your ad-supported pages get crawled for training and agent use by default, you are effectively letting AI systems consume your content. If you block them, you reduce that consumption, but you also risk the real business concern publishers have been living with: traffic loss in search, or confusion over what is blocked and what is merely governed.

Cloudflare is also layering additional commercial and measurement changes on top of the access gate. It is rebranding its “Pay Per Crawl” tollbooth as “Pay Per Use.” The idea, according to the source, is to reward publishers when their content creates value instead of just when it is fetched. To operationalize that, Cloudflare is partnering with Ceramic.ai, an API-based search business, so publishers get paid whenever their content appears in a Ceramic.ai search result. It is also working with You.com, a search engine for AI agents, to generate content payments whenever there is demand from an agent.

Finally, Cloudflare is introducing a new Business Insights Dashboard, designed to give publishers more visibility into how bots are consuming content and how much traffic AI models send. That matters because policy without measurement turns into noise. If you are a publisher, or a platform partner relying on publisher traffic, you need to know whether enforcement changes crawling behavior, whether that affects downstream distribution, and whether any payment mechanisms reflect real demand.

For executives across the web ecosystem, this is a governance story with distribution consequences. Cloudflare’s default change is effectively a bid to move from “robots.txt as suggestion” and “mixed-use bots as fait accompli” toward something more explicit: search access separated from training and agent use. Whether you run a site behind Cloudflare, sell ad inventory, build AI systems that rely on web content, or invest in the tooling in between, the direction is clear. The next 12 to 18 months are about learning how these controls interact with search discovery, AI training pipelines, and agent-driven demand. And if you are not planning for September 15, 2026, you may find your content is suddenly gated in ways you did not intend.