New AI crawler classifications, AI bot analytics, and Pay Per Use help publishers manage AI access, improve discoverability, and monetize their content for the Agentic Internet.
Key Updates for the Agentic Internet
- New AI crawler classifications to distinguish search, AI training, and agent use.
- Default AI crawl settings for ad-supported websites from September 15, 2026.
- Attribution Business Insights dashboard to track AI bot activity and referral traffic.
- Answer Engine Optimization (AEO) insights to help publishers understand how AI cites their content.
- Pay Per Use replaces Pay Per Crawl, allowing publishers to be compensated when their content is used by AI.
- Partnerships with Ceramic.ai and You.com to support AI content licensing and monetization.
Cloudflare has announced new classifications, enhanced analytics, and commercial partnerships that bring together site owners and transparent AI companies to support the growing agentic Internet. Cloudflare’s new tools and integrations help site owners manage AI access, improve discoverability, and monetize their content.
According to Cloudflare, the fundamental nature of Internet traffic has shifted. Today, automated agents and bots drive more than half of all web requests. AI is becoming the new interface for accessing information and conducting commerce online, changing the rules for how content is discovered. Most website owners want their content in AI systems. But for businesses built on advertising or subscriptions, they are threatened by AI systems training on their content without compensation.
Cloudflare adds, if businesses need their content in AI systems, it should be an easy and efficient process; but if businesses do not want that, then they should have the tools to restrict AI’s access. To tackle this, Cloudflare is testing new default classifications, delivering deeper insights to customers, making AI search faster, and ensuring creators are compensated when their content powers an answer. Cloudflare says the changes are intended to improve crawling efficiency, protect intellectual property, and help creators monetize content used by AI systems.
“Last year we provided site owners with transparency and control over what bots access their content, and we are thrilled with the benefits it has had to the ecosystem,” said Matthew Prince, Co-founder and CEO of Cloudflare. “Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge. Cloudflare’s new tools and partnerships give website owners increased visibility and commercial opportunities and benefit AI companies that have bots with clear and transparent intent. We hope that our proposed default changes encourage mixed use crawlers to separate out search from agent use and training.”
A Deadline to End the Free Pass for Mixed-Used Crawlers
Cloudflare says that it has had thousands of conversations with AI companies and site owners, and most site owners want to be discoverable in AI. However, for those who monetize via ads and/or subscriptions, maximizing discovery shouldn’t mean giving away intellectual property for free. Transparent AI companies have enabled this by clearly separating and labeling their bots by intent.
Join the Techitup Executive Network
Join the Middle East’s inner circle of C-Suite Tech Leaders. Exclusive access to closed-door regional roundtables, priority VIP seating at our annual awards, and strategic industry briefings.
However, mixed crawler bots disadvantage both site owners and transparent AI companies by blending search, agent use, and training. This forces content owners on the internet to choose between being discoverable in an agentic era and giving away their most valuable content without compensation.
Over the next two months, Cloudflare will engage with the ecosystem, listen to feedback, and run tests to finalize new defaults and classifications with a deadline of September 15, 2026.
For new customers and for new sites for existing customers, on September 15, 2026, the defaults will be set to allow for search but block training and agent use for pages with ads. Mixed crawlers that do not give site owners the ability to choose between search, agent use, and training, will be blocked on all pages with ads. At any time these settings can be changed by customers in their dashboard.
On September 15, 2026 these changes will also be made for all existing free customers that have not changed their settings by September 15, 2026 in their dashboard. These changes aim to level the playing field between AI companies, who are redefining what it means to be discoverable on the agentic internet in an equitable and transparent way, and legacy search companies, adds Cloudflare.
Enhanced Analytics to Drive Visibility and Monetize Content
The market for monetized content is here, with publishers and AI platforms signing more than 50 major content licensing agreements over the past year alone. To accelerate these arrangements, Cloudflare is introducing the new Attribution Business Insights dashboard, giving publishers greater visibility into how AI companies use their content. Business stakeholders, not just security and IT teams, will now clearly see how AI bots consume their content and track how much human traffic specific AI companies actually send back.
But understanding how AI consumes content is only the first step. As discovery shifts from search engines to answer engines, customers need to perfect a new discipline: Answer Engine Optimization (AEO). AEO is about understanding not just who is crawling, but how your content is cited in AI-generated answers, how often, and how prominently, across models. Just as web analytics became essential infrastructure for the search era, Cloudflare aims to provide the definitive measurement and optimization layer for the AI era.
Smarter Search and Fair Compensation
Negotiated deals can work for the largest publishers, but they don’t scale to the rest of the web. Cloudflare is working on two fronts that help creators and AI companies alike: making AI search more efficient, and paying creators automatically when their work is used. Today, AI models and search engines repeatedly crawl millions of unchanged web pages searching for fresh data to answer user queries. According to Cloudflare, more than 50% of crawl traffic from AI crawlers is spent re-fetching unchanged pages.
The result is wasted bandwidth for publishers, wasted compute for AI companies, and no better answers to show for it. Because Cloudflare can see what has actually changed, it can signal when a page is worth re-fetching and when it isn’t — cutting needless crawls in the process, while delivering better results. Cloudflare is testing these signals with leading AI companies to measure how much they improve freshness and reduce wasted crawling, and plans to make them broadly available later this year.
A year ago, on the first Content Independence Day, Cloudflare launched the first version of Pay Per Crawl so publishers could charge AI companies to crawl their content. Today, Cloudflare is evolving Pay Per Crawl into Pay Per Use, so publishers are paid when their content actually creates value, not just when it’s fetched. The company is working with partners, including Ceramic.ai and You.com to to support new content licensing models.
With Ceramic.ai, when a publisher opts in, they are paid every time their content appears in Ceramic’s AI search results, and they get back the queries, citations, and ranking back. Other AI companies can bring the model that fits how they work. You.com, for instance, will let an agent pay on demand for a specific piece of premium content it needs, while Cloudflare provides the common layer beneath them all.
Cloudflare says the latest updates build on its efforts to give publishers greater control over AI access, improve transparency, and create new monetization opportunities for content used by AI systems.
