On July 1st local time, Cloudflare updated its crawler management rules, tagging various crawlers with labels such as search, AI proxy, and model training. September 15th, 2026 is set as the implementation date, after which AI proxies and crawlers used for training purposes will be blocked by default from accessing web pages with advertisements.

Most website administrators hope that their content can be indexed while protecting their intellectual property revenue. However, many hybrid crawlers on the market have both search and AI collection functions, making them difficult to distinguish and manage. The platform's crawler manager will identify all purposes of a crawler, and if any one of these activities is prohibited by the site, the entire hybrid crawler will be unable to scrape the website's content.

Cloudflare will launch a new data dashboard, adapting to the industry shift in network optimization from SEO, GEO to AEO (Answer Engine Optimization), making it easier for website administrators to view traffic sources.

The platform also introduced a page change monitoring feature, filtering repeated crawl requests for pages without updates, reducing server bandwidth consumption. At the same time, the platform implements a crawler billing model based on usage, allowing website owners to charge corresponding fees for AI content crawling, balancing the rights and interests between content creation and AI data collection.