Patreon has begun actively blocking artificial intelligence bots from scraping creator content for training purposes. The membership platform is now working with Cloudflare to implement AI Crawl Control technology, a shift from its previous reliance on robots.txt files. This new approach directly prevents unauthorized AI training bots from accessing content.
The platform's move away from a passive request system to active blocking addresses the growing sophistication of AI scraping. In testing, Patreon observed a significant reduction in unauthorized AI training crawler attempts, dropping from thousands per week to zero after implementing Cloudflare's system. This change aims to provide creators with greater control over how their work is utilized by AI companies.
Patreon's product chief, Drew Rowny, stated that creators should not have to accept AI training on their work simply to reach an audience, contrasting this with the broader internet landscape. The company emphasized that this measure targets bots specifically designed for AI training and will continue to permit legitimate search engine crawlers that direct users back to Patreon, thereby preserving discoverability.
The platform's previous method, using robots.txt files, relied on the voluntary compliance of AI crawlers, many of which ignored these instructions. Patreon first introduced measures to deter AI crawlers in 2023. However, the increasing sophistication of AI scraping and the introduction of new discovery features on Patreon, such as a redesigned Home Feed and "Quips," expanded the amount of content potentially accessible to automated crawlers.
Cloudflare's AI Crawl Control technology operates at the network level, actively blocking unauthorized AI bots. This system distinguishes between bots that help creators gain visibility, like search engine crawlers, and those designed to train AI models without permission. Patreon's partnership extends its existing work with Cloudflare to leverage these more direct enforcement tools.
This development aligns with broader industry trends where content creators and publishers are increasingly seeking to control the use of their work in AI training datasets. Cloudflare has also updated its own policies, including blocking "mixed-use" crawlers by default on ad-supported pages and introducing a "Pay Per Crawl" marketplace. Patreon's implementation prioritizes outright exclusion over monetization for training bots.
CEO Jack Conte framed the move around creator rights, stating that creators deserve consent, credit, and compensation for their work. He indicated that if these conditions are not met, AI crawlers should not access Patreon's content. Rowny added that Patreon offers a different vision where creators can grow their audience while maintaining control over how their work is used, a departure from the common internet practice of accepting AI training for audience reach.
During testing, Cloudflare's AI Crawl Control technology demonstrated effectiveness by reducing weekly access attempts from individual AI training crawlers from thousands to zero. This indicates that many AI scrapers previously disregarded voluntary exclusion requests made via robots.txt files.
Patreon's policy update signifies a more assertive stance against unauthorized AI data collection. The company clarified that it will continue to allow bots that index pages and organize information to direct users back to Patreon, differentiating these from crawlers that extract content for AI model training. This approach aims to balance the need for platform discoverability with the imperative of data control for creators.