@Captainant
San Francisco, CA, July 1, 2025 –Cloudflare, Inc. (NYSE: NET), the leading connectivity cloud company, today announced it is now the first Internet infrastructure provider to block AI crawlers accessing content without permission or compensation, by default. Starting today, website owners can choose if they want AI crawlers to access their content, and decide how AI companies can use it. AI companies can also now clearly state their purpose – if their crawlers are used for training, inference, or search – to help website owners decide which crawlers to allow. Cloudflare's new default setting is the first step toward a more sustainable future for both content creators and AI innovators.
For decades, the Internet has operated on a simple exchange: search engines index content and direct users back to original websites, generating traffic and ad revenue for websites of all sizes. This cycle rewards creators that produce quality content with money and a following, while helping users discover new and relevant information. That model is now broken. AI crawlers collect content like text, articles, and images to generate answers, without sending visitors to the original source – depriving content creators of revenue, and the satisfaction of knowing someone is viewing their content. If the incentive to create original, quality content disappears, society ends up losing, and the future of the Internet is at risk.
“If the Internet is going to survive the age of AI, we need to give publishers the control they deserve and build a new
https://www.cloudflare.com/press-releases/2025/cloudflare-just-changed-how-ai-crawlers-scrape-the-internet-at-large/