AI Crawl Control

Visibility and control for AI content scraping AI crawlers may scrape webpages thousands of times for every referral they send. AI Crawl Control lets you control when and how they can access your content. Understand which AI crawlers are scraping your site, how often they do it, and whether they’re following your rules. Set a crawler-by-crawler strategy for who should be blocked and who should be allowed through. Use machine learning, behavioral analysis, and fingerprinting based on Cloudflare’s visibility into 20% of all Internet traffic. Charge the AI crawlers of your choice for scraping your data [in private beta]. HOW IT WORKS Cloudflare sees and fingerprints more AI bots than any other provider, thanks to our broad visibility into global Internet traffic. AI Crawl Control uses this visibility to give you granular insight into AI crawler activity on your domains. It also lets you: Block or allow specific crawlers Charge certain crawlers for access to your content [in private beta] “While bots, data scraping, credential harvesting, and DDoS attacks continue to evolve, Cloudflare allows us to mitigate incoming threats, protecting millions of storefronts, and ensuring customers can complete their transactions without interruption.” Duncan Davidson VP Developer Productivity, Shopify Top AI Crawl Control use cases Decide which AI crawlers can access which domains across your digital environments Prevent repeated AI crawls from overwhelming your origin servers Keep AI crawlers from skewing web tracking with repeated scraping Documentation AI Crawl Control Press Cloudflare just changed how AI crawlers scrape the Internet-at-Large; Permission-based approach makes way for a new business model. Blog The crawl before the fall… of referrals: understanding AI’s impact on content providers.