Skip to main content
R
Cloudflare is offering to block crawlers scraping information for AI bots.

Tech giants are rewriting the rules on web scraping, blaming unnamed third parties for disregarding robots.txt, and seemingly claiming the right to reuse anything posted anywhere for AI.

Now, Cloudflare is telling customers on its CDN that it can find and block AI bots that try to get around the rules.

The upshot of this globally aggregated data is that we can immediately detect new scraping tools and their behavior without needing to manually fingerprint the bot, ensuring that customers stay protected from the newest waves of bot activity.


A line graph showing user agent matches for known AI bots over the last year.
The most popular AI bots seen on Cloudflare’s network in terms of request volume.
Image: Cloudflare