From ChatGPT to Gemini: how AI is rewriting the internet
See all Stories
R
Cloudflare is offering to block crawlers scraping information for AI bots.
Tech giants are rewriting the rules on web scraping, blaming unnamed third parties for disregarding robots.txt, and seemingly claiming the right to reuse anything posted anywhere for AI.
Now, Cloudflare is telling customers on its CDN that it can find and block AI bots that try to get around the rules.
The upshot of this globally aggregated data is that we can immediately detect new scraping tools and their behavior without needing to manually fingerprint the bot, ensuring that customers stay protected from the newest waves of bot activity.
Follow topics and authors from this story to see more like this in your personalized homepage feed and to receive email updates.
Loading comments
Getting the conversation ready...
Most Popular
Most Popular
- Midjourney goes from generating cat images to full-body ultrasound scans
- Apple’s weird anti-nausea dots cured my car sickness
- Amazon employees say they’re facing termination for backing data center limits
- This Ghost in the Shell keyboard makes me want to activate the hundred spidery robot fingers inside my regular fingers
- This robotic self-driving toilet comes to you
