My entire website is 44MB in size. Most of this are images of course.
Yesterday 1,2GB of data was transferred from my webserver, including EVERY image several dozen times.
Either a lot of people discovered my blog last week and spent the whole day reading ALL of my posts, or there's some AI scraping going on again.
I'd hate to do it, but I'm seriously considering putting some Anubis or Cloudflare protection in front of it. Stuff like this really pisses me off...
With images, you could keep them from loading if they don't have the HTTP referrer header set to something on the domain? That would prevent hotlinking and probably most scraping.
Hmm, what's your hosting setup like? Where/how do you host your site?
If you don't have access to the reverse proxy I don't think you'd be able to set up Anubis anyway. :(
Hm, there might be a way to do that hotlink protection thing in htaccess, though! Been a long time since I've written one, but it's the kind of thing they do well… some sort of rewrite rule for requests to certain file extensions (ie images) that don't have referrers?
You can definitely block access to certain user agents (ie known LLM crawlers) that way.
Neil Clarke's article has recommendations on blocking via .htaccess, worth a look. https://neil-clarke.com/block-the-bots-that-feed-ai-models-by-scraping-your-website/
Andreas (82mhz)
Amin Hollon 🏳