john22h ago
If I eventually move my (simple, static) website to a digital ocean droplet, do I need to do anything special in the awful modern internet to keep bandwidth from getting run out by AI scrapers immediately? I want to host game downloads, that's the big gain moving off github sites
Show threadsean

@farawaythyer for static content you mostly just need aggressive caching. most webhosts will do that for you automatically or give you an easy way to configure it, and many of them will already have comprehensive bot detection/blocking/rate limiting at the infrastructure level in order to support a large-scale network anyway

if you want to be more explicit, you can add your own block/allowlists to a robots.txt but that's basically a formality these days and the truly malicious bots won't respect it anyway

Mar 24 at 2:26pmWeb
Show threadsean22h ago
@farawaythyer for something like downloads you can also put `rel="nofollow"` on the links themselves (since a non-malicious bot would only benefit from crawling the referring page, not the download itself) but like robots.txt that's just a suggestion and ensuring it's cached is way more important
Show threadjohn21h ago
@sean gotcha, this is all really helpful! it looks like nginx has a lot of caching options, so after setting up the server I just need to make sure the right options are configured https://docs.nginx.com/nginx/admin-guide/content-cache/content-caching/
NGINX Content Caching | NGINX Documentation

Cache both static and dynamic content from your proxied web and application servers, to speed delivery to clients and reduce the load on the servers.

Show threadsean20h ago
@farawaythyer idk how digital ocean works specifically but you should prob confirm in their docs that the nginx server is where you actually want to cache things before diving too deep into the generic nginx docs (e.g. you may end up still paying for the bandwidth if they expect you to cache it further downstream, or may find a simpler option like standard cache-control headers if they expect you to cache it further upstream)
Show threadjohn20h ago
@sean yeah, that tracks, noted. I want to be careful with this stuff for sure