Given how aggressively AI crawlers (I presume) are hitting my servers (and of course ignoring the robots.txt), I need to either install some crawler-blocking solution or switch from my denylist for IP ranges to an allowlist…

Does anyone know a software that blocks crawlers that is compatible with traefik and ideally uses very little resources?

#self-hosting #AI-crawlers #self-hosting-headaches #I-literary-had-to-deny-an-entire-/16-subnet-yesterday-to-prevent-my-servers-from-crash-looping