@civodul

The documentation is a little bit out of date - I recently added support for doing detection within iocaine itself, it no longer relies on a reverse proxy to do that.

It still mostly relies on passive identification (user-agent and other headers), but that already gets rid of a very large part of the crawlers, in my experience.

Those legit looking user-agents? A lot of them can be caught, with a little bit of care! Of the 7-9 million requests my iocaine catches daily (@iocaine has some fun daily stats + dashboard snapshots, if you're curious), only about 300k get past my passive checks, and those meet the Cookie Monster. The Cookie Monster is a very simple proof of work hack: it uses JavaScript to set a cookie, then reloads the window. For a human visitor, this is barely noticable (if you visited my forge, you likely met the Cookie Monster, btw; hopefully you didn't notice!). In the past ~3 months or so, a handful of bots got past it, maybe, and I updated my rules when I noticed.

If you think Anubis is overkill - I agree, hence iocaine's far lighter PoW implementation.

If you think it would be valuable, I can chime in to the issue linked a few toots above. I'd love to see less Anubis, tbh. It gets the job done, but I can't view anything behind Anubis on my phone, and that does not spark joy.

@luj