CodebergAug 15, 2025

We apologize for a period of extreme slowness today. The army of AI crawlers just leveled up and hit us very badly.

The good news: We're keeping up with the additional load of new users moving to Codeberg. Welcome aboard, we're happy to have you here. After adjusting the AI crawler protections, performance significantly improved again.

Show threadCodebergAug 15, 2025
It seems like the AI crawlers learned how to solve the Anubis challenges. Anubis is a tool hosted on our infrastructure that requires browsers to do some heavy computation before accessing Codeberg again. It really saved us tons of nerves over the past months, because it saved us from manually maintaining blocklists to having a working detection for "real browsers" and "AI crawlers".
Show threadCodebergAug 15, 2025
However, we can confirm that at least Huawei networks now send the challenge responses and they actually do seem to take a few seconds to actually compute the answers. It looks plausible, so we assume that AI crawlers leveled up their computing power to emulate more of real browser behaviour to bypass the diversity of challenges that platform enabled to avoid the bot army.
Show threadCodebergAug 15, 2025

We have a list of explicitly blocked IP ranges. However, a configuration oversight on our part only blocked these ranges on the "normal" routes. The "anubis-protected" routes didn't consider the challenge. It was not a problem while Anubis also protected from the crawlers on the other routes.

However, now that they managed to break through Anubis, there was nothing stopping these armies.

It took us a while to identify and fix the config issue, but we're safe again (for now).

Show threadCodebergAug 15, 2025

For the load average auction, we offer these numbers from one of our physical servers. Who can offer more?

(It was not the "wildest" moment, but the only for which we have a screenshot)

Show threadDamonHDAug 15, 2025
@Codeberg In the days of single CPU servers (early 90s?) and an interesting filesystem problem, I think I may have seen ~400 at a client site!
Show threadmirabilosAug 15, 2025

@DamonHD @Codeberg got around 800 on a single CPU server (well actually VM) this week, I had blocked lots of Huawei networks but there’s no list of all of them.

I now block some more, but I also reduced MaxServer in Apache httpd, to put an upper cap on this. Getting into swapping is not fun.

Show threadDamonHDAug 15, 2025
@mirabilos @Codeberg Yes, that is bad. I now run servers with no disc swap and tightly capped limits on all services to avoid the dreaded OOM Killer being needed.
Show threadmirabilos
@DamonHD @Codeberg no swap at all is actually a bad thing on Linux. Put some in, if it is just 128 MiB. They designed it to work with it.
Aug 15, 2025 at 9:28pmWeb
Show threadDamonHDAug 16, 2025
@mirabilos @Codeberg my scheme has been working well for me (with a little zram) for well over a decade!
Show threadmirabilosAug 16, 2025
@Codeberg @DamonHD it works, sure
Show threadDamonHDAug 16, 2025
@mirabilos @Codeberg I run all my primary services on an RPi3 off grid. It has 1GB RAM and swapping to the memory card would be very slow and would wear it fast.
Show threadDamonHDAug 16, 2025
@mirabilos @Codeberg It also gives better Web response than I can get out of Cloudflare for my target audience. So it meets all my goals.
Show threadDamonHDAug 16, 2025
@mirabilos @Codeberg Those may well be different to your goals for a server.
Show threadseppel3210Aug 19, 2025
@mirabilos @DamonHD
AFAIK zram swap has the same positive effects as typical swap in terms of Linux being designed with swap in mind
Show threadmirabilosAug 19, 2025
@DamonHD @seppel3210 ah, good to know