CodebergAug 15, 2025

We apologize for a period of extreme slowness today. The army of AI crawlers just leveled up and hit us very badly.

The good news: We're keeping up with the additional load of new users moving to Codeberg. Welcome aboard, we're happy to have you here. After adjusting the AI crawler protections, performance significantly improved again.

Show threadCodebergAug 15, 2025
It seems like the AI crawlers learned how to solve the Anubis challenges. Anubis is a tool hosted on our infrastructure that requires browsers to do some heavy computation before accessing Codeberg again. It really saved us tons of nerves over the past months, because it saved us from manually maintaining blocklists to having a working detection for "real browsers" and "AI crawlers".
Show threadCodebergAug 15, 2025
However, we can confirm that at least Huawei networks now send the challenge responses and they actually do seem to take a few seconds to actually compute the answers. It looks plausible, so we assume that AI crawlers leveled up their computing power to emulate more of real browser behaviour to bypass the diversity of challenges that platform enabled to avoid the bot army.
Show threadCodebergAug 15, 2025

We have a list of explicitly blocked IP ranges. However, a configuration oversight on our part only blocked these ranges on the "normal" routes. The "anubis-protected" routes didn't consider the challenge. It was not a problem while Anubis also protected from the crawlers on the other routes.

However, now that they managed to break through Anubis, there was nothing stopping these armies.

It took us a while to identify and fix the config issue, but we're safe again (for now).

Show threadCodebergAug 15, 2025

For the load average auction, we offer these numbers from one of our physical servers. Who can offer more?

(It was not the "wildest" moment, but the only for which we have a screenshot)

Show threadJann HornAug 15, 2025
@Codeberg huh, that's a pretty kernel-heavy workload, so much red
Show threadJann Horn
@Codeberg what are they hitting that's so kernel-intensive, is this filesystem stuff or process executions or something?
Aug 15, 2025 at 11:48pmWeb
Show threadVinceAug 16, 2025

@jann @Codeberg

It’s io. If the system was cpu bound at 5000+ load you certainly wouldn’t be running anything interactive.

More like typing out the command to stop the web service / nft block web traffic waiting for it to appear on the screen and execute just to restore console interaction 😆