Hi everyone! We are still fighting with an ongoing DDoS attack against our service on the network level. We do not yet have any ETA on recovery time, as we're still investigating different mitigation strategies.

As always, we are very grateful for your support and patience.

We were able to restore IPv4 and IPv6 connectivity. Performance is still not optimal, and connections are occasionally dropping. We've taken measures to try and maintain uptime overnight (with a certain probability).

Routes protected by iocaine (crawler protection) may return 503 errors. Currently, it appears that these packets are also occasionally being lost.

It's 2:30am, and we require a good night's sleep before we can look into this further.

Good morning. Since a bit more than an hour ago, multiple people from our side begun working on restoring the availability of our services and will share further updates as they come.

Nothing that's impossible, it's all just a bit of a bother.

We believe that access has been restored for the time being, we'll continue monitoring the situation for now.
@codebergstatus hey folks, how can one help on sysadmin tasks?
@codebergstatus Could it be that codeberg.page is still down? Amazing work, very grateful that Codeberg exists :-)
@emmabastas i can also confirm that @Codeberg pages are down, at least https://lora.reseaulibre.ca/ is still not responding here, which points to lora-reseaulibre-ca.anarcat.codeberg.page. in DNS.
@codebergstatus Thanks for your hard work 👍 . As usual, you rock.
@codebergstatus @Codeberg I'm using codeberg pages with custom domains, I noticed one site was working while others not, so I desumed that AAAA record must be set to 2a03:4000:4c:e24:85e:10ff:fef8:a405 and not to 2a0a:4580:103f:c0de::2 like custom domain page is saying (https://docs.codeberg.org/codeberg-pages/using-custom-domain/)
Using custom domains | Codeberg Documentation

@codebergstatus I just wanted to add another thank you to the pile; y'all are doing so much work to keep Codeberg running and it's very appreciated!
@codebergstatus thank you all for your work and efforts!
@codebergstatus thanks for the hard work💜 Have a good and refreshing night of sleep 🥰
@codebergstatus Thank you for the overtime work. Much appreciated.
@codebergstatus well done! Much appreciated!
@codebergstatus
Well done and thank you.
@codebergstatus look after yourselves and don't burn yourselves out.
@codebergstatus thanks for all you bring to the community!
@codebergstatus Good work. I wonder is it an attempt to get “training data”… or just an old school take down.
@brianwdouglas The issue is currently not crawling-related.
@codebergstatus I bestow the curse of a never ending wipe on the attackers.

@LangerJan @codebergstatus

"May your arse forever itch and your arms be too short to reach."

@madsenandersc @LangerJan @codebergstatus I also add a casting of "manual blinking and breathing"
@codebergstatus I don't understand it; you're probably not causing relevant financial loss to anyone, so why pay for the attack?

Anyway I wish you peace of mind while dealing with the crap.
@vojta001 @codebergstatus M$ spent a big buck to own the community they are currently losing.
@vojta001 @codebergstatus but you can hurt others in hope of them going down just for the "fun" of it.
@codebergstatus Keep strong ❤️‍🔥
@codebergstatus, good luck to everyone in the team!! Thanks for your daily work on this great project.
@codebergstatus This is low, even for GitHub.
@sjm @codebergstatus does anything indicate that github would be the attacker?
@codebergstatus good work on getting ssh access back up. 💪
@codebergstatus best wishes, sounds rough out there
@codebergstatus Annoying.
Why not limiting access to users only when getting attacked?

@apnoe_soeren

DDoS attacks usually cannot be fought back on a per-user basis. These attacks typically floods the network with too much traffic, regardless of the target accounts.

What typically happens ... several thousand IP addresses tries to access a service on a server (say, https - TCP port 443) which opens up a response port on the server (the theoretical maximum is 65536 ports). The attackers continues to flood the network with such connection attempts, exhausting the response ports and keeping them busy. Which then results in the server not being able to receive any new connections, for example for https traffic. If the ssh port (TCP port 22) is not attacked, that may be accessible - until that port also gets attacked in similar ways.

Another approach is to do something similar, where the attacker trigger larger downloads - exhausting the available bandwidth to the server. Which again makes everything stop. And then keeping the server busy from thousands of different IP addresses.

Or a combination of these. The port exhaustion approach can also take a considerable large chunk of the available bandwidth, if there are enough IP addresses attacking.

The way to fight back DDoS attacks is usually to block the IP addresses flooding the server. Sometimes you can be lucky and it's enough to block IP address ranges for a certain country or a certain ISP. Other times you're not that lucky, as the attack comes from many ISPs and many countries. Or when you block one region, another region surfaces - in a true "whack a mole" game.

@codebergstatus

@codebergstatus Thanks a lot for providing this awesome service! May future attacks go towards bad actors instead of those working on a better world!
@codebergstatus
Thanks or your efforts, I will contribute asap with a new donation.