CodebergWe are currently having network issues. We are able to connect to our server's onboard recovery system, but the access is slow and unreliable.
We'll keep you updated.
A quick glimpse at the monitoring (until network access went away) indicated that it's either a grave network misconfiguration or an active DDoS going on. It looked like an inrush of junk data at the maximum link speed of our server.
We are waiting for our provider to help us out with our current situation.
Our provider confirmed an incoming DDoS. We have got access to our systems and currently considering next steps.
For your convenience, we have temporarily established a way to access Codeberg. This is heavily constrained and might go back offline at any time.
mostly working:
codeberg.org HTTPS (Read/Write)
not working:
SSH, CI, Weblate, Pages and all the rest.
SSH is also back. We expect overall performance to be rather low, though.
Looks like some of you are already back. (At least the regular spam is, too)
Just for clarification: Due to our current networking constraints, there are no webhooks and email notifications at this time.
We are optimistic to have a plan to restore services early afternoon (European time).
We are also looking forward to restoring the Codeberg logo (it is currently missing in some places 😂)
We (partially) restored service for:
Fully working:
- E-Mail
IPv6-only:
- Webhooks
- Codeberg Pages
- Weblate https://translate.codeberg.org
- static pages (blog, docs, the Codeberg logo)
IPv6-only and limited functionality:
- Woodpecker CI (should accept Webhooks, but no actual builds are running: Due to IPv6-only, the builds will not connect to GitHub and other prominent sources for dependencies, there is likely no use in having most builds fail due to this)
In the past minutes, we alternately broke IPv4 and IPv6 support - for equality.
IPv4 access to all services is now proxied. This should bring most functionality back.
Still unavailable:
- CI builds
- Codeberg Pages for people who hardcoded the A records instead of using ALIAS/CNAME.
We are working on restoring the last pieces.
⁂ Kyva
Slatian@Codeberg That’s awesome! 
smxi@Codeberg sorry you have to suffer this nonsense from toxic bad actors.
macfanpl
ronix 🏺@Codeberg Keep up your great work! Too bad, there are people out there, trying to sabotage FOSS.
@ronix Im affraid there will always be someone with ill-intent..... its unavoidable.
Readeck@Codeberg good luck and thanks for your hard work! ❤️
@Codeberg I wonder why? thanks for the hard work!
Michał "rysiek" Woźniak · 🇺🇦
Solinvictus 
André-LA 🎀 gamedev@Codeberg thanks for the work! ✨
Andre_601 
@Codeberg People DDoSing a service managed by a non-profit are some of the most morally bancrubt people I can think of...
K. Ryabitsev-Prime 🍁@Codeberg Waves of support from fellow git server admins!
Devil 
@Codeberg Why ddossing Codeberg! Really sh*t people
DHeadshot's Alt
Earl Warren@Codeberg thanks for taking care of this. Lots of good vibes until it is over ❤️
libreware sniffer@Codeberg will there be a ddos protection layer later down the road?
Linfeng@Codeberg there's a similar situation with sourcehut earlier(yesterday?) could be related?
James Seward
nev@Codeberg Just realized it was down now…thank you for updating us on here! Hope you get it sorted.
kik@Codeberg do you use iptables? If you do and manage to get a shell, this is what I'm using to fight DDoS:
iptables -A INPUT -p tcp -m tcp --dport 80 -m state --state NEW -m recent --update --seconds 2 --hitcount 100 --name DEFAULT --mask 255.255.255.255 --rsource -j DROP
The important parameters here are --seconds 2 and --hitcount 100. It says : if the same IP is making more than 100 requests in 2 seconds (on port 80), drop it. You can adjust to you're liking.
Calicosine@kik Might be a good idea to have a second one for port 443, too. I get attempted connections from random IP addresses on both ports in my UFW log every few minutes and I don't even have a server running yet, which is why I suggest this.
@DrewNaylor Yeah indeed, and any port open really. We used that to defend a friend's server against a 4chan DDoS more than 10 years ago (we didn't really used 443 back then :P ), hope it still works.
lenny_@kik if it's at the max link speed of the server I don't think there's much iptables can do to help
@lenny_ Is that common? My only experience with an actual DDoS is decade old, so yeah, it might be obsolete…
BitPirate@bitpirate @lenny_ Thanks for trying to educate me, but I'm quite aware of the difference between DoS and DDoS! 😂
I already mitigated a DDoS that way, for the simple reason that even when there are many hosts participating in an attack, it's still a given set making repeated requests, so this iptables rule is about identifying those and dropping them.
That being said, we used that in a simpler time when DDoS were done manually. Indeed I can see how a more elaborated one leveraging a big enough infected botnet could overload the hardware/infrastructure itself before even reaching the software, as @lenny_ was mentioning. I'd be interested in seeing any data on how common it is that attackers packets don't even reach the host.
@kik @lenny_ You can only prevent that traffic from flowing further, e.g. to your services on that machine or to other servers. Each packet will still arrive at your server, otherwise nftables/iptables wouldn't be able to filter it.
If you were able to mitigate an attack in this way, they lacked the bandwidth to overwhelm your server, or the attack was done by script kiddies.
The DDoS game is simple: the bigger pipe wins.
Latte macchiato 
@privateger @lenny_ Got it, thanks. 👍️ So yeah, seems like the client/server is condemned to centralize even more (by having big central points like Cloudflare), thanks to those assholes. Maybe it's time to take the p2p web more seriously (like what Secure Scuttlebutt is doing, or what the Beaker browser tried to do through Dat).