Codeberg.org
Codeberg is currently unreachable, apparently due to a network outage at our provider. We'll hit you up as soon as we know more.
Jul 3 at 9:38amWeb
Show threadCodeberg.orgJul 3

From a quick ping on various services, it seems the issue is not with our volunteer-driven rackspace provider, but affecting the whole datacenter we're located in.

Which might be good news, because it then doesn't rely on volunteers to fix the issue, but on professional staff 😉.

Show threadCodeberg.orgJul 3
We're back. It looks like all servers (all datacenter?) lost power, but we have no confirmed information on the background yet.
Show threadCodeberg.orgJul 3
We have fixed some remaining issues with Weblate (Codeberg Translate) and our hosted Woodpecker CI. Their web UI was working, but some operations were failing. The problem is resolved now. Please restart CI jobs or repository operations in Weblate if they failed due to this.
Show threadhappyborgJul 3
@Codeberg What's Weblate? I'm learning to use your CI but haven't come across that.
Show threadCodeberg.orgJul 3
@happyborg https://weblate.org hosted at https://translate.codeberg.org, documentation at https://docs.codeberg.org/codeberg-translate/
Weblate - web-based localization

Copylefted libre software, used by over 2,500 libre software projects and companies in over 165 countries.

Weblate
Show threadglynJul 3
@Codeberg Thanks but codeberg.org is unavailable again. Or is this just a DDOS thanks to users pushing after the outage?
Show threadCodeberg.orgJul 3
@underlap We're not sure yet, investigating.
Show threadGerard BraadJul 3

@Codeberg temperature affecting operation?

BTW, which datacenter?

I used to host in RedBus Amsterdam. Very reliable. We occupied several racks. Never had issues with them.... and they even provided an occasional 'reboot' for a machine.

Show threadCodeberg.orgJul 3
@gbraad We're using infrastructure provided by https://in-berlin.de in Berlin. They are another non-profit dedicated to network services. However, they only rent rackspace from another operator in Berlin, that seemed to have suffered from power outage.
Startseite

Willkommen beim IN-Berlin Willkommen beim Individual Network Berlin e.V., dem etwas anderen Provider. Wir bieten das Internet in seiner vollen Vielfalt, ohne Abstriche und Kompromisse - und alle sind eingeladen, mitzugestalten.

Individual Network Berlin e.V.
Show threadGerard BraadJul 3

@Codeberg power outages are more common during heatwaves.

Glad to see you guys up again.

Show threadSirWumpus 👾🍁Jul 3
@Codeberg That's not a good sign for a data centre you'd expect to provide 5-9s uptime. No industrial UPS, no generator? If a professional data centre looses power such that client machines shutdown (unless you're in the charity / staff rack that isn't covered) that's a worry.
Show threadCodeberg.orgJul 3
@sirwumpus The DC claims to have redundant power feeds into the building, UPS and generators. However, it's not the first time I have seen UPS systems fail, even in large datacenters. Often during maintenance and human error :) ~f
Show threadSirWumpus 👾🍁Jul 3

@Codeberg True, though when I was with TASFrance they would run the generator monthly and check the UPS charges and holds a charge. An unplanned outage during maintenance seems odd - has to be human error (like cooking toast or falling onto power mains terminals).

Thank you for the informative updates.

Show threadCodeberg.orgJul 3
@sirwumpus The outage immediately rang a bell for this German blog article from uberspace https://blog.uberspace.de/2021/06/ausfall-am-29-06-2021/ that was also once affected by a power outage. In short, during maintenance of the UPS, some component broke and resulted in a short interruption of 70ms, which was enough to cause headache recovering all network equipment for several hours.
Ausfall am 29.06.2021

Show threadCodeberg.orgJul 3
@sirwumpus
From the uptime report of our servers, they rebooted immediately after the power was lost, so the outage was probably very short (a few minutes at most, probably much less). But they were not reachable, probably because it took so long until all network equipment etc recovered.
Show threadSirWumpus 👾🍁Jul 3
@Codeberg Maybe someone was sucking all the power 🔋 out of the room 😁
Show threadCodeberg.orgJul 3
@sirwumpus Still no official information from the provider, but BCIX has the outage reported as well https://status.bcix.net/maintenance-windows/52472
BCIX Status - [BCIX] Incident at BCIX site A36s Speedbone

SUMMARY: We would like to inform you about the following Incident: Incident: 2025-6715 SEVERITY : high Device : edge-a36s-21.bcix.de, edge-a36s-31.bcix.de Location : Alboinstrasse 36 (Speedbone) Start Time : 2025-07-03 09:30:00 UTC (2025-07-03 11:30:00 Europe/Berlin) End Time : 2025-07-03 09:40:00 UTC (2025-07-03 11:40:00 Europe/Berlin) AFFECTED NETWORKS: ASN IPv4 IPv6 PEER ========================================================================= 112 193.178.185.112 2001:7f8:19:1::70:1 www.as112.net 250 193.178.185.23 2001:7f8:19:1::fa:1 AS250 917 193.178.185.165 2001:7f8:19:1::395:1 Misaka.io 9002 193.178.185.64 2001:7f8:19:1::232a:1 RETN 10310 193.178.185.110 2001:7f8:19:1::2846:1 Yahoo 10310 193.178.185.111 2001:7f8:19:1::2846:2 Yahoo 12732 193.178.185.97 2001:7f8:19:1::31bc:3 dhosting 12741 193.178.185.46 2001:7f8:19:1::31c5:1 netia 13020 193.178.185.124 2001:7f8:19:1::32dc:1 CCC Event 20562 193.178.185.147 2001:7f8:19:1::5052:1 Open Peering 20676 193.178.185.60 2001:7f8:19:1::50c4:1 Plusnet 29670 193.178.185.50 2001:7f8:19:1::73e6:1 INBERLIN 29670 193.178.185.51 2001:7f8:19:1::73e6:2 INBERLIN 31078 193.178.185.142 2001:7f8:19:1::7966:1 ase-network 34758 193.178.185.180 2001:7f8:19:1::87c6:1 Axera 39614 193.178.185.85 2001:7f8:19:1::9abe:1 Dawico 44194 193.178.185.82 2001:7f8:19:1::aca2:2 Freifunk 50472 193.178.185.105 2001:7f8:19:1::c528:1 ccc 198225 193.178.185.131 2001:7f8:19:1:0:3:651:1 MDDSL GmbH 201701 193.178.185.74 2001:7f8:19:1:0:3:13e5:1 ffrl 206313 193.178.185.81 2001:7f8:19:1:0:3:25e9:1 Freifunk Nordwest 206813 193.178.185.13 2001:7f8:19:1:0:3:27dd:1 Freifunk Guetersloh 211286 193.178.185.176 2001:7f8:19:1:0:3:3956:1 measurement.network 215250 - 2001:7f8:19:1:0:3:48d2:1 V4LESS DESCRIPTION: A power failure occurred, leading to disruptions across multiple systems. This incident affected peering services, resulting in downtime. Immediate investigation and remediation efforts are underway to restore functionality If you have any questions regarding this incident, please don't hesitate to contact us. Thank you -- BCIX NOC BCIX Management GmbH Albrechtstr. 110 12103 Berlin +49 30 2332195 55 noc@bcix.de https://www.bcix.de/

BCIX Status
Show threadbignoseJul 3
Thank you @Codeberg for being responsive to issues as they arise, and for keeping the community informed promptly.
Show threadStefan SchmidtJul 3
@Codeberg Short writeup by IN-Berlin now online at https://www.in-berlin.de/posts/statusupdates/2025/07/03/stromwackler-am-standort-ak/ [DE]
Stromwackler am Standort AK

Am Standort AK kam es heute gegen 11:33 Uhr zu einem kurzen Stromwackler. Ob dieser mit einem Problem mit der Mittelspannung von Stromnetz Berlin in der Nähe des Rechenzentrums zusammenhängt, wissen wir noch nicht. Eigentlich hätten die USVs übernehmen sollen, jedoch gab es laut Informationen des RZ-Betreibers ein Problem, wodurch sie 19 Sekunden lang keinen Strom lieferten. Dies muss nun durch den RZ-Betreiber im Nachgang mit der Wartungsfirma der USVs geklärt werden.

Individual Network Berlin e.V.
Show threadJens LinkJul 3
@Codeberg traceroute
shows that packages are circling between Paris and Madrid. Vacation time. ;-)
Show threadCc 🇪🇺🏳️‍🌈💚🇺🇦💉Jul 3
@Codeberg
But it is not a remote attack by idiots?