Saustrup

https://mstdn.dk/ was down for about half an hour this morning, as I stupidly and/or naively had enabled automatic updates for #Cilium, the software handling the #Kubernetes network layer. I'm unsure what exactly is broken in release 1.17.4, but deployments were failing with InvalidImageName and from the looks of it, the container layer was unable to resolve the tag to a specific image. Rolling back AND PINNING THE VERSION to 1.17.3 fixed the issue. Sorry about that.

Now back to breakfast.

#downtime #MastoAdmin

mstdn.dk

Just your average friendly Danish Mastodon server. New users tooting in Danish/English welcome. Administered from Denmark. Hosted on bare-metal Kubernetes in the EU.

Mastodon hosted on mstdn.dk
May 16 at 5:02amWeb
Show threadMark GjølMay 16
@saustrup I did notice, and honestly didn't expect it to be resolved for the remainder of the day, so thank you very much for the surprisingly quick resolution to the problem!
Show threadjoestringer 🐧🔀🐝May 16
@saustrup sorry about that! One of the steps in the Cilium release process hit an error while correlating the version number into the Helm chart, resulting in an invalid version being embedded into the DaemonSet. The test job intended to catch this also did not catch it. We've identified the reason for the latter, still looking into the former. We were able to catch this within a couple of hours, but any upgrade attempts during that initial period would fail.
Show threadSaustrupMay 16
@joestringer No worries - accidents happen, and people running production workloads really should pin versions. Note taken ;-)