For the last ~6 months, my immich Longhorn PVC wouldn't rebuild replicas across regions, and timeout instead
Today, I figured I had misplaced my MTU configuration for the Wireguard network under k3s...
So some packets were getting dropped silently...
Woops
#kubernetes #k3s #longhorn #network #networking #wireguard #wg #mesh #homelab #selfhosted #selfhosting #mtu
Had to do a maintenance on Immich, postgres cluster got disconnected from the operator (somehow?) and wouldn't scale past 1 replica anymore...
And we don't like single points of failure here
So first, descheduling the Immich pod, to avoid weird writes between last backup and actual DB shutdown
Triggering a manual backup
Deleting cluster
Recovering cluster from last backup
Woops, the shared library object name got renamed from vectorchord 0.3 to 0.4.3, now it's "vchord.so" instead of "vectors.so"
Quick fix and boom, everything comes back up
Restoring Argo auto sync and the cluster is all back up, distributed and shining
When Backups, DevOps and GitOps come back in a smooth and clear motion â
#postgresql #postgres #cnpg #psql #selfhosted #selfhosting #homelab #k3s #kubernetes #linux #opensource #immich #vectorchord #backup #devops #argocd #gitops
As Hetzner is deprecating dns configuration via the dns-console, I migrated my domains to the new Cloud API. Last piece of the puzzle was to create new tokens and move from the old cert-manager-webhook-hetzner (by vadimkim) to the official chart maintained by Hetzner.
Migrated my 7 kubernetes clusters (k3s, rke2, OpenShift) without major hiccups, only had to do some cleanup due to old acme challenge entries being leftover after the migration (as cert-manager could not remove them without the new webhook and API token).
Only things left are the machines without k3s using lego.
#homelab #hetzner #certmanager #dns #hellyeah #kubernetes #k3s #rke2