Had to do a maintenance on Immich, postgres cluster got disconnected from the operator (somehow?) and wouldn't scale past 1 replica anymore...

And we don't like single points of failure here

So first, descheduling the Immich pod, to avoid weird writes between last backup and actual DB shutdown

Triggering a manual backup

Deleting cluster

Recovering cluster from last backup

Woops, the shared library object name got renamed from vectorchord 0.3 to 0.4.3, now it's "vchord.so" instead of "vectors.so"

Quick fix and boom, everything comes back up

Restoring Argo auto sync and the cluster is all back up, distributed and shining

When Backups, DevOps and GitOps come back in a smooth and clear motion ⭐

#postgresql #postgres #cnpg #psql #selfhosted #selfhosting #homelab #k3s #kubernetes #linux #opensource #immich #vectorchord #backup #devops #argocd #gitops