https://www.cncf.io/blog/2026/03/17/when-kubernetes-restarts-your-pod-and-when-it-doesnt/

#K8s #k3s #crashloopbackoff #insights

I have services in my #homelab that need #postgres quite often. In my #k3s cluster I'm using #cloudnativepg to provision single-use databases for services. What should I do for my services running in #docker? Create a single VM that just runs Postgres and connect them all to it? Provision a single Postgres in k3s and expose it so that docker containers externally can use it?

Move anything that needs postgres to #kubernetes?

#selfhosting can get complicated 😔

Reducing my #homelab #k3s cluster to 1 agent node in an attempt to simplify things.

So now the homelab is down to 1 k3s "cluster", 1 VM to run docker containers and an external #traefik instance to reverse proxy to both.

#homelab and #k3s made further progress, they now run #cilium for networking.

Why? Because I like Cilium and eBPF.

Compare to my full blown #k8s cluster on 10 bare metal supermicro systems, this was a tremendous amount easier. In total I spent about two hours to get k3s and Cilium to play nice.

All of this is Ansible based. I am reusing the official k3s orchestration, and a personalized Cilium role.

#homelab progress has been made.

We have three #Asus PB60 installed, plus one #tuxedo pulse 15. All devices use Avahi service advertising and the PB60 s form a #k3s Cluster.

The fun part... It is all mDNS and Wi-fi 😅

I didn't expect that to work.

For the last ~6 months, my immich Longhorn PVC wouldn't rebuild replicas across regions, and timeout instead

Today, I figured I had misplaced my MTU configuration for the Wireguard network under k3s...

So some packets were getting dropped silently...

Woops

#kubernetes #k3s #longhorn #network #networking #wireguard #wg #mesh #homelab #selfhosted #selfhosting #mtu

Had to do a maintenance on Immich, postgres cluster got disconnected from the operator (somehow?) and wouldn't scale past 1 replica anymore...

And we don't like single points of failure here

So first, descheduling the Immich pod, to avoid weird writes between last backup and actual DB shutdown

Triggering a manual backup

Deleting cluster

Recovering cluster from last backup

Woops, the shared library object name got renamed from vectorchord 0.3 to 0.4.3, now it's "vchord.so" instead of "vectors.so"

Quick fix and boom, everything comes back up

Restoring Argo auto sync and the cluster is all back up, distributed and shining

When Backups, DevOps and GitOps come back in a smooth and clear motion ⭐

#postgresql #postgres #cnpg #psql #selfhosted #selfhosting #homelab #k3s #kubernetes #linux #opensource #immich #vectorchord #backup #devops #argocd #gitops