SaustrupFeb 28

I've been a little rough and irresponsible with my #baremetal #Kubernetes cluster, especially when it comes to randomly rebooting nodes. Today I fixed that.

I'm running a bunch of somewhat delicate workloads, including database clusters with CSIs like #Longhorn and #OpenEBS. Checking if everything is in working order has been demanding task and often something I've skipped before rebooting or upgrading nodes - occasionally with horrific results.

Last night I finally took the time and wrote a pretty thorough script that checks that everything is working and healthy, before politely cordoning off a node, draining it and applying upgrades.

I felt so confident today that I tested it by running this new safe upgrade script for all the nodes in the cluster - and it worked! All nodes are now fully upgraded and running kernel 6.12.73 on Debian 13.

This also fixes the outstanding issue caused by #Hetzner no longer supporting obtaining IP addresses through DHCP.

#Linux #MSTDNDK #K8s

Show threadSaustrupOct 21

A quick follow-up: Given that caching an #LVM volume requires setting up a cache and a cache meta volume per data volume you need cached, this just gets needlessly complicated when dealing with a storage provisioner that constantly creates new volumes. Caching seems absolutely necessary with the hardware I have available and LVM seems like the only way to go. There are alternatives, like #flashcache by #Synology, but they're not part of the main stream #kernel.

The solution? It ain't pretty, and I have still to test the performance implications of it. I've created a 4TB file on a filesystem cached by LVM, created a #loopback device from it, initialized that as an LVM physical volume, then added a volume group. The #OpenEBS Local LVM provisioner now uses this group to provision persistent volumes in #Kubernetes.

Not sure about the implications of nested LVM'ing, but it solves the #caching issue.

#linux #cache #hdd #nvme #ssd

SaustrupOct 19

I'm running my own #Kubernetes cluster on bare metal at #Hetzner. I'm going through different iterations of node configurations, and trying to figure out how to squeeze the most out of a hardware configuration with a 10TB+ HDD drive and two 512GB #NVME drives. #bcache has apparently been discontinued, which leaves me with #dmcache, which is nicely built into #LVM.

One catch though: you need to create NVME cache volumes for each of the data volumes you create. This is all well and good if you're using a single volume for everything, and tools like #Longhorn work well in this kind of setup. Others, like the #OpenEBS LVM provisioner don't. There is a version in the making that just provisions raw disk images as files on whatever filesystem you have, but it's still lacking some features that are vital to me - primarily the ability to offer up StorageClasses that provision on different paths of the node, so I can pick a fast or slow filesystem to store it on.

Tips for other solutions?

SaustrupOct 16

Having recently experienced a rather horrible #Kubernetes crash, I'm looking for #backup solutions. We're good with PostgreSQL since we're using #CNPG with remote transaction logs to an offsite #S3 bucket. I need something for volumes and maybe Kubernetes resources. #Longhorn offers S3 backups for it's own volumes, but for other #CSI like local #OpenEBS, maybe #Velero? Thoughts?

https://velero.io/

Velero

SaustrupOct 3
I've been doing things I shouldn't with #Kubernetes. We're using a replicated #MinIO cluster as the storage backend on #mstdndk, which requires a boat load of storage, especially if you forget to specify any kind of retention. So far, the quick workaround for a full disk, was just to expand the filesystem. Since we're replicating across nodes, we're using #OpenEBS #LVM for local storage. Poor partitioning means we're running out of storage on the volume group, but even worse - PVCs sizes were increased before checking if we had space for it. Kubernetes is now stuck in a most unfortunate situation - it can't grow the local filesystem, as the volume group is full and you're not allowed to decrease the size request. What then? Cue https://github.com/etcd-io/auger - a tools that allows you to edit #K8s resources directly in #etcd. Obviously you should never do this, but with steady hands and clinical precision, you can get yourself out of a pickle like mine. Size was reverted and PVCs were unstuck.
GitHub - etcd-io/auger: Directly access data objects stored in etcd by Kubernetes.

Directly access data objects stored in etcd by Kubernetes. - etcd-io/auger

GitHub
Show threadSaustrupAug 12, 2025
A unfortunate side affect of Longhorn experiencing I/O latency and saturation, is that volumes attached to pods become remounted read-only. This has very very unfortunate side effects on running databases, caches etc. Any tips on making #Longhorn behave would be greatly appreciated. I've looked briefly into OpenEBS' distributed, replicated storage but the requirements are currently not allowing it - specifically, replicated #OpenEBS needs an entire physical disk for itself.
Show threadEPChrisFeb 1, 2025
@fwaggle I tried setting up #longhorn a couple years ago and got frustrated, but maybe its worth another look? Or maybe #OpenEBS?
Jon 404Apr 25, 2024

In what I can only describe as a bad case of "proof of concept making it to production", my #homelab #kubernetes cluster cannot run replicated #openebs because lack of hugepages support.....

Because I rolled forward with PV VMs. And then built everything on top of that, without realizing that the workers are PV.

I'm assuming swapping to #HVM is going to be painful, so I'm contemplating wiping everything and starting over.

#xen #selfhosting

simplyblockMar 18, 2024
#OpenEBS got archived by the #CNCF, and some people expressed concerns and asked if they have to think about alternatives. Are you concerned for its future and if you can still use it?
Yes
100%
No
0%
Poll ended at .
viqFeb 23, 2024
#kubernetes question: our deployment stack is #skaffold plus #helm. We have #OpenEBS deployed, I'm trying to add their monitoring mixin to deployment. But they want to deploy whole kube-prometheus, which we already have. I have a script to generate their manifests and grab just those 4 files we're interested in, and have helm grab those. But here comes the question: how do I tie in the generation with the rest of the deployment?
[1/2]