Denny
JulianCalabyCursed homelab update:
I learned a _lot_ about Rook and how it manages PersistentVolumes today while getting my PiHole working properly. (Rook is managed Ceph in Kubernetes)
In Kubernetes, the expectation is that your persistent volume provider has registered a CSI driver (Container Storage Interface) and defined StorageClasses for the distinct "places" where volumes can be. You then create a volume by defining a PersistentVolumeClaim (PVC) which defines a single volume managed by a StorageClass. The machinery behind this then automatically creates a PersistentVolume to define the underlying storage. You can create PersistentVolumes manually, but this isn't explored much in the documentation.
In Rook, this system is mapped onto Ceph structures using a bunch of CSI drivers. The default configuration defines StorageClasses for RBD images and CephFS filesystems. There are also CSI drivers for RGW and NFS backed by CephFS. You then create PVCs the normal way using those StorageClasses and Rook takes care of creating structures where required and mounting those into the containers.
However there's a another mechanism which is much more sparsely mentioned and isn't part of the default setup: "static provisioning". You see, Ceph clusters are used to store stuff for systems that aren't Kubernetes and people tend to organise things in ways that the "normal" CSI Driver + StorageClass + PVC mechanism can't understand and shouldn't manage. So if we want to share that data with some pod, you need to create specially structured PersistentVolumes to map those structures into Kubernetes.
Once you set up one of these special PersistentVolumes and attach them to a pod using a PVC, then you effectively get a "traditional" "cephfs" volume mount, but using Rook's infrastructure and configuration, so all you need to specify is the authentication data and the details for that specific volume and you're done.
The only real complication is that you need a separate secret for this, but chances are you're referencing things in separate places to the "normal" StorageClass stuff and giving Rook very limited access to your storage, so this isn't a big deal.
So circling back around to the big question I wanted answers for: Does Rook mess with stuff it doesn't know about in a CephFS filesystem?
No.
If you use the CSI driver + StorageClass mechanism it will only delete stuff that it creates itself and won't touch anything else existing in the filesystem, even if it's in folders it would create or use.
If you use a static volume, then you're in control of everything it has access to and the defaults are set so that even if the PersistentVolume is deleted, the underlying storage remains.
So now onto services that either should be using CephFS volumes or need to access "non-kubernetes" storage, starting with finding a way to make Samba shares in a container.
Michael
OpenInfra Foundation
Worteks
beyondwatts
benzogaga33 
sebulon
HostingViet