RachelJun 23, 2024
Currently wishing I had a better way to manage multiple redis and minio storages

The minio operator is missing features that would streamline it (looked at rook a bit but I still can't justify it I think)

The redis operator basically doesn't exist, idk if a valkey one will happen. Maybe dragonflydb? Their docs seem not amazing for their operator tho

#kuberbetes #homelab
Show threadRachel
Huh there was a proposal for a native object storage framework but it doesn't look like it got traction:

https://github.com/kubernetes/enhancements/tree/master/keps/sig-storage/1979-object-storage-support

edit: not dead yet! (https://github.com/kubernetes/enhancements/issues/1979)
enhancements/keps/sig-storage/1979-object-storage-support at master · kubernetes/enhancements

Enhancements tracking repo for Kubernetes. Contribute to kubernetes/enhancements development by creating an account on GitHub.

GitHub
Jun 23, 2024 at 8:09pmWeb
Show threadRachelJun 23, 2024
Oh no backend for on-prem ecept for ceph.... lmao at which point I could just run rook?

(https://container-object-storage-interface.github.io/)
Kubernetes COSI | Kubernetes COSI

Description will go into a meta tag in <head />

Show threadMichaelJun 24, 2024
@rachel Yupp, I also looked into this one briefly when setting up Rook, but went for Rook's own ObjectBucketClaim. I can't really recall why, but I think it was because COSI still felt a little bit too experimental?
Show threadRachelJun 24, 2024
@mmeier yeah not only experimental but turns out it doesn't even really do what I wanted it to do

Which was to feed it a regular persistent volume and have it serve objects from that

edit well it looks like it could do that but not currently implemented just big3 clouds + ceph
Show threadMichaelJun 24, 2024

@rachel And as much as I like Ceph and Rook - I think it would be a little bit much if you're only in it for the object storage.

On the other hand, if you'd also move your block storage over, I can very warmly recommend it. Ceph has just been rock solid for me, both baremetal and in Rook. I'm especially impressed with its stability. I recently woke up and found one of my two storage hosts dead, and everything was still up and once I rebooted the host, everything synced again without any issues.

Show threadRachelJun 24, 2024
@mmeier I am kiiiiiiiinda considering selling one of my PowerEdges and replacing it with a low power node so it can run an additional ProxMox and then get ceph running there... then I'd hook rook into that
Show threadMichaelJun 24, 2024
@rachel Oh, you mean you'd like to run Ceph on Proxmox, and then use that cluster an ExternalCluster with Rook?
Show threadRachelJun 24, 2024
@mmeier yup,

that way my VMs don't need to manage storage still

otherwise I'd have to like create a block device per k8s VM, or map disks to VMs? eeeeep.
Show threadMichaelJun 24, 2024
@rachel Do you plan to use Proxmox' internal Ceph (I think Proxmox itself can deploy/use Ceph?) or do you mean you want to run a couple "special" VMs on that Proxmox instance which will in turn form your Ceph cluster?
Show threadRachelJun 24, 2024
@mmeier Probably the internal ceph, no need to manage 3+ more OSes while I'm at it