chazwhiz

LF Suggestions on how to architect new setup, 5x22TB + 3x4TB NVME

https://lemmy.world/post/39019749

Nov 19 at 2:36pmWeb
Show threadAvid AmoebaNov 19
ZFS. It runs on whatever RAM you give it.
Show threadCondorWonderNov 19
For your second scenario - yes you can use md under bcache with no issues. It becomes more to configure but once set up has been solid. I actually do md/raid1 - luks - bcache - btrfs layers for the SSD cache disks, where the data drives just use luks - bcache - btrfs. Keep in mind that with bcache if you lose a cache disk you can’t mount - and of course if you’re doing write-back caching then the array is also lost. With write-through caching you can force disconnect the cache disk and mount the disks.
Show threadchazwhizNov 19
With write-back you’d only lose what was in cache right? Not the entire array?
Show threadCondorWonderNov 19
Bcache can’t differentiate between data and metadata on the cache drive (it’s block level caching), so if something happens to a write-back cache device you lose data, and possibly the entire array. I wouldn’t use bcache (or zfs caching) without mirrored devices personally to ensure resiliency of the array. I don’t know if zfs is smarter - presumably is can be because it’s in control of the raw disks, I just didn’t want to deal with modules.
Show threadCmdrShepard49Nov 19

Today I have a huge bottleneck in unpacking and moving. I’ve got 1gb fiber and can saturate it, getting a complete iso in just a few minutes, but then it’s another 30min plus waiting for that to actually be usable.

Are you doing this all manually or using the *arr suite? For me, this process takes a minute or two depending on the size of the files with Proxmox and ZFS but even previously on Windows 10 with SnapRAID it was quick.

Show threadchazwhizNov 19
Arr and Sab
Show threadWhyJiffieNov 19

Take the ZFS plunge - My only real concern is the overhead

you shouldn’t worry about ZFS overhead if you are planning to use mergerfs.

you can tune the memory usage of ZFS significantly. by default it targets using half of your RAM, but on a home setup that’s wasting resources, you should be able to limit the arc cache to 1-2 GB, maybe somewhat more depending on how you want to use it. It’s done with sysctl or kernel parameters.