Yay! This is finally documented and out [1]. It took me a while to consolidate my thoughts and write down the explicit motivation and steps to create this modern virtualized system architecture (Proxmox, Rootless Docker, VirtIO-FS, and ZFS).

[1]. https://du.nkel.dev/blog/2026-05-16_rootless_docker_virtiofs_proxmox/

#Proxmox #Docker #Homelab #Selfhosting #ZFS #Sysadmin #Linux

A modern Proxmox Docker architecture with disposable VMs, VirtIO-FS, and ZFS state separation - du.nkel.dev

A personal code notes blog

Die Selbsthilfegruppe CoW FS (#ZFS, #BTRFS, #bcachefs) war ein voller Erfolg, wir haben nur den Namen/Nick von dem Menschen mit hoher ZFS-Erfahrung, der früher weg musste nicht mehr, bitte melden.

https://cfp.gulas.ch/gpn24/talk/AJNBYY/

Auch die Tatsache, dass ich direkt vom CSD rüber musste noch in MX und ohne Laptop hat der Sache keinen Abbruch getan.

#gpn24 #gpn24cow

Linux CoW file system meetup – did your file system hurt you? 24th Gulaschprogrammiernacht

Which copy-on-write file system is right for your home use under Linux? Is more complexity in the file system always better? How does ZFS licensing affect the real-world use? What even is the Btrfs RAID5 write hole and why wasn't it fixed? What's up with all the Bcachefs-drama, does this now out-of-tree fs still hold any advantages? Join us and freely discuss over a Mate how your file system has hurt you, which slightly non-recommended settings have worked for you and all the creative ways you have managed to lose your TBs of Linux isos.

FreeBSD bug 263171 – add loader(8) and boot loader menu support for boot with OpenZFS-encrypted ROOT

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=263171

@teapot_ben

#FreeBSD #encryption #boot #ZFS #OpenZFS

263171 – add loader(8) and boot loader menu support for boot with OpenZFS-encrypted ROOT

New tool available: findbay

It’s still early days, but I got a wild hair today and began writing a new simple tool, findbay.

root@elden:~# zpool status | grep SHGP31 nvme-SHGP31-2000GM_ANA7N7[elided] root@elden:~# findbay ANA7N7 /dev/nvme0n1 root@elden:~# pv < `findbay ANA7N7` > /dev/null 47.0GiB 0:00:08 [6.97GiB/s] [> ] 2% ETA 0:04:59

In the above snippet, we can see me demonstrating that NVME drives have absurdly long device labels, showing you a chunk of one of mine, and then running findbay against a small chunk of that device label (usually, I’d recommend using the last four or five characters, because those are the most likely to be unique).

Run by itself, findoid [string] will either return the bare devicename (eg /dev/sda) associated with any entry in /dev/disk/by-id, or will complain and error if there’s either no result or multiple results (the latter meaning the string you chose was not even locally unique, eg using SEA when you have more than one Seagate drive).

But you can also chain it into a pv (or even cat) command, as shown directly after that, which will not only return the raw devicename but also light up the associated LED.

Future versions of the tool will support blipping the drive with reads DIRECTLY, and include an option to specify a pattern (for obnoxious bays that don’t show much difference between full saturation and light activity). But I think this is already pretty useful in the meantime.

You can find findbay in MASTER of the Sanoid project Github right now; it should make its way out to distributions with our next full release.

1 post - 1 participant

Read full topic


#zfs

New tool available: findbay

It’s still early days, but I got a wild hair today and began writing a new simple tool, findbay. root@elden:~# zpool status | grep SHGP31 nvme-SHGP31-2000GM_ANA7N7[elided] root@elden:~# findbay ANA7N7 /dev/nvme0n1 root@elden:~# pv < `findbay ANA7N7` > /dev/null 47.0GiB 0:00:08 [6.97GiB/s] [> ] 2% ETA 0:04:59 In the above snippet, we can see me demonstrating that NVME drives have absurdly long device labels, showing you a chunk of one of mine, and then running findbay against a...

Practical ZFS

FYI:

Ok, todays fun:

Finding out that the two disk images in Proxmox on #ZFS that I can't delete, are a bug in ZFS.

AI writes:
--------------
The root cause was a ZFS 2.4.x kernel deadlock in z_zvol — triggered by volmode=none racing with active zvol creation. Setting volmode=none before the reboot was the key that allowed the destroy to succeed cleanly afterwards.

The most relevant one to your exact stack trace is OpenZFS issue #17570 — "task z_zvol blocked for more than 6225 seconds", which shows the identical zvol_remove_minor_task → zvol_set_volmode_impl deadlock pattern you hit: GitHub
👉 https://github.com/openzfs/zfs/issues/17570
There's also a related one: issue #17696 — "zpool export hangs indefinitely if a zvol is in use", with the same D-state z_zvol thread and identical call trace: GitHub
👉 https://github.com/openzfs/zfs/issues/17696
Both are known regressions introduced around ZFS 2.2.x and still present in 2.4.x. You might want to add a comment to #17570 with your specific 2.4.2-pve1 experience.
--------------

So, this was the reason why the server were dog slow today as I was migrating VMs to get rid of those stuck disk images on my small NVMe storage.

To get rid off those disks I needed a reboot. To reboot, I needed to migrate the VMs to another host.

/ @ij

task z_zvol blocked for more than 6225 seconds · Issue #17570 · openzfs/zfs

System information Type Version/Name Distribution Name Rocky Linux Distribution Version 9.6 Kernel Version 5.14.0-570.25.1.el9_6.x86_64 Architecture x86_64 OpenZFS Version zfs-2.2.8-1 Describe the ...

GitHub

Freeze and unfreeze

I have 12 SATA disks in my pool. Everything is fine and it has good health. But I would like to document which disc serial number is in which slot. The easiest way I can think of is to run lsblk --scsi to see the disc designations and their serial numbers and print that to paper to create a list. Then, pop a disk out very slightly and Mark which slot it is in in the list. I’ve looked at the undocumented freeze and unfreeze commands, and I would like to know from any experts here how safe they are, and if they are applicable to my purpose. Thanks in advance.

3 posts - 2 participants

Read full topic


#zfs

Freeze and unfreeze

I have 12 SATA disks in my pool. Everything is fine and it has good health. But I would like to document which disc serial number is in which slot. The easiest way I can think of is to run lsblk --scsi to see the disc designations and their serial numbers and print that to paper to create a list. Then, pop a disk out very slightly and Mark which slot it is in in the list. I’ve looked at the undocumented freeze and unfreeze commands, and I would like to know from any experts here how safe they ar...

Practical ZFS

I spent a whole blog post doing #ZFS on-disk math by hand - just to corrupt one byte and watch #OpenZFS healing process.

Interested?

Feel free to join the journey into the on-disk jungle.

https://oshogbo.com/blog/90/

#storage #FreeBSD #Linux #Storage #Filesystem

Corrupting a ZFS File on Purpose

Most of the time, the whole point of ZFS is that your data does not get corrupted. But during development you sometimes need the opposite: a controlled, reproducible corruption, so you can watch self-healing kick in, see what a scrub reports, or just understand how a file maps onto the physical disk. There is no better exercise than breaking one byte on purpose and seeing ZFS notice.

oshogbo//vx

🇬🇧 Ever wondered how burningboard.net actually runs?

Fully self-hosted in Germany, no trackers, no external logging. Isolated FreeBSD jails, a strict firewall, ZFS and encrypted off-site backups (restore-tested twice a year). Sovereign and transparent. All the details: https://meta.burningboard.net/infrastruktur.html

🇩🇪 Wie läuft burningboard.net eigentlich?

Komplett selbst gehostet, in Deutschland, ohne Tracker und ohne externes Logging. Isolierte FreeBSD-Jails, strikte Firewall, ZFS und verschlüsselte Offsite-Backups (2x im Jahr per Restore getestet). Souverän und transparent. Alle Details: https://meta.burningboard.net/infrastruktur.html

#mastoadmin #freebsd #zfs #gdpr #digitalsovereignty #fediverse

Unsere Infrastruktur - burningboard.net

Wie wir burningboard.net betreiben: souverän, sicher und vollständig in eigener Hand. Segmentierte FreeBSD-Architektur, verschlüsselte Offsite-Backups und Hosting in Deutschland.

burningboard.net
Going to try out zfs.rent. You send them a hard drive or mirror pair and they put it in a machine in a data center. Cool way to zfs send snapshots and seems fairly inexpensive. Anyone using them?
#zfs