It appears Linux root-on-ZFS is a mess. No standard way to do it. Kernel updates require recompiling ZFS. Boot environments are a cornucopia of constantly evolving hacks.
If you're actually using #ZFS on root, on #Debian, what's your preferred hack to make that happen? #sysadmin
I suspect #openzfsmastery might need to assume root on extFS and data on ZFS, leaving root-on-ZFS for the advanced user or a terminal chapter. 
well, ⬆️ sparked discussion.
There are linuxes that integrate ZFS into the installer, like Alpine and Chimera, and support ZFS on root. Good for them!
In general, though, more most linuxes, ZFS on root is an edge case that requires above average expertise with the boot loader. (If you have that expertise you think it's not much, but for many folks it IS a big deal.) Properly covering ZFS on root on Linux would require I write a book on initrd, UEFI, and more. I don't want to write that book. The workarounds various folks use evolve constantly, and by the time the book comes out they would have changed.
Looks like Linux ZFS root with ZFSBootMenu will be relegated a final chapter for the advanced user. 
@mwl Yeah, this was a deliberate licence incompatibility from Oracle designed to hurt Linux specifically. Take it up with Larry Ellison and the rest of Trump's cabinet.
Ubuntu decided to just Do It Anyway, so there's that avenue for pre-built kernels, but my experience with that a decade ago left me with a bad taste in my mouth. Maybe they got better?
Yep, that's Oracle! 
:flan_actually_bloody_evil:
Ubuntu is heavily deprioritizing ZFS. https://openzfs.github.io/openzfs-docs/Getting%20Started/Ubuntu/Ubuntu%2022.04%20Root%20on%20ZFS.html
@spacehobo This is exactly the reason why it's still not supported. Btrfs spawned more or less as a response and might actually be the better filesystem in many cases.
@mwl I stole scripts from proxmox, which does a clever thing where they have the efi/boot partition on every disk in the vdev, and then they add a debian hook that installs the bootloader and copies the initrd to every single one of them whenever that changes, ensuring that any disk that survives an untimely death can boot the system correctly.
It also implies they have a lot of extra glue for all of these filesystems, and that in its absence, you have to deal with that yourself, which is annoyingly involved.
Which, I think, mostly reinforces your point
@mwl Or root-on-btrfs so you at least get CoW everywhere, even if they are managed in radically different ways.
But yes, root-on-ZFS on Linux is absolutely a mess. It’s the biggest reason I don’t consider Linux for anything I build and have to maintain.
@mrhamel Okay, fine, I left the word “usable” out. Linux lacks *usable* ZFS. Which, to me, is *lacking ZFS*.
I don’t have the patience for the traditional Linux “Tamagotchi mode”.
@mwl Not Debian, but a downstream derivative: I run Mint with ZFS root on my primary work computer and have for years, zero problems with kernel updates, Mint upgrades, etc. It all "just works".
They removed it from the installer a few versions ago but if you install an older Mint and upgrade it still works fine.
@mwl Zorin linux used to have an experimental root on ZFS option, maybe they took it from upstream Ubuntu? not sure.
But it was gone the last time I checked.
@mwl I’ve had success just following the OpenZFS Debian instructions: https://openzfs.github.io/openzfs-docs/Getting%20Started/Debian/Debian%20Trixie%20Root%20on%20ZFS.html .
Been running for multiple years across two distro updates with only minimal issues - ( and the worst ones were self inflicted).
It’s labor intensive to do the initial install but otherwise fine in my experience.
@mwl I compile my own ZFS kernel module + userspace packages from source for every specific kernel, then I pin the kernel.
Yes, I've documented how to do this. Yes, it's online.
This sent me looking to your blog, of course. After seeing your latest post I'd like to say: you saying you like my books is especially meaningful, because I know you'll say when you don't like a book. 
@mwl Oh dear. I don't know what you saw.
I have a number of ZFS-on-root tutorials for a few different Linux distributions on my old blog, at least one for Void Linux IIRC, but these days I have scripts to do it for me.
I followed your profile, xenotrope.blogspot.com. ;-)
thanks for the link, I like scripts!
@mwl @gnomon My personal feeling as a long time user of ZFS and also of ZFS on Linux specifically, is that root on ZFS is definitely "danger, experts only" territory today unless you can find a distribution where it's officially fully supported so it's tested, and maintained by them.
My personal view is that ext4 root + ZFS homedir+etc pool is the best and easiest way to go (and how I do it on my desktops). You lose some space, but it's much easier to deal with.
@mwl @gnomon For typical Linux scenarios I'm not sure ZFS root gets you very much (and it does get you pain and worry). ZFS integrity checking is somewhat redundant with good package managers being able to verify checksums for most everything (and then you can reinstall anything broken). But maybe there are people out there who've really used rollback after upgrades.
(Having flexible space usage would be nice, but ... modern disks have gotten big.)
@Tubsta @mwl @gnomon My personal view is that rollbacks are such a blunt hammer and so badly or not supported by software¹ that it would have to be something massive to get me to do a system-wide rollback instead of reverting a package back. And if I don't catch it right away, a rollback will also probably roll back other things I want to keep, and etc.
¹ eg Firefox explicitly doesn't guarantee that you can use a slightly older version of Firefox on a profile that's been touched by a newer one.
Its tragic.
There is no SINGLE Linux distribution with installer that would allow you to install Linux with Root on ZFS - with ZFS Boot Environments - and with ZFSBootMenu integrated ... NONE. ZIP. Nada. Zero.
Not to even mention that it would also support LUKS encryption on Linux - no way.
On FreeBSD you have fully protected setup with ZFS Boot Environments supported in the loader and with GELI or not included.
The ZFS Boot Environments feature on Linux - the thing that is DESIGNED to PROTECT you from any disasters (that it works on FreeBSD and Illumos) is one that is most fragile and uncertain on Linux.
@EF that helps, ty!
looks like you still need to add some fit and polish for it, but not too bad: https://wiki.alpinelinux.org/wiki/ZFS
@mwl you do but that applies to a lot of #alpinelinux. Not for newbies but it is reliable and fast even through updates. Run multiple machines with native ZFS encrypted roots.
Chimera (BSD userland) and Postmarket (mobile devices) both use Alpine as a base but never used them and not sure how integrated ZFS in them.
@mwl The boot environment for Linux has always been a mess of ugly and constantly evolving stop gap measures. If you want some fun of diving into this rabbit hole, check out the Linux from scratch stuff.
But yeah, that is far from the only disk related mess. Frankly everything disk related and booting basically is.
If you want to run screaming, check out the partial software RAID implementation md.
IMHO the whole disk subsystem needs re-writen from scratch to work akin to how GEOM does on FreeBSD.
@mwl
NixOS offered ZFS on root out of the box too.
In my case, as an opensuse user on non-servers (specifically its rolling Tumbleweed) - it's a fully manual non-standard install and then very careful upgrades (with locking the kernel version at times), but it's still better than not having ZFS, for the root FS screenshots included.
But yeah, it's a marginal "expert-ish" group within Linux users at all.
@mwl I’m sad for them.
zroot is the basis of Boot Environments.
BE allows upgrade in place with greatly reduced risk; there is a known good fallback position.
Hopefully, one day they can get their shit together.
@mwl fwiw my personal experience has been that initial installation is an unpleasant experience, but after that I have had pleasant experiences on Debian 11/12/13 (upgraded, never reinstalled) with ZFS root and /bot. This is using kernel & zfs packages from debian.org.
Initial installation: Run a very large series of commands in the terminal of a live CD to install, you can't use the interactive installer at all. It really is very daunting and I don't fault anyone for falling out of the funnel at this point!
grub compatibility: modern zfs has a set of compatibility flags for grub2. Use them.
kernel compatibility: zfs-dkms is automatically built when a new kernel is installed. In the rare case that the build fails for some reason, the previous kernel has remained bootable, allowing me to boot and fix the problem. If this had not been the case, things could have been rather difficult.
other problems:
*I made the pool containing /boot way too small, but that's life.
It's rather sad that at one point, Debian's installer had zfs support. I used it to install debian kfreebsd with all zfs filesystems. However, it has never been usable with the linux kernel since there's not a zfs kernel module udeb available and has likely been removed by maintainers (but I haven't gone checking anything about this)
Michael W Lucas 
Jens
Michael Dexter
Alex Rosenberg
ClaudioM
Space Hobo
Jason Tubnor 🇦🇺
Ryan Hamel
Emil Jacobs - Collectifission
mr_daemon
sungo
Simon Richter
Tom Rini
Zimmie
Jim P.
OpenTechFan 🇳🇱🇪🇺🇺🇳
Stefano Marinelli
Matt Bond | zxaos
Nelson Minar
AcidDensity
Shae Erisson
-dsr- (not a dog)
Jeff
chop wood, carry water
Guda Blues
Toby
itorres
Chris Siebenmann
vermaden
Carl C
AMS
chesheer
VVelox
gemelen
Dan Langille
Stylus