Yesterday morning, I pulled open my laptop to send a quick email. It had a frozen black screen, so I rebooted it, and… oh crap.
My 2-year-old SSD had unceremoniously died.
This was a gut punch, but I had an ace in the hole. I'm typing this from my restored system on a brand new drive.
In total, I lost about 10 minutes of data. Here's how. (Spoilers: #zfs #zrepl)
I don’t back up my drives, I replicate them.
Last winter, I set up my first serious home network storage. Part of this project was setting up periodic backups of the computers I do creative work on. After surveying the options, one approach stood out: ZFS incremental replication.
One of the flagship features of ZFS is the ability to take efficient point-in-time snapshots while it’s running. You can then send only the changed data to other machines...
To automate taking snapshots and sending them to my NAS, I’m using a really cool piece of software called zrepl (by @problame). I configured it to snapshot and send my entire filesystem every 10 minutes.
Since the snapshots are incremental, this is fine to run in the background on my home network to keep the replica up to date. The last run took 14 seconds to transfer and sent about 64 MiB.
Restoring the system was a learning process, and unfortunately quite manual. I let the 625 GiB ZFS receive operation run overnight.
My snapshots are encrypted by the original computer (this is cool because the NAS can’t read them!). So I also needed to restore the encryption “wrapper key” to be able to use the backups.
Not gonna lie, it was pretty terrifying until I had my first confirmation I could decrypt the data.
To rebuild my system, I followed the OpenZFS guide for setting up a filesystem from scratch via Ubuntu 22.04 live USB:
This was a priceless resource for getting back up and running. It’d intimidated me in the past, but it’s *so* thorough, and I learned a ton going through the process. This is the best hand-on guide I’ve seen for modern partitioning and chrooting in a Debian environment.
The end result was a beautiful moment: my laptop booted back up to right where I’d left it. Even my browser tabs restored my unfinished work from the previous night.
There’s this classic series of Chromebook ads from 12 years ago where computers are repeatedly destroyed in elaborate ways, and the host picks up a new machine and picks up where they left off, with no data lost:
Chrome UX designer Glen Murphy demonstrates some advantages of using a Chrome notebook. 25 computers were harmed in the making of this video. Fortunately, no...
That ad has been in my imagination for over a decade. I finally achieved my dream of having a similar disaster recovery plan. And it worked!
Setting ZFS up initially had a really high starting cost: it took a full filesystem swap. Maintaining it takes fairly knowledge-heavy and manual processes. But it certainly has unique benefits.
This is the first time I can recall losing an SSD in over 15 years of using them. It was fantastic luck that I’d set up replication before my first one failed. 😇
Btw, if you’re curious, the offending drive was a WD_BLACK SN850 from my original Framework order. I’d heard unsettling stories on the Framework forums of this drive spontaneously dying or becoming unbootable. I guess it was my turn to roll some unlucky numbers.
Amazon shipped me a new SK Hynix P41 SSD and a Sabrent NVMe enclosure in about 3 hours yesterday, which was phenomenal. I usually try not to order tech from there if I can avoid it, but credit where credit’s due.
@kmartino Thanks! I'm currently running a pretty bog standard Ubuntu setup since it's what I'm most familiar with. I'm using a Sabrent 5 drive USB 3.2 enclosure with some Seagate Exos drives I got on Cyber Monday for about $15/TB new.
TrueNAS is another popular alternative (though I haven't dabbled with it). Happy hacking!
@pirafrank @kmartino I haven't seen IO errors, but ZFS occasionally panics with:
[48259.172886] VERIFY3(c < SPA_MAXBLOCKSIZE >> SPA_MINBLOCKSHIFT) failed (36028797018963967 < 32768)
[48259.172896] PANIC at zio.c:341:zio_data_buf_alloc()
The Sabrent enclosure itself has been very nice. Well built, compact, and USB 3.2 should be sufficient for 4 drives without bottlenecking.
You are better prepared than other WD and Sandisk SSD users.
https://www.theverge.com/22291828/sandisk-extreme-pro-portable-my-passport-failure-continued
https://news.ycombinator.com/item?id=37042587
https://www.theverge.com/23837513/western-digital-sandisk-ssd-corrupted-deleted-questions
Just a heads up. Do not buy a WD Black SN770 model for use with ZFS. Possibly avoid any drives without a cache. https://github.com/openzfs/zfs/discussions/14793
@gnomon @laffer1 Oooh! WDS200T1X0E-00AFYO. Thanks for the link, I'll enjoy reading up.
FWIW, I didn't experience a notable amount of crashes or instability for the ~2 years I used it. It just suddenly stopped being an NVMe drive yesterday.
In my case it was a SN770, not the SN850.
Model was WDS200T3X0E
Has firmware: 731100WD
@laffer1 @chromakode right, with a critical differentiation that the ZFS issues reported by you and elsewhere were on NVMe drives without DRAM, which this SN850 definitely has.
(I know you know this already, just mentioning it for the sake of anyone else seeing the thread but not yet the detailed bug report)
@chromakode uh oh, I ordered that drive with my Framework (batch 2 AMD 13")
I'm now wondering if I should risk it, or remove the SSD from the order and get something else.
@piepants Oof, I hadn't heard about that with the AMD boards. First of the line struggles. I'm keenly waiting to see what the battery life looks like once they land 😁
I upgraded my 11th gen to 12th, swapped the lid, and replaced the heatsink (I clumsily dropped my phone on it while showing a friend). The repairability of these machines is awesome.
@oook @piepants That's a cool idea! For my own purposes I'm a little skeptical of the storage expansion cards. I'd rather not sacrifice a port for persistent storage.
A commenter on the orange site mentioned they use an external NVMe in an enclosure or a hot drive spare. I'd expect that to support better write speeds than a storage expansion card, and then if the primary dies you can swap it in.
If I was doing more road warrioring I'd explore this!
@chromakode @piepants I like your idea to plug an nvme as external enclosure of your nas.
For a while I used a work laptop that was provided with windows. I didn't want to risk wiping the windows install so I installed linux on an usb enclosure that was velcroed to the laptop. Now my work laptop has dual nvme port so I just plugged it in the second port and voilà, cleaner.
@thomastospace Beyond the restore process, a few things which have taken time and mindspace:
- Managing encryption on boot and saving backup copies of wrapper keys (not specific to ZFS)
- Setting up zrepl on several hosts and monitoring it in case it breaks
- Re-learning how to fix grub w/ ZFS root
- My wife's system has had intermittent zpool scrub failures which I spent a bunch of time debugging but haven't figured out
- My NAS seems to have some failure modes where the kernel panics :(
@stayprivate Amen! I tested about 6mo ago when I set up zrepl, but I could have run through the restore flow better -- especially around how to manage Ubuntu's tricky LUKS-nested-in-Zvol key wrapper. To be honest I expected to lose a file or directory for my first real world restore, not an entire drive!
Also agreed that the system files are a convenience rather than a necessity (though they could come in handy in edge case-y forensic situations)
@chromakode @problame I totally agree with you - zrepl is fantastic and I use it across my little ISP to have each hypervisor make a block device snapshot of running VMs and copying those offsite to two disk tanks.
I've managed to repair not one but several VMs and even hypervisors that became incapacitated!
@lordgaav I use raw sends because I want the opposite: I don't want my destination to be able to read the original data.
I haven't had much experience with the flow you're looking for, but I believe it's possible!
@chromakode #ZFS is just awesome!
And this is why everyone should use it!
Cyclist
Max Goodhart
kmartino
brianwilson
Francesco Pira @ FOSDEM 24
xet7
Ben Zanin
Lucas Holt
Andrew Cook
Oook
ThomasToSpace
Stay Private
IPng Networks
Nick Douma
rifter
Kevin Karhan 
ElectroFetish
tyil
Thibaultmol 🌈
naufraghi 🌈send feature that I tried and it seems to work quite well.@lamp @chromakode that's why #backups are essential.
#ZFS makes these easy: You just do a snapshot and transfer it away with zfs-send ...
Malte Petersen
Terin Stock
Lamp
Betoniusz