My command for syncing #ZFS snapshots (i.e. backing up servers) is getting more complicated by the day. Now we're at:
Guruprasad 
Tobias BΓΆlz/usr/local/bin/retry -t 3 -- /bin/timeout -k 30s 20m /usr/local/bin/syncoid --pv-options="-q" --no-privilege-elevation --no-sync-snap -r $SERVER:zroot zpool/data/backup/$SERVER 2>&1
Today I had cause to revisit my home lab setup, and that in turn caused me to take a new look at my backup configuration and validate that everything is still in order.
If anyone is curious about setting up 3-2-1 style backups with #ZFS #snapshots using #Sanoid and #Syncoid, perhaps my writeup may be of some service.
https://oxcrag.net/blog/2025/11/16/zfs-backup-strategy-with-sanoid-and-syncoid.html
@kolev I store data on servers backed by #Ubuntu #KVM hypervisors with #ZFS.
#Sanoid gives me hourly snapshots for 36 hours, daily snapshots for a month, and weekly snapshots for three months.
#Syncoid ensures the snapshots are replicated to a separate pool in my main machine, plus (over SSH) to a separate backup server.
Of course if you run your Borg server as a VM on such a setup, you can easily keep secure backups of your actual workstations even if they run lesser file systems. I do something like this for the Macs in my family: the TimeMachine server is one of the VMs that get the Sanoid+Syncoid treatment. .
My ZFS snapshot and replication setup on Ubuntu ft. sanoid and syncoid
I have known about ZFS since 2009, when I was working for Sun Microsystems as a campus ambassador at my college. But it wasnβt until I started hearing Jim Salter (on the TechSNAP and 2.5 Admins podcasts) and Allan Jude (on the 2.5 Admins podcast) evangelize ZFS that I became interested in using it on my computers and servers. With Ubuntu shipping ZFS in the kernel for many years now, I had access to native ZFS!,
Here is an overview of my setup running Ubuntu + ZFS before I explain and document some of the details.
David Cantrell π
Paul Wilde 
Mikael Hansson
Daniel Tihanyi
Guruprasad Lcube β A headless server running Ubuntu 24.04 LTS (at the time of writing) with ZFS on root and a lot of ZFS storage powered by mirror vdevs. Has sanoid for automatic snapshots.sanoid for automatic snapshots.Sanoid configuration
On my personal computers, I use a minimal sanoid configuration that looks like
############# datasets #############[zroot] use_template = production recursive = zfs############## templates ##############[template_production] frequently = 0 hourly = 26 daily = 30 monthly = 3 yearly = 0 autosnap = yes autoprune = yes[template_ignore] autoprune = no autosnap = no monitor = noOn servers, the sanoid configuration has some additional tweaks, like the following template to not snapshot replicated datasets.
[template_backup] frequently = 0 hourly = 36 daily = 30 monthly = 3 yearly = 0 # don't take new snapshots - snapshots # on backup datasets are replicated in # from source, not generated locally autosnap = noPre-apt snapshots
While sanoid provides periodic ZFS snapshots, I also wanted to wrap apt transactions in ZFS snapshots for the ability to roll back any bad updates/upgrades. For this, I used the following shell script,
#!/usr/bin/env bashDATE="$(/bin/date +%F-%T)"zfs snapshot -r zroot@snap_pre_apt_"$DATE"with the following apt hook in /etc/apt/apt.conf.d/90zfs-pre-apt-snapshot.
// Takes a snapshot of the system before package changes.DPkg::Pre-Invoke {"[ -x /usr/local/sbin/zfs-pre-apt-snapshot ] && /usr/local/sbin/zfs-pre-apt-snapshot || true";};This handles taking snapshots before apt transactions but doesnβt prune the snapshots at all. For that, I used the zfs-prune-snapshots script (from https://github.com/bahamas10/zfs-prune-snapshots) in a wrapper cron shell (schedule varies per computer) script that looks like
#!/bin/sh/usr/local/sbin/zfs-prune-snapshots \ -p 'snap_pre_apt_' \ 1w 2>&1 | logger \ -t cleanup-zfs-pre-apt-snapshotsSnapshot replication
The cube server has sufficient disk space to provide a replication target for all my other personal computers using ZFS. It has a pool named dpool, which will be referenced in the details to follow.
For automating snapshot replication, I chose to use syncoid from the same sanoid package. To avoid giving privileged access to the sending and the receiving user accounts, my setup closely follows the path in https://klarasystems.com/articles/improving-replication-security-with-openzfs-delegation/.
On my personal computer, I granted my unprivileged (but has sudo π€·ββοΈ) local user account the hold and send permissions on the root dataset, zroot.
sudo zfs allow send-user hold,send zrootzfs allow zroot---- Permissions on zroot --------------------------------------------Local+Descendent permissions: user send-user hold,sendOn the cube server, I created an unprivileged user (no sudo permissions here π) and granted it the create,mount,receive permissions temporarily on the parent of the target dataset, dpool.
Then I performed an initial full replication of a local snapshot by running the following commands as the unprivileged user.
zfs send \ zroot@snapshot-name | ssh \ remote-user@cube \ zfs receive -u \ dpool/local-hostnamezfs send \ zroot/ROOT@snapshot-name | ssh \ remote-user@cube \ zfs receive -u \ dpool/local-hostname/ROOTzfs send \ zroot/ROOT/os-name@snapshot-name | ssh \ remote-user@cube \ zfs receive -u \ dpool/local-hostname/ROOT/os-namezfs send \ zroot/home@snapshot-name | ssh \ remote-user@cube \ zfs receive -u \ dpool/local-hostname/homeThe -u flag in the zfs receive commands above will prevent it from trying to mount the remote dataset. The target remote dataset must not exist when running this initial full replication.
As it is not a good practice to allow unprivileged users to mount filesystems, I disabled automatic mounting by running
zfs set mountpoint=none dpool/local-hostnameas the sudo user on the target server.
Then I narrowed down the permissions of the receiving user to only its own dataset by running
zfs unallow remote-user \ create,mount,receive dpoolzfs allow remote-user \ create,mount,receive dpool/local-hostnameon the target server.
Next, I tried to test the snapshot replication by running syncoid manually like
syncoid -r \ --no-privilege-elevation \ --no-sync-snap \ zroot \ remote-user@cube:dpool/local-hostnameand it replicated all the other snapshots all on the local datasets (we had only replicated one snapshot previously).
The sanoid package in Debian and Ubuntu does not ship with a systemd timer for syncoid. So I created a user service and a timer that look like the following examples.
# ~/.config/systemd/user/syncoid.service[Unit]Description=Replicate sanoid snapshots[Service]Type=oneshotExecStart=/usr/sbin/syncoid -r --no-privilege-elevation --no-sync-snap zroot remote-user@cube:dpool/local-hostname# ~/.config/systemd/user/syncoid.timer[Unit]Description=Run Syncoid to replicate ZFS snapshots to cube[Timer]OnCalendar=*:0/15Persistent=true[Install]WantedBy=timers.targetThen I reloaded systemd, enabled and started the above timer to have everything working smoothly.
Just in case I run into the following error again:
CRITICAL ERROR: Target zpool/data/backup/somemachine/var/db/something exists but has no snapshots matching with zroot/var/db/something!
Replication to target would require destroying existing
target. Cowardly refusing to destroy your existing target.
This can be fixed by simply destroying the target filesystem with zfs destroy -r zpool/data/backup/somemachine/var/db/something and then running syncoid again.