HereIAm

Create a retention period for online backup storage

https://lemmy.world/post/38742977

Create a retention period for online backup storage - Lemmy.World

I finally got some online storage to backup my home server to, and I can finally sleep easy when my house inevitably goes up in flames. My initial push has been with rclone using the sync command. I plan on running a daily or weekly push. I’ve gone with Hetzner and they have a variety of options on how to connect to their storage box as they call it. They have an option for automatic snapshotting of, as I understand it, the entire storage box that you can then recreate. So my thoughts are around rclones sync and copy commands, and the storage snapshots. Copy) If I use copy, it won’t clear up deleted files automatically that I actually want gone. Sync) With sync I’m worried that say a drive failed (my set up is a bit wonky currently until I can build a proper RAID), it will do a sync, see an entire directory is gone, and go ahead and delete it. Snapshots) Here I’m worried that any new data that’s been added in-between the snap shot and a server failure would be lost. And I don’t you can retrieve partial data from a snapshot. What I have in mind is to use rclone’s copy command, but then somehow have a retention period on files that no longer exist on my server. So a deleted file might exist in my storage bucket for a month before being removed from there as well. How do you guys manage this? Are there any ready made solutions, or do I need to whip up a script that goes through and looks at the last modified dates and removes the old ones?

Nov 13 at 2:57pmWeb
Show threadEager EagleNov 13

I think you need something like restic with a retention policy

restic.readthedocs.io/en/stable/060_forget.html#r…

–keep-{hourly,daily,weekly,monthly,yearly}

other solutions that implement similar policies are kopia and rustic

the advantage of using an off the shelf solution is that it’s almost certainly more reliable than what anyone can come up with in a few hours, and, it works with incremental backups, so your space requirements are drastically reduced depending how often you run it.

Removing backup snapshots — restic 0.18.1 documentation

Show threadolivierNov 13
you might want to add borg, restic and - more recently - plakar to that list
Show threadJason2357Nov 14

Restic is great, and the de-duplication between snapshots is amazingly good. Same content in different files (e.g. tar files of linux systems) take very little space like magic). Backrest is a nice web frontend for it.

Note that you should use some retention features of your provider to manage the risk of ransomware deleting your backups.

Show threadnon_burglarNov 13

Scripting it isn’t that tricky.

I’m old and use rsync, with mtime and not(!) operands and I’m able to keep 7 daily, 4 weekly, and 4 monthly backups. Runs every day, the rest of the backups are pruned in this rolling window.

Show threadpi3r8Nov 13
I have been trying out duplicati for backups the last few weeks and so far it’s doing what I need and I think would offer the features you are looking for.
Show threadHereIAmNov 14
Yeah that looks cool. I will give it a closer look tomorrow :)
Show threadpassengerNov 14
Use duplicati, or if you want command line, use borg.
Show threadnomadNov 15

If you wanna backup your device as well, I can recommend Resticopia

It’s based on Restic and you can choose retention rules 👍

Resticopia - Android Backup Made Simple