A faster way to copy SQLite databases between computers – alexwlchan

LinkA faster way to copy SQLite databases between computers
https://example.com/faster-sqlite-copy

📌 Summary:
本文介紹了一種更快速且穩定地在不同電腦間傳輸 SQLite 資料庫的方法。作者面臨傳統使用 rsync 直接複製 SQLite 資料庫的瓶頸,特別是在資料庫規模增大且含有多個索引時,複製時間顯著增加且容易出錯。解決方案是利用 SQLite 的 .dump 指令將資料庫導出為包含建表及插入指令的純文字檔,該檔案大小往往比原始資料庫小許多,主要因為索引只以指令形式存在,不重複儲存數據。再將這個文字檔壓縮成 gzip 檔案,透過 ssh 傳輸到本機後解壓,再依序執行文字檔內容重建資料庫。這樣不僅減少了傳輸檔案大小,也避免了傳輸過程中資料庫被異動導致損壞的問題,進而提升傳輸速度與穩定性。最後作者分享實際例子,一個 3.4GB 的資料庫壓縮後文字檔僅約 240MB,縮小約 14 倍,顯著節省傳輸時間及資源。

🎯 Key Points:
→ ★ 問題描述:隨著 SQLite 資料庫及索引規模增大,使用 rsync 複製資料庫耗時且容易出錯。
→ ★ 主要瓶頸:索引會額外複製重複資料,導致檔案體積膨脹。
→ ★ 解決方案:利用 sqlite3 的 .dump 指令將資料庫轉成 SQL 指令的純文字檔,內容包含建表、插入資料以及索引建立語法。
→ ★ 優點:純文字檔大小常小於原始資料庫,gzip 壓縮後更為顯著,減少資料重複複製。
→ ★ 實務流程:① 在遠端伺服器透過 ssh 指令產生壓縮的純文字檔,② 使用 rsync 傳輸該檔案到本地,③ 解壓縮並利用 sqlite3 將 SQL 指令匯入重建資料庫,④ 清理暫存文字檔。
→ ★ 穩定性提升:先產生純文字轉存版本,避免在傳輸過程中資料庫被更新造成不一致導致打開錯誤。
→ ★ 範例數據:3.4GB 的 SQLite 資料庫,轉 dump 後文字檔約 1.3GB,用 gzip 壓縮後約 240MB,縮減近 14 倍。
→ ★ 實務價值:大量或多索引的 SQLite 資料庫能透過此方法更有效率且安全地傳輸與備份。

🔖 Keywords:
#SQLite #資料庫備份 #資料庫傳輸 #gzip壓縮 #rsync

A faster way to copy SQLite databases between computers

Dumping a SQLite database to a text file can make it much smaller, which means you can download it faster.

🚀 Hey tech wizards, did you know copying a database just got a whole minute faster? 😂 Apparently, #rsync has been discovered in 2025! 🎉 Prepare to be blown away by cutting-edge revelations from the future! 🚀📂
https://alexwlchan.net/2025/copying-sqlite-databases/ #technews #futuretech #databaseinnovation #speedupdates #hackernews #HackerNews #ngated
A faster way to copy SQLite databases between computers

Dumping a SQLite database to a text file can make it much smaller, which means you can download it faster.

Curious if it is truly as simple as running #rsync across the VPSs both running  Debian 11 like and resting up the DNS name server IPs like this guide claims

https://www.man42.net/blog/2017/07/how-to-migrate-a-debian-server/
How to migrate a Debian server (with rsync) - man42 blog

How to migrate a Debian server (with rsync)'

man42 blog
@frantictdrinker And a question on a personel pain point: what software do you use for #backup in #windows? I have home-cooked #rsync #bash #scripts (a couple of lines only, does the job) for #Linux, but I am at a loss what tool to use for incremental backup for Windows (I need a GUI app).

@thoralf
Ich verstehe eh nicht, warum #NextCloud so beliebt ist. Vielleicht weil es viele Features hat und daher in der Theorie auch sehr viele use Cases abbilden kann. In der Praxis läuft es halt wie von dir beschrieben.

Für File Sync gibt es im #Linux/Open-Source-Bereich viele gute Lösungen. Die Frage ist immer wer oder was auf der anderen Seite ist. #Syncthing #Seafile #Filen oder #Unison #rsync

Mit Web-Anwendungen habe ich weniger Erfahrung, aber Webmailer gibt es auch bessere.

Fliegender Synchronisationswechsel: Apple ersetzt rsync durch openrsync

Im Rahmen von macOS 15.4 wurden im Terminal Veränderungen vorgenommen. Wohl aufgrund von Lizenzänderungen verschwindet rsync.

heise online

¿Alguna distro tendrá intención de presentar una versión de GNU/Linux en la que el método predeterminado para copiar archivos sea con `rsync` en lugar de `cp`?

#GNU #Linux #rsync #cp

I didn’t know about openrsync until recently — been using rsync for years and somehow missed it!

openrsync is a lightweight rsync replacement developed by the OpenBSD team.
It focuses on security, simplicity, and a small footprint.

While not yet feature-complete compared to GNU rsync, it supports common
use cases like recursive copying, preserving permissions, and syncing over SSH.

Originally introduced in OpenBSD 6.5 (May 2019)
Authored by Kristaps Dzonsons
🔗 https://www.openrsync.org/manual.html

Thinking about writing a SlackBuild for it — I’ll give it a proper test first. 🙂

#openbsd #openrsync #slackbuild #rsync
OpenRsync: Manual Pages

OpenRsync Manual Pages

One thing not mentioned is that #OpenRsync's origin is the #OpenBSD project (at least as far as I am aware! Correct me if I am wrong :-) )!

https://derflounder.wordpress.com/2025/04/06/rsync-replaced-with-openrsync-on-macos-sequoia/

#openbsd #rsync #macos #openrsync

rsync replaced with openrsync on macOS Sequoia

On many Unix-based operating systems, rsync is a command line tool for transferring and synchronizing files on a computer, either between storage attached directly to the computer or between anothe…

Der Flounder
Thanks to a helpful suggestion from ryandesign, I modified the Portfile to stop passing the --prefix styled parameter using your suggestion for the oconfigure style and submitted a Pull Request to add openrsync to MacPorts here:

https://github.com/macports/macports-ports/pull/28096

GitHub Continuous Integration checks passed OK!

Here's hoping it gets merged without issues (I included some of the warnings from running %port test and maybe some other MacPorts contributors will have some helpful suggestions for clearing those up too).

Regardless, thank you for your efforts! I'm grateful to be able to downstream it in my own limited capacities. I realize, not as helpful as it might be if Apple were more collaborative, but who knows wtfh Tim "Apple" is up to besides removing headphone jacks and selling off stocks and racing devices out of India before tariffs impact their bottom line, or whatever it is that they do in Cupertino.

#OpenRSYNC #OpenBSD #MacPorts #rsync

CC: @ParadeGrotesque@mastodon.sdf.org @pertho@bsd.cafe @dexter@bsd.network
openrsync: New port by artkiver · Pull Request #28096 · macports/macports-ports

closes https://trac.macports.org/ticket/72311 Description Type(s) bugfix enhancement security fix Tested on macOS 15.4 24E248 arm64 Command Line Tools 16.3.0.0.1.1742442376 Verification H...

GitHub