📦 Efficient file syncing for massive datasets? Forget classic rsync!

⚡ #BLAKE3 hashes instead of timestamps – hash 70GB in just ~20 seconds thanks to multithreading across all CPU cores

#DevOps #Linux #SysAdmin #OpenSource #FileSync

📊 Algorithm comparison for 70GB:
- BLAKE3: ~20 seconds
- MD5: ~1.5 minutes
- SHA256: ~4.5 minutes

🔍 The trick: Create a hash baseline, compare with `comm -13`, then use `rsync --files-from` to transfer only actually changed files

🧵 👇

🦀 Rust weekly log

This week:

📡 RustPulse
First OpenTelemetry + Jaeger integration in place (phase 1/3).
https://github.com/VinEckSie/rustpulse

🔐 Sealed in Rust
New chapter on cryptographic hash functions: SHA-2, BLAKE3, and beyond.
https://vinecksie.github.io/sealed-in-rust/02-core-primitives/02-02-crypto-hashes.html

#Rust #SystemsProgramming #Observability #OpenTelemetry #Jaeger #Cryptography #BLAKE3

Research topic of the day: Choosing a #hashfunction for 2030 and beyond - https://kerkour.com/fast-secure-hash-function-sha256-sha512-sha3-blake3

TLDR: #blake3 is cool & let's hope it picks up steam, #sha512 is the winner for integrating into current projects.

#research

Content addressable storage is simple and elegant right up to the point when you need to support multiple hash functions and/or digest sizes. After that, it becomes a much more difficult beast to tame.

But tame that fucking beast I will, goddamnit!

I'm really trying to learn from Git's difficulty in migrating to something other than sha1:
https://git-scm.com/docs/hash-function-transition/

One thing I'm doing differently is supporting multiple hash functions and digest sizes from the Git-go (hehe).

#Git #Rust #BLAKE3