Job SnijdersDec 6

I'd like to store one billion variable-length binary objects & get them by SHA256(obj) key. The median object size is 2 kilobyte. Low read/write volumes.

What I've tried so far: NFS with the hash's first few octets as nested directory names, it works but it is a bit slow and I also tried ZeroFS (also too slow).

Under considerations: DuckDB, RocksDB, BerkeleyDB, SQLite3, lmdb, something bespoke

Recommendations? Things/papers I should be reading?

Show threadAlexander Bochmann
@job This kinda sounds like an application for S3 storage? IDK if any self-hostable solution would provide better performance than what you've already tried though. Also adds an S3 API somewhere in between, but then that's common enough to usually have good language support.
Dec 6 at 10:29amWeb
Show threadJob SnijdersDec 6
@galaxis tiny objects like these are terrible for s3 style storage. S3 works great if your objects are around 10MB, kinda okayish at 1MB each, and terrible at 2KB