Mastodawn

I'd like to store one billion variable-length binary objects & get them by SHA256(obj) key. The median object size is 2 kilobyte. Low read/write volumes.

What I've tried so far: NFS with the hash's first few octets as nested directory names, it works but it is a bit slow and I also tried ZeroFS (also too slow).

Under considerations: DuckDB, RocksDB, BerkeleyDB, SQLite3, lmdb, something bespoke

Recommendations? Things/papers I should be reading?

@job define slow

@ydroneaud had trouble consistently reading faster than 2500 obj/sec on NFS; on ZeroFS it goes slower and I seem to be maxing out the one CPU core it uses for some task.

Show thread

Tim Chase Dec 6

@job

I suspect that a lot of that was file-system overhead of things like finding the file and checking permissions. A single-file solution will remove that overhead doing all that once when opened and then only using fseek(3)s to jump around within that file (see also: why games like Doom used .wad files, creating effectively a big read-only k/v store)

So of your original list, bdb & lmdb both fill about the same space and should be pretty interchangeable. I'd start there since they have a bit less overhead than sqlite, where sqlite (would be my fallback if bdb/lmdb don't meet your needs) offers more flexibility but involves things marshaling in and out of textual data, and query-parsing that could introduce performance issues.

Show thread

Tim Chase Dec 6

@job also, I don't know if licensing considerations matter.

bdb: APGL
lmdb: OpenLDAP public license (permissive)
sqlite: ostensibly public domain

Show thread

Tim Chase Dec 6

@job okay, apparently I can't shut up or formulate all my thoughts in a single reply 🙃

I'm unfamiliar with RocksDB but it sounds like a reasonable option too (local, performant, comfortably licensed)

Unless you see the need to access the data from multiple machines (like an actual MySQL/PostgreSQL type install), I'd bias away from DuckDB for this use case.