Karsten Schmidt#ReleaseSaturday — This week I've been working on extracting, refactoring & generalizing the minimal column store database I've been using for my personal knowledge/media management toolset, and I'm happy to share it with the world now:
This is an in-memory column store database with:
- Customizable column storage types with configurable min/max cardinality, support for optional and/or tuple-values, default values
- Support for custom column type implementations
- Optional dictionary encoding of column values (memory & filesize saving)
- Powerful extensible multi-term query engine with built-in OR/AND/NOR/NAND operators and predicate-based matchers (column, row, partial row). Queries can be pre-built and then executed as standard JS iterables
- Optional bitfield indexing for dramatic query acceleration (esp. for complex multi-term queries)
- Dynamic adding/removing of columns
- JSON serialization with optional RLE compression (in my PKM dataset with ~20k items, the RLE compressed version is only 29% of the normal JSON serialization)
I hope the readme and code examples give a decent overview for now... I've been using the overall system for a couple of years now, but this new packaged version is still marked as _alpha_. Everything's still being worked on.
Also, for those wondering what's the point of this all and why not using SQLite etc. — I find there're many use cases for a which a pure JSON-based approach is more than sufficient (without requiring extra tools and interfacing layers). The structure/storage model and the bitfield optimizations enable very fast query performance (compared to other JSON db's I've tried in the past)...
(Including all dependencies [only some other thi.ng packages], the entire DB package is ~6KB brotli'd, 19KB uncompressed...)
#ThingUmbrella #TypeScript #JavaScript #JSON #Database #QueryEngine #RLE #SmallWeb
