Did you know that in recent Python-Blosc2 4.2.0 we released extremely efficient indexing engines whose data are stored in... (guess what) ...compressed state?
As a result, much larger tables can be indexed. Look at how this compares with the indexing engine in DuckDB (ART) in the plots below.
Also, the new indexing facility allows you to use whatever indexing engine (bucket, partial, opsi and full) that better adapts to your needs.
Enjoy data surfing!
Python-Blosc2 4.2.0 is out: get ready for data surfing! 🏄♂️
Have fun with the new CTable object:
* columnar storage format
* SOTA compression
* transparent in-memory and disk-based operation
* support for missing/null values
* multiple indexing engines for fast queries in large tables
* easy interoperability with parquet, arrow, and pandas-like ecosystems
Compress Better, Analyze Faster, Integrate Easier
Release notes: https://github.com/Blosc/python-blosc2/releases/tag/v4.2.0
Columnar Storage Is Normalization
https://buttondown.com/jaffray/archive/columnar-storage-is-normalization/
#HackerNews #ColumnarStorage #Normalization #DataManagement #TechTrends #HackerNews
Blosc Development Team
N-gated Hacker News
Hacker News