Ben HoustonZstdify: A Pure TypeScrpt Zstd Re-Implementation
Pure JavaScript/TypeScript zstd compression/decompression library. No native dependencies, works in Node.js and browsers.
@donmccurdy I did some benchmarks and this is 80% as fast as the zstddec package but 1/5 the size. Zstdify is 7.67 KiB bundled/compressed while zstddec is 40.66 KiB.
Arseny Kapoulkine@BenHouston3D @donmccurdy The claims *really* did not add up for me - zstd does not decompress this quickly, unless your data is just not compressible at all! - so I tried this, and on a non-zstd UASTC ktx2 file I'm getting ~100 MB/s decode performance. Native zstd decompresses at 1.7 GB/s. zstddec is about 1 GB/s. Your binary benchmarks are not using compressible data.
@BenHouston3D @donmccurdy also was this mostly written by an LLM? the commit sequence makes no sense for a human.
@zeux @donmccurdy yeah it is LLM. I don’t try to hide that fact: https://benhouston3d.com/blog/zstd-in-pure-javascript. My robust and feature filled hdr/exr library was created the same way: https://hdrify.benhouston3d.com/. And same with the forthcoming KTX library. I do think we have to judge based on the outcome, thus the extensive test coverage and benchmarking.
@BenHouston3D @donmccurdy Ok thanks for confirmation; this was missing from the repository page, so I wasn't sure until I looked at the history. I think your actively misleading benchmark numbers are a direct consequence of you not understanding the code that is in your repository, but of course you're free to choose the development approach that produces results you deem acceptable.
@zeux @donmccurdy you and me we’ve both been doing this a long time. (https://benhouston3d.com/about ). What matters is the end result - does it work? Is it fast? Is it reliable? Is it correct? That is what we are all aiming for. I view LLMs as a more high level compiler.
Rupert Rawnsley@BenHouston3D @zeux @donmccurdy perhaps version 1.0 has issues that will need to be resolved with time, but I’m grateful that we have something we didn’t have before: a pure JS version of a useful library. The more rigorous the testing and evaluating the better, but how it was created is incidental.
@rawnsley @BenHouston3D @donmccurdy https://github.com/101arrowz/fzstd exists for decompression and runs 1.8x-2.6x faster. The compressor here is quite weak (I see results that are nowhere close to zstd level 1 ratio-wise) and runs at ~2 MB/s.
If you need a smaller decoder, take zstddec, and:
- Enable EMMALLOC during building (saves 3 KB)
- Use wasmpack (meshoptimizer/tools/wasmpack.py) instead of Base64, saves 7 KB
The result is a 20 KB (post-gzip) file for non-stream version that's still as fast.
@zeux @rawnsley @donmccurdy you are correct of course. I am adding better real world benchmarks for both decompression, compression speed and compression ratio to prove your point during breaks from the kids.
@zeux @rawnsley @donmccurdy per our discussion I've added full benchmarks (compression ratio, decompress/compress speeds) to the README. I also did some optimization and make a new release.
https://github.com/bhouston/zstdify?tab=readme-ov-file#throughput-mbs
