Do you use #neuropixels or #highdensity probes? Are your recordings filling up your hard drives?
We got you covered!
In the first preprint from
@AllenInstitute
for Neural Dynamics, we looked at ways to reduce the footprint of #ephys data.
https://www.biorxiv.org/content/10.1101/2023.05.22.541700v2
(1/n)
We developed a framework based on @zarr_dev to benchmark lossless and lossy compression of #Neuropixels and similar data. The benchmark datasets included NP1 and NP2 recordings, available on Registry of Open Data on
@AWS
https://registry.opendata.aws/allen-nd-ephys-compression/
(2/n)
We started with #LosslessCompression. Across a range of general-purpose (GP) compressors, we found that #Zstandard with
@Blosc2 achieves the best compromise between compression ratio and decompression speed!
NP1: compressed size ~36%
NP2: compressed size ~52%
(3/n)
We then investigated two #LossyCompression strategies: bit truncation and WavPack Hybrid mode. Lossy compression can dramatically boost compression performance, but we must first assess how it affects downstream analysis (i.e., spike sorting).
(5/n)
Using simulated data with known ground truth spike times, we used #Kilosort 2.5 to evaluate spike sorting performance. WavPack Hybrid does not affect spike sorting accuracy, even at maximum compression levels (~14% file size).
(6/n)
We repeated spike sorting on experimental data, this time counting the number of units passing or failing quality control (QC). Again, we observed minimal changes in the results when using WavPack Hybrid.
(7/n)
Waveform shapes are also important for downstream analysis, e.g., cell-type classification. On simulated data, we found that WavPack Hybrid nicely preserves three commonly used waveform features.
(8/n)
WavPack Hybrid is promising, but we observed subtle differences in spike trains before and after compression. We need better methods for comparing spike sorting results to make sure we’re not losing any critical info. Until then, we’ll be using lossless compression.
(9/n)
Do you want to apply compression right away? All compression options can be readily deployed with a few lines using
@spikeinterface
, so you can easily try them out of your own data! This will also make it easy to benchmark new compression methods as they become available.
(10/n)
At @AllenInstitute
for Neural Dynamics we value fairness and reproducibility in science. All figures of the manuscript can be reproduced with
@codeocean:
https://codeocean.com/capsule/3822095/tree/v1
(11/n)
Finally, kudos to all co-authors!
Olivier Winter, David Bryant, David Feng, @svoboda314 and Josh Siegle, and thanks to
@alleninstitute for sponsoring this work!
(12/n)
Alessio Buccino
Chris Rodgers