Mastodawn

Your yearly remainder that @sozip exists (even if I've lost the credentials for the account. oops) and you can use it to offer a cloud native experience (and even a local one) to consume huuuge files compressed as ZIP. More at https://github.com/sozip/sozip-spec

Show thread

Chris Toney 3d ago

@EvenRouault @sozip I use it all the time and share with colleagues, mainly from gdalraster::addFilesInZip(). We used it yesterday on a set of huge CSVs of x, y, model value (I wasn't involved in deciding to create those CSVs but they're GDAL-readable and compressed now :)

https://firelab.github.io/gdalraster/reference/addFilesInZip.html

Create/append to a potentially Seek-Optimized ZIP file (SOZip) — addFilesInZip

addFilesInZip() will create new or open existing ZIP file, and add one or more compressed files potentially using the seek optimization extension. This function is basically a wrapper for CPLAddFileInZip() in the GDAL Common Portability Library, but optionally creates a new ZIP file first (with CPLCreateZip()). It provides a subset of functionality in the GDAL sozip command-line utility (https://gdal.org/en/stable/programs/sozip.html). Requires GDAL >= 3.7.

Show thread

Even Rouault 3d ago

@ctoney @sozip The SOZIP optimization will not bring anything to CSV-in-ZIP, since a CSV is consumed only by sequential reading, which regular ZIP is good at.
Quite funny that half of my posts about SOZIP are to explain how useless it is for various mentionned used cases.
Huge GeoPackage in ZIP is mostly the canonical use case.

Show thread

Chris Toney 3d ago

@EvenRouault @sozip Right, I don't expect it to add anything for the CSVs but addFilesInZip() defaults to using it (AUTO at least). I use it for most zip files generated by scripts since there doesn't seem to be any real need to disable it? The geospatial case is generally GPKG which the CSVs will probably become. They were generated in the first place by statisticians who don't code for geospatial.

Show thread

Even Rouault 3d ago

@ctoney @sozip No this doesn't hurt in a significant way to enable SOZIP optim, unless you're trying to achieve smallest zip size. SOZIP regular deflate stream flushing slightly degrades the compression rate, and there's obviously the size for the index ( 16 bytes per each 32 KB of uncompressed data, for default chunk size )

Show thread

Chris Toney

@EvenRouault @sozip yep, I've noticed they're slightly larger. The difference in overall compression ratio that I've seen is pretty negligible like ~0.01 range but probably varies a little.