Kicked off a HPC job, 64 CPUs, converting global 11 variable netcdf to single var cog, from and to the local object storage.

There's 120K files to be created, and 20 minutes or so in it's up to 37K done, each is 2880x1440, Int32 (zstd, predictor 2)

Blows my mind how fast stuff is when obstacles and complications are removed
...thanks to #GDAL
and #rstats {gdalraster}, and {mirai} via {purrr}

The daily (since 1993) NetCDFs *change resolution* in 2024, so I'm standardizing the entire set so I can avoid on the fly warp (even though that's fine for R users), I also have something sane to create xarray/Zarr from
This is easier-to-me, than tooling up around the producer's Zarr in Europe,
... at least for now it's easier, and whoa I was warming up to the task didn't expect to finish it this evening

I've been working on the required expertise on the HPC for over a year, it's overall been very positive and enabling and consistent with other cloud computing involving object storage 👌

Highly recommend getting a project going on #Pawsey

hello world please welcome 85000 new tiffs born in the last 40min