Adam H. Sparks 

RE: https://mas.to/@CRANberriesFeed/116606326823968998

We'll try for a third time. This has been archived 2X already. This time I just removed all caching capability and basically rewrote the whole package for a major release that breaks backward capability but simplifies using it. Now it only has two single functions, one creates a data.table the other a {terra} SpatRaster of the CRU CL 2.0 climate data either by download or from local files.
https://docs.ropensci.org/getCRUCLdata/

#RStats #ClimateData

May 20 at 10:19amWeb
Show threadMichael SumnerMay 20

@adamhsparks what's the download format, how big is it in total and what's the update schedule?

Just, maybe time to reformat it to an updating new store rather than downloading/caching? It's a hard boundary to cross but really nice when the new store is fast and lithe

We did this with GHRSST ~8000 daily 500Mb COGs, it's also a good collective topic going fwd. We copy to source.coop, S3 or bare url. No egress cost..the lambda $ about $10 a month, but it could run on hpc/nectar.

Show threadAdam H. Sparks May 20
@mdsumner .gz files. It’s not changing, static files. Just large enough to not want to download over and over for the full set but it is doable.
Show threadAdam H. Sparks May 20
@mdsumner oh, and the files are just text. Not geospatial
Show threadMichael SumnerMay 20
@adamhsparks oh even better 😅 xyz or .asc or something else? (I will.look at some point)
Show threadAdam H. Sparks May 20
@mdsumner no. Just plain text files compressed into a .gz file. 😅
Show threadMichael SumnerMay 20
@adamhsparks 🙏 it's not an easy one. Been trying heaps of stuff with mixed success...I can manage Zarr now but will I commit to creating them ... maybe not. If you do reformat source.coop is a great home and can be bundled into virtual.zarr - that's the dream, table of COGs or nc, plus the Zarr for the other world on the same bytes
Show threadOlivier LeroyMay 20

@mdsumner @adamhsparks

I do that for private packages and did that for a public package: download everything, reformat/clean a bit of the stupidity, use a better format/compression, wrap in duckdb (for spatial I am excited about gdal vrt but never have a use case).

that being said I do not need to bother with CRAN and the data is not updated frequently so a random shell script do the job of all the above.

Show threadOlivier LeroyMay 20

@mdsumner @adamhsparks

(and use objects storage)