Davide Madrisan 🚴♂️🎵Most likely the last update in the year 2024 of my #HadCRUT5 global temperature data plot.
#GitHub project @ https://github.com/madrisan/HadCRUT5
anlomedad#HelloWorld Underwhelming result
The picture is but a copy paste of this code https://stackoverflow.com/questions/61527498/how-to-insert-txt-data-into-netcdf-in-python
Amending my data to fit the example took 2 minutes.
But getting rid of the numerous error messages beforehand took 2 full days of fiddling with #python and #NetCDF4 versions on my Mac .
Glad I documented the process from the very beginning so I can backtrace my install configuration in case I run into more errors. Which I probably will, having had to install old versions of almost everything as one of the workarounds.
Incredible, the problems one needs to solve before you can <start> using a new language like python.
What do we see? A screenshot of a Panoply map plot with Germany in the center. And coloured squares which stand for the difference in January maximum precipitation 1960-1990 versus 2000-2023, averaged on 1°x1° fields.
Where it is blue – in the western half of Germany – January had higher maximum precipitation in mm from 1960-1990 compared to 2000-2023.
And in the East, the January maxima used to be lower than they are nowadays.
I won't explain more right now regarding methods and data.
Am just tooting this as a hello-world-sort-of-thing. With a big sigh of relief that I made it to this project milestone, at all. Wanted to give up more than once in the past 48 hours...phew!
gautehhidefix (https://github.com/gauteh/hidefix) can now be used as an #xarray backend: more than 6x faster on a 8 CPU machine!
This reads a full variable of about 370mb.
Yann Büchau 
@roadskater Compressed #CSV is (in my case) actually smaller than #NetCDF4. Trade IO speed for size and data approachability.
@thfriedrich I benchmarked the different compression algorithms in #HDF5 once if you're interested: https://gitlab.com/-/snippets/2043808
With the metric I use there (distance to optimum 'fast and small'), blosc:lz4 is the best compromise.
I still hit a wall with #HDF5 at some point though, I guess the compression prevented something from being done, I don't remember...
Also, #NetCDF4 is a subtype of #HDF5 so you'll feel familiar.
Thomas Friedrich@nobodyinperson thanks for the insight. 🙂 I’ll have a look at #MQTT and #NetCDF4 which are both new to me. Working in science, so far I used mostly #hdf5 for larger data, which I think has a + for #openData & #openScience since there’s easy interfaces for most programming languages. Compression is decent I believe.
@thfriedrich Also, I've had problems with #NetCDF4 bindings not being thread-safe, so I couldn't parallelize operations very well. With compressed CSV, just throw threads (or processes) onto the problem. Files just work, no weird library in between 🙂
@thfriedrich Multidimensional would definitely be #NetCDF4, but if you are on a multidimensional scale with model-size outputs, then that's a different task than handling measurement device timeseries (what I was referring to).