OblomovFeb 2, 2023

The Twitter API collapse is reigniting my desire to set up a bot to “compete” with @jen's @GIFEarth, based on #EUMETSAT's MSG #SEVIRI imagery.

Problem is: if I were to do full-resolution, complete, full disk GIFs, one per day, we'd be taking about sequences of 96 (1 every 15 minutes) 3712×3712 32-bit RGBx (10 bits per channel) images. I doubt most instances would appreciate something like that, even if I were to support them with donations.

Show threadOblomovFeb 2, 2023

Given the size of things, I actually think that a video might be more appropriate than a GIF, but I'm not sure if there is a PeerTube equivalent of bots in space.

To get an idea, this is the level of the stuff we're taking about: https://youtu.be/fP41uKRM7kM

Of course, the saner idea would be to downscale/downgrade the images when building the animation —which is probably a good idea regardless, to avoid infringing EUMETSAT's intellectual property and risking the bot being taken offline, or worse.

The Earth from space

YouTube
Show threadOblomovFeb 2, 2023
I've looked a bit more into this, at least as far as Mastodon's media support goes. My understanding is that the closest thing would be a GIFV (aka MP4 with no audio channel). So the first thing to do would be check how big such a file is on a typical “day on Earth” from SEVIRI, and check if it fits against Mastodon's limits or some kind of rearranging is needed (again, this is unless somebody knows of a bot-friendly PeerTube instance).
Show threadOblomovFeb 2, 2023

The other obvious question is the framerate of the resulting video/animation. Real-time is 1 frame every 15 minutes, which is around 0.00(1) FPS, obviously too slow. At 1FPS we're talking about squeezing a full day (24 hours = 96 images) into 96 seconds ok 1:36 minutes. 10FPS (minimum to get an animation, perceptually) means that the whole animation would last 9.6 seconds.

Decisions, decisions.

Show threadOblomovFeb 4, 2023
OK I verified that a full-resolution full-disk full-day MP4 produced using ffmpeg at default settings produces a ~13MB file. Honestly I expected worse. Also 10FPS is an acceptable compromise if a bit jerky. Of course that's still for a video that is 3712×3712 which, I believe, is a resolution that could only be displayed full screen in all is glory on 8K monitors, so it's definitely a waste for a GIFV-producing #bot on the Fediverse. The next question is: how much downscaling?
Show threadOblomovFeb 4, 2023

Actually, I just realized that this could be an excellent use case for #JpegXL, I'll have to test how big the resulting file is. Of course it still wouldn't work on the Fediverse, or even the web at large, especially considering Google's petty decision to pull support for it from Chrome.

JFC Google is really trying its best to be hated. And #Mozilla doesn't seem to have the spine to stand against them anymore. But I digress.

Show threadOblomovFeb 4, 2023
Damn, imagemagick on the machine that would handle this is too old.
Show threadOblomovFeb 4, 2023

Oh wow, no version of Ubuntu ships ImageMagick 7, so even upgrading the system would be useless.

Well, since this isn't for production anyway, I can just pipe the processed images to my machine here … this is going to be fun (not).

Show threadOblomovFeb 4, 2023

While I wait for the image processing to finish, the question remains: is there a #bot-friendly #PeerTube instance (to wit, something akin to https://botsin.space) or is lowering the image size and keep the output as an (animated) image format to put on Mastodon still my best bet?

#askFedi

botsin.space

Show threadOblomov
(And yes, it was stupid of me to not keep the intermediate processed images instead of piping the output directly to ffmpeg, given how much it takes on this machine to convert the raw satellite data to standard image formats. Also I should improve this process.)
Feb 4, 2023 at 7:37amWeb
Show threadOblomovFeb 4, 2023
If you wonder why the process takes a long time: there's a lot of data even in a single scan. #SEVIRI samples are 10 bits per channel, and in the “native” #SEVIRI format data is stored in packed format, so that each channel scanline is 4640 bytes long (3712*10/8). The first step is to unpack the firs three channels, and since apparently ImageMagick doesn't support RGBx data formats (3 10-bit channels packed into 32 bits per pixel), all data has to be promoted to 16-bit.
Show threadOblomovFeb 4, 2023

This of course increases memory/storage requirements (my NAT file parser outputs an uncompressed MIFF) to ~83MB per image (these would be just 55MB in RGBx format, if my math is correct). With 96 scans per day, that's ~8GB per day. You can easily see why the process takes _some_ time.

The MIFF sequence can be compressed to ~1.5GB with xz -9, which is excellent compression ratio, but still a lot of data to move around and more processing time to pack/unpack.

Show threadOblomovFeb 4, 2023
You can see now why I was impressed with MP4 shrinking that thing down to only around 13MB. It's still too much for the bot, but still an incredible reduction in size. You can guess why I'm curious to see how well JPEG XL can handle the thing.
Show threadOblomovFeb 4, 2023
BTW, fun fact, SEVIRI doesn't actually have channels in the Blue and Green range: the three shortest wavelength channels are nominal 0.6µm, 0.8µm and 1.6µm; EUMETSAT classified the first two as VIS (visible) and the third as NIR (Near Infra-Red), but the technically the only visible band is the first one, in the orange spectrum, and 0.8 is already considered Near Infra-Red, while 1.6 would typically be considered SWIR (Short-Wave Infra-Red).
Show threadOblomovFeb 4, 2023

(That being said, classification of Infra-Red bands is a bit subjective; you can read more about it on Wikipedia <https://en.wikipedia.org/wiki/Infrared>.)

Point is, the images my bot I'm working on will produce are in fake colors, with everything shifted down (I'm mapping the 0.6 from Orange to Blue, 0.8 from NIR to Green, and 1.6 from SWIR to Red). Still, as you can see from the video posted above, the results are quite … credible.

Infrared - Wikipedia

Show threadOblomovFeb 4, 2023
Things would be (will be?) different when the MTG (Meteosat Third Generation) satellite data becomes available: the onboard FCI (Flexible Combined Imager) has 5 “nominally” visible bands (VIS), with center wavelengths of 0.444µm, 0.51µm, 0.64µm, 0.865µm and 0.914µm. The first three correspond approximately to violet/blue, cyan/green and orange/red, so they could be used for a “truer” RGB image.
Show threadOblomovFeb 4, 2023
OTOH, these FCI have a nominal nadir resolution of 1km (compare with SEVIRI ones with a 3km nadir pixel width), so we can expect an order of magnitude more pixels … think about my complains about processing the data and multiply that by 10. It's going to be fun.
Show threadOblomovFeb 4, 2023
Anyway, in the meantime I've managed to do some tests with #JPEGXL, and the results are quite interesting. The entire 96-frame thing can be packed into a ~865MB #JXL file, which is a pretty amazing 1:10 ratio over the original data, considering the format has no inter-frame prediction. The size drops to 75MB at the default (lossy) compression setting, with no perceptual differences.
Show threadOblomovFeb 5, 2023
Aside from size, one of the things that was bothering me with these images was the time it took to actually get the data out from the original format and into something that could be manipulated by ImageMagick.
The code I wrote to parse the native format was correct, but wasn't exactly optimized. Luckily, the process itself is relatively simple, but I was spending an inordinate amount of time into conversion from the original packed representation into something more amenable to manipulation.
Show threadOblomovFeb 5, 2023
As I mentioned, the original data has a 10-bit depth per channel, but in contrast to the RGBx formats used in graphics, the data is stored in BIL (Band Interleave by Line) encoding: the first scanline is stored for each band, followed by the second scanline, etc. Within each band scanline, data is packed with 5 bytes encoding 4 samples (5*8 = 40 = 4*10).
While reading, I first de-interleave the data, so that each band is its own array. The next step is the unpacking.
Show threadOblomovFeb 5, 2023
Now, another interesting tidbit: not only data is packed: it's also _flipped_ compared to how we're used to visualize the planet, because storage follows the direction of the SEVIRI spin & scan directions, so during unpacking I also need to change the sample order.
Show threadOblomovFeb 5, 2023
Now, my original code did the “flip and unpack” sample by sample, in a way that wasn't easy to parallelize (due to the packing). So the first thing to do was convert the code to be scanline-oriented, which was relatively easy to do because each scanline is an integer multiple of both the packed representation and the expanded representation (3712 samples span 4640 bytes). Parallelization was done the lazy way (OpenMP with a simple #pragma omp parallel for) as it gives good enough gains.
Show threadOblomovFeb 5, 2023
Another minor optimization came from unrolling the unpacking loop, i.e. the loop that turns 5 bytes into 4 samples doing a lot of shifting and masking of byte pairs. The gains due to this were … impressive, honestly, although not entirely unexpected.
The final bottleneck was, a bit surprisingly, in the _output_ phase, and despite the improvements I've made, it remains a bit annoying for me, but I think this is mostly due to my ignorance of the way #ImageMagick works.
Show threadOblomovFeb 5, 2023
So, my problem with #ImageMagick. The script I'm using to feed the unpacked, reordered channel data to it was originally intended to simply take any selection of the channel and output them in so-called BSQ (Band SeQuential) encoding, where each band is output as a separate “plane”, scanline by scanline, followed by the next band: the multispectral equivalent to color separation. This is very fast because after flipping and unpacking each band, can just write that out as is.
Show threadOblomovFeb 5, 2023
Writing out a full array as-is (optionally after some plain text header holding the necessary metadata) is very fast. Having to manipulate the data before writing is not. An example of such manipulation would be to upscale each 10-bit sample to cover the full 16-bit range (multiply by 65535, divide by 1023), so that #ImageMagick could interpret the data correctly out of the box.
Show threadOblomovFeb 5, 2023
Another example would be having to output data in BIP encoding (Band Interleaved by Pixel), which is the standard form used for images (e.g. Red, Green, Blue values for the first pixel, followed by the RGB of the second pixel, etc). This takes time (and memory) because data needs to be rearranged.
To avoid wasting this time (and memory) I would need a way to tell #ImageMagick the format I would prefer using. And I haven't found how to do this.
Show threadOblomovFeb 5, 2023
I'm passing image information to #ImageMagick by outputting a MIFF header
https://imagemagick.org/script/miff.php
before the binary data. Problem is:
1. I haven't found a way to pass the “quantum scale” or “quantum range” information through the header (i.e. the fact that the maximum value is 1023 and not 65535)
2. I haven't found a way to tell it that the upcoming binary data would be band sequential, which would allow me to not interleave it in BIP form.
Maybe I can #askFedi if this can be done, and how?
ImageMagick

Create, Edit, Compose, or Convert Digital Images

ImageMagick