Paul etnomailgaT
I spent a few minutes writing down some commentary about a 
Marcus Müller@paul oh that's very relevant to my interests – thanks for writing it down; this is something I'll really need to read in-depth.
@paul background: Have my own format in the making, which is TLV-alike in that it's a RIFF(64) dialect, and metadata tags like your frequency and location encoding need to be handled in a more general way (I'd like to encapsulate a sensible subset, maybe all of GR3's PMT, and the most useful sigMF annotations).
We might want to sit down at some point, after all this plays out, and think about a compressed container format. I picked @rygorous's brain on high-compression for complex numbers …
@paul … years ago (I remember he casually discussed that while he was playing a game live on stream, slightly legendary that he is), but I really need to think about something that a beefy x86_64 can throw at > 120 MS/s, and a less beefy cortex-A52 could still throw at 8 MS/s, maybe at different compression ratios. (And ideally, also something that'd lent itself to being done in an SDR's FPGA. Stretch goal.)
That's also the point where I need to build the encoder such that it can segment well …
That's also the point where I need to build the encoder such that it can segment well …
@paul … enough so that I can reliably have a RIFF chunk with seeking information ("you have reached sample 12345678") every 2^24 bytes or so.
@funkylab word!
I published it because I'm using it a lot locally, but if this winds up sparking some useful convos that turn into something a few of us make more broadly useful I'd be excited!
I've played a lot with compressing IQ with some mixed but interesting results. It's fantastic for generated IQ, rx'd IQ has been a lot more miss than hit. The float complex types compress best. Old versions of ARF had a bit flag on the byte order that would pack 12 bit IQ samples (pluto, ettus)
@funkylab I went deep on compression for a while, https://github.com/facebook/zstd/blob/dev/contrib/seekable_format/zstd_seekable_compression_format.md was my favorite approach to what you were looking for there with seeking onto compressed files
@funkylab but that being said I just yanked all the compression from the spec because I wasn't using it, and in a bunch of cases wound up increasing or negligible decreases for a lot of extra code branches in the decoder
@paul re: recorded vs generated IQ: Well, true entropy :)
compression methodoligically, for float it might make sense to first simply go for conversion from I,Q to |z|, arg(z) and going block float on |z| and very simple linear predictive + adaptive error representation on arg(z); or maybe just LPC for the complex values as they are.