d@nny disc@ mc²#[derive(Debug, Snafu)]
#[snafu(display("needed {}, remaining {}", needed, remaining))]
pub struct RemainingError {
pub needed: usize,
pub remaining: usize,
backtrace: Option<Backtrace>,
}
wrote a good amount of code like this for the zstd impl that stopped when i realized zstd was weird and bad
the idea is so basic it's literally:
.tar in two like a magicianthis is my enum for their block type header
#[inline(always)]
fn block_type(&self) -> BlockType {
match (self.0 & 0b110) >> 1 {
0 => BlockType::RawBlock,
1 => BlockType::RLEBlock,
2 => BlockType::CompressedBlock,
3 => BlockType::Reserved,
_ => unreachable!("block type is limited to two bits"),
}
}
two whole bits:
the problem with prefix codes as i understand it now is that they require at least one bit to signal an output symbol (which is typically a byte--i also think this is incredibly important to parameterize). however prefix codes seem to be ideal in every other way (collet mentions performance)
and for extremely biased distributions i.e. ones that would benefit from less than one bit for some highly frequent output symbols, we have (actually a whole field here, but i like) duda's tANS. it's cute. it can be sized to a precise memory region. you follow the path of the blocks wherever they lead
i could cite yann collet on this here but i won't cause he doesn't. and i cited everyone here https://codeberg.org/cosmicexplorer/corporeal/src/branch/main/literature/README.md
the compressed block is where it becomes clear zstd is Just Fucking Deflate Again: https://en.wikipedia.org/wiki/Deflate
BTYPE (next two bits): Block type
00: No compression (sometimes called stored). Any bits up to the next byte boundary are ignored. The rest of the block consists of 16-bit LEN, 16-bit NLEN (one's complement of LEN), and LEN bytes of uncompressed data, i.e. up to 65,535 (216 − 1) bytes. Useful for incompressible data (e.g. high-entropy, random, or already compressed), adding minimal overhead (i.e. ~5 bytes per block).01: A static Huffman compressed block, using a pre-agreed Huffman tree defined in the RFC.10: A dynamic Huffman compressed block, complete with the Huffman table supplied.11: Reserved (error).shameless shit. they even kept the reserved block
more from the DEFLATE page
Most compressible data will end up being encoded using method 10, the dynamic Huffman encoding, which produces an optimized Huffman tree customized for each block of data individually. Instructions to generate the necessary Huffman tree immediately follow the block header. The static Huffman option is used for short messages, where the fixed saving gained by omitting the tree outweighs the percentage compression loss due to using a non-optimal (thus, not technically Huffman) code.
one thing i haven't seen ANYONE fucking talk about is generating compression blocks that understand the boundaries of file contents
https://www.7-zip.org/faq.html hmmmmm
Can I use the source code of 7-Zip in a commercial application?
he then provides maybe the most concise description of the LGPL requirements i've ever seen. i am team 7zip for life as of this moment
providing tarballs at your own site is cool but why you would also provide them from github and sourceforge? well, github has the lock-in effect.
very very funny repo though https://github.com/ip7z/7zip
some kind of orange shapetarlz archiver does https://www.nongnu.org/lzip/manual/tarlz_manual.html#Amendments-to-pax-format@astraleureka https://datatracker.ietf.org/doc/html/rfc9659
so in addition to saying "oh yeah skippable frames are for watermarking you can just remove them"
there's also these fucking zero-length fields everywhere
and this is a format that works extremely hard to save every goddamn bit in its fucked up headers
they tell you several times BE CAREFUL!!! SOMEONE MIGHT TELL YOU TO USE A BIG WINDOW! which is an indicator that the window size hint is either monopolistic orrrrrrrrrrrrrrr
so then the new update 9659 its ENTIRE fucking deal is saying BE CAREFUL!!!!! OF BIG WINDOWS!!!!!!
Deployments of Zstandard, or "zstd", can use different window sizes to limit memory usage during compression and decompression. Some browsers and user agents limit window sizes to mitigate memory usage concerns, thereby causing interoperability issues. This document updates the window size limit in RFC 8878 from a recommendation to a requirement in HTTP contexts.
the vessel of morganna
Timo the timo