alice pellerini don't see enough people with one of the best tool improvements i've ever made for reverse engineering, so i had to write a blog post about it!
d@nny disc@ mc²@simonomi i'm curious about the statement here:
the bitstream is much more colorful and chaotic because good compression algorithms output data that looks visually random.
not disputing its correctness but this is a very nontrivial claim described in visual terms that are somewhat removed from the discussion just above regarding prefix codes. i'm curious about how you arrived at this and in particular if your reverse engineering work motivated this intuitive description
@simonomi i think prefix trees are supposed to be pretty standard nowadays but i'm particularly under the impression that older formats employed hand-rolled heuristics and i'm wondering if this is what you're speaking to with the discussion of visual randomness here
@simonomi very much not a reverse engineering expert but have done some binary parsing and been frustrated with the expressiveness of languages for this task. scheme has some interesting work in this area but rust is my experience and could stand to do better
@hipsterelectron it mostly came from the intuition of having look at so many different types of binary. stuff with really high information density (compressed, executable, media, etc) tends to look very busy, because there's simply more information squished into fewer bytes
i've ended up writing my own whole fancy binary parsing system for my tool carbonizer. it's pretty specialized to the patterns used in the game files for Fossil Fighters, but i'm reasonably happy with it overall
@hipsterelectron i linked to some examples in the article, specifically https://github.com/simonomi/carbonizer/blob/6c311b6a2801576033cd42a8ba95461cee2ac6d1/Sources/Carbonizer/files/ff1/KPS.swift#L4-L25 and https://github.com/simonomi/carbonizer/blob/6c311b6a2801576033cd42a8ba95461cee2ac6d1/Sources/Carbonizer/files/ff1/DAL.swift#L20-L107 and https://github.com/simonomi/carbonizer/blob/6c311b6a2801576033cd42a8ba95461cee2ac6d1/Sources/Carbonizer/models/Texture.swift#L5-L28
@simonomi oooh swift!! this totally looks like the patterns i've arrived at. i do feel reflection is super helpful here it's the one point rust makes super difficult
@simonomi also glad to see this precise use of preconditions
@simonomi thx so much super cool stuff. i don't do too much hex reading but if i impl something similar for emacs i'll def link to this piece. great argument
@hipsterelectron i actually don't use any reflection at all >.< 100% macro magic
@simonomi i also can't rly say much about reflection but it's just a subject of some yearning. rly wish rust could have seen this take life https://soasis.org/posts/a-mirror-for-rust-a-plan-for-generic-compile-time-introspection-in-rust/
