AndresFreundTecI accidentally found a security issue while benchmarking postgres changes.
If you run debian testing, unstable or some other more "bleeding edge" distribution, I strongly recommend upgrading ASAP.
I was doing some micro-benchmarking at the time, needed to quiesce the system to reduce noise. Saw sshd processes were using a surprising amount of CPU, despite immediately failing because of wrong usernames etc. Profiled sshd, showing lots of cpu time in liblzma, with perf unable to attribute it to a symbol. Got suspicious. Recalled that I had seen an odd valgrind complaint in automated testing of postgres, a few weeks earlier, after package updates.
Really required a lot of coincidences.
Glyph@AndresFreundTec I feel both confident and also kind of queasy when saying this: it seems extremely likely that this is not the first time something like this has happened, it's just the first time we have been lucky enough to notice.
LisPi@glyph @AndresFreundTec That is true.
Binary artifacts have no business existing in Free Software (or near-binary considering how auditable pre-generated config scripts end-up being). The way it was compromised in this case is almost certain to have happened before and reminds me of the SourceForge malware debacle (so arguably that's another famous example of it happening before).
I"m not sure if many other projects do like Guix and record the checksum of the whole repository so as to ensure reproducibility purely from source.
Binary artifacts have no business existing in Free Software (or near-binary considering how auditable pre-generated config scripts end-up being). The way it was compromised in this case is almost certain to have happened before and reminds me of the SourceForge malware debacle (so arguably that's another famous example of it happening before).
I"m not sure if many other projects do like Guix and record the checksum of the whole repository so as to ensure reproducibility purely from source.
@lispi314 @AndresFreundTec In general this is reasonable, but this there are some clear exceptions for test vectors in cryptographic libraries and compression libraries (which this was).
@glyph @AndresFreundTec In this case the actual malicious vector was the near-binary injected code in the practical binary of the unaudited autotools vomit (always autoreconf) which was then bundled in the actual binary artifact that was the compromised tarballs.
None should have ever been part of the project.
As for the test files, I still think that having a hex dump with comments explaining what flaws particular parts test would be desirable in a lot of cases.
None should have ever been part of the project.
As for the test files, I still think that having a hex dump with comments explaining what flaws particular parts test would be desirable in a lot of cases.
l@lispi314
#guix still takes tarballs with `configure` scripts "precompiled":
https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/compression.scm?id=4b23fd7adbddc1bc18b209912c0f3ef369da2f24#n499
Same for #nix, they take distribution tarball with autoreconf-generated files.
Using autoreconf and not trusting distribution tarballs is apparently not as easy as pointing to git repo (which is now down) and using autoreconf because autoreconf itself has to be built before xz during bootstrapping then.
@link2xt @glyph @AndresFreundTec What you're pointing at is a maintainer screw-up in not using best practices.
Why aren't best practices made the only possible practice? Because it would make it impossible to support packages whose source-code is never distributed in any other form. So unfortunately for that the best that can be done is improve maintainer knowledge and documentation.
Why aren't best practices made the only possible practice? Because it would make it impossible to support packages whose source-code is never distributed in any other form. So unfortunately for that the best that can be done is improve maintainer knowledge and documentation.
🍒🌳 Hartmut Goebel> I"m not sure if many other projects do like Guix and record the checksum of the whole repository so as to ensure reproducibility purely from source.
If the packager chooses to use the official tarball as "the source", validating the checksum would not have helped. :-( Also whether it's always possible to run running autoreconf depends on the content of the tarball.
Which brings me to the (preliminary) conclusion that we'd better use repos as source of trust
@lispi314 @AndresFreundTec @glyph
@kirschwipfel @glyph @AndresFreundTec > If the packager chooses to use the official tarball as "the source", validating the checksum would not have helped. :-(
Unfortunately, yeah.
> Also whether it's always possible to run running autoreconf depends on the content of the tarball.
Of course if it isn't a C project then it probably isn't. If it is such a project, then one should have such tooling installed.
> Which brings me to the (preliminary) conclusion that we'd better use repos as source of trust
That is more sensible generally, as the history of an object and its belonging to a project is a reified (and verifiable) relationship under code versioning sytems, unlike arbitrary buckets of files.
Unfortunately, yeah.
> Also whether it's always possible to run running autoreconf depends on the content of the tarball.
Of course if it isn't a C project then it probably isn't. If it is such a project, then one should have such tooling installed.
> Which brings me to the (preliminary) conclusion that we'd better use repos as source of trust
That is more sensible generally, as the history of an object and its belonging to a project is a reified (and verifiable) relationship under code versioning sytems, unlike arbitrary buckets of files.
Marcelo Magallon@lispi314 @AndresFreundTec @glyph we have been using docker images for what? 10+ years now, and "everyone" seems OK with the fact that the vast majority of them are not reproducible because the corresponding dockerfile is not generally available.
@mem @glyph @AndresFreundTec That is incidentally one of the *many* reasons I don't use Docker or similar container infrastructure.
It is... unfortunately necessary to accept that some things aren't reproducible because the tooling fails to do it (Common Lisp implementations tend not to dump reproducible images, there are patches pending for a few of them but...). That being said, the reasonable answer in that scenario is to instead ensure that all of the inputs are identical and rebuild everything from source, as Guix does (or for Common Lisp to just run everything from source, as one should do anyway, compilation & caching, if any, should be transparently handled by the implementation).
And indeed, you *can* use this procedure with Guix to generate Docker images.
So for those images where the Dockerfile necessary to rebuild it from scratch isn't available? They're garbage and unusable.
It is... unfortunately necessary to accept that some things aren't reproducible because the tooling fails to do it (Common Lisp implementations tend not to dump reproducible images, there are patches pending for a few of them but...). That being said, the reasonable answer in that scenario is to instead ensure that all of the inputs are identical and rebuild everything from source, as Guix does (or for Common Lisp to just run everything from source, as one should do anyway, compilation & caching, if any, should be transparently handled by the implementation).
And indeed, you *can* use this procedure with Guix to generate Docker images.
So for those images where the Dockerfile necessary to rebuild it from scratch isn't available? They're garbage and unusable.