comexMaybe I should reconsider my policy of running Debian unstable, on both desktops and servers…
Andrew Zonenberg@comex Random breakage is my biggest reason for running stable. But yes, this is a side benefit.
Daniel Micay@comex Having updates delayed for years provides strong assurance you'll have lots of serious security vulnerabilities unfixed. Most security relevant fixes don't actually receive CVE assignments and therefore don't get backported. Delaying changes for a a month and backporting patches for serious issues might make sense but delaying for years is not going to bring you security. It's also possible that a tiny portion of the huge stream of security vulnerabilities being fixed were intentional.
@comex The person who did this had become the maintainer of the project. The issue was only caught at all because it caused performance issues someone looked into. It's entirely possible and perhaps even likely that there are similar, much longer term successful attacks similar to this one.
https://grapheneos.social/@AndresFreun[email protected]/112180083704855572
Since the attack was successful against Debian unstable, it's entirely possible it was used to compromise a lot of open source developers / projects and through that other projects.
AndresFreundTec (@[email protected])
I accidentally found a security issue while benchmarking postgres changes. If you run debian testing, unstable or some other more "bleeding edge" distribution, I strongly recommend upgrading ASAP. https://www.openwall.com/lists/oss-security/2024/03/29/4
@comex Many Debian developers likely use Debian unstable. The payload specifically targeted Debian. It relied on Debian adding liblzma as an indirect dependency of sshd through classic Debian downstream patching adding libsystemd as a dependency of sshd which pulled in liblzma. This avoided them needed to do more easily detected cross-process nonsense. If they hadn't caused a performance issue with their code, there's a high chance they would have gotten away with this for much longer.
@comex They could have targeted Arch Linux, NixOS, etc. too but they didn't bother making their payload work on them. They could have done something in systemd itself instead of targeting sshd indirectly. It didn't work on Arch Linux because sshd isn't patched to add notify support and it also seems the attacker hard-wired certain bits for Debian and Fedora. There was also a sockpuppet campaign to get it included into Debian and Fedora ASAP along with the maintainer pushing for it and helping.
@DanielMicay I know.
@DanielMicay …so to be clear, I’m not blaming Debian, just reconsidering stable versus unstable. And yes, lacking upstream fixes is a real issue. But on days like today it sure would have been nice to be hiding behind others rather than bearing the full force of the winds of regressions. Heck, days like that are not uncommon; it’s just that the regressions usually aren’t security-related…
@comex My point is largely that Debian stable users were only shielded from the first phase of this attack. The attack is also quite probably still ongoing since many people will have updated to the backdoored sshd but not yet updated to patch out the backdoor. It's very easy to scan every port across the whole IPv4 space and take advantage of this against every server with it exposed. I don't think it's unlikely that many servers were compromised and that there's another phase happening.
@comex It's quite scary for open source supply chain security as a whole right now because Debian sid/unstable are very widely used across the internet. There are surely many sshd instances exposed both on the default port and other ports which could have been easily detected by the attacker(s). My concern is mainly what happens next. What did they do with the attack after they succeeded? They could have started popping servers with sshd as soon as it shipped in Debian sid.
@DanielMicay Maybe. But actually using the backdoor does substantially increase the risk of detection, especially if you modify files or otherwise try to achieve persistence or backdoor other software. I suspect an attacker patient enough to set up a persona over a span of years – who probably did not expect the backdoor to be discovered so soon – would have avoided using it at all until it reached stable. But that’s just a guess. And of course, now that it’s public all bets are off.
@comex They also had a bunch of clear sockpuppet personas. At least one of those personas was reused a bunch of times to advocate for including their patches and then to advocate for the project adding them as a maintainer. They had commits in other projects too.
They were contributing to other projects beyond xz from the Jia Tan persona and it's quite possible they did more from other personas.
We don't really know the extent of their attacks. They may have had past successes already.
@DanielMicay Well, that’s largely orthogonal to the issue of stable distros lacking upstream vulnerability fixed, unless a backdoor is removed by accident without people realizing it’s a backdoor, which is possible I suppose.
@comex My main point is that if an attacker can get a backdoor into Debian unstable, they can almost certainly get it into Debian stable from there. Many Debian developers likely use Debian unstable on machines they use to develop Debian. It's not clear what the ultimate goal was of this attack but I would expect whoever is behind it took advantage of their backdoor being shipped even if it didn't reach their ultimate targets directly. They were playing the long game:
https://boehs.org/node/everything-i-know-about-the-xz-backdoor
@comex I think pretty much anyone with a lot of awareness of software security knew that this was possible. It's still pretty scary seeing a successful attack where they nearly got away with it for likely much longer if only they had optimized their code. It's scarier seeing it happen with core infrastructure, and we all know a whole lot of these projects are barely maintained if at all. As an example, every major distro has unzip/zip last updated in 2009 with dozens of downstream CVE patches.
@DanielMicay I have no illusion that stable will save me from every instance of a sophisticated backdoor like this, but to the extent a hypothetical backdoor exists in both stable and unstable, it wouldn’t affect the choice between them.
@DanielMicay I do wonder how lucky we got. It could have easily lasted longer than a month; but would it have lasted years? It was discovered by chance, but given how many different engineers there are doing different weird things, how long until someone should be expected to find it by chance? A better-designed backdoor would have made it (much?) harder. On the other hand, this is a core component present on systems with relatively small userlands.
@comex It's lucky to have discovered it so soon but we don't know how much they've compromised with it. It's very easy to mass compromise hosts via an easily exploitable vulnerability in sshd, nginx, etc. It's a concern we've had about having sshd exposed to the internet and is part of why we have our build infrastructure local without sshd publicly exposed. We expect compromising OVH, Hetzner, etc. would be easier than compromising sshd so it's not much of a concern for our public services.
@DanielMicay It will sure be spicy if a large hosting provider is ever revealed to have been backdoored! Especially if it’s named AWS. Just imagine.