HT to @wdormann here - somebody has backdoored the open source project XZ which has downstream impacts.
For example, although OpenSSH doesn’t use XZ, Debian patch OpenSSH and introduced a dependency which translates as the XZ changes introducing a sshd authentication bypass backdoor it appears.
One dude bothered to investigate in his free time about why ssh was running slow, so it was caught fairly early - i.e. hopefully before distros started bundling it.
As I said, the impact here will be very limited due to how quick it was caught. Everybody owes the finder a beer.
NOTE: THIS IS AN ATTEMPT AT INCLUDING A BACKDOOR. THIS IS LEFT FOR HISTORICAL PURPOSES ONLY AND MUST NOT BE DONE. Please sync xz-utils 5.6.1-1 (main) from Debian unstable (main) Hello! I am one of the upstream maintainers for XZ Utils. Version 5.6.1 was recently released and uploaded to Debian as a bugfix only release. Notably, this fixes a bug that causes Valgrind to issue a warning on any application dynamically linked with liblzma. This includes a lot of important applications. This cou...
Originally reported on Google Code with ID 342 What steps will reproduce the problem? 1. Testcase is attached. Compile with GCC with -fsanitize=address option. 2. Run. 3. What is the expected outpu...
Back in 2022 a host of characters appeared and basically bullied the creator of the XZ project to hand it over to somebody else - at the time the guy cited mental health issues around not updating the project quickly.
At the time he was already talking about maybe handing over to the account who years later introduced the backdoor.
In mid 2023 said account introduced a change to Google’s OSS Fuzzer to weaken detection for XZ.
Somebody played a years long game of Jenga and lost.
Before everybody high fives each other, this is how the backdoor was found: somebody happened to look at why CPU usage had increased in sshd, and did all the research and notification work themselves. By this point the backdoor had been there for a month unnoticed.
I’ve made the joke before that if GCHQ aren’t introducing backdoors and vulns in open source that I want a tax refund. It wasn’t a joke. And it won’t be just be GCHQ.
Another two thoughts on XZ -
- sshd itself has no dependency on the XZ utils library. The streams got crossed in a way I don’t think anybody understood (except the threat actor).
- had that backdoor been performant with sshd, I don’t think anybody would have spotted it.
The way this played out opens a window of opportunity to go back and look at both issues.
Really good timeline of what is known to have happened so far. It looks like the rogue developer deliberately introduced a vulnerability in other package, too - I haven’t seen anybody else mention this.
Reading the dev’s GitHub history, they’ve been making changes to other open source projects too around compression. It also appears they/somebody involved has other accounts, too.
https://boehs.org/node/everything-i-know-about-the-xz-backdoor
How far the rabbit hole goes - back in 2021 they deliberately introduced a risky change in the compression library libarchive. Nobody noticed. This is shipped in a ton of systems:
https://github.com/libarchive/libarchive/pull/1609
Whoever the threat actor is knows what they are doing as they’ve gone after chained dependencies around compression.
Added the error text when printing out warning and errors in bsdtar when untaring. Previously, there were cryptic error messages when, for example in issue #1561, the user tries to untar an archive...
If anybody thinks this kind of thing is unique, it isn’t.
Example - CVE-2021-44529 in Ivanti Endpoint Manager. The cause?
Backdoor in open source code, was there for 7 years.
XZ Embedded Linux kernel module for IoT devices, 10 days ago had a change submitted to add Jia Tan (backdoor author) as a maintainer.
https://lore.kernel.org/lkml/202403201[email protected]/
Linux kernel documentation: https://docs.kernel.org/staging/xz.html
The GitHub repository for XZ Embedded kernel module has also been disabled: https://github.com/tukaani-project/xz-embedded/
Original maintainer of XZ repos has posted a short update:
https://tukaani.org/xz-backdoor/
HT @SamantazFox
The poor original maintainer of xz is on it now, and has already found another "fun" thing: https://git.tukaani.org/?p=xz.git;a=commitdiff;h=f9cf4c05edd14dedfe63833f8ccbe41b55823b00 . The configure check for enabling the Landlock sandboxing facility was subtly broken, so that Landlock support would never get enabled. The original malicious commit landed around the same timeframe as the main backdoor, also at an abnormal time of day compared to the new maintainer's historical activity pattern.
Also since there’s a lot going on here, up thread I mentioned a 2015 minor bug in Google’s OSS Fuzzer (security testing tool) - the threat actor deliberately introduced the bugged function into XZ, then used that to get an exception in OSS Fuzzer’s code to stop scanning of XZ.
I’ve just been looking at the actual backdoor for a few hours with greater minds than me, it’s incredibly complex - it basically piggy backs RSA key RCE inside sshd as a Trojan horse. Somebody/bodies spent $$ on this.
I don't know which is the biggest crime: naming a product or service with a word of the English language, like "upstream", or mentioning that product or service in an article without capitalizing, changing the font, or qualifiying it -- as "software from upstream" rather than "software from Upstream" or "software from the /upstream/ site".
@GossiTheDog The whole Tukaani Project on GitHub has been disabled - all of its code repositories:
https://github.com/tukaani-project/
The mirrors work, though:
@bontchev @GossiTheDog We have an official communication from the original developer, Lasse Collin, btw:
It should have been, may the source be with you!
On the other hand. On closed source Windows people generally run all kinds of crummy kernel level drivers to maximize a game framerate or because their hardware brand uses some fancy service management tool that is riddled with every security malpractice that you can throw a book at, just to make sure that you get the brand™ experience.
And the source code leaked on multiple occasions, and is perfectly open book to the larger nation states. So, give me open source.
@GossiTheDog The very recent zstd fork branch updates agree with the assessment that the compression ecosystem as a whole was the domain this threat actor was playing in:
@fl @GossiTheDog “Jia Tan” was active on libera IRC (under the username “jiatan”). If you have access to historic client logs from the libera IRC, their IP shows up. Someone was kind of enough to share their logs with Open source investigators.
From that investigator:
“Jia Tan connected to Libera via a VPN out of Singapore.”
Not sure if this has been publicly mentioned yet.
We do not know yet where this guy comes from for now, do we?
@GossiTheDog wait, are we at "implementing features to get plausible deniability for evading vuln scanning?"
Because if so this feels quite novel. At the very least I can't remember a comparable supply chain attack of this sophistication from the top of my head
@fuomag9 @GossiTheDog no, it's actually harmless
The PR he made can be found here: https://github.com/MicrosoftDocs/cpp-docs/pull/4716/files
No code change, just documentation
How long before the numpties begin combining this with their Baltimore bridge cyber conspiracy theories?
"Oops, I just stopped a massive operation that could have brought down everything under the sun, accidentally." *flexes*
@GossiTheDog indeed we do.
This exploit was carefully aimed at Linux using systemd launched SSH. We non Linux folks can breathe a hair easier but who knows what else is going on. Don't panic but definitely be cautious.
Kevin Beaumont
Jorge Stolfi
0daystolive
okanogen VerminEnemyFromWithin
Luis Villa
VessOnSecurity
Samantaz Fox
Joakim Fors
CassandraVert
count
Dave Wilburn 
Simon B
Euph0r14
Alan Langford 🇨🇦🧤🧊摏
Hans-Cees 🍋🌲🦔🦦🐝🦋🐛🚅🇸🇳🇵🇾🇹🇬🇹🇲
Gerrit 🇪🇺🌍🍉🔻
Steve's Place
Deirdre Saoirse Moen
zl2tod
Sevoris
Daniel
FL | エフル
Bruno BEAUFILS
Kevin Lyda
uoxc
fuomag9
David Penfold 
jaark
Ben Aveling
NosirrahSec 🏴☠️ guillotine enthusiast
Tryst 🏴 🇮🇪 
Dan McDonald
Toasterson (Till Wegmüller)
Future Sprog
Ed Maste
Alex