Mastodawn

https://boehs.org/node/everything-i-know-about-the-xz-backdoor

Evan B🥥ehs Mar 29, 2024

I have begun a post explaining this situation in a more detailed writeup. This is updating in realtime, and there is a lot still missing.

#security #xz #linux

Everything I know about the XZ backdoor

Please note: This is being updated in real-time. The intent is to make sense of lots of simultaneous discoveries

Evan B🥥ehs Mar 29, 2024

Holy shit.

cpw Mar 31, 2024

Glyph

@eb I really hope that this causes an industry-wide reckoning with the common practice of letting your entire goddamn product rest on the shoulders of one overworked person having a slow mental health crisis without financially or operationally supporting them whatsoever. I want everyone who has an open source dependency to read this message https://www.mail-archive.com/xz-devel@tukaani.org/msg00567.html

Re: [xz-devel] XZ for Java

@eb "I never thought a sophisticated APT would backdoor *my* volunteer-maintained infrastructure that I got for free" sobs entire industry who voted for the "volunteer-maintained infrastructure that I get for free with no defense against sophisticated APTs" party

Phil M0OFX Mar 30, 2024

@glyph @eb gee what a shame they didn't - hang on, I need to check my notes - "pay a fair price for, or otherwise support the developers of the tools they use".

Johannes Bechberger Mar 30, 2024

@philpem @glyph @eb My company uses Java and I'm a full-time open-source JDK developer, so some companies give back. Especially if they can good support and priority fixes for their problems.

tessarakt Mar 30, 2024

@glyph @eb I guess we will get another corporate advisory to check for this version (well, first, that a taskforce has been formed to investigate the problem), and then everyone moves on as if nothing had happened ...

@glyph @eb I'm frustrated that big tech's efforts to increase core library security are "your project is too popular, you must use 2FA" and "the best reverse engineers in the world will find your bugs and put you on a 90 day disclosure deadline" and not "here is $100K/year and benefits to keep doing what you're doing at your own pace."

@geofft @eb I mean, por que no los dos, but one of these is clearly more important than the other

@geofft @glyph @eb (sobs in Tidelift)

@luis_in_brief @glyph @eb I should try harder to figure out what a Tidelift is and how to convince my employer to sign up. But also... IMO Microsoft or Google (whom I am subtooting) etc. can singlehandedly employ all the maintainers of `ldd sshd` and that would get results that fractionally paying for the commons never will.

Like this should be the job of a distro, and RH/SUSE/Canonical/Oracle kinda do this, but clearly none of them actually saved their customers (or the world) from this.

Geoffrey Thomas Mar 30, 2024

@luis_in_brief @glyph @eb Re "I should try harder to convince my employer to sign up for Tidelift": can Tidelift tell us whether .... *looks at /proc/$(pgrep sshd)/maps* ... libcap-ng or zlib has a sustainable maintenance story and what it would take to give it a sustainable maintenance story?

Looks like libcap-ng's maintainer is at RH and zlib's maintainer is (probably comfortably) retired, but
both seem to be one-person jobs, which as @ehashman points out is a problem.

@geofft @glyph @eb @ehashman sadly we do that for just about everything *except* C and C++, because our valuation metrics rely on customer usage and lol/sob C/C++ packaging is (relatively) poor and unstandardized.

@geofft @glyph @eb @ehashman but note also that the goal is not "support this one package", it's support as much of the stack as possible. If we focus on "tell you about one package" then we just replicate the problem (noted elsewhere in multiple threads here) that famous packages get support, and others (which data would surface but 'namebrands' don't) get zero.

Geoffrey Thomas Mar 30, 2024

@luis_in_brief @glyph @eb @ehashman I hear you re systemic problems but also - as a concrete example, is anyone (Tidelift or otherwise) doing anything about https://github.com/libexpat/libexpat/blob/master/expat/Changes ?

Like I suspect a large fraction of these core C libraries that we try not to think about actually do have institutional support from Red Hat etc. or perhaps Google etc., and there are a relatively small number of projects desperately screaming for help that would be worth focusing on.

libexpat/expat/Changes at master · libexpat/libexpat

:herb: Fast streaming XML parser written in C99 with >90% test coverage; moved from SourceForge to GitHub - libexpat/libexpat

GitHub

Geoffrey Thomas Mar 30, 2024

@luis_in_brief @glyph @eb @ehashman Is there anything we-the-community (or a customer) can do to help with C/C++ deps? Like can you find them from `dpkg -l`? Would tools to analyze configure.ac files or to instrument builds be helpful?

(This is basically an SBOM question, right? Like we would want to be able to answer "OK where are we using the bad xz versions," including vendored copies and OS packages.)

e. hashman 🇵🇸Mar 30, 2024

@luis_in_brief @geofft @glyph @eb I suspect that the things that are hardest to track are the ones most likely to be the source of problems, because they're otherwise not getting attention. This is, fwiw, why I think leaning on distros to track the dep/build-dep chains makes sense in this case—distros exist *because* C/C++ distribution and management sucks

@ehashman @geofft @glyph @eb that's... not our experience. Plenty of very trackable and nevertheless completely financially unsupported and problematically maintained packages in all the modern stacks.

@ehashman @geofft @glyph @eb which isn’t to diminish the very, very real and wide-spread problems in C-land, especially (as you pointed out somewhere) in build tools. That’s a problem we’ve always wanted to tackle but have needed more growth first :/

Glyph Mar 30, 2024

@luis_in_brief @ehashman @geofft @eb I really feel like this is a “yes, and” situation. There is more (a lot more) than one problem here

@glyph @ehashman @geofft @eb 💯 🎯 👍🏽

Colin Watson Mar 30, 2024

@geofft @luis_in_brief @glyph @eb Now that I'm >3 months out from working there: I do think that Canonical is essentially well-meaning in many ways, but I would not recommend it as a place to fund upstream maintenance; that was never particularly its priority, and the amount of overhead imposed on experienced engineers has gone through the roof in recent years

@geofft @glyph @eb I'm certainly not disputing that it's a real problem that that doesn't happen more often, but isn't there some precedent for big tech companies hiring people to work on specific open source projects? So it's not totally unheard of

@diazona @geofft @eb there's actually quite a bit of effort trying to address this problem, but it is a big collective action problem and … well, just look at the email, and tell me if that couldn't be just about any maintainer, on any project, anywhere. and xz is *extremely* core infrastructure, so the fact that this problem was this severe in this context is discouraging for the state of the rest of the industry

@glyph @geofft @eb Oh of course. I guess I just wanted to acknowledge being in a state of "a tiny bit of progress" rather than "zero progress". (I have an optimistic streak that comes out sometimes)

@diazona @geofft @eb it's appreciated, total despair is not a particularly useful affect.

(Also, as you will see in something longer-form I will post hopefully later today, this is extremely on my mind at the moment.)

@glyph @geofft @eb Indeed, and there seems to be more than enough total despair on social media for those who want it 😂

Rajnhildo Ursidino

Mar 30, 2024

@glyph @diazona @geofft @eb total despair is why all my open-source projects have fallen unmaintained

@diazona @geofft @glyph @eb there’s a lot of precedent for hiring maintainers of top-level programs whose brand (for lack of a better term) has reached the level of awareness of a C-level with a hiring budget. Collectively pooling money to help the projects C-levels have never heard of… has a much weaker track record. We’ve been trying to tackle it at Tidelift for a while and suffice to say I’ve definitely had a lot of “but it can’t happen to me” conversations.

@luis_in_brief @geofft @glyph @eb Gotcha, makes sense.

TBH I never quite understood what Tidelift was doing until now, so thank you for making that clear 🙂

@diazona @geofft @glyph @eb I increasingly wonder if we aren’t due for some “defragging” of a lot of core infra, with many projects pooled together, maintained, and funded more collectively, like Ruby Together.

@luis_in_brief @diazona @geofft @glyph @eb i doubt we can. Multiple people try but the expertise runs too niche and too big. At least with current tools.

Hell even just running the build system of a project easily takes months to learn.

@luis_in_brief @diazona @glyph @eb Yeah that resonates with my experience. People like GvR get hired (which is great!) but there's a whole dependency stack underneath. Their maintainers often have a strong résumé to get hired for a normal big tech job at a company that uses the language/ecosystem/etc. but not necessarily for maintaining the project as their job. Sometimes the job is even "build something similar for an internal non-OSS ecosystem."

@geofft @diazona @glyph @eb yup. Or they get hired with the promise that they’ll get 20% time to work on it, and that goes away for reasons (sometimes good, sometimes bad), or…. Etc etc

@geofft @luis_in_brief @diazona @eb there are layers and layers to this. Famous maintainers get hired more than critical maintainers. And maintenance is important but how do you pay for the commons of *new* projects? The tidelift model gets us part of the way there, because these costs need to be aggregated and there needs to be some kind of oversight, but even if they were universally adopted (and that is far from true) there are so many missing pieces

@glyph @geofft @diazona @eb “Famous maintainers get hired more than critical maintainers.” Owwwwwwww.

Sumana Harihareswara Mar 30, 2024

@luis_in_brief @glyph

I am way overdue in finishing and publishing my negative review of Eghbal's "Working in Public" but one of my critiques is that she basically concludes that maintainers need to become famous and use Substack/Patreon to crowdfund (from individual donors) in order to sustain their work. Which really doesn't fit what we have found in critical FLOSS infrastructure IMO.

@geofft @diazona @eb

@brainwane @luis_in_brief @glyph @geofft @diazona @eb yep i never published my own review because of that. The Road and Bridges report was great. The book felt like a massive PR piece for GitHub sponsor feature and a way to hide the problem.

Sumana Harihareswara Mar 30, 2024

@Di4na

If you have an unfinished or unpublished draft review I would very much like to read it. My own critique will/would expand on what I wrote in https://www.harihareswara.net/posts/2022/what-you-miss-by-only-checking-github/ as well as my comment at the top of https://www.metafilter.com/191414/Free-as-in-free-puppy-not-free-as-in-free-beer .

@luis_in_brief @glyph @geofft @diazona @eb

@brainwane @luis_in_brief @glyph @geofft @diazona @eb nah it stayed in my head. And i have far too many blogpost ideas in my "todo" list that have been more urgent.

Realistically most people already don't read Road and Bridges so that book basically has fallen into my "to forget" bin

Lars Wirzenius Mar 30, 2024

@brainwane @luis_in_brief @glyph @geofft @diazona @eb Patreon and other donation based approaches to funding open source dev and maint are a no-go for those of us living in Finland, where "appealing to the public for donations" requires permission from the police, which is often capriciously denied.

benwis 🦀Mar 30, 2024

@brainwane @luis_in_brief @glyph @geofft @diazona @eb @djc How to monetize open source is such an interesting question.

@benwis @brainwane @luis_in_brief @glyph @geofft @diazona @djc there’s some website (I forget what it is) that basically you pay x amount of dollars and it audits your entire dependency tree and attempts to pay maintainers proportionally. Unfortunately iirc it was kinda flawed but I think it’s a solid idea

@benwis @brainwane @luis_in_brief @glyph @geofft @diazona @djc might have been https://thanks.dev/home but I remember a different one with a retro theme

@benwis @brainwane @luis_in_brief @glyph @geofft @diazona @djc https://www.stackaid.us/

StackAid - Fund all your open source dependencies

@eb @benwis @brainwane @glyph @geofft @diazona @djc you’re thinking of Back Your Stack, probably.

On a more sustainable (read: commercial) basis, I co-founded https://tidelift.com to do exactly this.

Tidelift | Reduce security risk from bad open source packages

Reduce security risk from bad open source packages and ensure the packages you rely on keep getting better.

@luis_in_brief @benwis @brainwane @glyph @geofft @diazona @djc oh that’s sick, it’s so funny that you never know who you’re speaking to on here lol. Congrats on how successful tidelift has been :)

@eb @benwis @brainwane @glyph @geofft @diazona @djc Thanks! admittedly on a day like today, mostly I'm focused on how many projects we can't yet cover.

So, yeah, send people our way!

benwis 🦀Mar 31, 2024

@luis_in_brief @eb @brainwane @glyph @geofft @diazona @djc

Does Tidelift support Rust projects?

Sumana Harihareswara Mar 31, 2024

@benwis

it does!

@luis_in_brief @eb @glyph @geofft @diazona @djc

Luis Villa Mar 31, 2024

@benwis @brainwane we do, though sadly not a ton of customer demand yet so not a ton of money going into that ecosystem yet.

Sumana Harihareswara Mar 31, 2024

@luis_in_brief @benwis yeah. Example: https://tidelift.com/lifter/search/cargo/cargo

Tidelift | Reduce security risk from bad open source packages

Reduce security risk from bad open source packages and ensure the packages you rely on keep getting better.

@luis_in_brief @brainwane

benwis 🦀Mar 31, 2024

That’s fine, a lot of critical infrastructure is being rewritten it, but more importantly for me I write a lot of Rist

@glyph @luis_in_brief @diazona @eb Yes, e.g., what if the current maintainer is genuinely unavailable/uninterested? As may well have happened with xz even with a job offer.

Funding a new maintainer is by itself defensible, but doing so will drastically change both the pressure on the current maintainer and the choice of who becomes maintainer (e.g. there's now a bias in favor of those who have US work authorization).

I'm curious if either Tidelift or the commercial distros have norms for this.

Cassandrich Mar 29, 2024

@geofft @glyph @luis_in_brief @diazona @eb A relationship with a critical FOSS dependency maintainer is very clearly classifiable as independent contractor, and SHOULD or even MUST be for the sake of project integrity. There should be no reason to need US work authorization.

@dalias @geofft @glyph @luis_in_brief @eb Legally speaking I think it could be set up either way. Although if an OSS project maintainer is employed (not contracted) by a company to maintain the project, it is kind of as if the company is acting as the maintainer, which certainly raises questions about their motivation....

@luis_in_brief @diazona @geofft @glyph @eb i had a blogpost in the work all week around this but like.

By my (cursed) maths based on both Tidelift report and the Synopsys one, at least 50% of every single sloc in commercial project (averaged) is foss maintained by a "weekend maintainer".

Only 10% is commercially maintained.

And the rest is mostly "partially paid for maintenance" which means "sometimes i maybe get paid a few hours on a contract to do it".