Mastodawn

https://boehs.org/node/everything-i-know-about-the-xz-backdoor

Evan B🥥ehs Mar 29, 2024

I have begun a post explaining this situation in a more detailed writeup. This is updating in realtime, and there is a lot still missing.

#security #xz #linux

Everything I know about the XZ backdoor

Please note: This is being updated in real-time. The intent is to make sense of lots of simultaneous discoveries

Evan B🥥ehs Mar 29, 2024

Holy shit.

@eb I really hope that this causes an industry-wide reckoning with the common practice of letting your entire goddamn product rest on the shoulders of one overworked person having a slow mental health crisis without financially or operationally supporting them whatsoever. I want everyone who has an open source dependency to read this message https://www.mail-archive.com/xz-devel@tukaani.org/msg00567.html

Re: [xz-devel] XZ for Java

Geoffrey Thomas Mar 29, 2024

@glyph @eb I'm frustrated that big tech's efforts to increase core library security are "your project is too popular, you must use 2FA" and "the best reverse engineers in the world will find your bugs and put you on a 90 day disclosure deadline" and not "here is $100K/year and benefits to keep doing what you're doing at your own pace."

David Zaslavsky

@geofft @glyph @eb I'm certainly not disputing that it's a real problem that that doesn't happen more often, but isn't there some precedent for big tech companies hiring people to work on specific open source projects? So it's not totally unheard of

@diazona @geofft @eb there's actually quite a bit of effort trying to address this problem, but it is a big collective action problem and … well, just look at the email, and tell me if that couldn't be just about any maintainer, on any project, anywhere. and xz is *extremely* core infrastructure, so the fact that this problem was this severe in this context is discouraging for the state of the rest of the industry

@glyph @geofft @eb Oh of course. I guess I just wanted to acknowledge being in a state of "a tiny bit of progress" rather than "zero progress". (I have an optimistic streak that comes out sometimes)

@diazona @geofft @eb it's appreciated, total despair is not a particularly useful affect.

(Also, as you will see in something longer-form I will post hopefully later today, this is extremely on my mind at the moment.)

@glyph @geofft @eb Indeed, and there seems to be more than enough total despair on social media for those who want it 😂

Rajnhildo Ursidino

Mar 30, 2024

@glyph @diazona @geofft @eb total despair is why all my open-source projects have fallen unmaintained

@diazona @geofft @glyph @eb there’s a lot of precedent for hiring maintainers of top-level programs whose brand (for lack of a better term) has reached the level of awareness of a C-level with a hiring budget. Collectively pooling money to help the projects C-levels have never heard of… has a much weaker track record. We’ve been trying to tackle it at Tidelift for a while and suffice to say I’ve definitely had a lot of “but it can’t happen to me” conversations.

@luis_in_brief @geofft @glyph @eb Gotcha, makes sense.

TBH I never quite understood what Tidelift was doing until now, so thank you for making that clear 🙂

@diazona @geofft @glyph @eb I increasingly wonder if we aren’t due for some “defragging” of a lot of core infra, with many projects pooled together, maintained, and funded more collectively, like Ruby Together.

@luis_in_brief @diazona @geofft @glyph @eb i doubt we can. Multiple people try but the expertise runs too niche and too big. At least with current tools.

Hell even just running the build system of a project easily takes months to learn.

Geoffrey Thomas Mar 29, 2024

@luis_in_brief @diazona @glyph @eb Yeah that resonates with my experience. People like GvR get hired (which is great!) but there's a whole dependency stack underneath. Their maintainers often have a strong résumé to get hired for a normal big tech job at a company that uses the language/ecosystem/etc. but not necessarily for maintaining the project as their job. Sometimes the job is even "build something similar for an internal non-OSS ecosystem."

@geofft @diazona @glyph @eb yup. Or they get hired with the promise that they’ll get 20% time to work on it, and that goes away for reasons (sometimes good, sometimes bad), or…. Etc etc

@geofft @luis_in_brief @diazona @eb there are layers and layers to this. Famous maintainers get hired more than critical maintainers. And maintenance is important but how do you pay for the commons of *new* projects? The tidelift model gets us part of the way there, because these costs need to be aggregated and there needs to be some kind of oversight, but even if they were universally adopted (and that is far from true) there are so many missing pieces

@glyph @geofft @diazona @eb “Famous maintainers get hired more than critical maintainers.” Owwwwwwww.

Sumana Harihareswara Mar 30, 2024

@luis_in_brief @glyph

I am way overdue in finishing and publishing my negative review of Eghbal's "Working in Public" but one of my critiques is that she basically concludes that maintainers need to become famous and use Substack/Patreon to crowdfund (from individual donors) in order to sustain their work. Which really doesn't fit what we have found in critical FLOSS infrastructure IMO.

@geofft @diazona @eb

@brainwane @luis_in_brief @glyph @geofft @diazona @eb yep i never published my own review because of that. The Road and Bridges report was great. The book felt like a massive PR piece for GitHub sponsor feature and a way to hide the problem.

Sumana Harihareswara Mar 30, 2024

@Di4na

If you have an unfinished or unpublished draft review I would very much like to read it. My own critique will/would expand on what I wrote in https://www.harihareswara.net/posts/2022/what-you-miss-by-only-checking-github/ as well as my comment at the top of https://www.metafilter.com/191414/Free-as-in-free-puppy-not-free-as-in-free-beer .

@luis_in_brief @glyph @geofft @diazona @eb

@brainwane @luis_in_brief @glyph @geofft @diazona @eb nah it stayed in my head. And i have far too many blogpost ideas in my "todo" list that have been more urgent.

Realistically most people already don't read Road and Bridges so that book basically has fallen into my "to forget" bin

benwis 🦀Mar 30, 2024

@brainwane @luis_in_brief @glyph @geofft @diazona @eb @djc How to monetize open source is such an interesting question.

@benwis @brainwane @luis_in_brief @glyph @geofft @diazona @djc there’s some website (I forget what it is) that basically you pay x amount of dollars and it audits your entire dependency tree and attempts to pay maintainers proportionally. Unfortunately iirc it was kinda flawed but I think it’s a solid idea

@benwis @brainwane @luis_in_brief @glyph @geofft @diazona @djc might have been https://thanks.dev/home but I remember a different one with a retro theme

@benwis @brainwane @luis_in_brief @glyph @geofft @diazona @djc https://www.stackaid.us/

StackAid - Fund all your open source dependencies

Luis Villa Mar 30, 2024

@eb @benwis @brainwane @glyph @geofft @diazona @djc you’re thinking of Back Your Stack, probably.

On a more sustainable (read: commercial) basis, I co-founded https://tidelift.com to do exactly this.

Tidelift | Reduce security risk from bad open source packages

Reduce security risk from bad open source packages and ensure the packages you rely on keep getting better.

@luis_in_brief @benwis @brainwane @glyph @geofft @diazona @djc oh that’s sick, it’s so funny that you never know who you’re speaking to on here lol. Congrats on how successful tidelift has been :)

Luis Villa Mar 30, 2024

@eb @benwis @brainwane @glyph @geofft @diazona @djc Thanks! admittedly on a day like today, mostly I'm focused on how many projects we can't yet cover.

So, yeah, send people our way!

benwis 🦀Mar 31, 2024

@luis_in_brief @eb @brainwane @glyph @geofft @diazona @djc

Does Tidelift support Rust projects?

Sumana Harihareswara Mar 31, 2024

@benwis

it does!

@luis_in_brief @eb @glyph @geofft @diazona @djc

Luis Villa Mar 31, 2024

@benwis @brainwane we do, though sadly not a ton of customer demand yet so not a ton of money going into that ecosystem yet.

Sumana Harihareswara Mar 31, 2024

@luis_in_brief @benwis yeah. Example: https://tidelift.com/lifter/search/cargo/cargo

Tidelift | Reduce security risk from bad open source packages

Reduce security risk from bad open source packages and ensure the packages you rely on keep getting better.

@luis_in_brief @brainwane

benwis 🦀Mar 31, 2024

That’s fine, a lot of critical infrastructure is being rewritten it, but more importantly for me I write a lot of Rist

Geoffrey Thomas Mar 29, 2024

@glyph @luis_in_brief @diazona @eb Yes, e.g., what if the current maintainer is genuinely unavailable/uninterested? As may well have happened with xz even with a job offer.

Funding a new maintainer is by itself defensible, but doing so will drastically change both the pressure on the current maintainer and the choice of who becomes maintainer (e.g. there's now a bias in favor of those who have US work authorization).

I'm curious if either Tidelift or the commercial distros have norms for this.

Cassandrich Mar 29, 2024

@geofft @glyph @luis_in_brief @diazona @eb A relationship with a critical FOSS dependency maintainer is very clearly classifiable as independent contractor, and SHOULD or even MUST be for the sake of project integrity. There should be no reason to need US work authorization.

@dalias @geofft @glyph @luis_in_brief @eb Legally speaking I think it could be set up either way. Although if an OSS project maintainer is employed (not contracted) by a company to maintain the project, it is kind of as if the company is acting as the maintainer, which certainly raises questions about their motivation....

@luis_in_brief @diazona @geofft @glyph @eb i had a blogpost in the work all week around this but like.

By my (cursed) maths based on both Tidelift report and the Synopsys one, at least 50% of every single sloc in commercial project (averaged) is foss maintained by a "weekend maintainer".

Only 10% is commercially maintained.

And the rest is mostly "partially paid for maintenance" which means "sometimes i maybe get paid a few hours on a contract to do it".

@luis_in_brief @diazona @geofft @glyph @eb

And that is going to the conservative side. I think it is more than 50%.

And realistically, we are not going to massively move the needle here. This is what makes opensource win. If we change that equation, people will revert to this.

We need to find ways for weekend maintainers to do more with less effort on their part too. We are not going to pay everyone soon enough. We need to work on both angles.

JP Mar 29, 2024

@diazona @geofft @glyph @eb there is also precedent for corps to pay a proportion of their income into a shared pool that can be used to pay for creating and maintaining public goods, but that idea is weirdly unpopular for some reason