Seth Larson3d ago
Brett Cannon

RE: https://fosstodon.org/@jni/116287554201659198

I said digital attestations and `pylock.toml` would have helped with the litellm attack. People asked for more details, so I wrote a blog post explaining why. It also hopefully acts at motivation for people to use:

- Trusted publishing
- Digital attestations
- Lock files, and `pylock.toml` specifically

https://snarky.ca/why-pylock-toml-includes-digital-attestations/

So yes, @jni , I have a "human-readable intro" because I wrote one for you (and the other folks asking me questions on the subject). 😁

Mar 26 at 4:05amWeb
Show threadJuan Nunez-Iglesias3d ago

@brettcannon Amazing, thank you! Indeed thank you to all the PyPA folks who implemented trusted publishing, thanks to whom napari automagically ticked box 2 after ticking box 1:

https://pypi.org/project/napari/0.7.0/#napari-0.7.0-py3-none-any.whl

and thank you very much for the blog post which made it easy for go check that!

We still use requirements.txt files for our CI constraints, though, so we'll look into migrating to pylock.toml!

Client Challenge

Show threadJuan Nunez-Iglesias3d ago
@brettcannon One thing I'm not clear on: we don't use lock files for "library" use, ie if you pip install napari, you get whatever new version of the dependencies is up there. So this wouldn't help in that scenario, right? Is your assertion that it would have helped people catch on faster, or that we should all be locking our dependencies? (Which I think is generally an antipattern in Python as it leads to mutually incompatible libraries...?)
Show threadAlmar3d ago

@jni @brettcannon That's what I thought as well. I guess the final responsibility if for the end user to make sure their packages are sound, and a lockfile would help there.

Though perhaps we library maintainers can play a role by using a lockfile only for CI, and update it regularly to spot upstream compromised packages? Preferably in an automated way?

Show threadJuan Nunez-Iglesias2d ago

@almar @brettcannon "using a lockfile only for CI" We do this for napari! But we use requirements.txt based constraints, not pylock.toml, so when a version is updated, we kinda just bump it. 😅 (Once a week.) It does mean in most cases we have a built-in cooldown period.

But yes, this doesn't really help end users. I think the best thing for users is that uv/pip etc should have cooldown periods by default, and a --unsafe-get-absolute-latest opt-in flag for getting "today's" release.

Show threadBrett Cannon1d ago
@jni @almar it's more of a "catch on" for the community than expecting libraries to use lock files. What would have been also useful is a cooldown period before a new release is used in resolving dependencies.
Show threadRyan Hiebert3d ago
@brettcannon this seems like it really puts new platforms at a disadvantage if security only comes from having enough money to convince pypi that you’re good enough to be a trusted publisher. If codeberg can’t get it even though many people are using it, that really makes it difficult.
Show threadRyan Hiebert3d ago
@brettcannon I get that we wouldn’t want bad actors as trusted publishers, but I do wonder if we’re being too careful on allowing new trusted publishers, since we do require the trusted publisher to be authorized by the specific project, inherently limiting the damage. Re-upping the recent thread on the discussion board, it seems like there is at least some sentiment in that direction.
Show threadCarlton Gibson3d ago
@ryanhiebert @brettcannon yes… 🤔 I’m really happy to jump through any kind of local signing flow, such that “yep, that was definitely me” but delegating releases to the oligopoly is not progress. Alas.
Show threadSeth Larson3d ago

@ryanhiebert @brettcannon yeah, we need to figure out how we can provide some link to tie an archive on PyPI back to a claimed source code reference (whether that's git or a URL). Especially now that many folks are self-hosting their infrastructure.

What comes to my mind is focusing on build reproducibility, submitting a "claim" within a package (source code ref, build backend version, tool versions, etc) and having that claim verified by another party.

Show threadFlorian Schulze3d ago
@brettcannon as far as I understand the developer account was compromised and the bad payload was commited to the repository, so trusted publishing would have created a valid release and you wouldn’t have noticed anything. Did I miss something?
Show threadAlyssa Coghlan3d ago

@fschulze @brettcannon This particular attack was an exfiltration of a long lived PyPI API key, with packages uploaded from a separate origin. If I understand correctly, there *was* a dev account compromise, but that was of the GitHub Action that was used to perform the key exfiltration (that is, trivy was compromised first, then used to attack LiteLLM et al).

Edit to link the specific analysis my understanding is based on: https://futuresearch.ai/blog/litellm-hack-were-you-one-of-the-47000/

LiteLLM Hack: Were You One of the 47,000?

The litellm 1.82.7 and 1.82.8 supply chain attack on PyPI hit 47,000 downloads in 46 minutes. We analyzed all 2,337 dependent packages - 88% had version specs that allowed the compromised versions.

FutureSearch
Show threadTamir Bahar3d ago
@brettcannon @jni thanks! This makes things a lot clearer for me!
Show threadTimo Zimmermann3d ago

@brettcannon if I understand the workflow and requirements correctly I’d need to move CI and signing to one of the few blessed systems such as Microsoft’s falling apart slop dumpster fire.

If this is correct I honestly hope it causes the opposite reaction and people start looking into / working on alternatives instead of using the system as is.

Show threadBrett Cannon1d ago

@fallenhitokiri First, I work at MS; I don't expect you to necessarily agree with the trajectory of the company but it did pay for that blog post, so please keep the insults civil.

Two, you can work with whomever you have hosting your code to become trusted enough to use trusted publishing. Or you can work to make a system where trusted publishing isn't a prerequisite.

Show threadBrett Cannon1d ago
@fallenhitokiri as well, check out disulciss.python.org where there was a recent discussion as to why Codeberg isn't a trusted publisher yet and what it takes.
Show threadTimo Zimmermann1d ago

@brettcannon per the thread there doesn’t seem to be a way to establish any new providers outside of the blessed ones when 7 years is considered “too young”. The technical / operational concerns are valid IMHO.

The maturity argument also falls very short when you objectively look at the trajectory of a blessed provider with outages piling up.
The example with SourceForge in the discussion is a perfect example for that and also why blessed providers are not the way to go.

Show threadBrett Cannon21h ago
@fallenhitokiri trusted != uptime; it just means PyPI expects that security is at a certain level around identity
Show threadTimo Zimmermann1h ago

@brettcannon sorry, I wasn't very clear with my comment. I was referring to this comment https://discuss.python.org/t/new-oidc-providers-for-trusted-publishing/106334/5

> [...] 7 years isn’t a large amount of time to be able to look back on [...]

New OIDC providers for Trusted Publishing

Not speaking authoritatively here, just my personal opinion, though I am a PyPI administrator so I’ve got some sense of what the PyPI admins might think 😉 . To be clear though, this is a quick look and not any sort of official Yes or No. I believe the key requirement beyond the technical, is the reliability and security of a given OIDC provider. These providers effectively have the ability to release as any project, and thus their ability to secure and reliably operate the software and inf...

Discussions on Python.org
Show threadTimo Zimmermann1d ago

@brettcannon Based on the discuss. thread I agree that a system not requiring trusted publishing is what we should be working towards to.

With the arguments in the thread it’s just hard to see how. Many of the properties of a “trusted” way of doing things discussed are inherently incompatible with any new development.

Show threadTimo Zimmermann1d ago

@brettcannon also sorry for the less civil part of the comment.

I didn’t know you are at MS and that the blog post was sponsored / written as part of your job, I usually try to be more factual when engaging with content like this.

Show threadBrett Cannon21h ago
@fallenhitokiri not sponsored from a "brought to you by MS" perspective (still my personal opinion), just done on work time
Show threadHugo van Kemenade3d ago

@brettcannon Thanks for this writeup!

"What can you do as a person if you don't have code to check that things line up (which isn't a lot of code; the lock file should have the index server for the package, so you follow the index server API to get the digital attestation for each file and compare)?"

Let's build this into common tooling! Directly into installers like pip and uv? And pip/uv audit?
#Python #security

Show threadBrian Koopman3d ago
@hugovk @brettcannon I had the same thought when reading this (not being familiar with pylock.toml) -- "why doesn't pip do this?"
Show threadBrett Cannon1d ago
@brian @hugovk pip doesn't support installing from a lock file yet, so there's nothing for it to check ATM. And I think it makes sense to work out some common library that does the best practices here, it's just the usual questions of who and where?
Show threadsaucoide3d ago
@brettcannon @jni what am i missing? I dont see any publisher information on `pylock.toml`, generated with any of `uv`, `pip` or `pdm`, are the tools choosing not to include it at the moment or am I doing something wrong?
Show threadJuan Nunez-Iglesias2d ago
@saucoide @brettcannon based on other conversations I'm having, it seems the tooling hasn't quite caught up with the standard…
Show threadBrett Cannon1d ago
@jni @saucoide I wouldn't expect pip to have it as its pylock.toml support is basically a fancy `pip freeze` and this info isn't recorded at install time. As for PDM and uv, if you verified a project on PyPI has the requisite data then they haven't added support for digital attestations.
Show threadsaucoide1d ago
@brettcannon @jni yes I was trying to replicate it with the `packaging` example, but it does looks like the tooling doesn't support it yet: https://github.com/astral-sh/uv/issues/9122