Hugo van Kemenade21h ago
Brett Cannon

RE: https://fosstodon.org/@jni/116287554201659198

I said digital attestations and `pylock.toml` would have helped with the litellm attack. People asked for more details, so I wrote a blog post explaining why. It also hopefully acts at motivation for people to use:

- Trusted publishing
- Digital attestations
- Lock files, and `pylock.toml` specifically

https://snarky.ca/why-pylock-toml-includes-digital-attestations/

So yes, @jni , I have a "human-readable intro" because I wrote one for you (and the other folks asking me questions on the subject). 😁

Mar 26 at 4:05amWeb
Show threadJuan Nunez-Iglesias1d ago

@brettcannon Amazing, thank you! Indeed thank you to all the PyPA folks who implemented trusted publishing, thanks to whom napari automagically ticked box 2 after ticking box 1:

https://pypi.org/project/napari/0.7.0/#napari-0.7.0-py3-none-any.whl

and thank you very much for the blog post which made it easy for go check that!

We still use requirements.txt files for our CI constraints, though, so we'll look into migrating to pylock.toml!

Client Challenge

Show threadJuan Nunez-Iglesias1d ago
@brettcannon One thing I'm not clear on: we don't use lock files for "library" use, ie if you pip install napari, you get whatever new version of the dependencies is up there. So this wouldn't help in that scenario, right? Is your assertion that it would have helped people catch on faster, or that we should all be locking our dependencies? (Which I think is generally an antipattern in Python as it leads to mutually incompatible libraries...?)
Show threadAlmar21h ago

@jni @brettcannon That's what I thought as well. I guess the final responsibility if for the end user to make sure their packages are sound, and a lockfile would help there.

Though perhaps we library maintainers can play a role by using a lockfile only for CI, and update it regularly to spot upstream compromised packages? Preferably in an automated way?

Show threadRyan Hiebert1d ago
@brettcannon this seems like it really puts new platforms at a disadvantage if security only comes from having enough money to convince pypi that you’re good enough to be a trusted publisher. If codeberg can’t get it even though many people are using it, that really makes it difficult.
Show threadRyan Hiebert1d ago
@brettcannon I get that we wouldn’t want bad actors as trusted publishers, but I do wonder if we’re being too careful on allowing new trusted publishers, since we do require the trusted publisher to be authorized by the specific project, inherently limiting the damage. Re-upping the recent thread on the discussion board, it seems like there is at least some sentiment in that direction.
Show threadCarlton Gibson23h ago
@ryanhiebert @brettcannon yes… 🤔 I’m really happy to jump through any kind of local signing flow, such that “yep, that was definitely me” but delegating releases to the oligopoly is not progress. Alas.
Show threadSeth Larson16h ago

@ryanhiebert @brettcannon yeah, we need to figure out how we can provide some link to tie an archive on PyPI back to a claimed source code reference (whether that's git or a URL). Especially now that many folks are self-hosting their infrastructure.

What comes to my mind is focusing on build reproducibility, submitting a "claim" within a package (source code ref, build backend version, tool versions, etc) and having that claim verified by another party.

Show threadFlorian Schulze1d ago
@brettcannon as far as I understand the developer account was compromised and the bad payload was commited to the repository, so trusted publishing would have created a valid release and you wouldn’t have noticed anything. Did I miss something?
Show threadAlyssa Coghlan1d ago

@fschulze @brettcannon This particular attack was an exfiltration of a long lived PyPI API key, with packages uploaded from a separate origin. If I understand correctly, there *was* a dev account compromise, but that was of the GitHub Action that was used to perform the key exfiltration (that is, trivy was compromised first, then used to attack LiteLLM et al).

Edit to link the specific analysis my understanding is based on: https://futuresearch.ai/blog/litellm-hack-were-you-one-of-the-47000/

LiteLLM Hack: Were You One of the 47,000?

The litellm 1.82.7 and 1.82.8 supply chain attack on PyPI hit 47,000 downloads in 46 minutes. We analyzed all 2,337 dependent packages - 88% had version specs that allowed the compromised versions.

FutureSearch
Show threadTamir Bahar1d ago
@brettcannon @jni thanks! This makes things a lot clearer for me!
Show threadTimo Zimmermann22h ago

@brettcannon if I understand the workflow and requirements correctly I’d need to move CI and signing to one of the few blessed systems such as Microsoft’s falling apart slop dumpster fire.

If this is correct I honestly hope it causes the opposite reaction and people start looking into / working on alternatives instead of using the system as is.

Show threadHugo van Kemenade21h ago

@brettcannon Thanks for this writeup!

"What can you do as a person if you don't have code to check that things line up (which isn't a lot of code; the lock file should have the index server for the package, so you follow the index server API to get the digital attestation for each file and compare)?"

Let's build this into common tooling! Directly into installers like pip and uv? And pip/uv audit?
#Python #security

Show threadBrian Koopman18h ago
@hugovk @brettcannon I had the same thought when reading this (not being familiar with pylock.toml) -- "why doesn't pip do this?"
Show threadsaucoide18h ago
@brettcannon @jni what am i missing? I dont see any publisher information on `pylock.toml`, generated with any of `uv`, `pip` or `pdm`, are the tools choosing not to include it at the moment or am I doing something wrong?