So You Want to Solve Python Packaging: A Practical Guide
First, the technical: Python is used by vastly different groups of people, some that don't identify as "developers". Those groups often have disparate expectations about how packaging should work. Some don't even know what a package is.
Some don't even know they're using Python! Here's some examples: Python's in the Linux Standard Base and bunch of critical Linux stuff is written in Python. Distros gotta package those & their deps into their package database (deb/rpm).
Most distros want nothing to do with the language-specific package manager. They want to manage everything though rpm/deb/portage/whatever and they don't want you fucking around with system packages. Ever got burned by Python included with macOS? Yeah, same deal
So OS vendors want Python to be invisible to the user. They want it for system purposes, and they want to distribute python apps, scripts, and packages on their own terms. Cool. Let's pick another group: academics and researchers.
They want to do their research. They don't want to program Python. They want to work their data, create visualizations, and very importantly: they want fellow researchers to be able to use their code. These folks don't really want to think about packaging.
That's because web devs have different expectations. They *expect* to work with a packaging tool. They expect to find and install dependencies. They don't expect to work with a ton of native dependencies. They don't have the same problems.
This only scratches the surface of the technical complexity here. The reason there are so many tools for managing Python dependencies is because Python is not a monoculture and different folks need different things.
I'm still upset that Python 3 broke Python 2.7 compatibility, I've still got systems built and running Python 2.7 and I always will.
@stargirl We're doing something related to what you wrote in this threat at ActiveState, but our target audience is mostly enterprise customers.
I agree that having a single packaging solution that works great for every use case is naive, but I don't see why each of them can't be "solved" "individually".
(NixOS developer, maintain some Python packages in nixpkgs and has knowledge of the issues we encounter classically with the Python packaging ecosystem wrt to native dependencies. I don’t represent the NixOS project or community.) I read the whole thread and found the remarks of @steve.dower very interesting. Firstly, I have to say we know that Nix ecosystem is probably a small subset of Python users and probably the one for which Python packaging works more or less the best in our closed garde...
> my experience is that it gets weird-and-broken pretty quick in ways that are confusing,
This usually happens somewhere in the vicinity of loading drivers (e.g. libcuda or libGL), which currently is only accommodated by wrappers (nixglhost, nixGL) and, yes, sucks. It's hard to improve upon because of how other dsitributions mix userspace drivers and normal libraries, but there are some cursed ideas being talked about involving libcapsule...
@nobody @whack @stargirl It's a fair critique, though, and one that spawns from the *same* memetic issue as in Python packaging, so it's worth examining.
Sure, a Nix package *should* only rely on being located in /nix/store next to its dependencies. But as you've listed, there's piles of ambient authorities and mutable state which also must be correctly configured in order for some packages to run.
This is *exactly* analogous to Python packages which aren't pure statically-imported Python modules. In a simpler world, we would deploy Python apps by tossing all Python modules into a single ZIP file, but because CPython allows extension modules and exposes a C API, a practical Python app must also depend on C runtime and C linkage.
This isn't hypothetical. RPython, the toolkit for building PyPy, is pure Python 2.7. My rpypkgs flake https://github.com/rpypkgs/rpypkgs "installs" RPython packages by unpacking all of them to a common build directory.
@stargirl Rephrasing @corbin, I'd say that "Python Packaging" is actually solved. Mostly. Everything-pyproject.toml (maybe with PEP-725) is comprehensive for what's truly in scope for "python packaging". There will necessarily be some native stuff that is absolutely out of scope for "python packaging", in fact most of it. There will always have to be a "system layer" under "python packages".
Nixpkgs is one consumer that integrates PEP-compliant "python packages" with its "native" stuff
TLDR; I wanted to get feedback on a potential feature that may be added to nixpkgs that allows multiple versions of the same python package to be installed in the same PYTHONPATH. This is a general approach that is not specific to nixpkgs and could be used in other package managers. The only nix specific part is the tooling to allow for the building of these specialized packages. All of the materials/demo is in this repo https://github.com/costrouc/python-multiple-versions. Sorry discourse preve...
@nobody @stargirl I'll note that this suffers from Zooko's triangle, a trilemma in nominal logic: https://en.wikipedia.org/wiki/Zooko%27s_triangle
Imagine every Python module were content-addressed somehow, and packages too. This only kicks the can down the road, because you'd probably rather import from a petname than from a SHA-256 hash, so somebody must now set up a petname-to-hash map, and all of the existing political issues of the Cheeseshop reappear upon the maintenance of that map. (This is why politics is an inevitable part of maintaining large ports trees too.)
@nobody @stargirl Sure. Those hints would be populated from a mapping either maintained via Cheeseshop or via local choices. Either way, there's politics going on; each individual mapping is a policy choice. In terms of Zooko's triangle, this would be the "centralized" option, which means that the import keyword would now functionally depend on whatever build/runtime configuration you've imagined.
Not that this is a bad thing! Check out PEP 302 for the hooks you'd need to prototype an actual implementation of your idea. Allen Short explained how to use PEP 302 here https://web.archive.org/web/20180411011138/http://washort.twistedmatrix.com/2011/01/introducing-exocet.html and I used it to add hot-reload plugin functionality to a Python Minecraft server.
@corbin @stargirl Nice, I'm not sure how I missed this. I suppose the PEP-302 example you shared is exactly the "local choices" implementation. I also now noticed this: https://github.com/mitsuhiko/multiversion. Similarly local.
> means that the import keyword would now functionally depend on whatever build/runtime configuration you've imagined.
Well it does today too
Dr. Kai Blin
Stargirl
rastilin
sbszine
Jeff Forcier
Pēteris Krišjānis
Mark Gardner
Scott Cain
Graham Sutherland / Polynomial
Mike Garuccio
Haelwenn /элвэн/ 
the_curiostech
kryptec
Zorro Notorious MEB 😡
Greg
Sky Leite
Corbin
Raito Bezarius
Jordan Sissel
Else, Someone
Mark Tomczak
Jason Bowen 🇺🇦
AlgoCompSynth by znmeb 🇺🇦
moonchild
Clément Robert 
Gabriel Pettier
matt
Matt "msw" Wilson
