So You Want to Solve Python Packaging: A Practical Guide

First, the technical: Python is used by vastly different groups of people, some that don't identify as "developers". Those groups often have disparate expectations about how packaging should work. Some don't even know what a package is.

Some don't even know they're using Python! Here's some examples: Python's in the Linux Standard Base and bunch of critical Linux stuff is written in Python. Distros gotta package those & their deps into their package database (deb/rpm).

Most distros want nothing to do with the language-specific package manager. They want to manage everything though rpm/deb/portage/whatever and they don't want you fucking around with system packages. Ever got burned by Python included with macOS? Yeah, same deal

So OS vendors want Python to be invisible to the user. They want it for system purposes, and they want to distribute python apps, scripts, and packages on their own terms. Cool. Let's pick another group: academics and researchers.

They want to do their research. They don't want to program Python. They want to work their data, create visualizations, and very importantly: they want fellow researchers to be able to use their code. These folks don't really want to think about packaging.

The packages they use, however, are complex fucking monsters. They're a mix of C, C++, FORTAN, Haskell, Julia, and god knows what else. They don't want to waste time installing build tools and compiling these things. Their packages need to be precompiled and ready to go.
Precompilation is *hard*, especially for high-performance libraries. You can't just distribute a build will all the fancy vector extensions enabled cause someone on a different processor won't be able to use it. You wanna see a nightmare? Look at TensorFlow.
Fundamentally these users do not want to think about this shit, and they're a *huge* group of users. You know who does think about this shit? Web developers, and every time someone comes along with "Python packaging sucks and someone should fix it" they're a web dev.

That's because web devs have different expectations. They *expect* to work with a packaging tool. They expect to find and install dependencies. They don't expect to work with a ton of native dependencies. They don't have the same problems.

This only scratches the surface of the technical complexity here. The reason there are so many tools for managing Python dependencies is because Python is not a monoculture and different folks need different things.

@stargirl

For the bioinformatics world, the "solution" is to use Conda. Even for R.