Mastodawn

I published a new blog post explaining how virtual environments in #Python work. I also used the post to announce my lastest project: microvenv.

https://snarky.ca/how-virtual-environments-work/
https://pypi.org/project/microvenv/

How virtual environments work

After needing to do a deep dive on the venv module (which I will explain later in this blog post as to why), I thought I would explain how virtual environments work to help demystify them. Why do virtual environments exist? Back in my the day, there was no concept

Tall, Snarky Canadian

matt Mar 12, 2023

@brettcannon "When thing to notice here ..." sounds like should be "One thing to ..."
---
I've not used venv or variants, simply by historical accident of discovering conda first on my haphazard journey. I'm greatly attracted to the simplicity of what you describe with microvenv though. It takes so long to set-up a new conda env just to try something out. And then have them accumulate in an ever growing pile mismash pile of chaff and gold.

matt Mar 12, 2023

@brettcannon Conda still primary for me though, 'cause of non-python packages like gdal. I wonder: can multiple venv/microvenv work profitably and safely inside conda?

Brett Cannon Mar 13, 2023

@maphew I wouldn't mix conda environments and virtual environments. Usually the reason things take long with conda is the resolver and then the subsequent installation.

@brettcannon Interesting write-up! I'll be curious to see where microvenv gets adopted.

You mention that Windows handles creation of the python binary for the venv a little differently (presumably due to the lack of symlinks on Windows). How is it handled? I've always wondered what those .exe files were in the Scripts directory, since they don't appear to be standalone copies of the various Python binaries.

Brett Cannon Mar 13, 2023

@joshburnett The code for `venv` can be found in https://github.com/python/cpython/blob/main/Lib/venv/__init__.py . And it does support symlinking on Windows when requested, but it symlinks different things.

Jürgen Gmach Mar 12, 2023

"When thing to notice here" -> "One..."

"virtualen" -> "virtualenv"

"single-file virtual environment creator" -> you can use virtualenv as a zipapp, this is how I bootstrap a new #Ubuntu machine, and no, it is not small 😉

And #Ubuntu not distributing #pip by default is a blessing, as then there is no chance that I install anything globally or into my user folder by mistake.

Brett Cannon Mar 13, 2023

@jugmac00 Actually Ubuntu just needs to adopt https://peps.python.org/pep-0668/ to prevent you from installing globally into the interpreter

PEP 668 – Marking Python base environments as “externally managed” | peps.python.org

A long-standing practical problem for Python users has been conflicts between OS package managers and Python-specific package management tools like pip. These conflicts include both Python-level API incompatibilities and conflicts over file ownership.

Python Enhancement Proposals (PEPs)

Jürgen Gmach Mar 14, 2023

@brettcannon I had a quick chat with my colleagues, and it looks like 23.04 will be the first #Ubuntu version to support #PEP668.

Marius Gedminas Mar 13, 2023

@brettcannon Re: "And before anyone asks why we don't just use virtualenv", virtualenv depends on venv and so breaks on Debian where python3-venv is not installed, so it would not solve the problem anyway.

(Does anyone know _why_ Debian splits out venv from the default install?)

Pamela Fox Mar 13, 2023

@mgedmin @brettcannon That was also my question after reading the post- why Debian doesn't include venv- I didn't realize that was the case.

Brett Cannon Mar 13, 2023

@pamelafox @mgedmin Because Debian considers `venv` an app, and Debian has a policy of "one install -> one app".

Brett Cannon Mar 13, 2023

@mgedmin Actually virtualenv can use venv, but it's optional: https://virtualenv.pypa.io/en/latest/user_guide.html#creators

User Guide - virtualenv

@brettcannon wow what a terrible situation re the virtualenv thing. Can't MSFT just bring in Clippy "it looks like you are attempting to run python on a curmudgeonly Debian distribution who left out basic tools for 'reasons', shall I apt-get those libraries for you?"

Brett Cannon Mar 13, 2023

@zzzeek We actually contemplated telling users what command to run, although they would have to run it as sudo which is always a little icky to tell users to do. But when I realized it was simple enough to create microvenv, I just went ahead created the library instead and covered ourselves for any other odd situation where it might be missing on Unix.

Hynek Schlawack Mar 13, 2023

@brettcannon one thing re relocatable venvs: there’s one use-case that’s kinda painful rn and that’s packaging venvs into OS packages where the build root is usually not the FS root. I’ve solved it by building them in Docker containers within distinct root subdirs (eg /opt/app), but it’s kinda annoying to keep that special case around just for Python app Debs.

So it’s not just about speed. And pex/shiv won’t allow me to install multiple CLI commands.

Brett Cannon Mar 13, 2023

@hynek I'm not quite seeing where the virtual environment comes into play with the .deb file (probably because I don't know the file format details). Are you slurping up the entire virtual environment to create a .deb to install somewhere else? Is the issue you're building sdists into the venv and then assumptions break when you move them (or build into them later)? And I take it .deb files don't have any post-install step to patch things up?

Hynek Schlawack Mar 14, 2023

@brettcannon I think you're thinking too complicated. Yes, I'm slurping a whole venv with all installed deps into a package, because that's the closest we have of a pre-built build fragment. The problem is that if I want to install an app to `/opt/app`, I also have to build it at `/opt/app` which is uncommon. There are post install hooks, but I really don't want to package a broken thing and hope to fix it on installation.

Here's an old article by me on that topic: https://hynek.me/articles/python-app-deployment-with-native-packages/

Python Application Deployment with Native Packages

Speed, reproducibility, easy rollbacks, and predictability is what we strive for when deploying our diverse Python applications. And that’s what we achieved by leveraging virtual environments and Linux system packages.

Hynek Schlawack

Brett Cannon Mar 14, 2023

@hynek OK, so it's mainly the building of sdists that can't be relocated which are making things troublesome based on the "have to build" comment.

If that's true then I don't think that's a problem due to virtual environments not being relocatable but the build tool not creating something that can be relocated (unless you're also relying on the `python` symlink).

Hynek Schlawack Mar 14, 2023

@brettcannon I’m starting to suspect we’re talking about different things, because I wouldn’t know what sdists have to do with anything? Imagine simpler: you create a venv on a build server, you install your app and its dependencies. Now you want to put that venv on multiple servers eg by creating a zip archive of the venv that you unpack on said servers.

Unless the absolute path on the build server is equal to the absolute path on the target servers, the venv is broken.

Brett Cannon Mar 14, 2023

@hynek So it's the symlinks that are breaking for you? So are you having the app executed based on the `python` in the venv that gets put into the .deb and that's what breaks?

Hynek Schlawack Mar 15, 2023

@brettcannon the first line of broken are the symlinks – I never investigated further, because it all felt janky (even with --relocatable – I'm not arguing for that argument that never really worked, my point was that it's not about performance only).

Here's the bug where ppl complain about it on the virtualenv bug tracker after they dropped it – that's probably a better medium than Mastodon: https://github.com/pypa/virtualenv/issues/1549

--relocatable alternatives · Issue #1549 · pypa/virtualenv

I used the --relocatable flag. The new release 20.0 does not have this flag. How do I now create relocatable environments?

GitHub

Brett Cannon Mar 15, 2023

@hynek I think that's more of an argument for relocatable binaries (which I totally support having BTW).

I'm also curious whether WASI could somehow lead to statically linking extension modules with Python in a platform-independent `.wasm` file which then just needs a WASI runtime to run?

Hynek Schlawack Mar 16, 2023

@brettcannon I just noticed I miswrote: I meant shebang lines 😅