Brett Cannon
For those of you who install your virtual environments into a global directory (e.g. virtualenvwrapper users, `$WORKON_HOME`), what's the primary reason you do that instead locally into your workspace (e.g. `.venv` directory next to your code)?
Easy to delete all environments
7.4%
Environment sharing/reuse
24.2%
No reason/habit/tool default
52.6%
Other (w/ follow-up comment to explain)
15.8%
Poll ended at .
Dec 20, 2022 at 6:31pmWeb
Show threadTamir BaharDec 20, 2022
@brettcannon poetry does this by default and I never bothered to change it.
Show threadpeterDec 20, 2022
@brettcannon Tool default (pipenv) - an npm-like solution (PEP 582) would be really nice.
Show threadSebastian RaschkaDec 20, 2022
@brettcannon conda default. But it’s also nice when you do a yearly spring cleaning (virtual envs with PyTorch can be multiple gigabytes these days). Sometimes I override the defaults depending on the project.
Show threadBrett CannonDec 20, 2022
@SebRaschka I actually specifically left out conda environments and asked about virtual environments. 😁
Show threadSebastian RaschkaDec 20, 2022
@brettcannon but conda creates & manages virtual environments ^^. So you were wondering about pure Python tools like virtualenv/venv/pyenv/poetry in particular?
Show threadBrett CannonDec 20, 2022
@SebRaschka I would argue conda creates conda environments, which are not the same as virtual environment (i.e. what the `venv` module creates): https://docs.conda.io/projects/conda/en/latest/user-guide/concepts/environments.html . For instance, conda environment require you activate them while virtual environments do not.
Conda environments — conda 22.11.1.post15+d63470c41 documentation

Show threadThomas A Caswell Dec 20, 2022

@brettcannon @SebRaschka I'm with Sebastian on this, the difference between conda envs vs venvs vs {the other tools} is thin.

I'm 95% sure you can get away without activating conda envs in most cases (if you do not have any activate scripts or extra env exports). You lose some of the self-description that you _are_ using a conda installed Python, but 🤷

I somewhat regularly do `/path/to/conda/env/bin/python blah` and have not had any issues (but now I'm a bit worried....)

Show threadBrett CannonDec 20, 2022
@tacaswell @SebRaschka My experience with VS Code suggests skipping activation is not an option in general. For example, Python 3.10.0 was released for conda which would not load without an activated environment (couldn't load zlib I think; got fixed within a month). People also expect to be able to configure things via `conda activate`.
Show threadThomas A Caswell Dec 20, 2022

@brettcannon @SebRaschka 😱

I remember some early discussion about those start up scripts / envs with Aaron Meurer arguing they were a Bad Idea ™️ .

Seems he was right (and I was wrong).

Show threadSebastian RaschkaDec 20, 2022
@brettcannon @tacaswell I remember putting the Python executable in my SLURM script and was able to run code without having to activate like Thomas said. Regarding the definition m, I see “conda activate” as a syntax/implementation-details whereas a virtual environment is more of a broader concept for having a separate virtual space next to your main installation.
Show threadThomas A Caswell Dec 20, 2022

@SebRaschka @brettcannon The problem is conda packages can install on activate scripts and ENVS exports so while not strictly needed for _Python_ to work (modulo the zlib bug), if you are a tool like vscode it will break a lot of users expectations that their conda installed custom-what ever well be run!

I can totally see people shipping things like ENVS for access / configuration in a conda package that exports them on activate (because I, ah, tried to do that..my coworker talked me out of it)

Show threadSebastian RaschkaDec 21, 2022
@tacaswell @brettcannon Thanks for the explainer!
Show threadCarlton Gibson 🇪🇺Dec 20, 2022
@brettcannon two reasons: nice to see all the venvs that are around, and I don’t want the venv in the project folder (I have cdvirtualenv and cdsitepackages for the times I do actually want to rummage)
Show threadBrett CannonDec 20, 2022
@carlton What would you think if there was a symlink from `.venv` back to the virtual environment? How about a file that listed the directory of the virtual environment?
Show threadCarlton Gibson 🇪🇺Dec 20, 2022
@brettcannon I’d probably find a symlink noisy, but I’d have no problem at all with a file (a dot file) saying where the venv was. (That would help tools with discovery I’d imagine)
Show threadBrett CannonDec 20, 2022
@carlton That's one of my thoughts on this; see https://fosstodon.org/@brettcannon/109547636668367144 for a discussion I'm having on this very idea.
Show threadCarlton Gibson 🇪🇺Dec 20, 2022

@brettcannon Hmm 404 there right now.

(It's my 🛌 time so I'll check in the morning.)

Thanks.

Show threadBrett CannonDec 20, 2022
@carlton If it keeps 404'ing for you, it's a conversation with Simon that came off the poll, so it's easy to find
Show threadCarlton Gibson 🇪🇺Dec 21, 2022
@brettcannon i think the visibility must be set to mentioned only. No worries. I’ll imagine 😜
Show threadBrett CannonDec 21, 2022
@carlton It's basically your suggestion: a file that points to the environment. Toyed with reusing the `.venv` name and `.venv-path`.
Show threadשי ברגר   Shai BergerDec 20, 2022
@brettcannon @carlton
I'm putting my followup comment for "other reason" here:
WDYM *The* virtual environment?
In some of my projects, I use more than one (e.g. a long term project where I've had to work on env upgrade while the old env is still in use in production).
That need grew into a general habit, I'm used to workon. The idea of switching a virtualenv because I changed directory seems scary.
Show threadBrett CannonDec 21, 2022

@shaib @carlton There will be a follow-up poll around multiple environments per project; one thing at a time. 😉

As for switching per directory, that's entirely how I work and I think it's great since I never have to worry about using the wrong environment for the project. I have one primary environment I directly dev in and then let nox handle testing against other versions. If I have to create a quick environment then it's cheap since the wheels are already cached from nox installing them.

Show threadשי ברגר   Shai BergerDec 21, 2022
@brettcannon @carlton
I should maybe try that some day
Show threadCarlton Gibson 🇪🇺Dec 21, 2022
@shaib @brettcannon Yes, this too. I use tox (and the devenv option) a lot for this, but also have named venvs managed with virtualenvwrapper.
Show threadCarlton Gibson 🇪🇺Dec 20, 2022
@brettcannon also I use mktmpenv (which does what you imagine) a lot, and then it is handy for them to be in a know place (for when I forget to deactivate to clean them up)
Show threadNeil TenenholtzDec 20, 2022
@brettcannon workon, deactivate, mkvirtualenv, and rmvirtualenv have nicer ergonomics than venv IMO.
Show threadBrett CannonDec 20, 2022
@ntenenz So it's because of a tool default then.
Show threadNeil TenenholtzDec 20, 2022

@brettcannon Largely yes, though I'd probably do the same even if weren't, assuming easy configurability, due to environment sharing and inability to accidentally commit the venv.

In other words, I'm a vote for "combination of multiple choices."

Show threadSimon WillisonDec 20, 2022
@brettcannon I keep all of my active projects in a Dropbox folder, and I don't want the virtual environments to get backed up there - so I use "pipenv shell" for them purely as a tool to keep the virtual environments in a separate are of my disk where Dropbox can't see them
Show threadSimon WillisonDec 20, 2022
@brettcannon Dropbox here is acting as my insurance policy against my laptop being stolen before I've managed to commit and push to GitHub!
Show threadDoug HellmannDec 20, 2022
@brettcannon most of the virtualenvwrapper features were easier to implement with the envs all in one place
Show threadBrett CannonDec 20, 2022
@doughellmann That is definitely a niche reason for someone to have 😉
Show threadDawidDec 20, 2022

@brettcannon I was annoyed when I forgot to activate venv in terminal, or switch to another one. So I wrote a simple tool for activating and deactivating venv automatically basen on current path. And I just decided to keep all those venvs in one place.

Here's link for powershell version: https://github.com/dejvidq/autoenv-ps1
I need to merge this repo with second one I created for zsh version.

GitHub - dejvidq/autoenv-ps1

Contribute to dejvidq/autoenv-ps1 development by creating an account on GitHub.

GitHub
Show threadBrett CannonDec 20, 2022
@dawid This question is for https://github.com/brettcannon/python-launcher as I'm trying to make it so people don't have to do these sort of one-off solutions for this problem.
GitHub - brettcannon/python-launcher: Python launcher for Unix

Python launcher for Unix. Contribute to brettcannon/python-launcher development by creating an account on GitHub.

GitHub
Show threadDawidDec 20, 2022
@brettcannon I love the idea! I definitely need to check it! 🙂
Show threadChip WardenDec 20, 2022
@brettcannon With `.venv` in the project directory, someone will sometime, somehow, no matter how many precautions are taken, check the virtual environment into version control. I have seen this happen multiple times at multiple employers.
Show threadBrett CannonDec 20, 2022

@lgw4 We have seen this for the Python extension for VS Code. This is why we drop a `.gitignore` file containing just `*` when the extension creates a virtual environment via its `Create Environment` command.

Does the tool you use tie the environment back to the project directory somehow? The motivator for me asking this is how does one know what environment to use for a project directory?

Show threadChip WardenDec 20, 2022

@brettcannon For personal use, I switch between `virtualenvwrapper` and `virtualfish`. My team at work selected Poetry, but configured to use `~/.virtualenvs` for virtual environments.

I guess that is a long way of saying "yes."

Show thread64kbJan 7, 2023
@brettcannon @lgw4 virtualenvwrapper has a way of associating it.
In zsh I often use an extension to automatically set it on cd... It only bites me in the times when I use a secondary environment for the same source directory and navigate in and out of it
Show thread64kbJan 7, 2023
I guess I do the reverse with virtualenvwrapper, workon will send me to the project dir.
I think there is a file like .project that gets put in the virtualenv.
Show thread64kbJan 7, 2023
@lgw4 @brettcannon yep, having it in the source dir leads to pain at some point in the process.
Show threadAruDec 20, 2022

@brettcannon two reasons:

1. I frequently use environments outside of the root directory for a project. `workon project` saves me from needing to remember where I am to source the venv.
2. Relatedly, when I use `git worktree` I don't need to have a venv/ dir per root.

Show threadThomas A Caswell Dec 20, 2022

@brettcannon Similar to @ogimoore I tend to keep a couple of environments with different versions of Python around.

I also do mostly library development so I'm interested in how changes to library A affects library B and how both interact in application C.

Under this workflow it in very non-obvious which of those source checkouts should "own" the env and installing things N**2 times (each project in each other projects env) seems overkill.

Maybe this should have been "sharing/reuse", but 🤷

Show threadThomas A Caswell Dec 20, 2022
@brettcannon @ogimoore I also make a lot of one-off envs (in /tmp 😱 so reboot clears them out for me) to test _very_ specific versions sets to try and reproduce bug reports. I do not want those to mess with day-to-day environment or any of the work I am doing.
Show threadpowersoffourDec 20, 2022
@tacaswell @brettcannon @[email protected] Hey, this is a really helpful trick! Thank you for sharing it and I am so glad I saw it!
Show threadBrett CannonDec 20, 2022
@tacaswell @ogimoore Ditto on /tmp usage. It's actually how `pipx run` does things.
Show threadBrett CannonDec 20, 2022

@tacaswell I would have put that under "sharing/reuse" as that's the primary reason you have structured things this way.

Does this mean you typically have a checkout associated with a single environment? Or at least only one environment per Python version? IOW how could a tool ever know which environment you meant to be used (potentially by Python version)?

BTW I'm also happy with, "it's not possible" as you're obviously a power user. 😅

Show threadThomas A Caswell Dec 20, 2022

@brettcannon I tend to have my "bleeding" environment with I build from the main branch of as many things as I can (starting at CPython and going up), and then a bunch `dd3x` environments per python version (via Arch + AUR I have py34 - py311 available).

I think of envs as a per-terminal thing and tend to have multiple terminals open. I will (editable) install a checkout into more than one venv if I have to and select the venv by what terminal I am on (independent of the cwd).

Show threadThomas A Caswell Dec 20, 2022
@brettcannon I guess the tool knows what env I want because I tell it 🤣
Show threadBrett CannonDec 20, 2022
@tacaswell Yep, you're ending up in the "I can't help make your life easier" camp 😉
Show threadThomas A Caswell Dec 20, 2022

@brettcannon That is one step better than I am doing to myself were I keep trying to make my life harder: https://github.com/tacaswell/build_the_world

But seriously, please do not make this sort of workflow _worse_.

Super opinionated tools are great until they are utterly unusable. Assuming all Python development is fundamentally application-like and propagating those patterns as general Python "best practices" worries me (and I do not think writing off all library developers as "power users" is fair either).

GitHub - tacaswell/build_the_world: Scripts to build much of the Scipy ecosystem from source.

Scripts to build much of the Scipy ecosystem from source. - tacaswell/build_the_world

GitHub
Show threadBrett CannonDec 20, 2022

@tacaswell I'm not sure what hat you think I'm wearing here, but it's for https://github.com/brettcannon/python-launcher which is a personal project, so you don't have to use it. And even in context of the Python Launcher for Unix, I'm thinking of https://github.com/brettcannon/python-launcher/discussions/168 and https://github.com/brettcannon/python-launcher/discussions/222 which increases the flexibility to potentially support you.

But I also can't write code to read minds, so I'm trying to figure out how much people's workflows require thought vs just happen to lack some easy automation.

GitHub - brettcannon/python-launcher: Python launcher for Unix

Python launcher for Unix. Contribute to brettcannon/python-launcher development by creating an account on GitHub.

GitHub
Show threadThomas A Caswell Dec 21, 2022
@brettcannon Your wearing the Brett Cannon hat so lots of people will listen to you :)
Show threadBrett CannonDec 21, 2022

@tacaswell Mastadon might have more characters to use than some of us are used to, but it still isn't infinite to always explain which 🤠 I'm wearing. 😉

I'm unfortunately also being a bit vague on purpose as I want to conceptually understand the environment workflow people have instead of what tools people use. I want the "what do I need" answers instead of "what I use".

Show threadDan SloanDec 20, 2022

@brettcannon I tend to avoid putting venv in-project since I started doing some work on windows; sometimes I’m in WSL, sometimes I’m in windows native. Keeping the venv globally avoids conflicts.

I also find myself using multiple python versions for one project lately, so I often use pyenv’s virtualenv wrapper, which stores venvs globally and lets you refer to it with an in project .python-version file

Show threadBrett CannonDec 20, 2022
@LucidDan Does a project associate with a single environment (can be one per Python version)? I.e. how could a tool know what environment you wanted to use with your project?
Show threadDan SloanDec 20, 2022

@brettcannon with pyenv, the .python-version file points to the venv name (which is a directory somewhere in $HOME/.pyenv). Pyenv auto activated the right venv when you cd into the project where the python-version file is.

If I want to switch to another venv with another python ver I just “pyenv use <venv name>” which temporarily activates that instead. This works for me as I usually just want to use other versions a briefly to test things

Show threadDan SloanDec 20, 2022
@brettcannon if I’m not using pyenv, I’m generally relying on Pycharm to keep track for me. And then I have to switch interpreter in the project settings to change versions. I still tend to use a global dir for those envs though, partly out of habit I guess…
Show threadBrett CannonDec 20, 2022
@LucidDan So either you're using a tool default for pyenv or you just have to do it manually.
Show threadDan SloanDec 20, 2022
@brettcannon yes. I would make the distinction (based on the poll wording) that I use the tool default because it suits my purposes rather than just because it’s the tool default. Hence the vote for “other”.