Ben Lindsay#Python question: When installing python using the official downloader on python.org, it sets up a symlink from /usr/local/bin/python3 to /usr/local/bin/python3.11, but it doesn't add a symlink from /usr/local/bin/python to /usr/local/bin/python3. Does anyone know why not? I'm an experienced python user, but this kind of threw me for a loop, because after doing the official python installation, `which python` still shows /usr/bin/python
For people who use this method, (@brianokken I think I've heard you promote it?) do you add that python ->python3 symlink manually or just type python3 everywhere or something else? I feel like I shouldn't have to type python3, since that's the default assumption these days, but I'm not sure if adding a symlink in /usr/local/bin will have side effects with the python installer or #homebrew or something else
Ed 👾🎮💻@benlindsay I usually create an alias to solve this problem.
@ede That makes sense. The part that gets me though is that I keep hearing the recommendation to use the official installer as the thing to recommend to beginners to help them fall into the "pit of success". Needing to create an alias for "python" to work instead of "python3" doesn't seem to fit that story
@benlindsay I think the problem is Python versioning and what ships with operating systems by default.
Many older programs may assume /usr/bin/python to be version 2 whereas /usr/bin/python3 is typically where Python 3 is installed and how it’s referenced. This includes PIP as well.
Typically, specific Python 3 versions (ie. 3.11) will symlink from /usr/bin/python3.11 to /usr/bin/python3.
Many older programs may assume /usr/bin/python to be version 2 whereas /usr/bin/python3 is typically where Python 3 is installed and how it’s referenced. This includes PIP as well.
Typically, specific Python 3 versions (ie. 3.11) will symlink from /usr/bin/python3.11 to /usr/bin/python3.
@ede That makes sense. But I think it's fair to assume that people working with legacy python2 stuff are more likely to be experienced devs who can figure out all the symlinking and version management stuff themselves, while new users are likely working with newer code where python 3 should be the default. The installation wizard should make python point to python3 by default and allow users to opt out
@benlindsay I think there may be some dependencies that may break if that happens.
Python 3 is not backwards compatible with 2 or earlier, so if a dependency exists in the environment, it may cause unexpected failures.
For example, macOS ships with Python 2.x. Replacing the symlink may cause a break for any macOS dependencies on Python 2.x.
Python 3 is not backwards compatible with 2 or earlier, so if a dependency exists in the environment, it may cause unexpected failures.
For example, macOS ships with Python 2.x. Replacing the symlink may cause a break for any macOS dependencies on Python 2.x.
Brian Okken@ede @benlindsay yep. I’m pretty sure that’s the reason. Newer macs don’t have the problem, so maybe a request to Python is in order.
It “should” only be an issue when creating virtual environments. Once in an active venv, the python link exists.
I actually like the absence of a global symlink, because it acts as a warning that I’m not in a virtual environment.
And even for new users, we should be teaching virtual environments, IMHO
It “should” only be an issue when creating virtual environments. Once in an active venv, the python link exists.
I actually like the absence of a global symlink, because it acts as a warning that I’m not in a virtual environment.
And even for new users, we should be teaching virtual environments, IMHO
@ede @benlindsay However, @brettcannon or other core devs might have better insight into the missing python symlink on macs.
@brianokken @ede @brettcannon Agreed about the importance of using and teaching virtual environments. Not sure I agree that the missing symlink is a useful warning...without the symlink, "python -m pip install" will install to the system python, and I think that's maybe more important to provide warnings or protections to avoid
Exploratory Data Alex@benlindsay @brianokken @ede @brettcannon I guess I'm a contrarian on virtual envs, but I stopped using them a year ago and the sky hasn't fallen for me. And for beginners, I think they add more complexity to the experience than the problem they solve is worth. Meanwhile pip has also gotten a *lot* better over the last couple years.
@benlindsay @brianokken @ede @brettcannon But this thread does make me wonder how much of the motivation to use virtual envs is coming from Mac-specific issues.
Robert Ritz@alexkyllo I'm on Mac (Apple Silicon) after switching from Windows. I used conda on PC and have used conda on Mac.
What got me was when JupyterLab Desktop came out. Simplified my life massively.
If I need something that doesn't work on M1 I have Docker Desktop or Deepnote.
@robertr @alexkyllo I use containers as well. It simplifies my development workflow and fixes dependency issues.
@alexkyllo @ede they are definitely overkill. Would be nice to get full M1 support for most libraries. Alas the issue isn’t Python but the many underlying C++ libraries popular Python libraries are built on.
Takes time and it will work. I would love to never use containers.
@alexkyllo @robertr I treat containers as a development environment, and use Tilt.dev for real-time coding and testing.
It’s really nice because it only rebuilds specific portions of my container when needed instead of having to rebuild everything every time I make a small change.
It’s really nice because it only rebuilds specific portions of my container when needed instead of having to rebuild everything every time I make a small change.
@alexkyllo @robertr I was writing a detailed reply and ran out of characters.
The key highlights are:
- Clean dev environment
- Versioning
- Automation
- Testing
- Portability
- Transferability
The key highlights are:
- Clean dev environment
- Versioning
- Automation
- Testing
- Portability
- Transferability
@alexkyllo @robertr A little bit of everything. My biggest use case ATM is security automation, but I write and maintain a lot of various Python-based tools.
Brett Cannon@alexkyllo @benlindsay @brianokken @ede Virtual environment are most definitely **not** motivated by macOS specifically. You've just been lucky that it hasn't bit you yet (speaking as someone who has been coding in Python before environments existed).
And I'm not touching the education angle on this one as there are arguments on both sides. Either you want to keep it simple and assume they are going to break their install anyway, or you want to start with best practices.
@alexkyllo @benlindsay @brianokken @ede If by "coming from Mac-specific issues" you meant to say "Unix", then that is _a_ motivator because Windows never shipped Python to everyone like every other OS on the planet and so you can't screw up something other software on the machine relies on (which, depending on who you talk to, consider a good thing). 😉
@brettcannon @benlindsay @brianokken @ede I've hit incompatibilities before, but I know how to resolve them and will use a virtualenv when necessary. It just usually isn't a problem for me and I don't think the reason is luck, but that I mostly use the same small set of compatible packages in each project, I want to use the latest releases of them, and most of my projects don't need long-term maintenance. I think these criteria apply to a lot of data scientists due to the nature of our work.
@brettcannon @benlindsay @brianokken @ede In other words, I don't really need like 87 copies of various versions of jupyter, numpy, pandas, matplotlib, seaborn, statsmodels, and scikit-learn on my system. These are stable packages that are designed to be compatible. And if they deprecate a function I used in an old project, it's fine because I'm willing to update my code if I need to use it again.
@alexkyllo @benlindsay @brianokken @ede If it's disk space you're concerned about then now you get to blame Windows as pip can't assume symlink support and so it has to install everything separately 😉
@brettcannon @benlindsay @brianokken @ede Disk space is part of it, but the bigger issue was the clutter of having dozens of venvs scattered around my machine, time spent waiting for installs, and overhead of making sure to always have the right ones activated in my IDE and terminal for each of the several projects I have open at any given time. Using them has benefits but also costs, so I don't blanket recommend others always use them--to me, it depends on the nature of your projects.
@alexkyllo @benlindsay @brianokken @ede If you would like your environments centralized there are various tools to support that workflow.
And I hear there's an editor that will simply remember the environment you select so it's a one-time cost (if not zero-cost if you keep the environment in your workspace) 😉
But I get it, environments are an extra step. Totally fine that it works for you, I just don't want to promote not using environments as a general solution for everyone.
@alexkyllo @benlindsay @brianokken @ede I think the fact you know how to resolve them also automatically makes you advanced. And as you said, the nature of your work may make you less susceptible to breakage, but I personally can't make that assumption is true for most Python users, nor do I want someone to come back to me later and say, "why didn't you warn me this could happen?!?"
@brianokken @ede @benlindsay https://peps.python.org/pep-0394/ is the original reason, but for macOS specifically, it probably would have broken the old Python 2 install or code that relied/assumed they got the system install. And that system install only just disappeared and I believe the installer supports older versions of macOS that still have a system install. Probably worth opening an issue at https::/github.com/python/cpython to see if it can intelligently add the symlink when safe to.
PEP 394 – The “python” Command on Unix-Like Systems | peps.python.org
This PEP outlines the behavior of Python scripts when the python command is invoked. Depending on a distribution or system configuration, python may or may not be installed. If python is installed its target interpreter may refer to python2 or python3. ...
@brettcannon @brianokken @ede Thanks for pointing me to where that issue would make sense to be opened and for confirming that it might be worth opening!