Revisiting PEP 394

With the uptake of Python 3 (and the imminent end of life for Python 2.7), there is a question of which version of Python a user should get when they type "python" at the command line or have it as part of a shebang ("#!") line in a script. Back in 2011, PEP 394 ("The 'python' Command on Unix-Like Systems") was created as an informational PEP that relayed the recommendations of the Python core developers to Linux distributions and others in a similar position about which version to point python to. Now, Petr Viktorin, one of the authors of the PEP, would like to revisit those recommendations, which is something that is suggested in the PEP itself.

Viktorin is concerned that the recommendations essentially require a distribution to include Python 2 if it wants to install a "python", in addition to explicit "python2" and/or "python3" commands. The current recommendations are that, if there is a python, it should point to the same place as python2, and that scripts which are compatible with both Python 2 and Python 3 may continue to use python on their shebang lines. That effectively means a distribution with shiny new Python 3 scripts (which also support Python 2) needs to ship Python 2, which is not what some distributions want.

The main issue is that some distributions are supported for ten or more years, but Python 2, in the form of Python 2.7, will reach its end of life on January 1, 2020—just ten months away. That leaves those distributions with some tough choices. They can either break with PEP 394 by making the version invoked by python configurable, for example. Or they have to ship scripts that could happily work on Python 2 with shebang lines that will invoke Python 3 (e.g. #!/usr/bin/python3).

As Victor Stinner pointed out, this problem has been discussed before, including at the 2018 Python Language Summit. Both Viktorin and Stinner work for Red Hat, so their perspective is largely focused on Fedora and Red Hat Enterprise Linux (RHEL). Stinner also noted a pull request (PR) that Viktorin filed for PEP 394 that would clarify some of the language in it. In a comment on the PR, Viktorin said that where python points is configurable for the RHEL 8 beta:

The eventual intent, at least according to a comment by Guido van Rossum on an earlier PR, is that python won't actually point anywhere because it won't exist; users will need to explicitly choose either python2 or python3. That is part of what Viktorin's PR is aiming for as well. He wants to make two changes to the PEP: allow installing a "python" (or, an "unversioned Python", as he calls it) to be optional for a distribution and to recommend that scripts supporting both Python 2 and Python 3 use python3 in their shebang lines. That last is a bit counterintuitive, since it means that those scripts will really only run under Python 3, regardless of their ability to run under either version. It is, Viktorin said, "the least bad option".

Some would like to see the unversioned Python point to Python 3, either now or soon, however. Antoine Pitrou pointed out that users of the conda package management tool for Python have been using unversioned Python as Python 3 for several years. Stinner said that he advocates unversioned Python be the same as the latest Python version. There is a problem with installing Python without providing an unversioned binary or link, he said: the documentation refers to python in lots of places.

Barry Warsaw agreed that it was desirable to have the unversioned Python be the latest Python 3 version, though it may be too soon for the PEP to recommend that. But both Stinner and Warsaw also brought up another potential solution: the Python Launcher for Unix that Brett Cannon is working on. It is modeled on the "py" launcher for Windows, which was specified in PEP 397 ("Python launcher for Windows"). py finds the "right" Python interpreter to invoke based on its configuration, command-line arguments, and any shebang line.

Cannon's version of the launcher is written in Rust (and is available as a crate). Since there is no legacy of using it on Unix systems, distributions would be able to point it at the version that makes the most sense for them. Users and administrators could still change the default behavior as they wished as well. Cannon said that py is still missing some basic functionality, but he is working on rectifying that. He intends to keep working on it even if others do not find it all that useful:

Unlike nearly everyone else commenting in the thread, Gregory P. Smith did not think py is a solution to the problem. In his view, any distributions that are not installing some unversioned Python, when the language is used for its core functionality (e.g. various administrative scripts), has "done something undesirable for the world at large". That is because that makes it impossible to write a shebang line that invokes the latest version of Python installed and will work across all of the distributions. There are "hacks" that can be done to work around that problem and those should be adopted, he said. As he forecast, though, that wasn't a wildly popular approach.

As he did at the language summit, Matthias Klose represented the Debian and Ubuntu Python packagers in the thread. He described the plans for unversioned Python; for Debian, python is never planned to point to Python 3, while Ubuntu has not made a final decision, but it currently does not install an unversioned Python. Debian will continue to have python point to Python 2 until it no longer ships Python 2, then remove it entirely. Klose is trying to ensure that, for upcoming distribution releases starting in 2020 or 2021, Python scripts in both Debian and Ubuntu packages use explicit shebang lines.

Klose does not think there is any way to solve the problem that Smith is complaining about: "there is *no* way to have a sane way to have what you want". Smith more or less concurred:

Nick Coghlan agreed with Smith that not having an unversioned Python "does indeed break the world (search for Fedora 28 and Ubuntu 16.04 Ansible Python issues for more)". But he thinks it is time to consider changing the stance of the PEP. The only thing that is stopping Fedora from doing what it thinks is the right thing (pointing the unversioned Python at Python 3) is the PEP:

He pointed out that Viktorin is not asking to change the recommendation to follow Fedora's lead, "just for the PEP to acknowledge it as a reasonable choice for distros to make given the looming Python 2 End of Life". Warsaw thought that made sense:

Both Coghlan and Warsaw, along with Cannon, Van Rossum, and Carol Willing, are members of the new steering council, which means they will be participating in determining the outcome of the changes to this PEP. More importantly, perhaps, both are also co-authors of the PEP (with Viktorin and Kerrick Staley), so their opinions clearly matter. So far, Van Rossum has not commented in the thread or on the PR. It has been almost a year since he was opposed to the last effort to change the PEP. A lot of water has gone under the bridge since then, so he might be more inclined to accept these changes (or a subset of them)—or simply to defer to the other members of the council.