Python and deprecations redux

The problem of how to deprecate pieces of the Python language in a minimally disruptive way has cropped in various guises over the last few years—in truth, it has been wrangled with throughout much of language's 30-year history. The scars of the biggest deprecation, that of Python 2, are still rather fresh, both for users and the core developers, so no one wants (or plans) a monumental change of that sort. But the language community does want to continue evolving Python, which means leaving some "baggage" behind; how to do so without leaving further scars is a delicate balancing act, as yet another discussion highlights.

We looked in on some discussion of the topic back in December, but the topic pops up frequently. There is a policy on handling deprecations that is described in PEP 387 ("Backwards Compatibility Policy"), but the reality of how they are handled is often less clear-cut. Python has several warnings that can be raised when features slated for deprecation are used: PendingDeprecationWarning and DeprecationWarning. The former is meant to give even more warning for a feature that will coexist with its replacement for multiple releases, while the latter indicates something that could be removed two releases after the warning is added—effectively two years based on the relatively recent annual release cycle.

But, as noted in that earlier discussion, the deprecation period is for a minimum of two release cycles. There are concerns that time frame is being treated as a deadline of sorts—to the detriment of some parts of the ecosystem. So on January 18, Victor Stinner, Tomáš Hrnčiar, and Miro Hrončok proposed postponing some deprecations that had been scheduled for Python 3.11, which is due in October. The message referred to an early January posting by Hrnčiar to the Python discussion forum that described the problems Fedora had encountered when building its packages using a development version of 3.11.

In particular, two specific sets of deprecations were causing the most trouble for Fedora packages. Removing deprecated aliases from the unittest module (bug 45162) and getting rid of deprecated pieces from the configparser module (bug 45173) led to the bulk of the problems that Fedora encountered. The unittest deprecation caused 61 Fedora packages to break, while the configparser changes broke another 28. In the proposal, Stinner said that they and others had reported the problems upstream and often contributed a fix, but that there is still a lengthy process before the changes actually reach the distribution:

The problem is that fixing a Fedora package requires multiple steps:

1. Propose a pull request upstream
2. Get the pull request merged upstream
3. Wait for a new release upstream
4. Update the Fedora package downstream, or backport the change in Fedora (only needed by Fedora)

Reverting those two changes, which caused most of the problems Fedora has run into in its testing of the new version of Python, will allow for "more time on updating projects to Python 3.11 for the other remaining incompatible changes". As reported by Hrnčiar, four other changes led to problems building Python packages, but those were fewer in number.

Silencing deprecations

In a reply to the proposal, Antoine Pitrou wondered whether it showed "that making DeprecationWarning silent by default was a mistake?" He is referring to the changes to the visibility of DeprecationWarning that have occurred over the years. While DeprecationWarning is useful for the developers of a Python package, it is often seen by users, who may not be in a position to do much about it. The warnings were made invisible by default for Python 2.7 and 3.2 (in 2010 and 2011), but that policy was changed for Python 3.7 in 2017 with PEP 565 ("Show DeprecationWarning in __main__").

Guido van Rossum did not think that the evidence was quite that clear, but deprecations are tricky:

At best it shows that deprecations are complicated no matter how well you plan them. I remember that "noisy by default" deprecation warnings were widely despised.

Some ideas of further tweaks that could be made to the visibility of the warnings were raised. Richard Damon suggested having them only be visible when running unit tests. It turns out that pytest already enables those warnings, as Brett Cannon pointed out. That is something of a double-edged sword, though, Christopher Barker noted: "It's really helpful for my code, but they often get lost in the noise of all the ones I get from upstream packages." Gregory P. Smith pointed out that the standard library unit tests enable the warnings as well; "Getting the right people to pay attention to them is always the hard part."

Fixing deprecations

There was a bit of discussion about how to silence warnings from imported modules, possibly semi-automatically, but Steven D'Aprano had a bit of a warning about that approach:

If we use a library, then we surely care about that library working correctly, which means that if the library generates warnings, we *should* care about them. They are advanced notice that the library is going to break in the future.

Of course I understand that folks are busy maintaining their own project, and have neither the time nor the inclination to take over the maintenance of every one of their dependencies. But we shouldn't just dismiss warnings in those dependencies as "warnings I don't care about" and ignore them as Not My Problem.

Like it or not, it is My Problem and we should care about them.

In the world of open-source software, the lines between users and "vendors" of software are blurred, he said. Users often have the ability, and certainly have the legal right, to change the code based on observing problems of this (or any other) nature, but there is something of a social problem, "and you cannot fix social problems with technology". Ignoring warnings breaks some assumptions about how open source works:

The open source mantra about many eyes making bugs shallow doesn't work when everyone is intentionally closing their eyes to the warnings of pending bugs.

Barker said that he does try to submit fixes upstream when he notices problems of that sort, as did others in the thread. There is still the problem, mentioned by Stinner, that even once fixes are contributed, releases including them may still take a while; as Stephen J. Turnbull put it: "even if you submit a patch, there's no guarantee that the next version (or three) will contain it".

With regard to silencing DeprecationWarning, Steve Dower said that it was not necessarily a mistake to do so:

If we'd gone the other way, perhaps we'd be looking at massive complaints from "regular" end users about all the noisy warnings that they can't fix and saying that making it noisy was the mistake.

He was not opposed to reverting the changes as proposed, though he thought it might be "a bit premature" to do so now, roughly nine months before the release. They can be reverted closer to the release if the packages in question still are not fixed (and released). If they do get reverted now, because "they cause churn for no real benefit", that would be reasonable; those who are opposed can argue that the benefit is real, however, "as long as they also argue in favour of the churn". He also made a broader point:

We shouldn't pretend to be surprised that something we changed causes others to have to change. We *know* that will happen. Either we push forward with the changes, or we admit we don't really need them.

Stinner pointed to two different examples of the kinds of problems that Fedora has found by testing with development versions of upcoming Python releases. There are advantages to finding these problems as early as possible: "If issues are discovered earlier, we get more time to discuss and design how to handle them." He thinks it makes sense to revert these particularly problematic deprecations now because it will help flush out more problems further down in the dependency chain:

In Fedora, if a frequently used dependency is broken, a long list of packages "fail to build". (In Fedora, the package test suite must pass to build a package successfully.) If it takes 9 months to fix this dependency, we will likely miss other issues before the Python final version in dependent packages.

Sebastian Rittau said "that some (semi-) automated way to actively test and notify important projects of deprecations/removals before a release would be a great addition to the Python ecosystem", though he acknowledged that it might be difficult to do. Stinner replied that, in effect, Fedora is already doing that, albeit with "changes already merged in Python". He has done some work on ways to automatically test Python with patches applied, to test upcoming or proposed changes, but it turned out to be rather complicated.

Smith was also in favor of the reversions; he thanked the Fedora team for helping bring these problems to light, and noted that being proactive is a better way forward:

Deprecation removals are hard. Surfacing these to the impacted upstream projects to provide time for those to integrate the changes is the right way to make these changes stick in 3.12 or later. [...]

As you've done the work to clean up a lot of other OSS projects, I suggest we defer this until 3.12 with the intent that we won't defer it again. That doesn't mean we can't hold off on it, just that we believe pushing for this now and proactively pushing for a bunch of cleanups has improved the state of the world such that the future is brighter. That's a much different strategy than our passive aggressive DeprecationWarnings.

Toward the end of the original proposal message, Stinner had some thoughts on being even more proactive in the future. He suggested that before making an incompatible change, doing a search of the Python Package Index (PyPI) for uses of the feature in question "and try to update these projects *before* making the change". Once the number of affected projects has been reduced to some low number (he suggested 15), the change could be made in Python.

The Python ecosystem is huge, with an amazing number of projects, libraries, packages, tools, and so on, subsets of which are gathered up together into Linux (and other) distributions. All of those packages support differing ranges of Python versions, which makes the job of distributions that much harder, since they typically settle on one Python version to maintain throughout the life of a particular distribution release. Deprecating pieces along the way makes that ever more difficult, of course.

There are other software projects that take a different approach; the Linux kernel somewhat famously almost never deprecates something unless it truly can no longer be supported (e.g. ancient hardware or an API that leads to a security hole), but Python (and some other languages) have not chosen that course. There are certainly advantages to leaving things behind, especially when replacing them with something emphatically and unquestionably better, but it does have its downsides as well. It would seem that Python is drawing closer to finding the right balance when the deprecation route is taken, though there are always likely to be bumps along the way.