Adding auditing to pip

A tool to discover known security vulnerabilities in the Python packages installed on a system or required by a project, called pip-audit, was recently discussed on the Python discussion forum. The developers of pip-audit raised the idea of adding the functionality directly into the pip package installer, rather than keeping it as a separately installable tool. While the functionality provided by pip-audit was seen as a clear benefit to the ecosystem, moving it inside the pip "tent" was not as overwhelmingly popular. It is not obvious that auditing is part of the role that the package installer should play.

Background

In late July, Dustin Ingram proposed adding pip-audit to pip, as a subcommand (i.e. pip audit). As part of Ingram's work on the Google open-source security team, he and a group from Trail of Bits developed pip-audit, which was created with the hope it could eventually be merged into pip itself. Doing so would help meet the goals they set out with:

Our ultimate goal is to make a) useful vulnerability tooling that is b) free to use, c) community owned and operated and and d) canonical and easily available to every Python user. We've already achieved a) and b) and to some extent c) (the project is open-source, and we plan to request a transfer to the PyPA [Python Packaging Authority] org) but we think the most effective way to achieve d) is by making pip-audit a subcommand of pip itself, due to pip's wide user base.

The post was seeking feedback from the pip developers and Python community on the tool, the idea of adding it to pip, and rough roadmap he had created for integrating pip-audit into pip. The tool was carefully designed with that integration in mind, so there should be minimal disruption if it were to happen, Ingram said. The command-line interface is compatible with pip and the pip-audit code could simply be "vendored" (copied) into pip repository; the latter would allow pip-audit development and maintenance to continue in its existing repository if that was deemed desirable.

The purpose of pip-audit is to look at the packages that have been installed from the Python Package Index (PyPI) and determine which of them have known security vulnerabilities. The PyPA maintains an advisory database that stores vulnerabilities affecting PyPI packages in YAML format. For example, pip-audit reported that the version of Babel on my system is vulnerable to PYSEC-2021-421, which is a local code-execution flaw. That PYSEC advisory refers to CVE-2021-42771, which is how the flaw is known to the wider world.

As it turns out, my system is actually not vulnerable to CVE-2021-42771, as the Ubuntu security entry shows. The pip-audit tool looks at the version numbers of the installed PyPI package to decide which are vulnerable, but Linux distributions regularly backport fixes into earlier versions so the PyPI package version number does not tell the whole story—at least for those who get their PyPI packages from distributions rather than via pip.

But pip-audit is clearly geared for installations where Python packages do come from PyPI, perhaps installed in multiple versions in separate virtual environments to support the needs of different applications, libraries, and frameworks. It can also be used to examine a project's "requirements" file, which lists the packages and versions the project depends on, to see if any of those packages (or those they depend on) have known vulnerabilities, which entails retrieving and unpacking the packages in question and, recursively, their dependencies. That check could be added as a step in the continuous integration (CI) pipeline, for example, or run as a test before a change is committed to a repository. When the tool finds vulnerable packages, it can even auto-upgrade them to fixed versions with pip‑audit ‑‑fix.

The tool is obviously useful, but, of course, it is not magic, as the security model section of the documentation points out. "TL;DR: If you wouldn't pip install it, you should not pip audit it." It is important to note that pip-audit cannot defend against malicious PyPI packages or their requirements files; using the -r file option to check the dependencies of a package may seem harmless, but:

In particular, it is incorrect to treat pip‑audit ‑r INPUT as a "more secure" variant of pip-audit. For all intents and purposes, pip‑audit ‑r INPUT is functionally equivalent to pip install ‑r INPUT, with a small amount of non-security isolation to avoid conflicts with any of your local environments.

Response

After Ingram's post, the response was quite favorable about the tool, but some did not see it as something that belonged in pip. Bernát Gábor said: "I personally feel that pip does one thing: discover, download and install packages. IMHO the audit feature sounds very useful but orthogonal to its goal." He noted that the auditing should not be dependent on the installation mechanism, so the tool should not really be tied to pip.

Both Pradyun Gedam and Donald Stufft were in favor of turning pip-audit into a pip subcommand; neither thought that maintaining it in its own repository was the right choice, however. Stufft said that it might make sense to look at creating a library for pip that other tools can use; right now, pip-audit uses pip-api which wraps the pip command line to work around the lack of an importable pip library. If pip-audit moves into pip, it can use the internal pip API, but there may be other tools that would benefit from such a library, he said.

User "fungi" guessed that the end goal is to do the auditing as part of the installation process, but for the functionality to also be available for audits after installation in order to catch vulnerabilities that are found later. If so, having the code in two different places would be cumbersome. But Gábor was strongly against tying installation and auditing together:

If I ask an installer to install a package, I'd not like to get a security audit, just install it. I'll handle security auditing in parallel when I need it. I'd rather not make pip even slower if that's possible.

Paul Moore wondered where the idea of running an audit at install time came from; he was opposed to that idea for the same reasons as Gábor. Beyond that, though, he was not really in favor of moving pip-audit into pip; he did not think the proposal described the advantages for pip all that well. He said: "pip is large and complicated enough already, it's up to you, if you're suggesting we add this to pip, to explain why we should be willing to consider it." If audit does get added to pip, however, he had another good reason to move the code into pip as well:

[...] trying to make it work as both standalone and part of pip will be suboptimal for both. Let's choose one and stick with it. Apart from anything else, I don't think we want to be in a position where pip-audit gets fixes added and released and they take up to 3 months to get into pip (which is what vendoring could result in) - that just gives a bad message about using the pip subcommand rather than the standalone version.

Ingram replied to multiple pieces of those messages. He noted that "running an audit at install time is explicitly not a goal for pip audit"; the npm package manager for JavaScript has a similar feature that has not worked out well. Meanwhile, the main reason for wanting the functionality in pip was mentioned in the goals (quoted above), he said; putting it in pip would make it readily available to every Python user right out of the box.

But Moore was not convinced that was a good argument for adding it to pip; "other pip maintainers may disagree, but I think pip is already overloaded with functionality and we should be streamlining, not adding more". Stufft noted that there are a lot of pip subcommands that already start down the path toward pip audit, however, such as pip list and pip show; he said that if those were added to pip, it is hard to see "an objective criterion" that would reject pip audit.

Stéphane Bidoul is concerned with further burdening the pip team with a large feature of this sort even though he sees pip-audit as a "very useful and important feature" that is worth making ubiquitously available. He wondered if there were other ways to make that happen, perhaps by automatically installing pip-audit with pip or by adding a plugin system to pip similar to the one Git has. There are dangers to adding it directly into pip, however:

My (very personal) feeling is that, at this point in time, the pip team is so small, and there is so much to do to just to make the existing pip features consistent with each other, to support new standards and new python versions, to deprecate legacy behaviours, etc… that I tend to cringe at the idea of adding new large features that could live outside, since just the review would divert us.

Moore agreed with that assessment; he said that more help in maintaining pip is needed, so that if the pip-audit maintainers wanted to pitch in, that might change the equation. The people working on a project obviously help shape its direction, but he would want new maintainers to "feel responsible for pip as a whole" and not only on the audit piece. Ingram said that he could potentially help out, but that he definitely had money available in his budget to pay for development and maintenance, which might be a path forward. He stressed that the pip-audit team intended to continue maintaining the code wherever it ended up landing and that the intent had always been to eliminate "any increased work on any existing pip maintainer (aside from what would be unavoidable to land the feature)".

There are plans for a Python Enhancement Proposal (PEP); the discussion and decision on that, once it appears, should help clarify things. As Sumana Harihareswara put it, the right location for the tool comes down to user expectations: "what do users want/expect when they're working with pip, and how much additional friction would it cause for them if they have to invoke the audit command one way versus another way?" Other package managers do have audit capabilities, as pip-audit co-maintainer William Woodruff pointed out, but there are additional arguments to bolster the case for pip audit:

I think of pip as a "package management system." It already has functionality that's strictly outside of package installation, but all current functionality (AFAIK) has something to do with querying, managing, or checking the state of Python packages and package distributions.

From there, I think pip-audit would qualify for subcommand inclusion on the basis that (1) it operates entirely on the same objects and state as pip ordinarily does (i.e., it does not broaden the scope of things pip concerns itself with, even though it adds new code), and (2) it exposes functionality that exists in other package management ecosystems, like npm and cargo.

I recognize, however, that "other package tools do it!" is not necessarily a precedent that the pip maintainers wish to establish. But I think the combination of prior art and the limited domain of interest make pip-audit a reasonable candidate in particular for inclusion.

We'll have to wait to see how things play out from here, but at a minimum, there is a new tool available to help developers and administrators find any known vulnerabilities in their PyPI packages (and the dependencies thereof). It needs to be installed separately for now, but that does not limit its usefulness, really, just its visibility. Meanwhile, there are a quite few discussions of proposals for new Python packaging functionality floating around on discuss.python.org these days, so it is an active area of development. Like every other project, though, PyPA, pip, and other tools all suffer from a lack of contributors, so some hard decisions on which to pursue are going to have to be made.