Adding auditing to pip
This article brought to you by LWN subscribers Subscribers to LWN.net made this article — and everything that surrounds it — possible. If you appreciate our content, please buy a subscription and make the next set of articles possible. |
A tool to discover known security vulnerabilities in the Python packages installed on a system or required by a project, called pip-audit, was recently discussed on the Python discussion forum. The developers of pip-audit raised the idea of adding the functionality directly into the pip package installer, rather than keeping it as a separately installable tool. While the functionality provided by pip-audit was seen as a clear benefit to the ecosystem, moving it inside the pip "tent" was not as overwhelmingly popular. It is not obvious that auditing is part of the role that the package installer should play.
Background
In late July, Dustin Ingram proposed adding pip-audit to pip, as a subcommand (i.e. pip audit). As part of Ingram's work on the Google open-source security team, he and a group from Trail of Bits developed pip-audit, which was created with the hope it could eventually be merged into pip itself. Doing so would help meet the goals they set out with:
Our ultimate goal is to make a) useful vulnerability tooling that is b) free to use, c) community owned and operated and and d) canonical and easily available to every Python user. We've already achieved a) and b) and to some extent c) (the project is open-source, and we plan to request a transfer to the PyPA [Python Packaging Authority] org) but we think the most effective way to achieve d) is by making pip-audit a subcommand of pip itself, due to pip's wide user base.
The post was seeking feedback from the pip developers and Python community on the tool, the idea of adding it to pip, and rough roadmap he had created for integrating pip-audit into pip. The tool was carefully designed with that integration in mind, so there should be minimal disruption if it were to happen, Ingram said. The command-line interface is compatible with pip and the pip-audit code could simply be "vendored" (copied) into pip repository; the latter would allow pip-audit development and maintenance to continue in its existing repository if that was deemed desirable.
The purpose of pip-audit is to look at the packages that have been installed from the Python Package Index (PyPI) and determine which of them have known security vulnerabilities. The PyPA maintains an advisory database that stores vulnerabilities affecting PyPI packages in YAML format. For example, pip-audit reported that the version of Babel on my system is vulnerable to PYSEC-2021-421, which is a local code-execution flaw. That PYSEC advisory refers to CVE-2021-42771, which is how the flaw is known to the wider world.
As it turns out, my system is actually not vulnerable to CVE-2021-42771, as the Ubuntu security entry shows. The pip-audit tool looks at the version numbers of the installed PyPI package to decide which are vulnerable, but Linux distributions regularly backport fixes into earlier versions so the PyPI package version number does not tell the whole story—at least for those who get their PyPI packages from distributions rather than via pip.
But pip-audit is clearly geared for installations where Python packages do come from PyPI, perhaps installed in multiple versions in separate virtual environments to support the needs of different applications, libraries, and frameworks. It can also be used to examine a project's "requirements" file, which lists the packages and versions the project depends on, to see if any of those packages (or those they depend on) have known vulnerabilities, which entails retrieving and unpacking the packages in question and, recursively, their dependencies. That check could be added as a step in the continuous integration (CI) pipeline, for example, or run as a test before a change is committed to a repository. When the tool finds vulnerable packages, it can even auto-upgrade them to fixed versions with pip‑audit ‑‑fix.
The tool is obviously useful, but, of course, it is not magic, as the security
model section of the documentation points out. "TL;DR: If you
wouldn't pip install it, you should not pip audit
it.
" It is important to note that pip-audit cannot defend against
malicious PyPI packages
or their requirements files; using the -r file option to check the
dependencies of a package may seem harmless, but:
In particular, it is incorrect to treat pip‑audit ‑r INPUT as a "more secure" variant of pip-audit. For all intents and purposes, pip‑audit ‑r INPUT is functionally equivalent to pip install ‑r INPUT, with a small amount of non-security isolation to avoid conflicts with any of your local environments.
Response
After Ingram's post, the response was quite favorable about the tool, but
some did not see it as something that belonged in pip. Bernát Gábor said:
"I personally feel that pip does one thing: discover, download and
install packages. IMHO the audit feature sounds very useful but orthogonal
to its goal.
" He noted that the auditing should not be dependent on the
installation mechanism, so the tool should not really be tied to pip.
Both Pradyun Gedam and Donald Stufft were in favor of turning pip-audit into a pip subcommand; neither thought that maintaining it in its own repository was the right choice, however. Stufft said that it might make sense to look at creating a library for pip that other tools can use; right now, pip-audit uses pip-api which wraps the pip command line to work around the lack of an importable pip library. If pip-audit moves into pip, it can use the internal pip API, but there may be other tools that would benefit from such a library, he said.
User "fungi" guessed that the end goal is to do the auditing as part of the installation process, but for the functionality to also be available for audits after installation in order to catch vulnerabilities that are found later. If so, having the code in two different places would be cumbersome. But Gábor was strongly against tying installation and auditing together:
If I ask an installer to install a package, I'd not like to get a security audit, just install it. I'll handle security auditing in parallel when I need it. I'd rather not make pip even slower if that's possible.
Paul Moore wondered
where the idea of running an audit at install time came from; he was
opposed to that idea for the same reasons as Gábor. Beyond that, though,
he was not really in favor of moving pip-audit into pip; he did not think
the proposal described the advantages for pip all that well. He said: "pip is large
and complicated enough already, it's up to you, if you're suggesting we add
this to pip, to explain why we should be willing to consider it.
" If
audit does get added to pip, however, he had another good reason to move
the code into pip as well:
[...] trying to make it work as both standalone and part of pip will be suboptimal for both. Let's choose one and stick with it. Apart from anything else, I don't think we want to be in a position where pip-audit gets fixes added and released and they take up to 3 months to get into pip (which is what vendoring could result in) - that just gives a bad message about using the pip subcommand rather than the standalone version.
Ingram replied
to multiple pieces of those messages. He noted that "running an audit at
install time is explicitly not a goal for
pip audit
"; the npm
package manager for JavaScript has a similar feature that has not worked out
well. Meanwhile, the main reason for wanting the functionality in pip
was mentioned in the goals (quoted above), he said; putting it in pip would
make it readily
available to every Python user right out of the box.
But Moore was not
convinced that was a good argument for adding it to pip; "other pip
maintainers may disagree, but I think pip is already overloaded with
functionality and we should be streamlining, not adding more
". Stufft
noted
that there are a lot of pip subcommands that already start down the path
toward pip audit, however, such as pip list
and pip show;
he said that if those were added to pip, it is hard to see "an objective
criterion
" that would reject pip audit.
Stéphane Bidoul is concerned
with further burdening the pip team with a large feature of this sort even
though he sees pip-audit as a "very useful and important feature
"
that is worth making ubiquitously available. He wondered if there were
other ways to make that happen, perhaps by automatically installing
pip-audit with pip or by adding a plugin system to pip similar to the
one Git has. There are dangers to adding it directly into pip, however:
My (very personal) feeling is that, at this point in time, the pip team is so small, and there is so much to do to just to make the existing pip features consistent with each other, to support new standards and new python versions, to deprecate legacy behaviours, etc… that I tend to cringe at the idea of adding new large features that could live outside, since just the review would divert us.
Moore agreed
with that assessment; he said that more help in maintaining pip
is needed, so that if the pip-audit maintainers wanted to pitch in, that
might change the equation. The people working on a project obviously
help shape its direction, but he would want new maintainers to "feel
responsible for pip as a whole
" and not only on the audit piece. Ingram said
that he could potentially help out, but that he definitely had money
available in his budget to pay for development and maintenance, which might
be a path forward. He stressed that the pip-audit team intended to
continue maintaining the code wherever it ended up landing and that the
intent had always been to eliminate "any increased work on any existing
pip maintainer (aside from what would be unavoidable to land the
feature)
".
There are plans for a Python Enhancement Proposal (PEP); the discussion and
decision on that, once it appears, should help clarify things. As Sumana
Harihareswara put
it, the right location for the tool comes down to user expectations:
"what do users want/expect when they're working with pip, and how much
additional friction would it cause for them if they have to invoke the
audit command one way versus another way?
" Other package managers do
have audit capabilities, as pip-audit co-maintainer William Woodruff pointed
out, but there are additional arguments to bolster the case for
pip audit:
I think of pip as a "package management system." It already has functionality that's strictly outside of package installation, but all current functionality (AFAIK) has something to do with querying, managing, or checking the state of Python packages and package distributions.From there, I think pip-audit would qualify for subcommand inclusion on the basis that (1) it operates entirely on the same objects and state as pip ordinarily does (i.e., it does not broaden the scope of things pip concerns itself with, even though it adds new code), and (2) it exposes functionality that exists in other package management ecosystems, like npm and cargo.
I recognize, however, that "other package tools do it!" is not necessarily a precedent that the pip maintainers wish to establish. But I think the combination of prior art and the limited domain of interest make pip-audit a reasonable candidate in particular for inclusion.
We'll have to wait to see how things play out from here, but at a minimum, there is a new tool available to help developers and administrators find any known vulnerabilities in their PyPI packages (and the dependencies thereof). It needs to be installed separately for now, but that does not limit its usefulness, really, just its visibility. Meanwhile, there are a quite few discussions of proposals for new Python packaging functionality floating around on discuss.python.org these days, so it is an active area of development. Like every other project, though, PyPA, pip, and other tools all suffer from a lack of contributors, so some hard decisions on which to pursue are going to have to be made.
Index entries for this article | |
---|---|
Security | Package repositories |
Security | Python |
Python | Security |
(Log in to post comments)
Adding auditing to pip
Posted Aug 10, 2022 12:42 UTC (Wed) by LtWorf (subscriber, #124958) [Link]
Adding auditing to pip
Posted Aug 10, 2022 16:13 UTC (Wed) by adam820 (subscriber, #101353) [Link]
Adding auditing to pip
Posted Aug 10, 2022 22:13 UTC (Wed) by iabervon (subscriber, #722) [Link]
Adding auditing to pip
Posted Aug 11, 2022 12:52 UTC (Thu) by Saphyel (guest, #160208) [Link]
* https://php.watch/articles/composer-audit
* https://docs.npmjs.com/cli/v6/commands/npm-audit
* https://lib.rs/crates/cargo-audit
If you don't like standards you can always create your own alias so don't make it harder for the rest of the world.
Adding auditing to pip
Posted Aug 12, 2022 13:42 UTC (Fri) by patrakov (subscriber, #97174) [Link]
Adding auditing to pip
Posted Aug 11, 2022 9:52 UTC (Thu) by cortana (subscriber, #24596) [Link]
* dnf updateinfo --list --security
* apt-listbugs
I'm not a fan of either of these...
Adding auditing to pip
Posted Aug 11, 2022 14:04 UTC (Thu) by dfc (subscriber, #87081) [Link]
Adding auditing to pip
Posted Aug 12, 2022 2:54 UTC (Fri) by pabs (subscriber, #43278) [Link]
Naming is Hard: These Subpackages Have CVE's
Posted Aug 14, 2022 15:02 UTC (Sun) by k3ninho (subscriber, #50375) [Link]
The action isn't an audit, like you say. It's "list CVE's in subpackages" so maybe we ask 'pip tell-me-CVEs'. Maybe it's "tell me the risk I'm exposed to" so we can call it 'pip risk-log'.
Other ecosystems don't describe (or prescribe) what you do with them information gleaned from their vulnerability databases. The workflow should be made plain for people new to vendoring-in huge codebases to make a system from other people's work: replace-and-upgrade, ignore-for-now, suggest mitigations, migrate-to-alternative and more. Reviewing some imported code is considered too time-consuming, but also the risk we all expose ourselves to makes it kinda 'too expensive to **not** review'.
K3n.
Can it find typosquatting and similar packages?
Posted Aug 10, 2022 14:28 UTC (Wed) by southey (guest, #9466) [Link]
Every package you install increases your risk
Posted Aug 11, 2022 19:08 UTC (Thu) by petiepooo (guest, #153884) [Link]
While I understand the value of the audit tooling, I feel the drive to make it ubiquitous by integrating it into pip could ultimately increase people's exposure to risk rather than decrease it. If you wish to audit a production environment, duplicate it in a lab and run your audit there. You already have a lab that duplicates your production environment, right?
Every package you install increases your risk
Posted Aug 14, 2022 15:22 UTC (Sun) by Wol (subscriber, #4433) [Link]
> If you wish to audit a production environment, duplicate it in a lab and run your audit there. You already have a lab that duplicates your production environment, right?
WHAT production environment? WHO is paying for the lab?
I know there are a lot of people who are paid to write this stuff, but there are a lot of people who USE it in a *personal* capacity, and there are a lot of people who *write* it in a personal capacity.
If I'm writing/maintaining this stuff, I would love that sort of functionality. I have ONE powerful workstation / home-server, and I don't have the time, or money, to faff about trying to sort out a duplicate.
One *massive* advantage of this tool is that it would enable - in short order - popular packages to eliminate dependencies on abandoned packages. By reducing the amount of work needed to identify them, and massively increasing their visibility. A lot of the time people don't care because caring costs too much. Reduce that cost, and suddenly people will see the effort as worth while.
Cheers,
Wol
Every package you install increases your risk
Posted Aug 14, 2022 16:13 UTC (Sun) by amacater (subscriber, #790) [Link]
Every package you install increases your risk
Posted Aug 14, 2022 16:30 UTC (Sun) by Wol (subscriber, #4433) [Link]
But my previous system it would have been a lot harder. My current system is still not quite how I want it - for personal reasons it's hard to find time - but the old system didn't really have the disk space or ability to run VMs or anything and I didn't have the knowledge ... I'm still learning :-)
But it's the assumption that the people who want and could make use of this tool are the same as the people who have all that tooling and resources that gets me - I know your "lone coder" is now much less of a reality than they were, but those people don't necessarily have those resources, and even me, I'm not officially a coder even though in practice I'm turning into a VBA guru (yuck! :-), and if I want those resources I have to pay for them out of my own cash. Unless you live in the West, how many people can afford that?
Cheers,
Wol
Adding auditing to pip
Posted Aug 16, 2022 10:29 UTC (Tue) by amarao (guest, #87073) [Link]
Adding auditing to pip
Posted Aug 16, 2022 12:37 UTC (Tue) by kleptog (subscriber, #1183) [Link]
If your security team can't accept that, then that's their problem. Not every CVE is equally important or equally relevant.
Adding auditing to pip
Posted Aug 16, 2022 13:52 UTC (Tue) by amarao (guest, #87073) [Link]
If there is CVE out there, how to make it 'not CVE'? I know no such process.
Adding auditing to pip
Posted Aug 17, 2022 0:31 UTC (Wed) by pabs (subscriber, #43278) [Link]
https://nvd.nist.gov/general/cve-process
https://nvd.nist.gov/vuln/vulnerability-status
Adding auditing to pip
Posted Sep 28, 2022 5:34 UTC (Wed) by onurozkan (guest, #153789) [Link]