Testing & Packaging

How to ensure that your tests run code that you think they are running, and how to measure your coverage over multiple tox runs (in parallel!).

src

I used to scoff when I saw Python projects that put their packages into a separate src directory. It seems unnecessary and reminded me of Java. But it also seemed to be dying out.

To my surprise, cryptography – one of Python’s most modern projects – adopted a src directory subsequently (if you’re not surprised, feel free to jump ahead to Combined Coverage)!

Quick interlude from 2021: while there is still disagreement on the src topic¹, a significant number of projects moved to src since this article has been published. From the four projects above that I used to demonstrate how uncommon it is, three switched to an src layout (Flask, Pyramid, and Twisted). It has become much more difficult to find a major project with a plain layout outside scientific packages.

Another interlude from 2021: NASA landed another robot on Mars, and they use src directories in their Python code. My job here is done.

Less than a year later, I received a bug report that my tox and Coverage.py setup works by accident: my tests didn’t run against the version of my app that got installed into tox’s virtual environments.

But this is not just about tests: the same is true for your application. The behavior can change completely once you package and install it somewhere else.

All of this makes you miss packaging issues which are especially frustrating to track down: ever forgot to include a resource (like templates or translation files) or a sub-package? Ever uploaded an empty package to PyPI? That GitHub issue demonstrated to me that there’s likely more that can go wrong than I thought and that isolating the code into a separate – un-importable – directory might be a good idea².

In hindsight, it looks better to have docs, src, and tests directories in your project root – in that order! – instead of a mumbo-jumbo of directories whose purpose is unclear.

To achieve that, you move your packages into a src directory and tell your packaging build backend about it. Modern tools like Hatch, Flit, or Poetry detect it on their own – the venerable setuptools needs some help.

If you use pyproject.toml:

[tool.setuptools]
package-dir = {"" = "src"}

[tool.setuptools.packages.find]
where = ["src"]

If you use setup.cfg:

[options]
package_dir=
    =src
packages=find:

[options.packages.find]
where=src

If you use setup.py:

setup(
    [...]
    packages=find_packages(where="src"),
    package_dir={"": "src"},
)

Coping with that minor issue exposed me to a more interesting problem: measuring coverage over multiple tox runs.

Combined Coverage

If you want to support multiple Python version, chances are you have branches that are specific to certain version. What you want is to measure the combined coverage computed over all your tox runs that contribute to overall coverage.

As of Coverage.py 4.2, this is easily achieved if you run your tests directly against your package directory (i.e. without an src directory in between) by running coverage in parallel mode:

$ coverage run --parallel -m pytest tests

Then add an environment to your tox configuration that will combine and report the coverage over all runs at the end.

[tox]
envlist = py39,py310,pypy3,coverage-report

# ...

[testenv:coverage-report]
deps = coverage
skip_install = true
commands =
    coverage combine
    coverage report

It gets more complicated if your tests run against the installed version of your package, though. Because now the paths of the actually executed modules look like this:

.tox/py39/lib/python3.9/site-packages/attr/__init__.py

As they should, because we want to test what’s installed by your package and not what’s lying around in your directory!

So, you end up with a very long coverage output with all site-packages of all tox environments.

Fortunately, there is a solution in Coverage.py which is the [paths] configuration section. It allows you to tell Coverage.py which paths it should consider equivalent:

[run]
branch = True
source = attr

[paths]
source =
   src
   .tox/py*/**/site-packages

Now coverage combine will fold the coverage data of these paths together, and you get what you expect:

Name                     Stmts   Miss Branch BrPart  Cover   Missing
--------------------------------------------------------------------
src/attr/__init__.py        17      0      0      0   100%
src/attr/_compat.py         15      0      2      0   100%
src/attr/_config.py          9      0      2      0   100%
src/attr/_funcs.py          35      0     18      0   100%
src/attr/_make.py          202      0     92      0   100%
src/attr/filters.py         15      0      3      0   100%
src/attr/validators.py      33      0     12      0   100%
--------------------------------------------------------------------
TOTAL                      326      0    129      0   100%

Bonus Coverage.py tip: set skip_covered = true in your [report] section to filter out all files with 100% coverage from the output.

Speeding Up With Parallelization

This approach works, but it relies on the order in which the environments are run: coverage-report must come after all environments that measure coverage.

Therefore, if you want to run your tox environments in parallel using the --parallel option (or -p for short, not related to Coverage.py’s option of the same name), you’ll have to make sure that reporting runs separately after all other environments are done.

You achieve that by listing all coverage-measuring environments in the depends option:

[testenv:coverage-report]
deps = coverage
skip_install = true
parallel_show_output = true
depends =
    py27
    py35
    pypy
commands =
    coverage combine
    coverage report

You can use the same patterns as in you envlist.

CIs

I’ve written down how to have combined coverage on GitHub Actions without any external services.

And I’ve published a GitHub Action that builds and checks your package against the most common mistakes: build-and-inspect-python-package.