navigation

testmon: selects tests affected by changed files and methods

pytest-testmon is a pytest plugin which selects and executes only tests you need to run. It does this by collecting dependencies between tests and all executed code (internally using Coverage.py) and comparing the dependencies against changes. testmon updates its database on each test execution, so it works independently of version control.

Installation and Quickstart

# install
pip install pytest-testmon

# build the dependency database and save it to .testmondata
pytest --testmon

# change some of your code (with test coverage)

# only run tests affected by the changes
pytest --testmon

Before testmon can select the right subset of your tests you have to run all your tests with the –testmon option.

If you want to read more about how testmon collects and compares changes we wrote about it here.

If you want to run your tests on every file save we recommend using testmon with pytest-watch.

Testmon always re-executes tests which failed last time.

All command line options

optionhelp
--testmonSelect tests affected by changes (based on previously collected data) and collect + write new data (.testmondata file). Either collection or selection might be deactivated (sometimes automatically). See below.
--testmon-noselectRun testmon but deactivate selection, so all tests selected by other means will be collected and executed. Forced if you use -k, -l, -lf, test_file.py::test_name
--testmon-nocollectRun testmon but deactivate the collection and writing of testmon data. Forced if you run under debugger or coverage.
--testmon-forceselectRun testmon and select only tests affected by changes and satisfying pytest selectors at the same time.
--no-testmonTurn off (even if activated from config by default). Forced if neither read nor write is possible (debugger plus test selector).
--testmon-env=<STRING>This allows you to have separate coverage data within one .testmondata file, e.g. when using the same source code serving different endpoints or Django settings.
--tmnetThis is used for internal beta. Please don’t use. You can go to https://www.testmon.net/ to register.

Configuration

If you need to keep .testmondata anywhere else than in pytest rootdir please use TESTMON_DATAFILE environment variable.

Persistent configuration for testmon can be held in pytest config section like this (e.g. pytest.ini):

[pytest]
addopts = --testmon # you can make --testmon a default if you want
# If you want to separate different environments running the same sources.

Troubleshooting - usual problems

Testmon selects too many tests for execution: Depending you your change it most likely is by design. If you changed a method parameter name, you effectively changed the whole hierarchy parameter -> method -> class -> module, so any test using anything from that module will be re-executed.

Tests are failing when running under testmon: It’s quite unlikely testmon influenced the execution of the test itself. However set of deselected and executed tests with testmon is highly variable, which means testmon is likely to expose undesired test dependencies. Please fix your test suite. We wrote down a couple of tips and tricks on how to tackle this challenge here.

You can also try if your test is influenced under pytest-cov (coverage) without testmon. For reporting a bug a repository with description of unexpected behavior is the best, but please don’t hesitate to report even if your project is closed source. We’ll try to fix it!

Test dependencies

There are many things influencing the outcome of a test. testmon keeps track of these dependencies:

  1. code inside the tested project itself, which presumably changes frequently
  2. environment variables (e.g. DJANGO_SETTINGS_MODULE), python version (you are able to separate testmon data by configuring run_variant_expression)
  3. third-party packages/libraries (their versions)

Testmon doesn’t know and doesn’t track changes of these categories:

  1. static files (txt, xml, other project assets)
  2. external services (reachable through network)

Later versions can implement a mechanism to help reacting to these changes.