Why You Should Document Your Tests

Some projects have the policy that all tests must have an explanatory comment – including all of mine. At first, I found that baffling. If that’s you right now, this article is for you.

I’m sure you’ve been there too: you code along, write an obvious test, that tests an obvious thing, that you give an obvious name. Fast forward a year or two. Nothing about that test is obvious anymore. It might be obvious what property is tested under what circumstances but you have no idea why you test it.

To illustrate that, let’s have a look at a simplified test from one of my Python projects:

def test_eq_false():
    @attr.s(eq=False)
    class C:
        pass

    c1 = C()
    c2 = C()

    assert hash(c1) != hash(c2)

What are we testing here? The code is crystal clear and requires no comments: we’re testing that if the eq argument is set to False, the hashes of the instances of the class differ¹. The test name suggests that the eq=False is important – but why?

In this case we test that setting eq=False makes the class hashable by object ID. The fact that their hashes differ is a side-effect of what we’re actually testing. This is quite common in testing: very often, you can’t ask questions directly. Instead you verify certain properties that prove that your code is achieving its goals.

If you don’t explain what you’re actually testing, you force the reader (possibly future you) to deduce the main intent by looking at all of its properties. This makes it tiring and time-consuming to quickly scan a file for a certain test or to understand what you’ve actually broken if a test starts failing.

Arguably, the example test name could be more descriptive. Maybe test_eq_false_makes_hashable_by_id. But the intent can’t always be summed up in a few words and you’re bound to some line length limit. Also, for regression tests I like to add links to Sentry or bug tracker issues – good luck putting those infos into a test name.

How you go about documenting your tests depends on your language of choice. For Python I use docstrings, for Go I usually have a descr field in my table tests. If in doubt, add a simple comment.

Writing good test docs however is an art by itself. Firstly, clear writing requires clear thinking. That might feel like it’s slowing you down in the beginning, but it will help you understand the problem better in the end. Secondly, you don’t want to have redundant linguistic boilerplate in your comments. I recommend this blog post by Jonathan Lange where he iterates from a useless “Test that input is parsed correctly.” to something valuable.

Enforcing Their Presence in Python

Missing test docstrings is one of the most common issues that I have to raise on pull requests. That is frustrating for my contributors which is bad for contributor retention and therefore bad for my projects.

Which is why I was delighted when my friend Lynn Root released interrogate: a tool that helps you to find methods, functions, classes, and modules that lack docstrings in Python files.

Enforcing that all test functions and test methods have a docstring is as simple as running

$ interrogate -vv --fail-under 100 --whitelist-regex "test_.*" tests

Get on board – your future you will be slightly less frustrated by you!