Testing the diff

Filipe Ximenes
July 27, 2021
<p>Good unit tests laser focus on the smallest possible scope and are  crafted to isolate the functionality from as much external interference  as possible. The way to do this is to write tests that self ensure they  are testing what we expect them to be testing.</p><p>Let's see this in practice. Suppose the following test for an endpoint filtering:</p><!--kg-card-begin: markdown--><pre><code>def test_filter_by_country(): client = TestClient() Store.create(country=&quot;AUS&quot;) response = client.get(&quot;/stores/?country=AUS&quot;) assert len(response.data) == 1 </code></pre> <!--kg-card-end: markdown--><p>At first sight, the above test seems to be correctly checking if our  endpoint is filtering by country. But as we will see, it's so fragile to  the point that it's almost useless. Here is the code for the store  listing endpoint we are testing:</p><!--kg-card-begin: markdown--><pre><code>def list_stores(request): stores = Store.get_all() country_filter = request.querystring.get(&quot;country&quot;, None) if country_filter: stores = stores.filter(country=country_filter) return stores.to_json() </code></pre> <!--kg-card-end: markdown--><p>Notice how the test we wrote will still pass even if we completely delete the filtering part of the endpoint as such:</p><!--kg-card-begin: markdown--><pre><code>def list_stores(request): stores = Store.get_all() return stores.to_json() </code></pre> <!--kg-card-end: markdown--><p>This might look like an unreasonable thing to happen, but this is  precisely the kind of thing we, developers, are constantly doing due to  changes in business rules, and it could even happen by accident. It's  tough to track down which tests need updating when we are making changes  [especially in big codebases], and it's easy to remove something we  assume is not being used [especially in legacy codebases].</p><p>Alright, how do we improve on this then? One easy way would be to add more fixtures to the test:</p><!--kg-card-begin: markdown--><pre><code>def test_filter_by_country(): client = TestClient() Store.create(country=&quot;AUS&quot;) Store.create(country=&quot;BR&quot;) response = client.get(&quot;/stores/?country=AUS&quot;) assert len(response.data) == 1 </code></pre> <!--kg-card-end: markdown--><p>Although that seems to solve the issue, we still cannot be sure that  what is causing the BR store not to show in the results is the action of  the <code>country=AUS</code> filter or some other business logic we are not aware of. It could still  be the case that we are not capturing that the filter is actually  filtering! The following code would still make the test to pass:</p><!--kg-card-begin: markdown--><pre><code>def list_stores(request): stores = Store.get_all().remove_LATAM() return stores.to_json() </code></pre> <!--kg-card-end: markdown--><p>Ugh! Again, this looks a bit too specific, but we are constantly making  decisions in our day-to-day that cause fragile tests to break in similar  ways.</p><p>So what can we do to make tests more reliable? Make them <strong>SPECIFIC</strong> and <strong>ISOLATE</strong> the functionality they are covering!</p><p><strong>The best technique I've encountered to help with this is to test the diff.</strong></p><p>Testing the diff means that our tests will pass or fail precisely  because of an intended and explicit change. Here is how this would work  in our example:</p><!--kg-card-begin: markdown--><pre><code>def test_filter_by_country(): client = TestClient() store = Store.create(country=&quot;BR&quot;) response = client.get(&quot;/stores/?country=AUS&quot;) assert len(response.data) == 0 store.update(country=&quot;AUS&quot;) response = client.get(&quot;/stores/?country=AUS&quot;) assert len(response.data) == 1 </code></pre> <!--kg-card-end: markdown--><p>The new setup gives us confidence of 2 very important assertions about the functionality we are writing:</p><ul><li>The filter includes the AUS items;</li><li>The only thing that is causing the filter to return the item is the fact that the store country is AUS;</li></ul><p>Notice we are still not ensuring that the filter is what is causing the  BR store to be left out of the results. So we can improve this even  further by doing:</p><!--kg-card-begin: markdown--><pre><code>def test_filter_by_country(): client = TestClient() Store.create(country=&quot;AUS&quot;) Store.create(country=&quot;BR&quot;) response = client.get(&quot;/stores/&quot;) # no filter assert len(response.data) == 2 response = client.get(&quot;/stores/?country=AUS&quot;) # country filter assert len(response.data) == 1 assert response.data[0][&quot;country&quot;] == &quot;AUS&quot; </code></pre> <!--kg-card-end: markdown--><p>We are now sure our tests are testing precisely the filter  functionality. Changes in unrelated features can still break this test,  but they will break the first assertion, and we will probably be able to  fix it by updating the Store objects creation lines (which brings us to the topic on the importance of having good fixtures, but that's subject  to another post 😉)</p><p>So, when can we use the "testing the diff" technique? There are many  situations where testing the diff can give us more reliable tests, but  to leave you with some food for thought, try to think how you would  write the tests and the benefits of testing the diff when testing  permissions and when doing performance tests.</p>