Django: Flush out test flakiness by randomly ordering QuerySets

Sometimes code depends on the order of a QuerySet whilst not specifying an order. This can lead to random, flaky test failures because databases can return rows in any order when none is specified. The problem is made worse by some databases, notably PostgreSQL, which nearly always return rows in insert order, but occasionally use a different order when a table has had recent deletions.

For example, take this test:

from django.test import TestCase

from example.models import Book
from example.tasks import import_books


class ImportBooksTests(TestCase):
    def test_success(self):
        import_books("sutherland.csv")

        books = Book.objects.all()
        assert books[0].name == "Transport for Humans"
        assert books[1].name == "Alchemy"

The assertion block depends on the order of books. Unless the model has a Meta.ordering attribute, the database can return rows in any order. This test will occasionally, even rarely, fail due to a mismatch of books[0].name.

The simplest fix is to specify an ordering:

 from django.test import TestCase

 from example.models import Book
 from example.tasks import import_books


 class ImportBooksTests(TestCase):
     def test_success(self):
         import_books("sutherland.csv")

-        books = Book.objects.all()
+        books = Book.objects.order_by("id")
         assert books[0].name == "Transport for Humans"
         assert books[1].name == "Alchemy"

David Winterbottom’s post Patterns of flakey Python tests lists unspecified QuerySet ordering as a known flakiness-inducing pattern. He gave a talk based on the post at The London Django Meetup last year.

At the time, my client Silvr also had a bunch of flaky tests. David’s talk inspired me to try and cut off causes of flakiness at the root. For the unspecified ordering problem, I came up with the below pytest fixture.

import patchy
import pytest


@pytest.fixture(scope="session", autouse=True)
def make_unordered_queries_randomly_ordered():
    """
    Patch Django’s ORM to randomly order all queries without a specified
    order.

    This discovers problems where code expects a given order but the
    database doesn’t guarantee one.

    https://adamj.eu/tech/2023/07/04/django-test-random-order-querysets/
    """
    from django.db.models.sql.compiler import SQLCompiler

    patchy.patch(
        SQLCompiler._order_by_pairs,  # type: ignore[attr-defined]
        """\
        @@ -9,7 +9,7 @@
                 ordering = meta.ordering
                 self._meta_ordering = ordering
             else:
        -        ordering = []
        +        ordering = ["?"]
             if self.query.standard_ordering:
                 default_order, _ = ORDER_DIR["ASC"]
             else:
        """,
    )

The fixture uses my package patchy to tweak the source code of Django’s internal ORM class SQLCompiler. This class is responsible for compiling a QuerySet into actual SQL. The patch makes it so that any QuerySet without an explicit ordering instead uses the "?" order_by shortcut to tell the database to order results randomly.

You should be able to install patchy and copy-paste this fixture into your project’s top-level conftest.py. If you use Django’s test runner instead of pytest, you should be able to put the same patchy.patch() call in a custom test runner class.

Ordering randomly means that tests that depend on a specific ordering will likely fail. With two objects, the results will only be in the expected order 1/2 = 50% of the time. And with more objects, the chance of being in the expected order continues to drop.

Hopefully, between running the test locally and on CI, ordering-dependent flakiness is discovered before merging into the main branch. If not, such flaky tests should still fail on the main branch within a few CI runs.

This has been the experience at Silvr. Ordering-dependent flaky tests do occasionally get merged into the main branch, but they are discovered and fixed quickly.

Read my book Boost Your Git DX to Git better.

One summary email a week, no spam, I pinky promise.

Related posts:

• Django: Parametrized tests for all model admin classes
• Django: A version of json_script for pre-serialized JSON strings
• Django: Avoid database queries in template context processors

Tags: django