Tiny struggles

Always hacking something 🧑‍🔬.

Django Rest Framework Recipes

Introduction

One of my favorite tools in my app development toolkit is django rest framework (drf), that makes developing REST APIs with python and django easy and fun. It’s easy to set up, extensible and saves so much time.

The documentation of django rest framework is pretty extensive, there is a great tutorial, api docs and plenty of examples. But reality is usually a bit more complex and you need to customize your use of the framework.

In this article I’m going to share a bunch of recipes from my use of django rest framework taken from my latest project. They all come from just a single ViewSet (set of related api endpoints)!

At the end of the article I will show how all those bits fit together.

This article doesn’t provide any introduction to django or the django rest framework. Please read the official docs.

Piece of API used for examples

I’ve been working on a feature in the invertimo app where users can add multiple different investment accounts. The api is consumed by a react frontend and it’s customized by the need of the frontend app.

I’m using drf ViewSet to provide the following:

  • list endpoint ‘/accounts/’ that lists the accounts based on the model and provides a bunch of additional fields within each model
  • very rich detail get endpoint ‘/accounts/id/’ for retrieving detailed account data that takes additional parameters (from_date, to_date)
  • create and update endpoints that only allow to set or touch limited number of parameters
  • delete endpoint that in this case is very standard

Examples are using python type hints and are coming from a project using python 3.8 (at the time of writing). If you want to know more about setting up typechecking for django rest framework I recommend this article.

Recipes structure

The recipes are sorted from most common to more complex and are grouped together when they relate to a similar class of problems. I will provide a bit of a context, general motivation for using a given recipe and an example.

My app supports multiple users and they each have their own private data. I don’t want one user to see other user’s private data.

Motivation:

  • private data visible only to the owning user

Implementation:

  • filter the queryset

The best place to filter the queryset is to override get_queryset method provided by the parent class.

class AccountsViewSet(viewsets.ModelViewSet):
    permission_classes = [permissions.IsAuthenticated]
    serializer_class = AccountSerializer
    basename = "account"

    def get_queryset(self) -> QuerySet[models.Account]:
        assert isinstance(self.request.user, User)
        queryset = models.Account.objects.filter(user=self.request.user)
        ...
        return queryset

You can get the user from the self.request.user .

assert isinstance(self.request.user, User) is totally optional and done for the sake of narrowing down the type. In this case I can assert that the user is indeed a User because I enforce that the user has to the authenticated with: permission_classes = [permissions. IsAuthenticated] .

Different serializers for different methods within a viewset

Serializers in drf allow easy serialization (e.g to json) and deserialization (to native python) in your API.

By default there is one serializer class for a single ViewSet, even if it contains multiple separate views.

This is often fine, but at times you want to do it differently. See a simple example below:

simple serializers

Notice that there are less fields in the create form than within serialized values.

Motivation:

  • rich display of related fields for read only version
  • fields that are computed based on more complex logic

Ways to do it:

  • reimplement each method you want to override the serializer for (repetitive)
  • override get_serializer_class (recommended!)
class AccountsViewSet(viewsets.ModelViewSet):
    permission_classes = [permissions.IsAuthenticated]
    serializer_class = AccountSerializer
    basename = "account"

    def get_serializer_class(self):
        if self.action in ("create", "update"):
            return AccountEditSerializer
        if self.action == "retrieve":
            return AccountWithValuesSerializer

        return AccountSerializer

If the only difference between serializers you have is that some fields are read only and shouldn’t be used in views that are updating the data, you might want to mark those fields are read only instead of changing the serializer.

Add fields on the fly that aren’t present on the model

Another common case I encountered while developing APIs was adding more data to the serialized model instances.

I present two recipes here:

  • using queryset annotation
  • using a method on a serializer

Based on queryset annotation

Motivation:

  • add a field based on the result SQL query, e.g. count of related entities

How to do it:

  1. annotate the queryset
  2. update the serializer to display new fields

Here I’m adding positions_count and transactions_count :


class AccountsViewSet(viewsets.ModelViewSet):
    permission_classes = [permissions.IsAuthenticated]
    serializer_class = AccountSerializer
    pagination_class = LimitOffsetPagination
    basename = "account"

    def get_queryset(self) -> QuerySet[models.Account]:
        assert isinstance(self.request.user, User)
        queryset = models.Account.objects.filter(user=self.request.user).annotate(
            positions_count=Count("positions", distinct=True),
            transactions_count=Count("positions__transactions", distinct=True),
        )
        return queryset

Define additional fields within the serializer by specifying the fields and adding them to the list in the meta. New fields here are positions_count and transactions_count .

class AccountSerializer(serializers.ModelSerializer[Account]):

    positions_count = serializers.IntegerField()
    transactions_count = serializers.IntegerField()

    class Meta:
        model = Account
        fields = [
            "id",
            "nickname",
            "description",
            "balance",
            "last_modified",
            "positions_count",
            "transactions_count",
        ]

The great thing about adding fields with queryset annotations is that it’s also efficient and prevents using an excessive number of SQL queries.

Useful links:

Based on a dynamic function

Sometimes you can’t express the additional field you need this way. For example you need to call a method on the instance of the object to get the value you need.

There is an easy way with drf to do this with SerializerMethodField .

Motivation:

  • add a field to serialized data that requires custom logic

How to do it:

  • define a field as serializers.SerializerMethodField()
  • add it to fields list in Meta
  • define get_myfieldname method

Here is an example where I define new field called values :

class AccountWithValuesSerializer(serializers.ModelSerializer[Account]):

    positions_count = serializers.IntegerField()
    transactions_count = serializers.IntegerField()
    currency = CurrencyField()
    values = serializers.SerializerMethodField()

    class Meta:
        model = Account
        fields = [
            "id",
            "currency",
            "nickname",
            "description",
            "balance",
            "last_modified",
            "positions_count",
            "transactions_count",
            "values",
        ]

    def get_values(self, obj):
        from_date = self.context["from_date"]
        to_date = self.context["to_date"]

        return obj.value_history_per_position(from_date, to_date)

In this example I use some additional data from self.context that brings me to my next recipe.

Pass additional data to the serializer

Motivation:

  • use additional data to generate and additional field
  • perform additional validation

How to do it:

  • override get_serializer_context
  • the data can come e.g. from self.request, e.g. self.request.user of self.request.query_params

Example from the ViewSet code:


    def get_serializer_context(self) ->  Dict[str, Any]:
        context: Dict[str, Any] = super().get_serializer_context()
        query = FromToDatesSerializer(data=self.request.query_params)
        context["request"] = self.request
        return context

and then the set value can be used inside the serializer:

request = self.context.get("request")

Use a serializer for the query parameters

Serializers transform data between formats such as json and native python, they also provide a good place to put your validation logic. Well, you can use a serializer to extract and validate data from the query parameters (also known as URL params).

Motivation:

  • use additional query parameters within a ViewSet and have them validated

How to do it:

  • define a custom Serializer for the data you expect in your query params (see that this serializer is not based on the ModelSerializer)
  • use it within your view:
    • initialize with MySerializer(data=self.request.query_params)
    • validate and extract the data

You can combine with a previous technique of passing additional data through the serializer context like follows:

class FromToDatesSerializer(serializers.Serializer[Any]):
    from_date = serializers.DateField(required=False)
    to_date = serializers.DateField(required=False)

Useful links:


class AccountsViewSet(viewsets.ModelViewSet):
    ...
    def get_serializer_context(self) ->  Dict[str, Any]:
        context: Dict[str, Any] = super().get_serializer_context()
        query = FromToDatesSerializer(data=self.request.query_params)
        context["request"] = self.request

        if query.is_valid(raise_exception=True):
            data = query.validated_data
            self.query_data = data
            context["from_date"] = self.query_data.get(
                "from_date",
                datetime.date.today() - datetime.timedelta(days=30),
            )
            context["to_date"] = self.query_data.get("to_date", datetime.date.today())
        return context

Use string value in an API for a field that has more efficient DB representation (enum)

If a field can only have limited number of options, enums are a great choice. Django provides TextChoices , IntegerChoices , and Choices

enumeration types to make it very easy.

My preferred field is the IntegerChoices because it will end up using much less space in the database even if the represented value is a string.

I have a currency field defined as follows:


class Currency(models.IntegerChoices):
    EUR = 1, _("EUR")
    GBP = 2, _("GBP")
    USD = 3, _("USD")
    GBX = 4, _("GBX")

def currency_enum_from_string(currency: str) -> Currency:
    try:
        return Currency[currency]
    except KeyError:
        raise ValueError("Unsupported currency '%s'" % currency)

def currency_string_from_enum(currency: Currency) -> str:
    return Currency(currency).label

class Account(models.Model):
    user = models.ForeignKey(User, on_delete=models.CASCADE)
    currency = models.IntegerField(choices=Currency.choices, default=Currency.EUR)
    nickname = models.CharField(max_length=200)
    description = models.TextField(blank=True)

There are only 4 different values and they are stored very efficiently in the database.

However, I don’t want my API to expect value ‘1’ for EUR. I would much rather have “EUR” to represent “EUR” and not expose that I use integers to represent it in the database.

Motivation:

  • Use different internal and external representation for a value, e.g. integer vs string

How to do it:

  • create a new serializer Field class inheriting from a field that would be suitable for the internal representation
  • define to_representation and to_internal_value methods
  • specify that field in the serializer explicitly by using the newly defined class
class CurrencyField(serializers.IntegerField):
    def to_representation(self, value):
        return models.currency_string_from_enum(value)

    def to_internal_value(self, value):
        return models.currency_enum_from_string(value)
class AccountEditSerializer(serializers.ModelSerializer[Account]):

    # Currency needs to be changed from string to enum.
    currency = CurrencyField()

Useful links:

Unique together with a user that is not set in the form

It’s a fairly common case to create objects for a user that is currently logged in. But what if you don’t pass the user in the form directly? And what if you want the objects e.g. name to be unique for a given user?

You can combine:

  • passing additional value to the serializer with the serializer context by overriding get_serializer_context
  • custom field validation (override the validate_myfieldname method)

In this example the Account model has a constraint that nicknames have to be unique for a given user:


    # In the model.

    class Meta:
        unique_together = [["user", "nickname"]]

Within a serializer nickname field validation is overridden and if the uniqueness constraint is not satisfied, the serializer raises serializers.ValidationError.


     class Meta:
        model = Account
        fields = [
            "id",
            "currency",
            "nickname",
            "description",
        ]

    def validate_nickname(self, value):
        # If user was also included in the serializer then unique_together
        # constraint would be automatically evaluated, but
        # since user is not included in the serializer the validation is
        # done manually.
        request = self.context.get("request")
        if request and hasattr(request, "user"):
            user = request.user
            if Account.objects.filter(user=user, nickname=value).count() > 0:
                raise serializers.ValidationError(
                    f"User already has an account with name: '{value}'"
                )
        return value

Useful links:

ViewSet using all these patterns

Well, if you are curious what is the monstrosity I’ve been working on, I’m presenting you the code of it, showcasing how all these recipes fit together.

(Tests are not included, even though they exist! The entire codebase can be found here.)

models.py :


class Currency(models.IntegerChoices):
    EUR = 1, _("EUR")
    GBP = 2, _("GBP")
    USD = 3, _("USD")
    GBX = 4, _("GBX")

def currency_enum_from_string(currency: str) -> Currency:
    try:
        return Currency[currency]
    except KeyError:
        raise ValueError("Unsupported currency '%s'" % currency)

def currency_string_from_enum(currency: Currency) -> str:
    return Currency(currency).label

class Account(models.Model):
    user = models.ForeignKey(User, on_delete=models.CASCADE)
    currency = models.IntegerField(choices=Currency.choices, default=Currency.EUR)
    nickname = models.CharField(max_length=200)
    description = models.TextField(blank=True)

    balance = models.DecimalField(max_digits=12, decimal_places=5, default=0)
    last_modified = models.DateTimeField(auto_now=True, null=True)

    def __str__(self):
        return (
            f"<Account user: {self.user}, nickname: '{self.nickname}', "
            f"currency: {self.get_currency_display()}>"
        )

    def value_history_per_position(self, from_date, to_date):
        results = []
        for position in self.positions.all():
            results.append(
                (
                    position.pk,
                    position.value_history_in_account_currency(from_date, to_date),
                )
            )
        return results

    class Meta:
        unique_together = [["user", "nickname"]]

views.py

class AccountsViewSet(viewsets.ModelViewSet):
    permission_classes = [permissions.IsAuthenticated]
    serializer_class = AccountSerializer
    pagination_class = LimitOffsetPagination
    basename = "account"

    def get_queryset(self) -> QuerySet[models.Account]:
        assert isinstance(self.request.user, User)
        queryset = models.Account.objects.filter(user=self.request.user).annotate(
            positions_count=Count("positions", distinct=True),
            transactions_count=Count("positions__transactions", distinct=True),
        )
        return queryset

    def get_serializer_context(self) -> Dict[str, Any]:
        context: Dict[str, Any] = super().get_serializer_context()
        context["request"] = self.request

        query = FromToDatesSerializer(data=self.request.query_params)

        if query.is_valid(raise_exception=True):
            data = query.validated_data
            self.query_data = data
            context["from_date"] = self.query_data.get(
                "from_date",
                datetime.date.today() - datetime.timedelta(days=30),
            )
            context["to_date"] = self.query_data.get("to_date", datetime.date.today())
        return context

    def get_serializer_class(
        self,
    ) -> Type[
        Union[AccountEditSerializer, AccountWithValuesSerializer, AccountSerializer]
    ]:
        if self.action in ("create", "update"):
            return AccountEditSerializer
        if self.action == "retrieve":
            return AccountWithValuesSerializer

        return AccountSerializer

    def retrieve(self, request, pk=None):
        queryset = self.get_queryset()
        queryset = queryset.prefetch_related("positions__security")
        account = get_object_or_404(queryset, pk=pk)
        serializer = self.get_serializer(account, context=self.get_serializer_context())
        return Response(serializer.data)

    def create(self, request, *args, **kwargs):
        serializer = self.get_serializer(
            data=request.data, context=self.get_serializer_context()
        )
        serializer.is_valid(raise_exception=True)
        assert isinstance(self.request.user, User)
        accounts.AccountRepository().create(
            user=self.request.user, **serializer.validated_data
        )
        headers = self.get_success_headers(serializer.data)
        return Response(
            serializer.data, status=status.HTTP_201_CREATED, headers=headers
        )

serializers.py :

class AccountSerializer(serializers.ModelSerializer[Account]):

    positions_count = serializers.IntegerField()
    transactions_count = serializers.IntegerField()
    currency = CurrencyField()

    class Meta:
        model = Account
        fields = [
            "id",
            "currency",
            "nickname",
            "description",
            "balance",
            "last_modified",
            "positions_count",
            "transactions_count",
        ]

class AccountEditSerializer(serializers.ModelSerializer[Account]):

    # Currency needs to be changed from string to enum.
    currency = CurrencyField()

    class Meta:
        model = Account
        fields = [
            "id",
            "currency",
            "nickname",
            "description",
        ]

    def validate_nickname(self, value):
        # If user was also included in the serializer then unique_together
        # constraint would be automatically evaluated, but
        # since user is not included in the serializer the validation is
        # done manually.
        request = self.context.get("request")
        if request and hasattr(request, "user"):
            user = request.user
            if Account.objects.filter(user=user, nickname=value).count() > 0:
                raise serializers.ValidationError(
                    f"User already has an account with name: '{value}'"
                )
        return value

class AccountWithValuesSerializer(serializers.ModelSerializer[Account]):

    positions_count = serializers.IntegerField()
    transactions_count = serializers.IntegerField()
    currency = CurrencyField()
    values = serializers.SerializerMethodField()

    class Meta:
        model = Account
        fields = [
            "id",
            "currency",
            "nickname",
            "description",
            "balance",
            "last_modified",
            "positions_count",
            "transactions_count",
            "values",
        ]

    def get_values(self, obj):
        from_date = self.context["from_date"]
        to_date = self.context["to_date"]

        return obj.value_history_per_position(from_date, to_date)

Recommendations

Django rest framework is surprisingly well designed and provides a lot of great places for convenient customization. Whenever you are doing something a bit unusual, instead of jumping straight to stack overflow spend some time looking at the https://github.com/encode/django-rest-framework/tree/master either on github or within your editor.

Happy coding! If you find this article helpful, please share it and feel free to follow me on twitter.

Check out my latest project Watchlimits
Watch time insights and limits that work for you!
This is my mathjax support partial