Using pyproject.toml in your (Django) project

Back in 2018, I wrote about using setup.py in your Django/Python project. Five years later, setup.py is being phased out in favor of pyproject.toml. I’m a big fan of this change. With setup.py you could really go off the rails making everything dynamic or even executing malicious code during the installation process. In contrast, pyproject.toml moves the ecosystem towards a configuration file that can be parsed without executing arbitrary code. You can read more about the rationale behind pyproject.toml in PEP-517, PEP-518, PEP-621, and PEP-660.

Creating your pyproject.toml file

If you’re using poetry, pdm, or any of the other newer Python build systems, you’re already using pyproject.toml. How about folks that are using plain old pip or pip-tools? You can still take advantage of this new file format and ditch setup.py and/or setup.cfg as well. Most third-party tooling supports configuration via the tool section defined in PEP-518.

To start, define the build system for your project. To avoid introducing new tools, we’re going to use good ol’ setuptools:

[build-system]
requires = ["setuptools>=61.0"]
build-backend = "setuptools.build_meta"

Next, define your project and its dependencies:

[project]
name = "myproject"
version = "1.0"
dependencies = [
  "dj-database-url",
  "django==3.2.*",
  "gunicorn",
  "psycopg2",
  "whitenoise",
]

[project.optional-dependencies]
dev = [
  "black",
  "django-stubs",
  "ipdb",
  "pip-tools",
  "ruff",
]

At this point, you can run this to bootstrap a local development environment:

pip install --editable .[dev]

Installing manage.py

In our previous post about using setup.py we showed a trick that would allow you to remove manage.py from your repo and have it installed as a “proper” script on the PATH. You can add the functionality to myproject/__init__.py like this:

import os
import sys

def django_manage():
    os.environ.setdefault("DJANGO_SETTINGS_MODULE", "myproject.settings")
    from django.core.management import execute_from_command_line
    execute_from_command_line(sys.argv)

Now add this to your pyproject.toml:

[project.scripts]
"manage.py" = "myproject:django_manage"

When you install your project, it will create a manage.py script on your path so you can run it like any other command, $ manage.py ....

With pip-tools dependency locking

What we have so far is all well and good, but you really should be locking/freezing your dependencies to ensure the exact same requirements are installed every time. pip-tools allows you to create these in a format where only pip is required to install them elsewhere. For your primary dependencies, you can replace where you might have previously used requirements.in with pyproject.toml:

pip-compile \
  --generate-hashes \
  --output-file requirements.txt \
  pyproject.toml

You could do the same with your dev requirements by using the --extra dev flag:

# don't do this... see below for a better solution
pip-compile \
  --generate-hashes \
  --output-file requirements-dev.txt \
  --extra dev \
  pyproject.toml

There’s an issue here, however. We want to ensure requirements-dev.txt isn’t installing packages that are incompatible with what’s in requirements.txt. pip-tools suggests a workflow for this involving the --constraint flag, but it is incompatible with pyproject.toml dependencies. Here’s an ugly workaround for this situation:

1. Add the --strip-extras flag when you build your requirements.txt file so it can be used as a pip constraint. Don’t worry, the same dependencies will be installed.

   pip-compile \
     --generate-hashes \
     --output-file requirements.txt \
     --strip-extras \
     pyproject.toml
   

2. Pass that constraint in when you generate your dev requirements. To avoid adding more files to our project, we pass it via stdin.

   echo "--constraint $(pwd)/requirements.txt" | \
     pip-compile \
     --generate-hashes \
     --output-file requirements-dev.txt \
     --extra dev \
     - \
     pyproject.toml
   

Tip: I like to hide these long commands in a Makefile or Justfile so developers only need to remember make requirements.txt.

To install dependencies from your lock file and also install your project, you can pass the --no-deps flag to pip to make sure it doesn’t try to reinstall the un-pinned dependencies in pyproject.toml:

pip install -r requirements-dev.txt
pip install --no-deps -e .

It’s nice to trade setup.py, setup.cfg, requirements.in, requirements-dev.in, pytest.ini, .coveragerc, etc. with one file which defines everything Python related for your project. I hope we get the same for lock files, but with the rejection of PEP 665 we’ll have to wait a bit longer for that.