Makefile tricks for Python projects

I like using Makefiles. They work great both as simple task runners as well as build systems for medium-size projects. This is my starter template for Python projects.

Note: This blog post assumes some basic knowledge of how make and Makefiles work.

Basic configuration

I like using bash as the default shell. Then set some flags to exit on error (-eu -o pipefail), warn about undefined variables and disable built-in rules.

SHELL := bash
.SHELLFLAGS := -eu -o pipefail -c
MAKEFLAGS += --warn-undefined-variables
MAKEFLAGS += --no-builtin-rules

Using the virtual environment

The next lines will create two aliases to run Python from an existing virtual environment if it exists

py = $$(if [ -d $(PWD)/'.venv' ]; then echo $(PWD)/".venv/bin/python3"; else echo "python3"; fi)
pip = $(py) -m pip

Now, inside any recipe, you can use $(py) to call Python. The call will be converted to .venv/bin/python3 if there’s a virtual environment. You may need to change $(PWD) and/or .venv depending on how you set up your virtual environments.

$(pip) will also translate to .venv/bin/python3 -m pip if there’s a virtual environment. Otherwise, it will translate to python3 -m pip.

The reason to use $$(some-command) instead of the built-in function $(shell some-command) is that the expression will be evaluated every time it is called. So, every time a recipe contains $(py), the call will be translated to python3 or .venv/bin/python3 depending on the current context. When using $(shell ...) the expression gets evaluated only once and reused across all the recipes.

PWD and the repo root

NOTE: You don’t need to change $(PWD), see “update” below.

I normally prefer overriding PWD. You can call makefiles from a different directory, which will change PWD. These two lines will set:

PWD: Location of the Makefile
WORKTREE_ROOT: The root of the git repo. If this Makefile is used outside of a worktree, the variable will be an empty string.

# Override PWD so that it's always based on the location of the file and **NOT**
# based on where the shell is when calling `make`. This is useful if `make`
# is called like `make -C <some path>`
PWD := $(realpath $(dir $(abspath $(firstword $(MAKEFILE_LIST)))))

WORKTREE_ROOT := $(shell git rev-parse --show-toplevel 2> /dev/null)

Update

I kind of forgot about the CURDIR variable. Thanks to @phoebos on lobsters for the tip. It’s very likely you want to use $(CURDIR), not $(PWD)¹. Even if this feature is specific to GNU Make, I’m mostly worried about the Makefile working in the (old) make version shipped by macOS (3.81 in my case), and it seems to work there too. That variable was initially added in 1998².

MAKEFILE_PWD := $(CURDIR)
WORKTREE_ROOT := $(shell git rev-parse --show-toplevel 2> /dev/null)

Default goal and help message

Just a recipe to run some regex over the makefile and print a help message

.DEFAULT_GOAL := help
.PHONY: help
help: ## Display this help section
	@awk 'BEGIN {FS = ":.*?## "} /^[a-zA-Z\$$/]+.*:.*?##\s/ {printf "\033[36m%-38s\033[0m %s\n", $$1, $$2}' $(MAKEFILE_LIST)

By setting .DEFAULT_GOAL := help, the help recipe will run when calling make without a target.

Now you can add a string to the recipes like this:

some_target:  ## This help message will get printed
	@echo hello
	@touch some_target

Injecting paths into `PYTHONPATH`

This is probably my favourite trick. Once you have all your third-party dependencies installed, instead of installing all your packages, you can append some file system paths to the PYTHONPATH environment variable. This will let Python search for packages in those paths without installing the packages.

# PY_PATHS := list with paths to packages

pypath := python3 -c 'import sys, pathlib as p; print(":".join([str(p.Path(x).resolve()) for x in sys.argv[1:]]))'

How does it work?

First, you need a list of paths to python packages. For example:

PY_PATHS := $(PWD)/pkgs/package1 $(PWD)/pkgs/package2

The pypath variable is a compressed version of this Python script:

import sys
from pathlib import Path

paths = [Path(x).resolve() for x in sys.argv[1:]]
paths = [str(x) for x in paths]

joined = ":".join(paths)

print(joined)

Using it

PY_PATHS := $(PWD)/pkgs/package1 $(PWD)/pkgs/package2
pypath := python3 -c 'import sys, pathlib as p; print(":".join([str(p.Path(x).resolve()) for x in sys.argv[1:]]))'

.PHONY: test-pypath
test-pypath: export PYTHONPATH = $(shell $(pypath) $(PY_PATHS))
test-pypath:
	@python3 -c 'import sys; print(sys.path)'

When you run this makefile, you’ll get an output like this:

['',
 '<YOUR PWD>/pkgs/package1',
 '<YOUR PWD>/pkgs/package2',
 '/opt/homebrew/Cellar/[email protected]/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python311.zip',
 '/opt/homebrew/Cellar/[email protected]/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11',
  '/opt/homebrew/Cellar/[email protected]/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/lib-dynload',
 '/opt/homebrew/lib/python3.11/site-packages']

Now, if the paths '<YOUR PWD>/pkgs/package1' and '<YOUR PWD>/pkgs/package2' contain a Python package, the interpreter will be able to import them without having to install them first. Just make sure all their dependencies are installed.

Why this is useful

This may be a bit unrelated to this blog post. I work using git worktrees to have multiple branches checked out at the same time. By using this trick, I can run different commands and make sure the imported package is from that branch, instead of having to python3 -m pip install -e . around all the packages.

I also use this to run scripts which use packages that have heavy dependencies. I can still the third-party dependencies only once and then point to different paths (branches) to run the same command. Each run will use the code from that branch, without needing to reinstall the different versions of the package in different virtual environments.

Adding more paths when calling `make`

If you want to allow passing extra paths when calling make, you can store the initial value in a variable and append it to the exported value.

PY_PATHS := $(PWD)/pkgs/package1 $(PWD)/pkgs/package2
pypath := python3 -c 'import sys, pathlib as p; print(":".join([str(p.Path(x).resolve()) for x in sys.argv[1:]]))'

# Store the current value
DEFAULT_PYPATH := $(shell echo $$PYTHONPATH)


.PHONY: test-pypath
test-pypath: export PYTHONPATH = $(shell $(pypath) $(PY_PATHS)):$(DEFAULT_PYPATH)
test-pypath:
	@python3 -c 'import sys, pprint; pprint.pprint(sys.path)'

Now you can do this:

PYTHONPATH=./some/extra/path make test-pypath

And PYTHONPATH will contain ./some/extra/path, $(PWD)/pkgs/package1 and $(PWD)/pkgs/package2.

Creating a virtual environment

This rule can change depending on what tools you use to manage your virtual environments. But I use something like:

.venv: requirements.txt
	$(py) -m venv .venv
	$(pip) install -U pip setuptools wheel build
	$(pip) install -U -r requirements.txt
	touch .venv

This recipe will create a new environment in a .venv folder. It will then update and/or install pip, setuptools, wheel and build. It will install the requirements in requirements.txt. The last command touch .venv ensures that the modified date of the .venv folder is more recent than requirements.txt, which will avoid accidental rebuilds.

The template

SHELL := bash
.SHELLFLAGS := -eu -o pipefail -c
# .DELETE_ON_ERROR:
MAKEFLAGS = --warn-undefined-variables
MAKEFLAGS += --no-builtin-rules


# Override PWD so that it's always based on the location of the file and **NOT**
# based on where the shell is when calling `make`. This is useful if `make`
# is called like `make -C <some path>`
PWD := $(realpath $(dir $(abspath $(firstword $(MAKEFILE_LIST)))))

WORKTREE_ROOT := $(shell git rev-parse --show-toplevel 2> /dev/null)


# Using $$() instead of $(shell) to run evaluation only when it's accessed
# https://unix.stackexchange.com/a/687206
py = $$(if [ -d $(PWD)/'.venv' ]; then echo $(PWD)/".venv/bin/python3"; else echo "python3"; fi)
pip = $(py) -m pip

.DEFAULT_GOAL := help
.PHONY: help
help: ## Display this help section
	@awk 'BEGIN {FS = ":.*?## "} /^[a-zA-Z\$$/]+.*:.*?##\s/ {printf "\033[36m%-38s\033[0m %s\n", $$1, $$2}' $(MAKEFILE_LIST)

.venv: requirements.txt  ## Build the virtual environment
	$(py) -m venv .venv
	$(pip) install -U pip setuptools wheel build
	$(pip) install -U -r requirements.txt
	touch .venv

Everything else

Now, depending on your tooling, you may want to add recipes to call pip-tools, poetry, black, ruff, etc. I didn’t include those because I think it depends on how each project is set up.

If you’re looking for a general make tutorial, I think this is a good place to start. The GNU Make manual is also a good place to go deeper into some topics.

rand[om]