We recently added Python support to Wasm Language Runtimes. This article provides an overview of how Python works in WebAssembly environments and provides a step by step guide on how to use it.

At VMware OCTO WasmLabs we want to grow the WebAssembly ecosystem by helping developers adopt this new and exciting technology. Our Wasm Language Runtimes project aims to provide up-to-date, ready-to-run WebAssembly builds for the most popular language runtimes.

We are happy to announce that we have a first build of Python for the wasm32-wasi target! It is based on the WASI support that is already available in CPython (the mainstream, C-based implementation of Python), augmented with additional libraries and usage examples to make it as easy to use as possible. Python joins PHP and Ruby in the list of supported languages.

About the build and artifacts

To build python.wasm we rely on the WASI build support that is already available in CPython. We are reusing zlib and libuuid from the singlestore-labs/python-wasi repository but are building libsqlite in-house to also enable support for the sqlite3 module. You can find the source code here.

We will build and release new binaries of Python for Wasm whenever a new upstream release is available. To find the latest release look for "python" releases in webassembly-language-runtimes on github.

Also, Docker+Wasm fans, we are providing a python-wasm container image.

The rest of this article will deal with examples of how to run python.wasm and leverage it to run your Python apps on WASI.

Hands on python.wasm

All of the examples below include a short explanation along with sample output, where relevant, so it is easier to read them through.

Prerequisites

To give it a try, you will need to have a few tools installed in advance.

Most notably, a shell that has enough Unicode support to show emojis. Yep, this is what we use as part of our examples 😄

python3 on your machine

Implementing pip for python.wasm is not universally possible, because WASI still does not offer full socket support. Downloading a package from the internet may not even work on some runtimes.

But that is OK for most scenarios we are interested in, as python.wasm is likely to be used as a runtime in Cloud or Edge environments rather than a generic development platform. We will start by using a native python3.11 installation to setup a sample applications. And then we will show how you can run it on python.wasm.

wget and zip

These tools are required to download and extract the released binary.

A WASI-compatible runtime

As python.wasm is built for WASI you will need to get a compatible WebAssembly runtime, such as Wasmtime. We also provide an additional binary that will run on WasmEdge, which offers extended socket support on top of a modified WASI API. Since Docker+Wasm uses WasmEdge, this is the binary you will need if you want to build a WASM container image to use with Docker, as explained later in the article.

Docker+Wasm

To try the examples with Docker you will need "Docker Desktop" + Wasm version 4.15 or later.

Setup

All of the examples below assume you are using the same working directory. Some of them build on top of each other. Where this is the case we have tried referencing the previous one that we step on.

First, prepare a temporary folder and download the python.wasm binary

mkdir /tmp/try-python-wasm
cd /tmp/try-python-wasm

wget https://github.com/vmware-labs/webassembly-language-runtimes/releases/download/python%2F3.11.1%2B20230127-c8036b4/python-aio-3.11.1.zip
unzip python-aio-3.11.1.zip
rm python-aio-3.11.1.zip

Taking a look at the unzipped files we see two versions of python.wasm - one that's WASI compliant and one that can run on WasmEdge with its slightly non-standard and extended socket API. Also, the usr/local/lib folder includes a zip of the standard libraries, a placeholder Lib/lib-dynload and a Lib/os.py. The last two are not strictly necessary but if omitted will cause dependency warnings whenever python runs.

tree
.
├── bin
│   ├── python-3.11.1-wasmedge.wasm
│   └── python-3.11.1.wasm
├── python-aio-3.11.1.zip
└── usr
    └── local
        └── lib
            ├── python3.11
            │   ├── lib-dynload
            │   └── os.py
            └── python311.zip

Now, let's get the sample script and the data it will work on

wget https://raw.githubusercontent.com/vmware-labs/webassembly-language-runtimes/main/python/examples/emojize_text.py
wget https://raw.githubusercontent.com/vmware-labs/webassembly-language-runtimes/main/python/examples/source_text.txt

First time running python.wasm

The Python standard libraries are packed into usr/local/lib and the python.wasm binary is compiled to look for this path in /

So, in order to run Python properly we need to pre-open the current folder as root within the sandboxed WASM environment, where python.wasm will be running. For Wasmtime this is done via --mapdir.

wasmtime run \
  --mapdir /::$PWD \
  bin/python-3.11.1.wasm \
  -- -c "import sys; from pprint import pprint as pp; \
         pp(sys.path); pp(sys.platform)"

['',
 '/usr/local/lib/python311.zip',
 '/usr/local/lib/python3.11',
 '/usr/local/lib/python3.11/lib-dynload']
'wasi'

We could do the same with the WasmEdge-compliant binary (note the slight differences in the CLI arguments).

wasmedge \
  --dir /:$PWD \
  bin/python-3.11.1-wasmedge.wasm \
  -c "import sys; from pprint import pprint as pp; \
      pp(sys.path); pp(sys.platform)"

['',
 '/usr/local/lib/python311.zip',
 '/usr/local/lib/python3.11',
 '/usr/local/lib/python3.11/lib-dynload']
'wasi'

Running the repl

If you want, you can play with the Python repl.

wasmtime run --mapdir /::$PWD bin/python-3.11.1.wasm

Python 3.11.1 (tags/v3.11.1:a7a450f, Jan 18 2023, 22:43:41) ... on wasi
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>>
>>> sys.platform
'wasi'
>>>
>>> sys.version_info
sys.version_info(major=3, minor=11, micro=1, releaselevel='final', serial=0)

Running an app with dependencies

Next, let's assume we have a Python app that has additional dependencies. For example emojize_text.py, which we downloaded as part of the setup.

Installing dependencies to the pre-compiled path

To set up the dependencies we will need pip3 (or python3 -m pip) on the development machine, to download and install the necessary dependencies. The most straightforward way of doing this is by running pip with --target pointing the path that is already pre-compiled into the python.wasm binary. Namely, usr/local/lib/python3.11/

pip3 install emoji -t usr/local/lib/python3.11/

Now we can run our text emojizer. Taking a look at the sample source text.

cat source_text.txt

The rabbit woke up with a smile.
The sunrise was shining on his face.
A carrot was waiting for him on the table.
He will put on his jeans and get out of the house for a walk.

We get this result from emojize_text.py

wasmtime run \
  --mapdir /::$PWD \
  bin/python-3.11.1.wasm \
  -- \
  emojize_text.py source_text.txt

The 🐇 woke up with a smile.
The 🌅 was shining on his face.
A 🥕 was waiting for him on the table.
He will put on his 👖 and get out of the 🏠 for a walk.

Using a virtual environment

Any more complex python application is likely to be using virtual environments. In that case, you will have a venv folder with all requirements pre-installed. All you need to leverage them is to:

• Make sure this folder is pre-opened when running python.wasm
• Add it to the PYTHONPATH environment variable

Let's take a look at how to do this.

We will start by creating a virtual environment within the same folder and installing 'emoji' in it.

python3 -m venv venv-emoji
. venv-emoji/bin/activate
pip3 install emoji
deactivate

With what we did so far we were mapping the current folder as root anyway (for the sake of usr becoming /usr) so we only need to set the PYTHONPATH variable accordingly.

wasmtime \
  --env PYTHONPATH=/venv-emoji/lib/python3.11/site-packages \
  --mapdir /::$PWD \
  bin/python-3.11.1.wasm \
  -- \
  emojize_text.py source_text.txt

The 🐇 woke up with a smile.
The 🌅 was shining on his face.
A 🥕 was waiting for him on the table.
He will put on his 👖 and get out of the 🏠 for a walk.

Passing an environment variable with WasmEdge is similar

wasmedge \
  --env PYTHONPATH=/venv-emoji/lib/python3.11/site-packages \
  --dir /:$PWD \
  bin/python-3.11.1-wasmedge.wasm \
  emojize_text.py source_text.txt
...

Running the Docker container

Docker+WASM uses the WasmEdge runtime internally. To leverage it we have packaged the python-3.11.1-wasmedge.wasm binary in a container image available as ghcr.io/vmware-labs/python-wasm:3.11.1-wasmedge.

Here is an example of running the Python repl from this container image. As you can see from the output of the interactive session, the container includes only python.wasm and the standard libraries from usr. No base OS images, no extra environment variables, or any other clutter. The Dockerfile for this image is available at webassembly-language-runtimes/images/python/Dockerfile.

docker run --rm \
  -i \
  --runtime=io.containerd.wasmedge.v1 \
  --platform=wasm32/wasi \
  ghcr.io/vmware-labs/python-wasm:3.11.1-wasmedge \
  -i

Python 3.11.1 (tags/v3.11.1:a7a450f, Jan 27 2023, 11:37:16) ...  on wasi
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>>
>>> sys.platform
>>> 'wasi'
>>>
>>> import os
>>> os.listdir('.')
['python.wasm', 'etc', 'usr']
>>>
>>> [k for k in os.environ.keys()]
['PATH', 'HOSTNAME']

You can also run the Docker container to execute a one-liner like this.

docker run --rm \
  --runtime=io.containerd.wasmedge.v1 \
  --platform=wasm32/wasi \
  ghcr.io/vmware-labs/python-wasm:3.11.1-wasmedge \
  -c "import os; print([k for k in os.environ.keys()])"

['PATH', 'HOSTNAME']

Running the Docker container with dependencies

The python-wasm container image comes by default just with the Python standard library, so if your project has extra dependencies you will need to take care of them. Let's reuse the venv-emoji environment in which we installed emoji in the example above.

We need to do three things

1. Ensure that the emoji module installed in the venv-emoji folder is mounted in the running python-wasm container
2. Ensure that it is also on the PYTHONPATH within the running python-wasm container
3. Ensure that the python program and its data (in this case emojize_text.py and source_text.txt) are also mounted in the container

A vital piece of knowledge here is that whatever you mount in the running container gets automatically pre-opened by the WasmEdge runtime. Same goes for all environment variables that you pass to the container when you run it.

One way of doing what we want is to just mount site-packages from venv-emoji over the site-packages folder of the pre-compiled path in /usr/local. This is what this could look like:

docker run --rm \
  -v $PWD/venv-emoji/lib/python3.11/site-packages:/usr/local/lib/python3.11/site-packages \
  -v $PWD/emojize_text.py:/emojize_text.py \
  -v $PWD/source_text.txt:/source_text.txt \
  --runtime=io.containerd.wasmedge.v1 \
  --platform=wasm32/wasi \
  ghcr.io/vmware-labs/python-wasm:3.11.1-wasmedge \
  -- \
  emojize_text.py source_text.txt

The 🐇 woke up with a smile.
The 🌅 was shining on his face.
A 🥕 was waiting for him on the table.
He will put on his 👖 and get out of the 🏠 for a walk.

An alternative would be to map the current folder as /opt and use PYTHONPATH like this:

docker run --rm \
  -v $PWD:/opt \
  -e PYTHONPATH=/opt/venv-emoji/lib/python3.11/site-packages \
  --runtime=io.containerd.wasmedge.v1 \
  --platform=wasm32/wasi \
  ghcr.io/vmware-labs/python-wasm:3.11.1-wasmedge \
  -- \
  opt/emojize_text.py opt/source_text.txt

The 🐇 woke up with a smile.
The 🌅 was shining on his face.
A 🥕 was waiting for him on the table.
He will put on his 👖 and get out of the 🏠 for a walk.

Wrapping it all in a new container image

This way of running your python application with the python-wasm container is too cumbersome. Luckily OCI and Docker already offer a way to package everything nicely.

Let's first create a Dockerfile that steps on python-wasm to package our emojize_text.py app and its venv into a single image.

cat > Dockerfile.emojize <<EOF
FROM ghcr.io/vmware-labs/python-wasm:3.11.1-wasmedge

COPY venv-emoji/ /opt/venv-emoji/
COPY emojize_text.py /opt

ENV PYTHONPATH /opt/venv-emoji/lib/python3.11/site-packages

ENTRYPOINT [ "python.wasm", "/opt/emojize_text.py" ]
EOF

Building the container is straightforward

docker build -f Dockerfile.emojize --platform=wasm32/wasi -t emojize.py-wasm .

And to run it we only have to mount and provide the data file.

docker run --rm \
  -v $PWD/source_text.txt:/source_text.txt \
  --runtime=io.containerd.wasmedge.v1 \
  --platform=wasm32/wasi \
  emojize.py-wasm \
  source_text.txt

The 🐇 woke up with a smile.
The 🌅 was shining on his face.
A 🥕 was waiting for him on the table.
He will put on his 👖 and get out of the 🏠 for a walk.

To recap

We have seen how to use python.wasm with pre-existing applications either directly, or as part of the python-wasm container image.

If your Python application has only pure Python dependencies now you know how to run it on any Cloud or Edge WASI-compatible platform, including Docker+WASM.

As we work on extending what python.wasm has to offer, we are happy to get your feedback and suggestions. Give us a star at WebAssembly Language Runtimes, drop us a comment in Github at Python.wasm roadmap #46 or follow us on Twitter at @vmwwasm.

If you are interested in general discussion about Python WASM and WASI support you could join the #python channel on the WASM Discord Server (click here for an invite), or take a look at the latest WASM discussions in the Python community.