Python and Sigstore

This critical role would not be possible without funding from the Alpha-Omega project. Massive thank-you to Alpha-Omega for investing in the security of the Python ecosystem!

I was a guest on the Open Source Security podcast this week talking about Sigstore and Python among other things I'm working on at the Python Software Foundation.

From this diagram you can see the two "sources" of identity provided by PGP and Sigstore both link back to release managers. PGP relies on private keys which are maintained and protected by individual release managers to create signatures of artifacts. Sigstore uses third-parties which support OpenID Connect such as GitHub and Google to bind a human-readable identity like "thomas@python.org" to a signing certificate and signature that can be later verified.

Why Sigstore?§

The problem is that securely maintaining PGP private keys is not an easy task and the burden rests on volunteers for a minimum of 7 years (1 year of pre-releases then 5 years of bug and security fixes across at least two consecutive releases). Contrast that experience to Sigstore where release managers only need to click a button to OAuth sign-in during the release process.

Sigstore externalizes the operational burden of maintaining long-lived signing keys from individual volunteers to teams of people who run an identity platform professionally and the public-good code-signing certificate authority Fulcio. This seems like a pretty good trade-off, considering we already trust these platforms' identity management teams for things like GitHub accounts.

For this reason, I've authored PEP 761 providing a deprecation and discontinuance plan of PGP signatures for CPython artifacts. You can join the discussion of this PEP on discuss.python.org. Big thank-you to Hugo van Kemenade, the release manager for Python 3.14 for sponsoring my first PEP and helping me with the process and thanks to William Woodruff for reviewing the PEP draft and explaining the nitty-gritty details of Sigstore.

This PEP deprecates the expectation that future CPython releases will provide PGP signatures and sets a timeline for discontinuance (Python 3.14) and a mechanism for that timeline to be extended by a vote of the Steering Council, if necessary.

What do verifiers need?§

Signatures are only useful if they are verified, so we must weigh the needs of verifiers!

CPython's expected downstream verifiers are primarily "distributions" of CPython, such as through a Linux distro (Debian, Fedora, Gentoo, etc) or in a container image. There are other distributions of Python such as pyenv and python-build-standalone.

These users of CPython's source code are great places to verify signatures, as they're likely to be high value targets themselves and can provide a consistent stream of verifications. If any one signature verification were to fail, it would signal that something is wrong with upstream CPython artifacts or python.org and would likely be investigated and remediated quickly.

This further constrains attackers looking to affect CPython downstream users, as compromising python.org would no longer be enough. Instead, attackers would need to compromise the build infrastructure or CPython source code.

From discussions, the requirements that I've gathered from verifiers are:

• Need a tool for verification that can be packaged by distros. The recommended tool for verifying with a CLI is either Cosign or sigstore-python, both of which have challenges for Linux distro packagers.
• OS packages would require Cosign to be packaged in those OS package managers. This isn't trivial as it requires the Go toolchain to build.
• Docker and other container images want verification tools to be available at the OS level. Needing to pull these externally (from Cosign's GitHub releases) would require multi-stage builds which they want to avoid if possible. Today Cosign is available in Alpine, but not yet in Debian (but there is the beginnings of support), Gentoo, or Fedora.
• Offline verification is important. Many package ecosystems ship the signatures inside the packages to enable "build from source" (such as Gentoo), and there's no guarantee that the package is being built online. It's okay that revocation or root of trust updates can't happen when offline. Cosign and other Sigstore verification tools support offline verification if the root of trust is "bundled" as a file locally.
• Online periodic updates to revocations and root of trust. Cosign and other Sigstore verifiers supports this use-case as the default behavior.

Overall, the availability of Cosign in OS package managers appears to be the biggest blocker to see adoption for verifying CPython's Sigstore signatures.

Why adopt a new signature method?§

So why have CPython's Sigstore signatures not seen adoption despite being available for multiple years? Here's a short list of self-reinforcing reasons:

• Verifiers don't need to adopt the new signature method (Sigstore), because the existing one (PGP) works and there's no expectation for discontinuance.
• Signers can't migrate away from the old signature method because there's apparent demand from verifiers for the old signature method.
• Verifiers don't try or test the new signature method, so the maintainers of signature tooling can't learn about or improve verifier use-cases.
• Concurrently supporting multiple signature methods is more work for both signers and verifiers.
• There are fewer available signatures using the new signing method, so the value of adopting the method as a verifier is less (but maybe this will change soon?).

Keep in mind that almost everyone involved in the above scenarios are volunteers. Doing work to adopt a new process when existing processes are working can feel like "busy-work", but I don't think that's the case for Sigstore.

Sigstore's benefits for ergonomics paired with its ability to use workload identity are two stand-out features compared to PGP. Workload identity being extra important now that many projects are moving to hosted build infrastructure for releases.

Workload Identity§

Sigstore supporting workload identity means that release manager accounts can no longer be hijacked to produce bad signatures. Artifacts get signed by the build infrastructure provider directly:

Switching to workload identity also means downstream verifiers no longer need to make changes when new release managers join the project, the expected identity would always be gh/python/release-tools/....

We still have a ways to go to adopt workload identity for CPython because our macOS and Windows release processes don't use hosted build platforms that support OpenID Connect and Sigstore. That means that for now we'll keep using release manager identities.

But this future may not be far off for Python packages hosted on PyPI...

Many more signatures are coming!§

William Woodruff and the team at Trail of Bits have authored PEP 740 which is provisionally accepted. The PEP specifies how attestations that can be verified by PyPI (like Sigstore) using workload identities specified with the secure publishing feature "Trusted Publishers" and then served alongside artifacts on PyPI.

There's a lot more to this story (but it's not for me to tell). Given Trusted Publishers' success, there clearly are exciting times are ahead. Subscribe to the PyPI blog to learn more once the project is complete.

That's all for this post! 👋 If you're interested in more you can read the last report.

Have thoughts or questions? Send them my way:

sethmlarson.99 (Signal)
sethmichaellarson@gmail.com
@sethmlarson@fosstodon.org

Want more articles like this one? Get notified of new posts by subscribing to the RSS feed or the email newsletter. I won't share your email or send spam, only whatever this is!

Want more content now? This blog's archive has 111 ready-to-read articles. I also curate a list of cool URLs I find on the internet.

Find a typo? This blog is open source, pull requests are appreciated.

Thanks for reading! ♡ This work is licensed under CC BY-SA 4.0