What's coming in Python 3.8

LWN.net needs you! Without subscribers, LWN would simply not exist. Please consider signing up for a subscription and helping to keep LWN publishing

The Python 3.8 beta cycle is already underway, with Python 3.8.0b1 released on June 4, followed by the second beta on July 4. That means that Python 3.8 is feature complete at this point, which makes it a good time to see what will be part of it when the final release is made. That is currently scheduled for October, so users don't have that long to wait to start using those new features.

The walrus operator

The headline feature for Python 3.8 is also its most contentious. The process for deciding on PEP 572 ("Assignment Expressions") was a rather bumpy ride that eventually resulted in a new governance model for the language. That model meant that a new steering council would replace longtime benevolent dictator for life (BDFL) Guido van Rossum for decision-making, after Van Rossum stepped down in part due to the "PEP 572 mess".

Out of that came a new operator, however, that is often called the "walrus operator" due to its visual appearance. Using ":=" in an if or while statement allows assigning a value to a variable while testing it. It is intended to simplify things like multiple-pattern matches and the so-called loop and a half, so:

    m = re.match(p1, line)
    if m:
        return m.group(1)
    else:
        m = re.match(p2, line)
        if m:
            return m.group(2)
        else:
            m = re.match(p3, line)
            ...

becomes:

    if m := re.match(p1, line):
        return m.group(1)
    elif m := re.match(p2, line):
        return m.group(2)
    elif m := re.match(p3, line):
        ...

And a loop over a non-iterable object, such as:

    ent = obj.next_entry()
    while ent:
        ...   # process ent
	ent = obj.next_entry()

can become:

    while ent := obj.next_entry():
        ... # process ent

These and other uses (e.g. in list and dict comprehensions) help make the intent of the programmer clearer. It is a feature that many other languages have, but Python has, of course, gone without it for nearly 30 years at this point. In the end, it is actually a fairly small change for all of the uproar it caused.

Debug support for f-strings

The f-strings (or formatted strings) added into Python 3.6 are quite useful, but Pythonistas often found that they were using them the same way in debugging output. So Eric V. Smith proposed some additional syntax for f-strings to help with debugging output. The original idea came from Larry Hastings and the syntax has gone through some changes, as documented in two feature-request issues at bugs.python.org. The end result is that instead of the somewhat cumbersome:

    print(f'foo={foo} bar={bar}')

Python 3.8 programmers will be able to do:

    print(f'{foo=} {bar=}')

In both cases, the output will be as follows:

    >>> foo = 42
    >>> bar = 'answer ...'
    >>> print(f'{foo=} {bar=}')
    foo=42 bar=answer ...

Beyond that, some modifiers can be used to change the output, "!s" uses the str() representation, rather than the default repr() value and "!f" will be available to access formatting controls. They can be used as follows:

    >>> import datetime
    >>> now = datetime.datetime.now()
    >>> print(f'{now=} {now=!s}')
    now=datetime.datetime(2019, 7, 16, 16, 58, 0, 680222) now=2019-07-16 16:58:00.680222

    >>> import math
    >>> print(f'{math.pi=!f:.2f}')
    math.pi=3.14

One more useful feature, though it is mostly cosmetic (as is the whole feature in some sense), is the preservation of spaces in the f-string "expression":

    >>> a = 37
    >>> print(f'{a = }, {a  =  }')
    a = 37, a  =  37

The upshot of all of that is that users will be able to pretty-print their debugging, log, and other messages more easily. It may seem somewhat trivial in the grand scheme, but it is sure to see a lot of use. F-strings have completely replaced other string interpolation mechanisms for this Python programmer and I suspect I am far from the only one.

Positional-only parameters

Another change for 3.8 affords pure-Python functions the same options for parameters that those implemented in C already have. PEP 570 ("Python Positional-Only Parameters") introduces new syntax that can be used in function definitions to denote positional-only arguments—parameters that cannot be passed as keyword arguments. For example, the builtin pow() function must be called with bare arguments:

    >>> pow(2, 3)
    8
    >>> pow(x=2, y=3)
    ...
    TypeError: pow() takes no keyword arguments

But if pow() were a pure-Python function, as an alternative Python implementation might want, there is no easy way to force that behavior. A function could accept only *args and **kwargs, then enforce the condition that kwargs is empty, but that obscures what the function is trying to do. There are other reasons described in the PEP, but many, perhaps most, are not things that the majority of Python programmers will encounter very often.

Those that do, however, will probably be pleased that they can write a pure-Python pow() function, which will behave the same as the builtin, as follows:

    def pow(x, y, z=None, /):
	r = x**y
	if z is not None:
	    r %= z
	return r

The "/" denotes the end of the positional-only parameters in an argument list. The idea is similar to the "*" that can be used in an argument list to delimit keyword-only arguments (those that must be passed as keyword=...), which was specified in PEP 3102 ("Keyword-Only Arguments"). So a declaration like:

    def fun(a, b, /, c, d, *, e, f):
        ...

Says that a and b must be passed positionally, c and d can be passed as either positional or by keyword, and e and f must be passed by keyword. So:

    fun(1, 2, 3, 4, e=5, f=6)          # legal
    fun(1, 2, 3, d=4, e=5, f=6)        # legal
    fun(a=1, b=2, c=3, d=4, e=5, f=6)  # illegal

It seems likely that most Python programmers have not encountered "*"; "/" encounter rates are likely to be similar.

A movable __pycache__

The __pycache__ directory is created by the Python 3 interpreter (starting with 3.2) to hold .pyc files. Those files contain the byte code that is cached after the interpreter compiles .py files. Earlier Python versions simply dropped the .pyc file next to its .py counterpart, but PEP 3147 ("PYC Repository Directories") changed that.

The intent was to support multiple installed versions of Python, along with the possibility that some of those might not be CPython at all (e.g. PyPy). So, for example, standard library files could be compiled and cached by each Python version as needed. Each would write a file of the form "name.interp-version.pyc" into __pycache__. So, for example, on my Fedora system, foo.py will be compiled when it is first used and __pycache__/foo.cpython-37.pyc will be created.

That's great from an efficiency standpoint, but may not be optimal for other reasons. Carl Meyer filed a feature request asking for an environment variable to tell Python where to find (and put) these cache files. He was running into problems with permissions in his system and was disabling cache files as a result. So, he added a PYTHONPYCACHEPREFIX environment variable (also accessible via the -X pycache_prefix=PATH command-line flag) to point the interpreter elsewhere for storing those files.

And more

Python 3.8 will add a faster calling convention for C extensions based on the existing "fastcall" convention that is used internally by CPython. It is exposed in experimental fashion (i.e. names prefixed with underscores) for Python 3.8, but is expected to be finalized and fully released in 3.9. The configuration handling in the interpreter has also been cleaned up so that the language can be more easily embedded into other programs without having environment variables and other configuration mechanisms interfere with the installed system Python.

There are new features in various standard library modules as well. For example, the ast module for processing Python abstract syntax trees has new features, as do statistics and typing. And on and on. The draft "What's New In Python 3.8" document has lots more information on these changes and many others. It is an excellent reference for what's coming in a few months.

The status of PEP 594 ("Removing dead batteries from the standard library") is not entirely clear, at least to me. The idea of removing old Python standard library modules has been in the works for a while, the PEP was proposed in May, and it was extensively discussed after that. Removing unloved standard library modules is not particularly controversial—at least in the abstract—until your favorite module is targeted, anyway.

The steering council has not made a pronouncement on the PEP, nor has it delegated to a so-called BDFL-delegate. But the PEP is clear that even if it were accepted, the changes for 3.8 would be minimal. Some of the modules may start raising the PendingDeprecationWarning exception (many already do since they have been deemed deprecated for some time), but the main change will be in the documentation. All of the 30 or so modules will be documented as being on their way out, but the actual removal will not happen until the 3.10 release—three or so years from now.

The future Python release cadence is still under discussion; currently Python 3.9 is scheduled for a June 2020 release, much sooner than usual. Python is on an 18-month cycle, but that is proposed to change to nine months (or perhaps a year). In any case, we can be sure that Python 3.8 will be here with the features above (and plenty more) sometime before Halloween on October 31.

Index entries for this article
Python	Releases

---

(Log in to post comments)

def messagef(fmt, *args, **kwargs):
    ...
    fmt.format(*args, **kwargs)
    ...

m = re.match(p1, line)
if m:
    return m.group(1)
m = re.match(p2, line)
if m:
    return m.group(2)
m = re.match(p3, line)

#m[01x_]

if (p[0] == 0m0xxx_xxxx) {
...
} else if (p[0] == 0m110x_xxxx && p[1] == 0m10xx_xxxx) {
...
} ...

    m = re.match(p1, line)
    if m:
        # do something with m
    else:
        m = re.match(p2, line)
        if m:
            # do something with m
        else:
            m = re.match(p3, line)
            ...

def test_bitmask(value: int, mask: str) -> bool:
    mask = mask.replace("_", "")
    testval: int = 1
    for bit in reversed(mask):
        if bit == '1' and not (value & testval):
            return False
        elif bit == '0' and (value & testval):
            return False
        elif bit != "x":
            raise ValueError('Invalid bitmask character: '+bit)
        testval <<= 1
    return True

assert( test_bitmask(0x80, '1xxx_xxxx') == True )
assert( test_bitmask(0x3f, '0xx1_xxxx') == True )
assert( test_bitmask(0x3f, '0xx1_xxx0') == False )

class Bitmask(object):
    def __init__(self, mask: str):
        self._mask = list(reversed(mask.replace('_', '')))

    def _test(self, value: int) -> bool:
        testval: int = 1
        for bit in self._mask:
            if bit == '1':
                if not (value & testval):
                    return False
            elif bit == '0':
                if value & testval:
                    return False
            elif bit != "x":
                raise ValueError(f'Invalid bitmask character: "{bit}"')
            testval <<= 1
        return True

    def __eq__(self, other):
        if type(other) == Bitmask:
            return other._mask == self._mask
        if type(other) != type(0):
            return NotImplemented
        return self._test(other)

tests = [
    "0x80 == Bitmask('1xxx_xxxx')",
    "0x3f == Bitmask('0xx1_xxxx')",
    "0x3f != Bitmask('0xx1_xxx0')",
    "0x40 != Bitmask('0000_xxxx')",
    "'0x80' == Bitmask('1xxx_xxx')",
    "None == Bitmask('1xxx_xxx')",
    "None != Bitmask('1xxx_xxx')",
    "Bitmask('1xxx_xxx') == Bitmask('1xxx_xxx')",
    "Bitmask('1xxx_xxx') == Bitmask('xxxx_xxx')",
    "Bitmask('1xxx_xxx') == Bitmask('0xxx_xxx')",
]

for test in tests:
    print(f'{test}: {eval(test)}')