Python is (mostly) made of syntactic sugar

Benefits for LWN subscribers The primary benefit from subscribing to LWN is helping to keep us publishing, but, beyond that, subscribers get immediate access to all site content and access to a number of extra site features. Please sign up today!

"Sugar" is, to a certain extent, in the eye of the beholder—at least when it comes to syntax. Programming languages are often made up of a (mostly) irreducible core, with lots of sugary constructs sprinkled on top—the syntactic sugar. No one wants to be forced to do without the extra syntax—at least not for their favorite pieces—but it is worth looking at how a language's constructs can be built from the core. That is just what Brett Cannon has been doing for Python, on his blog and in talks, including a talk at PyCon back in April (YouTube video).

Cannon began with a definition of "syntactic sugar" as "syntax that is not necessary to get the semantic results you want in the end". But, eliminating the sugar will not necessarily result in code that is as fast as it would be with those "optional" constructs. In fact, eliminating some of them will "abuse the way the language works" in various ways. The idea is that if a construct had not been added to the language by the core developers, the same result can be obtained, "but probably in a much messier, long-winded way".

He set some rules for himself in his investigation, including that any syntactic-sugar replacement needed to work in Python 3.8, which was "the newest hotness" at the time he started. He stuck to things that were in the language specification and avoided things that were CPython-specific. He also restricted his techniques to those that could be applied to a single Python file, as opposed to whole-program analysis; he wanted to be able to write a tool that would apply the de-sugaring to a file such that the output would be a drop-in replacement for the original.

He was willing to live with some oddities in highly dynamic parts of the runtime, such as in locals() and globals(). In fact, he said, that already happens; inside of a list comprehension, there is an "extra" entry in locals() for a "magic little variable where we construct that list for you". Since CPython cheats, he decided that he could too. Beyond that, he did not take performance into consideration at all; it was purely a theoretical exercise.

Required syntax

As it turns out, "there are only ten pieces of syntax that you cannot live without"; he was able to eliminate more than 80 sugary pieces. He thinks, though, that once attendees see what has to be done to avoid some of them, they will agree that "a lot of this syntactic sugar is really handy to have".

The first element that you need is integers, which are the building blocks of lots of other things. A float can be represented as integers, bytes are arrays of integers, and Unicode strings are just arrays of codepoints, which are ... integers. "You can build all of the data representations that we have in the language using just integers." Everyone likes string literals, though, so even though you can reduce them to integers, that's not a pleasant path; that's true for other representations, as well, of course.

Function calls are another required piece; you need some way to call code. The while statement can provide both looping and conditional constructs, which makes it more useful than either for or if; those can only handle one or the other. So while is required and the other two can be defined in terms of it.

The try/except statement is another required piece. He could actually get rid of that requirement if he was willing to do the whole-program analysis and rewrite all of the functions in the style of Rust, with return values that are either "ok" or "an exception got raised". The raise statement is also required because of the no-CPython-internals restriction he placed on his experiment.

def is required in order to create functions that provide the API that other parts of the program can call. Both return and yield are needed, though there are ways around that; he could have brought back lambda, which would have allowed eliminating them from the list. Cannon said that he was "too lazy" to do so when he learned how; it is quite a bit of work and eliminating class had already worn him out.

The final two pieces of required syntax are assignment, which is perhaps an obvious one, and nonlocal, which is decidedly less so. The original namespace picture for Python was much simpler, but less powerful; it consisted of local, global, and built-in namespaces. The addition of closures complicated that picture; Cannon needed to keep nonlocal as one of his ten core syntax pieces in order to be able to control access to other namespaces.

So, just those ten pieces of syntax are enough to reimplement every other part of Python, including import, addition, class, and more. Technically, you can write your Python code just using those pieces, "you'd just hate yourself because you'd be having to write the weirdest code in the world".

Essentially, those constructs make up a Turing machine "with some Python flourishes", Cannon said. A Turing machine is more or less "the scientific definition of a computer" that can read, write, and process data; it can also make conditional decisions based on the data. while is available to make decisions, def and function calls for processing data, assignment for reading and writing, and so on. Things like try and except add the Python flourishes.

Examples

On his blog, Cannon has unraveled over 80 other pieces of syntax, but was not able to do all of that in a talk. Instead, he would give some examples that would hopefully give attendees an idea of what it entails. In addition, he hopes that it gives people a look under the hood in order to better understand how Python operates.

He started with a simple example of attribute access for the expression X.Y. That can be replaced with:

    getattr(X, "Y")

Fundamentally, every attribute access in Python is reduced to a call to getattr(). It quickly gets tiring to write attribute accesses that way, however, especially for long chains of them; in fact, he would continue to use the sugary version for the other examples in his talk, he said.

Sometimes, the language definition makes things a bit interesting, as with handling decorators. A Python decorator wraps a function definition, generally to transform the function in some fashion:

    # pre-decorators
    def Y():
        pass
    Y = X(Y)

    # using decorators
    @X
    def Y():
        pass

The decorator syntax was added to make it clearer what was going on right at the top of the function definition; if the function is long, it may be easy to miss that it is getting wrapped and re-assigned. But, as it turns out, the unraveled version cannot be the same as the pre-decorator version above; instead it must look like the following:

    def _Y():
        pass
    Y = X(_Y)

The language specification specifically requires that the new function name (Y) not be visible until after the wrapping has occurred, which surprised him when he was working on this. He believes that the decorator syntax is far more readable and is a great example of why syntactic sugar is added to the language. He knows that Python developers do not always agree with changes of that sort, pointing to the mess around the "walrus" operator (:=) as an obvious case in point, but "we have our reasons" for doing so.

He moved on to "special methods" (or "magic methods"), which are all of the "dunder" (double-underscore) method names on objects, __add__(), __index__(), __init__(), and so on. These are an area where syntactic sugar comes up the most and is the easiest to unravel, because those methods fundamentally implement the operations of interest. For example, ~X (using the bitwise inversion operator) unravels as: type(X).__invert__(X). That may be a little surprising, but the type() call is actually an optimization mechanism; it is much faster to retrieve __invert__() from the class, than from the instance dictionary. In the C code, it just requires following a few pointers to make the call:

    (*Py_TYPE(X)->tp_as_number->nb_invert)(X);

Even though dictionary lookup is fast, the above is "stupid fast", he said.

He moved on to the "most complex example we're going to have in this whole talk: binary operators". He said that attendees may think those operators should be fairly simple to unravel, but they will be surprised. His example was a simple subtraction: X ‑ Y. The operator does not really matter as the process is the same; he chose subtraction because he wanted to show an operator where the order of the operands matters. For subtraction, the first step is:

    type(X).__sub__(X, Y)

That seems pretty straightforward, but what if X does not define __sub__() or if it returns NotImplemented for the type of Y? In that case, the second operand gets a chance to provide an implementation:

    type(Y).__rsub__(Y, X)

The language defines the __rop__() versions of the operators to handle this case. The dunder operators return NotImplemented if they are unable to work with the two objects passed. Another optimization is that if both operands have the same type, only the first dunder operator (__sub__() in this case) will be tried, since there is no point in asking again as it is already known that the two types cannot be accommodated.

But there are class hierarchies where a subclass needs to be able to override the parent class's operator, no matter which side of the operator it falls on. For example, if Y is a subclass of the class of X, the class of Y may want to ensure that subtraction always results in that type. There is a set of rules that Python follows to ensure that such a situation is properly handled: the two types must be different, the right-hand side (RHS), Y in this case, must be a subclass of the left, both operands must have an __rsub__(), and the __rsub__() for Y must be different than that of X. If all of those are true, the RHS can boss its parent around. "Got it all?", he asked with a smile.

That is quite a bit of work to do every time that a binary operator is executed. That is why various efforts are made to avoid that overhead, such as Python extensions that move the operations into C code and the work being done on specialization by the Faster CPython team.

As his last example, he deconstructed if:

    if X:
        Y

    # can use while instead
    while X:
        Y
	break

However, that while version uses break, which is not one of the ten anointed syntax constructs. But break can be unraveled as well:

    _DONE = False
    while not _DONE and X:
        Y
	_DONE = True

But that contains two more pieces of syntax (not and and) that are not on the list. not is similar to the invert operation; it is just a special method call. In the talk, he did not discuss and at all, but for this particular case, he simply avoided both operators by using exceptions:

    try:
        while X:
	    Y
	    raise _DONE
    except _DONE:
        pass

"Once again, syntactic sugar is really frickin' handy, because I sure as heck would not want to write that every time I have an if." He almost forgot to point out that pass was also not on the list, but it could be replaced here with various things, such as None or a constant. His blog posts on not and and show other ways this could be handled.

He ended his talk by thanking his wife Andrea for proofreading most of the blog posts; he stopped asking after a while, so she is not responsible for typos in the later posts, he said with a chuckle. He also thanked Guido van Rossum for proofreading the class unraveling post, "which was huge". Beyond that, Van Rossum was always happy to answer the random questions Cannon posted to the python-dev mailing list about why things are the way they are in the language. Cannon suggested that those who are interested read the blog posts, perhaps starting with the last, "MVPy: Minimum Viable Python", which outlines the goals of the project and lists his ten syntactic elements.

In answer to an inaudible question, which must have referred to names like False and None, Cannon said that those were originally simply variable names and not syntax in Python, so he went back to that mode for his project. Those names were only turned into syntax as an aid to developers so that they did not overwrite them by mistake. The final question was about the != operator used in the subtraction example, which is, of course, also not on the list. Cannon acknowledged that, and pointed attendees at the rich comparison operators post for how that can be unraveled.

Index entries for this article
Conference	PyCon/2023
Python	Language specification

---

(Log in to post comments)

for i in range(10):
    print(i)

i = 0
while i < 10:
    print(i)
    i += 1