Using JavaScript-style async promises in Python

Posted by
on under

When you compare Python and JavaScript async code, you may think they are very much alike, since both use the async/await pattern. In reality, there is a significant difference between the two, which in my opinion, makes the JavaScript async model superior to Python's.

Do you want to know what the difference is, and how to make async Python work like async JavaScript? Watch or read on for the details!

How Async Works in Python

Let's run a quick experiment in the Python shell. Instead of starting the shell by just typing python, use the following command, which allows you to use await directly in the Python prompt:

python -m asyncio

Let's write a quick async function to have something to test with:

async def f():
    print('f is running')
    return 42

The function has a print statement, so that we can see in the terminal when it runs. To keep things simple, this function just returns a number. A longer function that performs asynchronous tasks would work just as well, so feel free to make the function more complex with some await statements before the return value if you like.

Let's run the function and get its result:

>>> result = f()
>>> result
<coroutine object f at 0x109a9f1c0>

Interesting, right? Calling the function doesn't really run it. If the function had executed, we would see the printed text in the terminal, so we know that the function did not run. You can see that calling the async function just returned a coroutine object without actually running the function. The coroutine object is now stored in the result variable.

So how can we get the function to run? We have to "await" the coroutine:

>>> await result
f is running
42

In my opinion the way this works in Python is extremely counterintuitive. This is the main aspect of the Python async model that I dislike.

In practice, you will find that the function invocation is combined with the await, all in a single statement, which somewhat hides the odd behavior we just experienced:

>>> await f()
f is running
42

But this usage has a problem that bites me often. I frequently forget to add the await keyword before the function invocation, and when this happens the function doesn't run, but execution continues without any errors, because calling the function just to get a coroutine is a perfectly legal thing to do. To be fair, Python does print a warning when you fail to await a coroutine, but this happens when the process exits.

How Async Works in JavaScript

Should we repeat the experiment in JavaScript to learn about the differences in async implementations?

Here is how to start node with support for awaiting in the REPL prompt:

node --experimental-repl-await

Below you can see a similar f() function, with a print statement and a return of a number:

const f = async () => {
  console.log('f is running');
  return 42;
};

Let's call this function and see what happens:

> let result = f();
f is running
undefined

So this is interesting! In JavaScript, when you call an async function, the function executes, as you would expect. Note that the undefined is the return value of the let statement, not the value that was assigned to the result variable.

Let's take a look at result:

> result
Promise {
  42,
  [Symbol(async_id_symbol)]: 357,
  [Symbol(trigger_async_id_symbol)]: 5,
  [Symbol(destroyed)]: { destroyed: false }
}

So our result variable received a Promise, which is a core building block of the JavaScript async model.

In JavaScript, an async function is defined as a function that returns a promise. You can write it as a standard function (i.e. without the async keyword) and return the Promise object explicitly, or if you prefer, you can use the async qualifier and then the JavaScript compiler creates and returns the promise for you.

Unlike in Python, async JavaScript functions are normal functions. If you call them, they execute. This is a huge advantage, as it avoids having to treat async functions differently than regular ones.

So what is a promise, anyway? A promise is an object that is associated with an asynchronous task. An asynchronous function is supposed to launch the background work and immediately return a promise, without waiting for this work to complete. The caller can then use the promise object to determine when the work has completed.

How do you get the result from a promise? There are a couple of ways, but the most convenient is to await the promise:

> await result;
42

So here is what makes a lot of people think that JavaScript and Python have the same async solution. When you combine calling the function with the await, both languages look exactly the same. Check out how the JavaScript async function is called in a single line:

> await f();
42

This is identical to the Python solution. But of course, if you forget the await in JavaScript, the function still runs. You will get some concurrency issues because the function will run in the background while the calling code continues to run concurrently, but at least the function will execute, which in my view is a much better outcome.

The Promise class in JavaScript is extremely powerful. It provides lots of interesting ways to work with concurrency beyond async/await. For example, its then() method makes it possible to create chains of functions that execute asynchronously. Below you can see how to attach a callback (the console.log function) to our promise. JavaScript will run the provided function, with the result of the asynchronous function passed as an argument.

> result.then(console.log);
Promise {
  <pending>,
  [Symbol(async_id_symbol)]: 636,
  [Symbol(trigger_async_id_symbol)]: 357,
  [Symbol(destroyed)]: { destroyed: false }
}
42

Making Python Async More Like JavaScript Async

I always wondered if it would be possible to make Python use a promise-based async solution. The other day I spent a few hours trying to replicate promises in Python, and the result is a new package that I called promisio. You can install it with pip:

pip install promisio

Promisio includes a JavaScript-compatible Promise class, but most importantly, it comes with a promisify decorator that can convert any function (async or not) to a promise-based function that works as in JavaScript:

from promisio import promisify

@promisify
async def f():
    print('f is running')
    return 42

A function decorated with promisify can be called directly, and it runs, returning a promise that can be awaited to get the result:

>>> result = f()
f is running
>>> result
<promisio.TaskPromise object at 0x10bc5ad60>
>>> await result
42

The decorator also works on regular functions, because as in JavaScript, the only requirement is that the function returns a promise. There is no requirement that an asynchronous function is defined with the async keyword:

@promisify
def g(x):
    print('g is running')
    return x * 2

Regardless of the original function being async or not, the resulting promise-based function is always async and must be awaited to get a result:

>>> await g(21)
g is running
42

The Promise class that I implemented in the promisio package matches the JavaScript one in functionality, so all the concurrency features available in JavaScript can also be used in Python. It is especially interesting that these features do not require the use of async and await, opening the door to using async features in regular functions, something that is quite difficult in Python. Here is how to use the then() method to configure a completion callback that prints the result once it becomes available:

>>> result = g(21)
g is running
>>> result.then(print)
<promisio.Promise object at 0x10bc76280>
42

Conclusion

I'm quite happy with the promise implementation I made for Python, and I hope you decide to try it out. Please let me know what you think below in the comments!

Become a Patron!

Hello, and thank you for visiting my blog! If you enjoyed this article, please consider supporting my work on this blog on Patreon!

19 comments
  • #1 Jose Laurentino III said

    Hello Miguel:

    Very interesting work. Thanks for taking the time to provide the information and details.

    Oh PHP/JavaScript I over-used promise for images throughout my entire application. That gave me the security of not allowing direct access to any images in the application. On the top of that, it also helped as a way to block hackers by running their patience out with so much bits-n-bytes.

    I just got involved with Python and want to fast forward. I just learned a bit about Python-with-ajax (from Django Central) and believe that your library--promisio--will be of great value at some point as I learn more.

    Thank you and happy thanksgiving!

    Laurentino

  • #2 Jacob said

    Just a minor correction: Promises in JS don't run in parallel. They run out of order (outside of the synchronous flow (hence asynchronous). All that means is that the promise won't necessarily be evaluated in the order in which it was declared.
    The only way to achieve true parallelism in JS is with the platform's implementation of a worker thread.

    Just thought I'd mention it

  • #3 Miguel Grinberg said

    @Jacob: yes, but to be correct, promises don't run at all, parallel or not. A promise is just an object that holds the state of a task. Tasks are what runs. I have edited out the reference to running in parallel, since you are correct in that this word is used to describe a different type of concurrency that is not what this article is about. What I meant to say is that the async task will run at the same time as the calling task. This is true for both languages.

  • #4 RyomaHan said

    Very cool attempt.

  • #5 Duncan Booth said

    One minor point, while it's true that the async function returns a coroutine that won't run until you await it for result = f() you can instead do result = asyncio.create_task(f()) which convert the coroutine into a Task object. The Task object is a Future and behaves similarly to Javascript Promise, the coroutine is not a Future although it is awaitable.

    Once you've wrapped the coroutine in a Task you can do pretty much anything you could with a Promise in Javascript such as awaiting the same Task multiple times (a coroutine throws an exception if you await it a second time, Task just gives back the completed value each time) or calling .then on a completed Promise (in Python that's calling .add_done_callback() on a Future or Task).

  • #6 Miguel Grinberg said

    @Duncan: Of course. If you couldn't work in the style of promises in Python, it wouldn't be possible to create a library that emulates those promises! The point I'm trying to make is that the API to work in this way in Python is horrible. Promises have an elegant API (at least my opinion), unlike the Future and Task classes in Python which appear to be designed mostly to be used internally.

  • #7 adatron said

    Hi Miguel,
    Great package and post, thank you! :)
    I'm newbie at the Python, I'm not familiar with dynamic programming patterns, so it's possible the answer will be easy.

    How can I achive a pattern something like that (or in a another way but same result)? :

    class PromiseTest {
    
        getResult(){
    
            return new Promise ( (resolve) => {
    
                this.commit = ( _ => {resolve(1); });
                this.rollback = ( _ => {resolve(0); });
    
            });
    
        }
    
    }
    
    let promiseTest = new PromiseTest();
    
    console.log('result:',await promiseTest.getResult());
    
    setTimeout( async () =>  promiseTest.commit(), 1000);
    

    I tried that with your package and the asyncio but I did not find the solution.

    Thanks in advance!

  • #8 Miguel Grinberg said

    @adatron: I'm not sure what is the utility of your example, but in any case, here is a Python version of it:

    import asyncio
    from promisio import Promise
    
    class PromiseTest:
        def __init__(self):
            self.commit = None
            self.rollback = None
    
        def get_result(self):
            def f(resolve, reject):
                self.commit = lambda: resolve(1)
                self.rollback = lambda: resolve(0)
    
            return Promise(f)
    
    async def main():
        promise_test = PromiseTest()
    
        async def timer():
            await asyncio.sleep(1)
            promise_test.commit()
    
        asyncio.create_task(timer())
        print('result:', await promise_test.get_result())
    
    asyncio.run(main())
    
  • #9 adatron said

    Thank you very much @Miguel for a very quick solution!

    Works well!

    (By the way, this is a simplified snippet of code (for the sake of example).) Asynchronous tasks are handled in an array. This solution helps manage their life cycle).

    P.S .: You're fantastic that you didn't want to suggest another solution right away, instead, you might have thought it might make sense. This is very rare in these technical forums. :). Your new fan is born

  • #10 Sabine said

    Super cool explanation!

  • #11 Olivier said

    Hi Miguel

    I came across your package while looking how to convert the following javascript code (puppeteer context) to python, but I can't find the right way to do it . So I was wondering if you could help here ...

      const wait_for_frame = (page, frame_name) => {
        const check_frame = () => {
          const frame = page.frames().find(current_frame => current_frame.name() === frame_name);
          if (frame) {
            fulfill_promise(frame);
          } else {
            page.once('frameattached', check_frame);
          }
        };
    
        let fulfill_promise;
        const promise = new Promise(x => fulfill_promise = x);
        check_frame();
        return promise;
      };
    
    let myframe = await wait_for_frame(page, 'my frame name');
    
  • #12 Miguel Grinberg said

    @Olivier: This is a somewhat convoluted function that mixes async/await, promises and callbacks. Not sure this translates 1:1 into Python and I'm not even sure the Python version of puppeteer works in the same way. I recommend that you create a new function that waits for the frame by checking for it at regular intervals, for example, which is much simpler than this.

  • #13 Nathan M said

    hi Miguel,

    how to recreate this in python using promisio

    let promise1 = new Promise(resolve => {
        setTimeout(() => resolve(), 2000);
    });
    let promise2 = new Promise(resolve => {
        setTimeout(() => resolve(), 5000);
    });
    
    async function c() {
        console.log(2);
        await promise1;
        console.log(4);
        await promise2;
        console.log(5);
    }
    
    console.log(1);
    c();
    console.log(3); 
    

    key point is that "console.log(3);" runs when first promise gets awaited inside c

  • #14 Miguel Grinberg said

    @Nathan: Here is one way to do it:

    import asyncio
    from promisio import Promise, promisify
    
    @promisify
    async def setTimeout(timeout, resolve):
        await asyncio.sleep(timeout)
        resolve(0)
    
    @promisify
    async def c():
        promise1 = Promise(lambda resolve, reject: setTimeout(2, resolve))
        promise2 = Promise(lambda resolve, reject: setTimeout(5, resolve))
        print(2)
        await promise1
        print(4)
        await promise2
        print(5)
    
    async def main():
        print(1)
        p = c()
        print(3)
        await p
    
    asyncio.run(main())
    
  • #15 Nathan M said

    Miguel,

    sadly that outputs "1 3 2 4 5" as compared to JS version where it's "1 2 3 4 5". That's because in JS version, when you hit the first await inside c() , it returns immediately and next line of code is executed (which is console.log(3)), and then the rest of stuff inside c() is just fired once the timeouts expire, which is pretty much like scheduling a task to be executed in future. Not sure if that's due to differences between event loop/run time in JS and PY.

  • #16 Miguel Grinberg said

    @Nathan: Python and JS are not the same language. They have different schedulers so you can't expect identical behavior. I don't think the "2 3" vs "3 2" difference in your example has any significance, though.

  • #17 Nathan M said

    No they are obviously not. Yes it seems due to scheduler differences the way is works in JS with example above is that just by calling the async function c() the execution progresses towards first await inside it and then immediately returns back to executor in order to continue doing other stuff (console.log(3) or possibly many other lines of code after the c() call) and then the promises kick in with their results when they are done kinda in background after the c() call.

    In python it's more 'serial' in a sense that using await will keep you on that specific line of code where it's used until whatever behind that await completes, without this "jumping" as you see in JS. And just by calling the async func you just get an object (coroutine etc), with nothing done yet. You can workaround this (sort of) by creating tasks or manually scheduling Futures, well, or promises like we see here :-)

  • #18 Jeronimo said

    If it weren't a statement you could have called the decorator @async , so it sort of reads like "async def", too ;)
    Btw: does promisify behave equivalently to JS in that it does not wrap a return value into a Promise again if it already is one? So that you can directly return an existing Promise (Future/coroutine/...) without await'ing it explicitly like you have to in Python?

  • #19 Miguel Grinberg said

    @Jeronimo: Sure. As long as you return a promise from your promisified functions everything should work. No need to await it if all you need to do is return its result.

Leave a Comment