What is the use case for future.add_done_callback()? - python-asyncio

Question

I understand how to add a callback method to a future and have it called when the future is done. But why is this helpful when you can already call functions from inside coroutines?
Callback version:
def bar(future):
# do stuff using future.result()
...
async def foo(future):
await asyncio.sleep(3)
future.set_result(1)
loop = asyncio.get_event_loop()
future = loop.create_future()
future.add_done_callback(bar)
loop.run_until_complete(foo(future))
Alternative:
async def foo():
await asyncio.sleep(3)
bar(1)
loop = asyncio.get_event_loop()
loop.run_until_complete(foo())
When would the second version not be available/suitable?

Answer 1

In the code as shown, there is no reason to use an explicit future and add_done_callback, you could always await. A more realistic use case is if the situation were reversed, if bar() spawned foo() and needed access to its result:
def bar():
fut = asyncio.create_task(foo())
def when_finished(_fut):
print("foo returned", fut.result())
fut.add_done_callback(when_finished)
If this reminds you of "callback hell", you are on the right track - Future.add_done_callback is a rough equivalent of the then operator of pre-async/await JavaScript promises. (Details differ because then() is a combinator that returns another promise, but the basic idea is the same.)
A large part of asyncio is implemented in this style, using non-async functions that orchestrate async futures. That basic layer of transports and protocols feels like a modernized version of Twisted, with the coroutines and streams implemented as a separate layer on top of it, a higher-level sugar. Application code written using the basic toolset looks like this.
Even when working with non-coroutine callbacks, there is rarely a good reason to use add_done_callback, other than inertia or copy-paste. For example, the above function could be trivially transformed to use await:
def bar():
async def coro():
ret = await foo()
print("foo returned", ret)
asyncio.create_task(coro())
This is more readable than the original, and much much easier to adapt to more complex awaiting scenarios. It is similarly easy to plug coroutines into the lower-level asyncio plumbing.
So, what then are the use cases when one needs to use the Future API and add_done_callback? I can think of several:
Writing new combinators.
Connecting coroutines code with code written in the more traditional callback style, such as this or this.
Writing Python/C code where async def is not readily available.
To illustrate the first point, consider how you would implement a function like asyncio.gather(). It must allow the passed coroutines/futures to run and wait until all of them have finished. Here add_done_callback is a very convenient tool, allowing the function to request notification from all the futures without awaiting them in series. In its most basic form that ignores exception handling and various features, gather() could look like this:
async def gather(*awaitables):
loop = asyncio.get_event_loop()
futs = list(map(asyncio.ensure_future, awaitables))
remaining = len(futs)
finished = loop.create_future()
def fut_done(fut):
nonlocal remaining
remaining -= 1
if not remaining:
finished.set_result(None) # wake up
for fut in futs:
fut.add_done_callback(fut_done)
await finished
# all awaitables done, we can return the results
return tuple(f.result() for f in futs)
Even if you never use add_done_callback, it's a good tool to understand and know about for that rare situation where you actually need it.