How can I run asyncio library code on top of Twisted's asyncioreactor? - interop

Question

I've managed to import/install Twisted's asyncioreactor and execute a trivial asynchronous function:
from twisted.internet import asyncioreactor
asyncioreactor.install()
from twisted.internet import task
from twisted.internet.defer import inlineCallbacks
from twisted.internet.defer import ensureDeferred
async def sleepy(reactor):
print("SLEEPING")
await task.deferLater(reactor, 3.0, lambda: None)
print("done sleep")
return 42
#task.react
def main(reactor):
d = ensureDeferred(sleepy(reactor))
d.addCallback(print)
return d
I'd like to intermix an asyncio library in said code, for instance asyncio.sleep. I've tried the following:
from twisted.internet import asyncioreactor
asyncioreactor.install()
from twisted.internet import task
from twisted.internet.defer import inlineCallbacks
from twisted.internet.defer import ensureDeferred
import asyncio
async def sleepy(reactor):
print("SLEEPING")
await asyncio.sleep(3)
print("done sleep")
return 42
#task.react
def main(reactor):
d = ensureDeferred(sleepy(reactor))
d.addCallback(print)
return d
which produces the following error:
$ python test.py
SLEEPING
main function encountered error
Traceback (most recent call last):
File "test.py", line 16, in <module>
#task.react
File "/Users/blz/.pyenv/versions/3.6.0/lib/python3.6/site-packages/twisted/internet/task.py", line 908, in react
finished = main(_reactor, *argv)
File "test.py", line 18, in main
d = ensureDeferred(sleepy(reactor))
File "/Users/blz/.pyenv/versions/3.6.0/lib/python3.6/site-packages/twisted/internet/defer.py", line 823, in ensureDeferred
return _inlineCallbacks(None, coro, Deferred())
--- <exception caught here> ---
File "/Users/blz/.pyenv/versions/3.6.0/lib/python3.6/site-packages/twisted/internet/defer.py", line 1301, in _inlineCallbacks
result = g.send(result)
File "test.py", line 11, in sleepy
await asyncio.sleep(3)
File "/Users/blz/.pyenv/versions/3.6.0/lib/python3.6/asyncio/tasks.py", line 476, in sleep
return (yield from future)
builtins.AssertionError: yield from wasn't used with future
Fair enough, thought I, so I tried swapping await asyncio.sleep(3) with await ensureDeferred(asyncio.sleep(3)) and await asyncio.ensure_future(asyncio.sleep(3)), but I get exactly the same error.
How can I schedule an aio coroutine (and/or Future) to run on the same event loop as is used by asyncioreactor?

Answer 1

so I tried swapping await asyncio.sleep(3) with await
ensureDeferred(asyncio.sleep(3)) and await
asyncio.ensure_future(asyncio.sleep(3))
You were almost there, you should combine the two, and use Deferred.fromFuture instead of ensureDeferred:
await Deferred.fromFuture(asyncio.ensure_future(asyncio.sleep(3)))
The rule is, async def functions running in Twisted context (with ensureDeferred) can await only on Deferred, and async def functions running in asyncio context (with ensure_future) only on asyncio Future (one can always await on other coroutine objects (results from async def function calls), but the chain will ultimately lead to Deferred/Future). To convert from asyncio Future to Deferred use Deferred.fromFuture and to convert to asyncio Future use Deferred.asFuture.
One can switch contexts from one to another and back. In this (contrived) example we start with sleepy_twisted run in
Twisted context, which does Twisted sleep, then it switches to asyncio context to run sleepy_asyncio which does asyncio sleep, but then switches again to Twisted context for Twisted sleep:
from twisted.internet import asyncioreactor, task
from twisted.internet.defer import inlineCallbacks, ensureDeferred, Deferred
import asyncio
asyncioreactor.install()
async def sleepy_asyncio(reactor):
print("Sleep 2")
await asyncio.sleep(1)
print("Sleep 3")
await Deferred.asFuture(task.deferLater(reactor, 1, lambda: None), loop=asyncio.get_running_loop())
async def sleepy_twisted(reactor):
print("Sleep 1")
await task.deferLater(reactor, 1, lambda: None)
await Deferred.fromFuture(asyncio.ensure_future(sleepy_asyncio(reactor)))
print("done")
#task.react
def main(reactor):
return ensureDeferred(sleepy_twisted(reactor))

Answer 2

Wow, you've come across an interesting corner case!
By using asyncio.sleep() you have triggered some interesting behavior.
I think you might have uncovered a bug in
Twisted's integration with the Python 3 asyncioreactor and async/await.
You may wish to follow up with the Twisted developer's on the Twisted mailing list.
I'm not 100% sure, but here are my thoughts.
The implementation of asyncio.sleep() is tightly coupled to the Python 3
asyncio implementation. It uses the asyncio.Future (which is similar to Twisted's deferred), and it uses get_event_loop() (which is similar to Twisted's reactor).
asyncio.sleep is implemented like this:
#coroutine
def sleep(delay, result=None, *, loop=None):
"""Coroutine that completes after a given time (in seconds)."""
if delay == 0:
yield
return result
if loop is None:
loop = events.get_event_loop()
future = loop.create_future()
h = future._loop.call_later(delay,
futures._set_result_unless_cancelled,
future, result)
try:
return (yield from future)
finally:
h.cancel()
I changed your code example slightly to pass Twisted's
asyncioreactor event loop into asyncio.sleep():
from twisted.internet import asyncioreactor
asyncioreactor.install()
from twisted.internet import reactor
from twisted.internet import task
from twisted.internet.defer import inlineCallbacks
from twisted.internet.defer import ensureDeferred
import asyncio
async def sleepy(reactor):
print("SLEEPING")
await asyncio.sleep(3, loop=reactor._asyncioEventloop)
print("done sleep")
return 42
#task.react
def main(reactor):
d = ensureDeferred(sleepy(reactor))
d.addCallback(print)
return d
I still got the same error as you: builtins.AssertionError: yield from wasn't used with future
The stack trace looks like:
main function encountered error
Traceback (most recent call last):
File "b.py", line 16, in <module>
#task.react
File "/Users/crodrigues/twisted8/src/twisted/internet/task.py", line 908, in react
finished = main(_reactor, *argv)
File "b.py", line 19, in main
d = ensureDeferred(sleepy(reactor))
File "/Users/crodrigues/twisted8/src/twisted/internet/defer.py", line 823, in ensureDeferred
return _inlineCallbacks(None, coro, Deferred())
--- <exception caught here> ---
File "/Users/crodrigues/twisted8/src/twisted/internet/defer.py", line 1301, in _inlineCallbacks
result = g.send(result)
File "b.py", line 12, in sleepy
await asyncio.sleep(3, loop=reactor._asyncioEventloop)
File "/usr/local/Cellar/python3/3.6.0/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/tasks.py", line 478, in sleep
return (yield from future)
builtins.AssertionError: yield from wasn't used with future
I think the asyncio.sleep() is a coroutine that is supposed to run to completion
on the asyncio loop, but this is not happening here, hence the assertion.
I think the problem is being introduced by result = g.send(result).
I'm not sure you can send() to a coroutine like this and expect it to work.
I advise you to ask on the Twisted mailing list to get more detailed feedback.