I'm learning about asycio and trying to run this script I get this error:
ValueError: a coroutine was expected, got <async_generator object
mygen at 0x7fa2af959a60>
What am I missing here?
import asyncio
async def mygen(u=10):
"""Yield powers of 2."""
i = 0
while i < int(u):
yield 2 ** i
i += 1
await asyncio.sleep(0.1)
asyncio.run(mygen(5))
The asyncio.run function expects a coroutine but mygen is an asynchronous generator.
You can try something like this:
test.py:
import asyncio
async def mygen(u=10):
"""Yield powers of 2."""
i = 0
while i < int(u):
yield 2**i
i += 1
await asyncio.sleep(0.1)
async def main():
async for i in mygen(5):
print(i)
if __name__ == "__main__":
asyncio.run(main())
Test:
$ python test.py
1
2
4
8
16
References:
What does the "yield" keyword do?
PEP 525 -- Asynchronous Generators
Related
A simple decorate to calculate time a function takes to run:
import time
def decor(f):
starttime=time.time()
f()
print("child functoin run time is ", (time.time()-starttime)*1000, "ms")
return f
try to use it to decorate async functions:
async def sleep_and_print(seconds):
print(f"starting async {seconds} sleep 😴")
await asyncio.sleep(seconds)
print(f"finished async {seconds} sleep ⏰")
return seconds
#decor
async def main():
# using arguments
results = await asyncio.gather(sleep_and_print(3), sleep_and_print(6))
print(results)
asyncio.run(main())
I got RuntimeWarning: coroutine 'main' was never awaited error
If I change the decorator function to async and await, eg
async def decor(f):
starttime=time.time()
await f()
print("child functoin run time is ", (time.time()-starttime)*1000, "ms")
return f
asyncio.run(main()) failed with coroutine' object is not callable
Why main() becomes uncallable? Any suggestion on the work around?
Try to return async function from the decorator function:
import time
import asyncio
def decor(f):
async def _fn():
starttime = time.time()
await f()
print(
"child function run time is ",
(time.time() - starttime) * 1000,
"ms",
)
return _fn
async def sleep_and_print(seconds):
print(f"starting async {seconds} sleep 😴")
await asyncio.sleep(seconds)
print(f"finished async {seconds} sleep ⏰")
return seconds
#decor
async def main():
# using arguments
results = await asyncio.gather(sleep_and_print(3), sleep_and_print(6))
print(results)
asyncio.run(main())
Prints:
starting async 3 sleep 😴
starting async 6 sleep 😴
finished async 3 sleep ⏰
finished async 6 sleep ⏰
[3, 6]
child function run time is 6002.1820068359375 ms
Most of the time I am using asyncio API. But how would I create an async generator "from scratch"?
Say I have a classic generator, and I want to make it async. How can I do that?
I naively thought that I could do something like below, but it does not run asynchronously (the 3 "for loops" are running one after the other, instead of concurrently). My hope was to make some_loop() asynchronous by calling it from some_async_loop() and releasing the event loop after each iteration with asyncio.sleep(0):
#!/usr/bin/env python3
import asyncio
async def run():
task = asyncio.ensure_future(run_async_loop())
asyncio.ensure_future(run_async_loop())
asyncio.ensure_future(run_async_loop())
await task
async def run_async_loop():
async for i in some_async_loop():
print(i)
async def some_async_loop():
for i in some_loop():
yield i
asyncio.sleep(0)
def some_loop():
for i in range(10):
yield i
if __name__ == "__main__":
loop = asyncio.get_event_loop()
loop.run_until_complete(run())
I'm glad the fixing the call to asyncio.sleep() got things running.
I'm concerned partly about your run() function, because you're only await-ing on the first task, which means your code could exit before the other tasks are complete. I would suggest this:
async def run():
tasks = [run_async_loop(), run_async_loop(), run_async_loop()]
await asyncio.gather(*tasks)
I think you can also simplify your __main__ block:
if __name__ == '__main__':
asyncio.run(run())
How can I invoke f and reach the next line?
from SomeLib import f
f()
print('never reaches')
I would prefer not to mess with the internals of 'SomeLib', but f it's a quick fix I'll do it:
def f():
asyncio.get_event_loop.run_until_complete(ag())
async def ag():
async with websockets.client.connect(...) as websocket:
:
await wait_for_recv(...)
async def wait_for_recv(...):
while True:
message = await asyncio.wait_for(websocket.recv(), timeout=time_out)
process(message)
Calling ag directly is an option, but how to do it?
I've tried using a thread.
I've tried executors.
I've tried new_event_loop.
I'm out of ideas.
I figured out a solution. I suspect it is terrible, so I would appreciate feedback.
At the callsite:
worker_thread = threading.Thread(target=worker)
worker_thread.start()
def worker:
f()
And fiddling the library:
def f():
self.loop = asyncio.new_event_loop()
asyncio.set_event_loop(self.loop)
self.loop.call_soon_threadsafe(ag)
# added this
def kill(self):
self.loop.stop()
First example:
import asyncio
async def req():
print('request')
await asyncio.sleep(1)
async def run():
print(len(asyncio.Task.all_tasks()))
asyncio.ensure_future(req())
print(len(asyncio.Task.all_tasks()))
await asyncio.sleep(2)
print(len(asyncio.Task.all_tasks()))
loop = asyncio.get_event_loop()
loop.run_until_complete(run())
The result is:
1
2
request
1
Second example:
import asyncio
async def req():
print('request')
await asyncio.sleep(1)
async def run():
print(len(asyncio.Task.all_tasks()))
t = asyncio.ensure_future(req())
print(len(asyncio.Task.all_tasks()))
await t
print(len(asyncio.Task.all_tasks()))
loop = asyncio.get_event_loop()
loop.run_until_complete(run())
The result is:
1
2
request
2
So, why in first example the last call asyncio.Task.all_tasks() return 1 and in second example it's return 2?
In other words, why in first example the task, that wrap req() was deleted from a set of all tasks for an event loop, and why it is not true for the second example.
The task is removed from all_tasks() when it is destroyed.
Add a del statement:
[...]
await t
del t
print(len(asyncio.Task.all_tasks()))
and it will yield:
1
2
request
1
A client connect websocket and calls tail_log method, and new client can't connect
How to solve this problem
def on_message(self, message):
def tail_log(user,ip,port,cmd,log_path,url):
cmd = "/usr/bin/ssh -p {port} {user}#{ipaddr} {command} {logpath}" \
.format(user=user, ipaddr=ip, port=port, command=cmd, logpath=log_path)
f = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
while True:
line = f.stdout.readline().strip()
if line == '':
self.write_message('failed')
break
self.write_message(line)
tail_log(user=SSH_USER,ip=IP_ADDR,cmd=CMD,port=SSH_PORT,log_path=LOG_PATH,url=SOCKET_URL)
Your infinite loop must yield control back to Tornado's event loop, either by executing a yield, await, or by returning from the tail_log function. Since your infinite loop does not yield control to the event loop, the event loop can never process any more events, including new websocket connections.
Try using Tornado's own process module to read from your subprocess's stdout asynchronously. Something like this:
import tornado.ioloop
import tornado.process
import tornado.web
import tornado.websocket
class TailHandler(tornado.websocket.WebSocketHandler):
def open(self):
self.write_message(u"Tailing....")
self.p = tornado.process.Subprocess(
"tail -f log.log",
stdout=tornado.process.Subprocess.STREAM,
stderr=tornado.process.Subprocess.STREAM,
shell=True)
tornado.ioloop.IOLoop.current().add_callback(
lambda: self.tail(self.p.stdout))
tornado.ioloop.IOLoop.current().add_callback(
lambda: self.tail(self.p.stderr))
self.p.set_exit_callback(self.close)
async def tail(self, stream):
try:
while True:
line = await stream.read_until(b'\n')
if line:
self.write_message(line.decode('utf-8'))
else:
# "tail" exited.
return
except tornado.iostream.StreamClosedError:
# Subprocess killed.
pass
finally:
self.close()
def on_close(self):
# Client disconnected, kill the subprocess.
self.p.proc.kill()
class MainHandler(tornado.web.RequestHandler):
def get(self):
self.write("""<html><head><script>
var ws = new WebSocket("ws://localhost:8888/tail");
ws.onmessage = function (evt) {
document.write('<p>' + evt.data + '</p>');
};</script></head></html>""")
def make_app():
return tornado.web.Application([
(r"/", MainHandler),
(r"/tail", TailHandler),
])
app = make_app()
app.listen(8888)
tornado.ioloop.IOLoop.current().start()
If you're not on Python 3.5 yet, substitute #gen.coroutine for "async def", substitute "yield" for "await", and substitute "break" for "return".