While learning asyncio I was trying this code:
import asyncio
from asyncio.coroutines import coroutine
#coroutine
def coro():
counter: int = 0
while True:
print("Executed" + str(counter))
counter += 1
yield
loop = asyncio.get_event_loop()
loop.run_until_complete(coro())
loop.run_forever()
print("Finished!")
I was expecting the coroutine to be executed only once because it contains a yield and should have returned control to the caller. The output I was expecting was:
Executed 0
Finished!
I was expecting this behaviour because I thought the loop was going to run the coroutine forever once every "frame" returning to the caller after each execution (something like a background thread but in a cooperative way). But instead, it runs the coroutine forever without returning?. Output is the following:
Executed 0
Executed 1
Executed 2
Executed 3
...
Could anyone explain why this happens instead of my expectations?
Cheers.
You have a couple of problems. When you call run_until_complete, it waits for coro to finish before moving on to your run_forever call. As you've defined it, coro never finishes. It contains an infinite loop that does nothing to break out of the loop. You need a break or a return somewhere inside the loop if you want to move on to the next step in your application.
Once you've done that, though, your next call is to run_forever, which, just as its name suggests, will run forever. And in this case it won't have anything to do because you've scheduled nothing else with the event loop.
I was expecting the coroutine to be executed only once because it contains a yield and should have returned control to the caller.
Looking past the fact that your coroutine has no yield, awaiting (or yielding from depending on which syntax you choose to use) does not return control to the caller of run_until_complete or run_forever. It returns control to the event loop so that it can check for anything else that has been awaited and is ready to resume.
Related
Trying to create a event loop inside a thread, where the thread is initiated within the constructor of a class. I want to run multiple tasks within the event loop. However, having an issue whenever I try to run with the thread and get the error "NoneType object has no attribute create_task"
Is there something I am doing wrong in calling it.
import asyncio
import threading
Class Test():
def __init__(self):
self.loop = None
self.th = threading.Thread(target=self.create)
self.th.start()
def __del__(self):
self.loop.close()
def self.create(self):
self.loop = new_event_loop()
asyncio.set_event_loop(self.loop)
def fun(self):
task = self.loop.create_task(coroutine)
loop.run_until_complete(task)
def fun2(self):
task = self.loop.create_task(coroutine)
loop.run_until_complete(task)
t = Test()
t.fun()
t.fun2()
It is tricky to combine threading and asyncio, although it can be useful if done properly.
The code you gave has several syntax errors, so obviously it isn't the code you are actually running. Please, in the future, check your post carefully out of respect for the time of those who answer questions here. You'll get better and quicker answers if you spot these avoidable errors yourself.
The keyword "class" should not be capitalized.
The class definition does not need empty parenthesis.
The function definition for create should not have self. in front of it.
There is no variable named coroutine defined in the script.
The next problem is the launching of the secondary thread. The method threading.Thread.start() does not wait for the thread to actually start. The new thread is "pending" and will start sometime soon, but you don't have control over when that happens. So start() returns immediately; your __init__ method returns; and your call to t.fun() happens before the thread starts. At that point self.loop is in fact None, as the error message indicates.
An nice way to overcome this is with a threading.Barrier object, which can be used to insure that the thread has started before the __init__ method returns.
Your __del__ method is probably not necessary, and will normally only get executed during program shut down. If it runs under any other circumstances, you will get an error if you call loop.close on a loop that is still running. I think it's better to insure that the thread shuts down cleanly, so I've provided a Test.close method for that purpose.
Your functions fun and fun2 are written in a way that makes them not very useful. You start a task and then you immediately wait for it to finish. In that case, there's no good reason to use asyncio at all. The whole idea of asyncio is to run more than one task concurrently. Creating tasks one at a time and always waiting for each one to finish doesn't make a lot of sense.
Most asyncio functions are not threadsafe. You have to use the two important methods loop.call_soon_threadsafe and asyncio.run_coroutine_threadsafe if you want to run asyncio code across threads. The methods fun and fun2 execute in the main thread, so you should use run_coroutine_threadsafe to launch tasks in the secondary thread.
Finally, with such programs it's usually a good idea to provide a thread shutdown method. In the following listing, close obtains a list of all the running tasks, sends a cancel message to each, and then sends the stop command to the loop itself. Then it waits for the thread to really exit. The main thread will be blocked until the secondary thread is finished, so the program will shut down cleanly.
Here is a simple working program, with all the functionality that you seem to want:
import asyncio
import threading
async def coro(s):
print(s)
await asyncio.sleep(3.0)
class Test:
def __init__(self):
self.loop = None
self.barrier = threading.Barrier(2) # Added
self.th = threading.Thread(target=self.create)
self.th.start()
self.barrier.wait() # Blocks until the new thread is running
def create(self):
self.loop = asyncio.new_event_loop()
asyncio.set_event_loop(self.loop)
self.barrier.wait()
print("Thread started")
self.loop.run_forever()
print("Loop stopped")
self.loop.close() # Clean up loop resources
def close(self): # call this from main thread
self.loop.call_soon_threadsafe(self._close)
self.th.join() # Wait for the thread to exit (insures loop is closed)
def _close(self): # Executes in thread self.th
tasks = asyncio.all_tasks(self.loop)
for task in tasks:
task.cancel()
self.loop.call_soon(self.loop.stop)
def fun(self):
return asyncio.run_coroutine_threadsafe(coro("Hello 1"), self.loop)
def fun2(self):
return asyncio.run_coroutine_threadsafe(coro("Hello 2"), self.loop)
t = Test()
print("Test constructor complete")
t.fun()
fut = t.fun2()
# Comment out the next line if you don't want to wait here
# fut.result() # Wait for fun2 to finish
print("Closing")
t.close()
print("Finished")
I wanted to know what happens when i call asyncio.create_task and never actually calling await on this created task.
i have this simple program:
import asyncio
async def simple_task() -> None:
while True:
print("before sleeping")
await asyncio.sleep(2)
print("after sleeping")
async def test():
task = asyncio.create_task(simple_task())
if __name__ == "__main__":
asyncio.run(test())
I'm creating a task, which should run forever, and im never using await.
When running this, the output i get is:
before sleeping
Process finished with exit code 0
So my question is:
why did the task actually run when i never called await?
if it did start running, why did it stop after the sleep?
This code is written to say “run until test completes”. test creates a task to run simple_task and then ends. The event loop gives anything already scheduled against it one last chance to run before stopping.
At this point simple_task gets to start executing. It prints, and then yields control back to the loop via asyncio.sleep.
Nothing else was scheduled against the loop so it stops. asyncio.run then closes the loop.
If you schedule more things against the event loop you may see more iterations of simple_task, but there’s really no way to have asyncio run something forever without waiting for it to do so (loop.run_forever is probably the closest you’ll get).
The following simple Promise is vowed and I am not allowed to break it.
my $my_promise = start {
loop {} # or sleep x;
'promise response'
}
say 'status : ', $my_promise.status; # status : Planned
$my_promise.break('promise broke'); # Access denied to keep/break this Promise; already vowed
# in block <unit> at xxx line xxx
Why is that?
Because the Promise is vowed, you cannot change it: only something that actually has the vow, can break the Promise. That is the intent of the vow functionality.
What are you trying to achieve by breaking the promise as you showed? Is it to stop the work being done inside of the start block? Breaking the Promise would not do that. And the vow mechanism was explicitly added to prevent you from thinking it can somehow stop the work inside a start block.
If you want work inside a start block to be interruptible, you will need to add some kind of semaphore that is regularly checked, for instance:
my int $running = 1;
my $my_promise = start {
while $running {
# do stuff
}
$running
}
# do other stuff
$running = 0;
await $my_promise;
Hope this made sense.
The reason why you cannot directly keep/break Promise from outside or stop it on Thread Pool are explained here in Jonathans comment.
Common misuse of Promises comes from timeout pattern.
await Promise.anyof(
start { sleep 4; say "finished"; },
Promise.in( 1 )
);
say "moving on...";
sleep;
This will print "finished". And when user realize that the next logical step for him is to try to kill obsolete Promise. While the only correct way to solve it is to make Promise aware that its work is no longer needed. For example through periodically checking some shared variable.
Things gets complicated if you have blocking code on Promise (for example database query) that runs for too long and you want to terminate it from main thread. That is not doable on Promises. All you can do is to ensure Promise will run in finite time (for example on MySQL by setting MAX_EXECUTION_TIME before running query). And then you have choice:
You can grind your teeth and patiently wait for Promise to finish. For example if you really must disconnect database in main thread.
Or you can move on immediately and allow "abandoned" Promise to finish on its own, without ever receiving its result. In this case you should control how many of those Promises can stack up in background by using Semaphore or running them on dedicated ThreadPoolScheduler.
I'm using uvloop with websockets as
import uvloop
coro = websockets.serve(handler, host, port) # creates new server
loop = uvloop.new_event_loop()
loop.create_task(coro)
loop.run_forever()
It works fine, I'm just wondering whether I could run to some unexpected problems without setting the global asyncio policy to uvloop. As far as I understand, not setting the global policy should work as long as nothing down there doesn't use the global asyncio methods, but works with the passed-down event loop directly. Is that correct?
There are three main global objects in asyncio:
the policy (common to all threads)
the default loop (specific to the current thread)
the running loop (specific to the current thread)
All the attempts to get the current context in asyncio go through a single function, asyncio.get_event_loop.
One thing to remember is that since Python 3.6 (and Python 3.5.3+), get_event_loop has a specific behavior:
If it's called while a loop is running (e.g within a coroutine), the running loop is returned.
Otherwise, the default loop is returned by the policy.
Example 1:
import uvloop
asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())
loop = asyncio.get_event_loop()
loop.run_forever()
Here the policy is the uvloop policy. The loop returned by get_event_loop is a uvloop, and it is set as the default loop for this thread. When this loop is running, it is registered as the running loop.
In this example, calling get_event_loop() anywhere in this thread returns the right loop.
Example 2:
import uvloop
loop = uvloop.new_event_loop()
asyncio.set_event_loop(loop)
loop.run_forever()
Here the policy is still the default policy. The loop returned by new_event_loop is a uvloop, and it is set as the default loop for this thread explicitly using asyncio.set_event_loop. When this loop is running, it is registered as the running loop.
In this example, calling get_event_loop() anywhere in this thread returns the right loop.
Example 3:
import uvloop
loop = uvloop.new_event_loop()
loop.run_forever()
Here the policy is still the default policy. The loop returned by new_event_loop is a uvloop, but it is not set as the default loop for this thread. When this loop is running, it is registered as the running loop.
In this example, calling get_event_loop() within a coroutine returns the right loop (the running uvloop). But calling get_event_loop() outside a coroutine will result in a new standard asyncio loop, set as the default loop for this thread.
So the first two approaches are fine, but the third one is discouraged.
Custom event loop should be passed as param
If you want to use custom event loop without using asyncio.set_event_loop(loop), you'll have to pass loop as param to every relevant asyncio coroutines or objects, for example:
await asyncio.sleep(1, loop=loop)
or
fut = asyncio.Future(loop=loop)
You may notice that probably any coroutine/object from asyncio module accepts this param.
Same thing is also applied to websockets library as you may see from it's source code. So you'll need to write:
loop = uvloop.new_event_loop()
coro = websockets.serve(handler, host, port, loop=loop) # pass loop as param
There's no guarantee that your program would work fine if you won't pass your event loop as param like that.
Possible, but uncomfortable
While theoretically you can use some event loop without changing policy I find it's extremely uncomfortable.
You'll have to write loop=loop almost everywhere, it's annoying
There's no guarantee that some third-party would allow you to pass
loop as param and won't just use asyncio.get_event_loop()
Base on that I advice you to reconsider your decision and use global event loop.
I understand that it may be felt "unright" to use global event loop, but "right" way is to pass loop as param everywhere is worse on practice (in my opinion).
# Spawn workers to consume items from the iterator's enumerator based on the current concurrency level.
def spawn_workers
EM.next_tick(start_worker = proc{
if #workers < #concurrency and !#ended
# p [:spawning_worker, :workers=, #workers, :concurrency=, #concurrency, :ended=, #ended]
#workers += 1
#process_next.call
EM.next_tick(start_worker)
end
})
nil
end
I read this part of code from EM interator which is used by EM-sychrony#fiberd_interator.
I have some basic idea of Eventmachin, but I'm not very clear about this kind of recursive usage of next_tick, could any one give me a explaination about this plz?
In my opinion, it's just like a loop while it is handled by EM, not "while" or "for". Am I right? And why this?
It's not really a recursive call, think of it as "scheduling a proc to happen a moment later",
EventMachine is basically an endless loop that does stuff scheduled to happen in the next iteration of the loop (next tick),
Imagine next_tick method as a command queueing mechanism,
spawn_workers method schedules the start_worker proc to happen on the next iteration of the event loop.
In the next EM loop iteration start_worker proc will be ran and a #process_next.call will happen which I assume spawns the worker and thus it happens that the first worker is instantiated, the command
EM.next_tick(start_worker)
schedules the same block to happen in next iteration of the EM loop until all workers are spawned.
This means that, for example, if 8 workers need to be instantiated, one worker at a time will be spawned in next 8 ticks of the event loop