Imagine that I have a main program which starts many async activities which all wait on queues to do jobs, and then on ctrl-C properly closes them all down: it might look something like this:
async def run_act1_forever():
# this is the async queue loop
while True:
job = await inputQueue1.get()
# do something with this incoming job
def run_activity_1(loop):
# run the async queue loop as a task
coro = loop.create_task(run_act1_forever())
return coro
def mainprogram():
loop = asyncio.get_event_loop()
act1 = run_activity_1(loop)
# also start act2, act3, etc here
try:
loop.run_forever()
except KeyboardInterrupt:
pass
finally:
act1.cancel()
# also act2.cancel(), act3.cancel(), etc
loop.close()
This all works fine. However, starting up activity 1 is actually more complex than this; it happens in three parts. Part 1 is to wait on the queue until a particular job comes in, one time; part 2 is a synchronous part which has to run in a thread with run_in_executor, one time, and then part 3 is the endless waiting on the queue for jobs as above. How do I structure this? My initial thought was:
async def run_act1_forever():
# this is the async queue loop
while True:
job = await inputQueue1.get()
# do something with this incoming job
async def run_act1_step1():
while True:
job = await inputQueue1.get()
# good, we have handled that first task; we're done
break
def run_act1_step2():
# note: this is sync, not async, so it's in a thread
# do whatever, here, and then exit when done
time.sleep(5)
def run_activity_1(loop):
# run step 1 as a task
step1 = loop.create_task(run_act1_step1())
# ERROR! See below
# now run the sync step 2 in a thread
self.loop.run_in_executor(None, run_act1_step2())
# finally, run the async queue loop as a task
coro = loop.create_task(run_act1_forever())
return coro
def mainprogram():
loop = asyncio.get_event_loop()
act1 = run_activity_1(loop)
# also start act2, act3, etc here
try:
loop.run_forever()
except KeyboardInterrupt:
pass
finally:
act1.cancel()
# also act2.cancel(), act3.cancel(), etc
loop.close()
but this does not work, because at the point where we say "ERROR!", we need to await the step1 task and we never do. We can't await it, because run_activity_1 is not an async function. So... what should I do here?
I thought about getting the Future back from calling run_act1_step1() and then using future.add_done_callback to handle running steps 2 and 3. However, if I do that, then run_activity_1() can't return the future generated by run_act1_forever(), which means that mainprogram() can't cancel that run_act1_forever() task.
I thought of generating an "empty" Future in run_activity_1() and returning that, and then making that empty Future "chain" to the Future returned by run_act1_forever(). But Python asyncio doesn't support chaining Futures.
You say that things are difficult because run_activity_1 is not an async function, but don't really detail why it can't be async.
async def run_activity_1(loop):
await run_act1_step1()
await loop.run_in_executor(None, run_act1_step2)
await run_act1_forever()
The returned coroutine won't be the same as the one returned by run_act1_forever(), but cancellation should propagate if you've got as far as executing that step.
With this change, run_activity_1 is no longer returning a task, so the invocation inside mainprogram would need to change to:
act1 = loop.create_task(run_activity_1(loop))
I think you were on the right track when you said, "I thought about getting the Future back from calling run_act1_step1() and then using future.add_done_callback to handle running steps 2 and 3." That's the logical way to structure this application. You have to manage the various returned objects correctly, but a small class solves this problem.
Here is a program similar to your second code snippet. It runs (tested with Python3.10) and handles Ctrl-C gracefully.
Python3.10 issues a deprecation warning when the function asyncio.get_event_loop() is called without a running loop, so I avoided doing that.
Activities.run() creates task1, then attaches a done_callback that starts task2 and the rest of the activities. The Activities object keeps track of task1 and task2 so they can be cancelled. The main program keeps a reference to Activities, and calls cancel_gracefully() to do the right thing, depending on how far the script progressed through the sequence of start-up activities.
Some care needs to be taken to catch the CancelledExceptions; otherwise stuff gets printed on the console when the program terminates.
The important difference between this program and your second code snippet is that this program immediately stores task1 and task2 in variables so they can be accessed later. Therefore they can be cancelled any time after their creation. The done_callback trick is used to launch all the steps in the proper order.
#! python3.10
import asyncio
import time
async def run_act1_forever():
# this is the async queue loop
while True:
await asyncio.sleep(1.0)
# job = await inputQueue1.get()
# do something with this incoming job
print("Act1 forever")
async def run_act1_step1():
while True:
await asyncio.sleep(1.0)
# job = await inputQueue1.get()
# good, we have handled that first task; we're done
break
print("act1 step1 finished")
def run_act1_step2():
# note: this is sync, not async, so it's in a thread
# do whatever, here, and then exit when done
time.sleep(5)
print("Step2 finished")
class Activities:
def __init__(self, loop):
self.loop = loop
self.task1: asyncio.Task = None
self.task2: asyncio.Task = None
def run(self):
# run step 1 as a task
self.task1 = self.loop.create_task(run_act1_step1())
self.task1.add_done_callback(self.run2)
# also start act2, act3, etc here
def run2(self, fut):
try:
if fut.exception() is not None: # do nothing if task1 failed
return
except asyncio.CancelledError: # or if it was cancelled
return
# now run the sync step 2 in a thread
self.loop.run_in_executor(None, run_act1_step2)
# finally, run the async queue loop as a task
self.task2 = self.loop.create_task(run_act1_forever())
async def cancel_gracefully(self):
if self.task2 is not None:
# in this case, task1 has already finished without error
self.task2.cancel()
try:
await self.task2
except asyncio.CancelledError:
pass
elif self.task1 is not None:
self.task1.cancel()
try:
await self.task1
except asyncio.CancelledError:
pass
# also act2.cancel(), act3.cancel(), etc
def mainprogram():
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
acts = Activities(loop)
loop.call_soon(acts.run)
try:
loop.run_forever()
except KeyboardInterrupt:
pass
loop.run_until_complete(acts.cancel_gracefully())
if __name__ == "__main__":
mainprogram()
You can do this with a combination of threading events and asyncio events. You'll need two events, one to signal the first item has arrived. The thread will wait on this event, so it needs to be a threading Event. You'll also need one to signal the thread is finished. Your run_act1_forever coroutine will await this, so it will need to be an asyncio Event. You can then return the task for run_act1_forever normally and cancel it as you need.
Note that when setting the asyncio event from the separate thread you'll need to use loop.call_soon_threadsafe as asyncio Events are not thread safe.
import asyncio
import time
import threading
import functools
from asyncio import Queue, AbstractEventLoop
async def run_act1_forever(inputQueue1: Queue,
thread_done_event: asyncio.Event):
await thread_done_event.wait()
print('running forever')
while True:
job = await inputQueue1.get()
async def run_act1_step1(inputQueue1: Queue,
first_item_event: threading.Event):
print('Waiting for queue item')
job = await inputQueue1.get()
print('Setting event')
first_item_event.set()
def run_act1_step2(loop: AbstractEventLoop,
first_item_event: threading.Event,
thread_done_event: asyncio.Event):
print('Waiting for event...')
first_item_event.wait()
print('Got event, processing...')
time.sleep(5)
loop.call_soon_threadsafe(thread_done_event.set)
def run_activity_1(loop):
inputQueue1 = asyncio.Queue(loop=loop)
first_item_event = threading.Event()
thread_done_event = asyncio.Event(loop=loop)
loop.create_task(run_act1_step1(inputQueue1, first_item_event))
inputQueue1.put_nowait('First item to test the code')
loop.run_in_executor(None, functools.partial(run_act1_step2,
loop,
first_item_event,
thread_done_event))
return loop.create_task(run_act1_forever(inputQueue1, thread_done_event))
def mainprogram():
loop = asyncio.new_event_loop()
act1 = run_activity_1(loop)
# also start act2, act3, etc here
try:
loop.run_forever()
except KeyboardInterrupt:
pass
finally:
act1.cancel()
# also act2.cancel(), act3.cancel(), etc
loop.close()
mainprogram()
I am trying to implement a micropython Class to make a function runs forever at a certain time interval. For that I am trying to adapt the following code from this post:
class Timer:
def __init__(self, timeout, callback):
self._timeout = timeout
self._callback = callback
self._task = asyncio.create_task(self._job())
async def _job(self):
while True:
await asyncio.sleep(self._timeout)
await self._callback()
def cancel(self):
self._task.cancel()
async def timeout_callback():
await asyncio.sleep(0.1)
print('echo!')
async def main():
timer = Timer(2, timeout_callback) # set timer for two seconds
await asyncio.sleep(12.5) # wait to see timer works
asyncio.run(main())
The point is, I can't make it run indefinitely as a "normal" while loop. I have notice that the callback keeps running as per the time defines in the last await asyncio.sleep(secs) command... but this is not what I actually want to achieve. Any help would be hilly appreciated.
so basically I am trying to make a spam command for my discord bot, which takes in a custom message to spam. Here's the code:
#client.command(name='spam')
async def spam(ctx):
global stop
stop = 0
content = ctx.message.content[11:]
if ctx.author.guild_permissions.administrator or ctx.author.id in admins:
if lock == 1:
await ctx.send('Jesus bot is currently locked.')
elif lock == 0:
await ctx.send('Beginning spam..')
while not stop:
await ctx.send(content)
else:
await ctx.send('Sorry, but you do not have admin permissions in this server, or you are not a verified admin.')
For some reason, whenever I try to use this command, the bot doesn't respond. I'm not sure why this happens, and could use some help please.
Picture of bot not responding:
I have a spam command, but I only use it to mess around with my friends. I would not recommend using this as a public command, as you may get rate limited or banned for abuse or something like that. Anyway here is the code I have used for it.
#commands.command()
#commands.is_owner()
# If you want to use admin only, use this below
# #commands.has_permissions(administrator=True)
async def spam(self, ctx, amount, *, word):
int(amount)
await asyncio.sleep(2)
print(f"Starting to spam {word} in {ctx.guild.name}")
await ctx.message.delete()
await ctx.send(f"{ctx.author.mention}\nPlease note that this will clog up the bot's reaction time")
await asyncio.sleep(3)
count = 0
counting=True
while counting:
await ctx.send(word)
count = count + 1
if count == amount:
await asyncio.sleep(2)
await ctx.send("Spam complete")
print(Fore.GREEN + "Spam complete")
counting = False
At the top of your code, make sure you import asyncio as time.sleep will cause the whole bot to pause. Also the Fore.GREEN stuff is just colorama (import colorama).
Try using tasks instead of asyncio. It is made for such repetetive operations and it is easier and nicer because it is made by discord and is included in discord.ext. Something like this:
from discord.ext import tasks
#client.command(name='spam')
async def spam(ctx):
#get message and do all the ifs that you have there
spamLoop.start()
#client.command(name='stopSpam')
async def spamStop(ctx):
# stop the loop
spamLoop.cancel()
#tasks.loop(seconds=1)
async def spamLoop():
print("The message")
Actually quite a simple way of adding spam
import asyncio
#bot.command(name='spam', help= "Spam to your heart's delight")
async def spam(ctx, thing, amount):
count = 0
while count < int(amount):
await ctx.send(thing)
count += 1
if count < amount:
await asyncio.sleep(1)
I'm writing a client in asyncio and using q.get() to wait for responses from the server. When I receive a response from the server I put it on the queue. If the server connection is lost I will no longer being doing those puts and could have any number of await q.get()'s hanging around.
How should I cancel them? I noticed that when I delete the queue the await gets are still waiting.
Does this look like what you are trying to do? You have two options I think:
If you keep a count of outstanding gets then when you are done with the queue you can just put(None) that many times?
Or if None is a valid response then keep a list of the outstanding futures and call cancel on them yourself.
import asyncio
async def qget(q):
try:
x = await q.get()
q.task_done()
print("qget done ",x)
except asyncio.CancelledError as e:
print("qget cancel exception ",e)
except Exception as e:
print("qget exception ",e)
async def run():
q = asyncio.Queue()
futs = []
futs.append( asyncio.ensure_future( qget(q) ) )
futs.append( asyncio.ensure_future( qget(q) ) )
num = 2
await asyncio.sleep(0.1)
# Keep the number of outstanding gets and put None for each one
if 1:
for x in range(num):
q.put_nowait(None)
# Or keep the futures in a list and cancel them
if 0:
for f in futs:
f.cancel()
await asyncio.sleep(1)
print("run loop done")
asyncio.run(run())
If you look at the python code for the queue it does keep a list called _getters, but there is no public api for accessing it.
Consider a single-threaded Python program. A coroutine named "first" is blocked on I/O. The subsequent instruction is "await second." Is the coroutine "second" guaranteed to execute immediately until it blocks on I/O? Or, can "first" resume executing (due to the I/O operation completing) before "second" is invoked?
Asyncio implemented a way that second would start executing until it would return control to event loop (it usually happens when it reaches some I/O operation) and only after it first can be resumed. I don't think it somehow guaranteed to you, but hardly believe this implementation will be changed either.
If for some reason you don't want first to resume executing until some part of second reached, it's probably better explicitly to use Lock to block first from executing before moment you want.
Example to show when control returns to event loop and execution flow can be changed:
import asyncio
async def async_print(text):
print(text)
async def first():
await async_print('first 1')
await async_print('first 2')
await asyncio.sleep(0) # returning control to event loop
await async_print('first 3')
async def second():
await async_print('second 1')
await async_print('second 2')
await asyncio.sleep(0) # returning control to event loop
await async_print('second 3')
async def main():
asyncio.ensure_future(first())
asyncio.ensure_future(second())
await asyncio.sleep(1)
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
try:
loop.run_until_complete(main())
finally:
loop.run_until_complete(loop.shutdown_asyncgens())
loop.close()
Output:
first 1
first 2
second 1
second 2
first 3
second 3