BaseEventLoop.run_in_executor + slow_callback_duration - python-asyncio

Question

In the Python asyncio library, I am using the BaseEventLoop.run_in_executor function to schedule a long running blocking function on a separate thread:
yield from loop.run_in_executor(None, long_running_blocking_function)
When I have the PYTHONASYNCIODEBUG env var set to enable asyncio debug logging, I see the following warning printed regularly to the text ticker (line wraps added):
WARNING:asyncio:Executing <Task pending coro=<long_running_blocking_function()
running at C:/Projects/Blah/blah.py:52>
wait_for=<Future pending cb=[wrap_future.<locals>._check_cancel_other() at
C:\Python34-64\lib\asyncio\futures.py:401, Task._wakeup()] created at
C:\Python34-64\lib\asyncio\futures.py:399> created at
C:/Projects/Blah/blah.py:59> took 0.999 seconds
I'm surprised by this, as I thought the run_in_executor function is specifically to hand off blocking functions to another thread? Can anyone shed some light on this? Thanks
Edit: As mentioned in a comment to Nihal below, the problem seems to lie in the integration of some library code with asyncio using the executor. Here's some example code that helps to describe the problem:
def on_data(*args, **kwargs):
logger.info('Received data %s', args[1])
def blocking_function(t):
logger.info('Going to sleep for %s', t)
time.sleep(t)
executor = ThreadPoolExecutor(2)
#asyncio.coroutine
def update_session():
while True:
# session.Update causes on_data to be called when data is available
yield from loop.run_in_executor(executor, session.Update, -1)
#asyncio.coroutine
def sleep_short():
while True:
yield from loop.run_in_executor(executor, blocking_function, .01)
asyncio.Task(update_session())
asyncio.Task(sleep_short())
loop = asyncio.get_event_loop()
loop.run_forever()
When I comment out the Task that drives the update_session function, I see my sleep_short function called as expected every 0.01 seconds.
2015-07-27 17:57:09,570 [MainThread] Using selector: SelectSelector
2015-07-27 17:57:09,577 [Thread-1] Going to sleep for 0.01
2015-07-27 17:57:09,587 [Thread-1] Going to sleep for 0.01
2015-07-27 17:57:09,597 [Thread-2] Going to sleep for 0.01
However including that task seems to hijack both threads, so the sleep_short task only runs every second or so:
2015-07-27 17:58:21,618 [MainThread] Using selector: SelectSelector
2015-07-27 17:58:21,624 [Thread-1] calling session update
2015-07-27 17:58:21,625 [Thread-2] Going to sleep for 0.01
2015-07-27 17:58:21,625 [Thread-1] calling session update
2015-07-27 17:58:21,633 [Thread-1] Received data
2015-07-27 17:58:21,633 [Thread-1] calling session update
2015-07-27 17:58:22,603 [Thread-2] Going to sleep for 0.01
2015-07-27 17:58:22,603 [Thread-1] calling session update
I'm quite confused... am I running into the GIL perhaps?
Edit 2:
The delays are definitely caused by the GIL. I was expecting the library I am calling to be blocking on IO, but apparently this is not the case.

Answer 1

I will try to give some explanation, I am also quite new to asyncio.
One of the scenario where this can happen is - Let's say you have initiated some connection and the connection has not been closed before the event loop is closed. The loop in such a case should wait for closing the connection and then close itself. This used to frequently occur in asyncio's redis module and a bug reported here can be of reference - https://github.com/jonathanslenders/asyncio-redis/issues/56
I am not sure this answers the situation properly.