How to run multiple functions (input dependent on prev function) with python asyncio? - python-asyncio

Question

I have 2 dataframes which I need to process separately with replace() before merging them together. In my actual use-case, there are more functions to be applied to each dataframe before the final step which is merging the dataframes.
# replace function
def replace(df, old_value, new_value='above 60'):
df.age = df.age.replace(old_value, new_value)
return df
# dataframes
dfx = pd.DataFrame({
'age': ['11', '19', '22', '30', '24', '27', '67'],
'group': ['A', 'B', 'C', 'C', 'B', 'C', 'B'],
'count':[3,5,2,1,4,5,2]
})
dfy = pd.DataFrame({
'age': ['11', '19', '79', '30', '24', '27', '15'],
'group': ['D', 'D', 'D', 'F', 'E', 'D', 'F'],
'count':[7,5,6,1,5,5,8]
})
my code to run asynchronously:
import asyncio
async def do_after(delay, what):
await asyncio.sleep(delay)
print(what)
async def main():
task1 = asyncio.create_task(
do_after(1, replace(dfx, '67')))
task2 = asyncio.create_task(
do_after(2, replace(dfx, '79')))
final_task = asyncio.create_task(
do_after(3, pd.merge(dfx, dfy, how='inner', on='age')))
print(f"started at {time.strftime('%X')}")
# Wait until both tasks are completed (should take
# around 3 seconds.)
await task1
await task2
await final_task
print(f"finished at {time.strftime('%X')}")
return pd.DataFrame(final_task)
await main()
[OUTPUT]:
started at 09:25:18
age group count
0 11 A 3
1 19 B 5
2 22 C 2
3 30 C 1
4 24 B 4
5 27 C 5
6 above 60 B 2
age group count
0 11 A 3
1 19 B 5
2 22 C 2
3 30 C 1
4 24 B 4
5 27 C 5
6 above 60 B 2
age group_x count_x group_y count_y
0 11 A 3 D 7
1 19 B 5 D 5
2 30 C 1 F 1
3 24 B 4 E 5
4 27 C 5 D 5
finished at 09:25:21
Looking at the last table below, the tables have been merged without the replace() applied to them.
1. Why is that?
2. How do I return the merged table as a pandas dataframe?
Any help is appreciated.

Answer 1

As stated in the comments, this is not the typical workload that will benefit, at all, from async programming. One could use asyncio to orchestrate calls work in parallel dataframes in different subprocess, and maybe that could make a difference (but the most optimized operations in pandas, numpy echo-system will make use of all CPU cores, so even in this case, the gains might be negligible).
That said, the way to ensure ordered execution in asyncio workloads is not to create tasks at all, and simply use inline await expressions:
async def main():
...
await replace(dfx, '67')
await replace(dfx, '79')
await pd.merge(...)
print("done")
If any steps can actually be run in parallel, then one can await for an
asyncio.gather call into which all these steps passed at once.