I have two async function like below
async def get_job_details(key: str) -> dict:
...
return data
async def get_resource(data: dict) -> dict:
...
return data
I want to call both functions in loop using pydash flow like below
py_.flow(await get_job_details, await get_resource)(key)
Getting error TypeError: object function can't be used in 'await' expression
This works fine without async. Even if I call them without flow that works too
data = await get_job_details(key)
data = await get_resource(data)
But I want to use some looping here , since there is a possibility of having more function call and must be called sequentially since they are dependent .
Related
I'm creating multiple web workers making http calls. I have to limit the number of web workers and so I'm trying to wait for some of the workers to finish.
here's an example of what I thought might work using Promises:
anArray.map(async contact => {
await new Promise((res, rej) => {
const worker = new Worker('./http.worker', { type: 'module' });
worker.onmessage = () => {
res();
};
worker.postMessage(contact);
});
});
I thought this would wait for each promise to resolve before moving on to the next item... but it's not.
What could I do to make this work? or... I've also thought of building an array of workers and run a recursive loop that checks/waits for one to be available... I'm open to general suggestions for solving this.
.map() is not promise aware. It does not look at the return value from each iteration to see if it's a promise and then pause the loop. Instead, it just blindly runs all the iterations one after another. When you return promises from .map() which you are with the async callback, that just means that your .map() will produce an array of promises and all your loop iterations will be "in-flight" at the same time, not sequenced.
If you want to iterate a loop and pause the loop in each iteration until a promise resolves, then use a regular for loop:
async function someFunc() {
for (let contact of anArray) {
await new Promise((res, rej) => {
const worker = new Worker('./http.worker', { type: 'module' });
worker.onmessage = () => {
res();
};
worker.postMessage(contact);
});
}
}
FYI, http calls in Javascript are non-blocking and asynchronous so it's not entirely clear why you're doing them in WebWorkers. Unless you have CPU-intensive processing of the result, you can do the http requests in the main thread just fine without blocking it.
Also FYI, for a number of different options for processing an array, with only N requests in flight at the same time (where you decide what value N is), see these various answers:
runN(fn, limit, cnt, options): Loop through an API on multiple requests
pMap(array, fn, limit): Make several requests to an api that can only handle 20 at a time
rateLimitMap(array, requestsPerSec, maxInFlight, fn): Proper async method for max requests per second
mapConcurrent(array, maxConcurrent, fn): Promise.all() consumes all my ram
There are also features to do this built into the Bluebird promise library and the Async-promises library.
I have multiple api routes which return data by querying database individually.
Now I'm trying to build dashboard which queries above api's. How should I put api calls in the queue so that they are executed asynchronously?
I tried
await queue.put({'response_1': await api_1(**kwargs), 'response_2': await api_2(**kwargs)})
It seems as though data is returned while task is being put in the queue.
Now I'm using
await queue.put(('response_1', api_1(**args_dict)))
in producer and in consumer I'm parsing tuple and making api calls which I think I'm doing wrong .
Question1
Is there a better way to do it?
This is code I'm using to create tasks
producers = [create_task(producer(**args_dict, queue)) for row in stats]
consumers = [create_task(consumer(queue)) for row in stats]
await gather(*producers)
await queue.join()
for con in consumers:
con.cancel()
Question2 Should I use create_task or ensure_future? Sorry if it's repetitive but I can't understand the difference and after searching online I became more confused.
I'm using FastAPI, databases(async) packages.
I'm using tuple instead of dictionary like await queue.put('response_1', api_1(**kwargs))
./app/dashboard.py:90: RuntimeWarning: coroutine 'api_1' was never awaited
item: Tuple = await queue.get_nowait()
My code for consumer is
async def consumer(return_obj: dict, que: Queue):
item: Tuple = await queue.get_nowait()
print(f'consumer took {item[0]} from queue')
return_obj.update({f'{item[0]}': await item[1]})
await queue.task_done()
if I don't use get_nowait consumer gets stuck because queue may be empty,
but if I use get_nowait above error is shown.
I haven't defined max queue length
-----------EDIT-----------
Producer
async def producer(queue: Queue, **kwargs):
await queue.put('response_1', api_1(**kwargs))
You can drop the await from your first snippet and send the coroutine object in the queue. A coroutine object is a coroutine that was called, but not yet awaited.
# producer:
await queue.put({'response_1': api_1(**kwargs),
'response_2': api_2(**kwargs)})
...
# consumer:
while True:
dct = await queue.get()
for name, api_coro in dct:
result = await api_coro
print('result of', name, ':', result)
Should I use create_task or ensure_future?
If the argument is the result of invoking a coroutine function, you should use create_task (see this comment by Guido for explanation). As the name implies, it will return a Task instance that drives that coroutine. The task can also be awaited, but it continues to run in the background.
ensure_future is a much more specialized function that converts various kinds of awaitable objects to their corresponding futures. It is useful when implementing functions like asyncio.gather() which accept different kinds of awaitable objects for convenients, and need to convert them into futures before working with them.
I have an action which requires me to go to the server, get 2 chunks of data, and on getting those then dispatching an action with each chunk of data respectively.
This is to be done when action$.ofType(PROCESSDATA).
Pretty new to rxjs and I can't quite figure out the right combination.
epic = (action$) => {
return action$.ofType(PROCESSDATA)
//should I be using switchMap? It is for initialising data
//so I figure no point in carrying on with original request
//if a new one comes in
.switchMap(action=>{
//is merge right? return statement currently just gives me
//an error about my epic does not return a stream
const token = localStorage.getItem('token');
return Observable.merge(
Observable.ajax({
url:'/first',
headers:{Authorization:'Bearer ' + token}
})
.map(r=>{type:PROCESSFIRST, data:r.response, item:action.item}),
Observable.ajax({
url:'/second',
headers:{Authorization:'Bearer ' + token}
})
.map(r=>{type:PROCESSSECOND, data:r.response}),
)
})
}
The reason I am doing it this way (in case it doesn't make sense) is that I already have an app using redux-thunk (and axios) and I am trying to replace one part of it with redux-observable to get the hang of it, so that is why I want to issue the 2 actions (PROCESSFIRST and PROCESSSECOND) for redux-thunk to deal with as they are already catered for. So I am simply putting a step in the middle to see if I can work with redux-observable but I am very much a newbie to rxjs etc
Ultimately I want to trigger the continuation actions using the data returned and then finish at that.
Also, if you have time, how and where should errors be handled in this.
UPDATE:
OK, so I just realised I need a return statement at the top of the epic. That gets me out of the first error.
Then I was using mapTo instead of map, and I was using r.data where it should have r.response. Amazingly it all works now. BUT I am not handling errors correctly OR not calling it in the right way.
So, because of what I said about the existing project using redux-thunk, I have an actionCreator that controls this:
initState = (lang) => {
return dispatch => new Promise(resolve,reject){
const pr1 = dispatch({type:INIT_STATE});
//this was another thunk that returned a Promise which did the axios
//calls to the server, dispatched the relevant continuation action
//and then either resolved or rejected
//and then dispatched further actions to process the results
const pr2 = dispatch(processData());
Promise.all([pr1,pr2])
.then(r=>resolve(true))
.catch(e=>reject(e))
}
}
I have changed this where const pr2 = .... to:
const pr2 = dispatch({type:PROCESSDATA})
, thereby swapping the thunk for an epic and left everything else the same. This does seem to trigger the epic and process the data but for example if I use the wrong url, nothing happens (as expected) but more importantly I don't catch the error. So what do I need to change to catch that error?
thanks in advance
Is it possible to extract the function and arguments of a coroutine object in python3.6?
Context: currently I have something like this:
async def func(*args):
...
ret = await remotely(func, x, y)
Under the hood, remotely pickles func, x, and y, scp's that to a different server, where it unpickles them, executes func(x,y), pickles the result, scp's that back, and finally unpickles it into ret.
This API feels distasteful to me, I'd prefer to have:
ret = await remotely(func(x, y))
I could do this if I could pickle the coroutine object represented by func(x, y), but when I tried that, I get:
TypeError: can't pickle coroutine objects
So my alternate hope is that I can extract f, x, and y from f(x, y), hence this question.
So when you do ret = await remotely(func(x, y)), you actually construct the coroutine object for func. Fortunately, you can extract the information you need from coroutine objects, which you can send over for remote execution.
So first of all you can get the function name using the __qualname__ attribute. This will give you the fully qualified name, i.e. if the coroutine is nested, it will get you the full path to your function.
Next, you can extract the argument values from the frame object of the coroutine.
So this is how your remote function would look like
async def remote(cr):
# Get the function name
fname = cr.__qualname__
# Get the argument values
frame = cr.cr_frame
args = frame.f_locals # dict object
result = await ... # your scp stuff
return result
There is just one caveat. You should indicate that the function should be only used the way that you have posted, i.e.
ret = await remotely(func(x, y))
...in other words, the coroutine should be "fresh", and not half-way executed (which is almost not possible if you initiate it right passing it to remote). Otherwise, the f_locals value might include any other local variable that is defined before any awaits.
Found no clean solution. But we can find function as a referrer to coroutine's code object:
import gc
import inspect
def get_function_from_coroutine(coroutine: Coroutine) -> Callable:
referrers = gc.get_referrers(coroutine.cr_code)
return next(filter(lambda ref: inspect.isfunction(ref), referrers))
def get_kwargs_from_coroutine(coroutine: Coroutine) -> dict[str, Any]:
return coroutine.cr_frame.f_locals
async def foo(a: str, b: int):
return a * b
coro = foo("test", b=2)
print(get_function_from_coroutine(coro)) # <function foo at 0x7ff61ece9820>
print(get_kwargs_from_coroutine(coro)) # {'a': 'test', 'b': 2}
Is it possible to use =LOAD(...) with a function rather then controller/function string
e.g:
Controller:
def test():
print "test"
def index():
return dict(test=test)
View:
{{=LOAD(test, ajax=True)}}
rather then:
View:
{{=LOAD('controller', 'test', ajax=True)}}
The main reason being, I want to use lambda/generated functions which cannot be accessed this way.
No. But not because the syntax is not supported, because it is logically impossible: the LOAD() is executed in a different http request than the one in which the lambda would be executed and therefore the latter would be undefined. Moreover to perform the ajax callback, the called function must have a name, cannot be a lambda. We could come up with a creative use of cache so that LOAD stores the lambda in cache:
def callback():
""" a generic callback """
return cache.ram(request.args(0),lambda:None,None)(**request.vars)
def LOAD2(f,vars={}):
""" a new load function """
import uuid
u = str(uuid.uuid4())
cache.ram(u,lambda f=f:f,0)
return LOAD(request.controller,'callback',args=u,vars=vars,ajax=True)
def index():
""" example of usage """
a = LOAD2(lambda:'hello world')
return dict(a=a)
But this would only work with cache.ram and would require periodic cache cleanup.