Using asyncio and aiohttp, I have implemented an async function that triggers an API get request whenever a new record is inserted into database. If the request is successful, the status code has to be updated in database, otherwise the request should be retried 4 times and if it still fails, then the status code has to be updated in database.
To raise the exception on 404 status code, I have added raise_for_status flag to the aiohttp client session. When exception arises, the backoff decorator will retry the API call 4 times and when it still fails, it doesn't return any status code. This is what I have done:
# backoff decorator to retry failed API calls by "max_tries"
#backoff.on_exception(backoff.expo, aiohttp.ClientResponseError, max_tries=4, logger=logger)
async def call_url(language: str, word:str, headers:dict) -> bytes:
url = f"https://od-api.oxforddictionaries.com:443/api/v2/entries/{language}/{word.lower()}"
print(f"Started: {url}")
# Create aiohttp session to trigger 'get' dictionary API call with app_id, app_key as headers
async with aiohttp.ClientSession(headers=headers) as session:
# raise_for_status is used to raise exception for status_codes other than 200
async with session.get(url, raise_for_status=True) as response:
# Awaits response from dictionary API
content = await response.read()
status = response.status
print("Finished: ", status)
# Returns API status code to be updated in db
return status
I can't add the try-except clause because once exception is raised, it's handled by the try-except clause and the backoff doesn't retry the failed API call. Is there a way to make backoff decorator return the status code after maximum retries are attempted?
You say that "when it still fails it doesn't return any status code." But it has to either return something or else raise an exception. How about wrapping the call_url and handling the error situation in the wrapper function? The wrapper function always returns a status code. For instance:
async def my_call_url(language: str, word:str, headers:dict) -> bytes:
try:
return await call_url(language, word, headers)
except WhateverExceptionAiohttpRaises:
return some_status_code
If, instead of raising, the decorator returns None, you can make the appropriate changes.
Your code will now call this new function instead of the other one.
Related
I am trying to hook my websocket endpoint with rabbitmq (aio-pika). Goal is to have listener in that endpoint and on any new message from queue pass the message to browser client over websockets.
I tested the consumer with asyncio in a script with asyncio loop. Works as I followed and used aio-pika documentation. (source: https://aio-pika.readthedocs.io/en/latest/rabbitmq-tutorial/2-work-queues.html, worker.py)
However, when I use it in fastapi in websockets endpoint, I cant make it work. Somehow the listener:
await queue.consume(on_message)
is completely ignored.
This is my attempt (I put it all in one function, so its more readable):
#app.websocket("/ws")
async def websocket_endpoint(websocket: WebSocket):
print("Entering websockets")
await manager.connect(websocket)
print("got connection")
# params
queue_name = "task_events"
routing_key = "user_id.task"
con = "amqp://rabbitmq:rabbitmq#rabbit:5672/"
connection = await connect(con)
channel = await connection.channel()
await channel.set_qos(prefetch_count=1)
exchange = await channel.declare_exchange(
"topic_logs",
ExchangeType.TOPIC,
)
# Declaring queue
queue = await channel.declare_queue(queue_name)
# Binding the queue to the exchange
await queue.bind(exchange, routing_key)
async def on_message(message: IncomingMessage):
async with message.process():
# here will be the message passed over websockets to browser client
print("sent", message.body)
try:
######### Not working as expected ###########
# await does not await and websockets finishes, as there is no loop
await queue.consume(on_message)
#############################################
################ This Alternative code atleast receives some messages #############
# If I use this part, I atleast get some messages, when I trigger a backend task that publishes new messages to the queue.
# It seems like the messages are somehow stuck and new task releases all stucked messages, but does not release new one.
while True:
await queue.consume(on_message)
await asyncio.sleep(1)
################## one part #############
except WebSocketDisconnect:
manager.disconnect(websocket)
I am quite new to async in python. I am not sure where is the problem and I cannot somehow implement async consuming loop while getting inspired with worker.py from aio-pika.
You could use an async iterator, which is the second canonical way to consume messages from a queue.
In your case, this means:
async with queue.iterator() as iter:
async for message in iter:
async with message.process():
# do something with message
It will block as long as no message is received and will be suspended again after processing a message.
The solution was simply.
aio-pika queue.consume even though we use await is nonblocking, so
this way we consume
consumer_tag = await queue.consume(on_message, no_ack=True)
and at the end of connection we cancel
await queue.cancel(consumer_tag)
The core of the solution for me, was to make something asyncio blocking, so I used
this part of the code after consume
while True:
data = await websocket.receive_text()
x = await manager.send_message(data, websocket)
I dont use this code, but its useful as this part of the code waits for frontend websocket response. If this part of the code is missing, then what happens is that client connects just to get disconnected (the websocket endpoit is succefully executed), as there is nothing blocking
I have a specific use case where I need to send an HTTP request ASAP, but cannot wait for the HTTP response to come back before doing some necessary work.
Conceptually, I need to do this:
async with aiohttp.ClientSession() as session:
request = await session.send_request(method='GET', url='https://httpbin.org/get')
# do some necessary work
response = await request.get_response()
# process response...
The problem with the simple plain way is, I can send the HTTP request as soon as I want, but I cannot yield while waiting for the response:
async with aiohttp.ClientSession() as session:
# this blocks until both the request is sent AND response arrived
response = await session.request(method='GET', url='https://httpbin.org/get')
# process response...
I tried to spin up a new coroutine so as to not have to wait for the HTTP response to arrive:
async def foo(url):
async with aiohttp.ClientSession() as session:
response = await session.request(method='GET', url=url)
# process response...
asyncio.create_task(foo('https://httpbin.org/get'))
# do necessary work
but then since create_task() occurs at "the first chance the event loop gets", it's sometimes half a second or a second after I call create_task(), which is too slow for my purpose.
My question(s):
(a) is there a way to separate HTTP request sending from HTTP response waiting in aiohttp?
(b) if not, can you suggest an alternative way to send HTTP request ASAP but to await the response asynchronously?
Thanks!
Update #1
From #Isabi's suggestion in the comments, I tried only using await after the necessary work is done, but the HTTP request is never sent until the await is used, e.g.:
async with aiohttp.ClientSession() as session:
# send out an HTTP request that takes ~2 seconds before response comes back
request = session.request(method='GET', url='https://httpbin.org/delay/2')
await asyncio.sleep(4) # simulate 4 seconds of necessary work
# the following line still takes 2 seconds, indicating the request
# didnt go out before `await` is used
response = await request
# process response...
Update #2
I worked out a way that makes my application behave the way I want it (send the HTTP request ASAP, but don't block waiting for the HTTP response). The solution uses a call to asyncio.sleep(0), inspired from this thread. However, it is not aesthetically satisfying:
async def foo(url):
async with aiohttp.ClientSession() as session:
response = await session.request(method='GET', url=url)
# process response...
asyncio.create_task(foo('https://httpbin.org/get'))
await asyncio.sleep(0)
# do necessary work
It doesn't feel right to me that what should be a not uncommon use case requires such an inelegant solution. Am I missing something?
Are you sure the task is being run half a second or even one second later? Because that should not be the case unless the loop is busy and the loop shouldn't be busy unless it's under heavy load or you have blocking code running at the same time.
You can use logging to check exactly when the request is being sent and when it's received:
import asyncio
import aiohttp
import logging
logging.basicConfig(format="%(asctime)s.%(msecs)03d %(levelname)s %(message)s", datefmt="%Y-%m-%d %H:%M:%S", level=logging.INFO)
async def foo(url):
async with aiohttp.ClientSession() as session:
logging.info("request started")
response = await session.request(method="GET", url=url)
logging.info("response received")
text = await response.text()
logging.info(f"response read {len(text)} bytes")
# process response...
async def test():
logging.info("sleep 0.1")
await asyncio.sleep(0.1)
logging.info("create task")
asyncio.create_task(foo("https://httpbin.org/get"))
logging.info("task created, sleep 2")
await asyncio.sleep(2)
logging.info("finished")
if __name__== "__main__":
asyncio.get_event_loop().run_until_complete(test())
Output:
2020-06-10 10:52:00.017 INFO sleep 0.1
2020-06-10 10:52:00.118 INFO create task
2020-06-10 10:52:00.118 INFO task created, sleep 2
2020-06-10 10:52:00.119 INFO request started
2020-06-10 10:52:00.621 INFO response received
2020-06-10 10:52:00.622 INFO response read 308 bytes
2020-06-10 10:52:02.121 INFO finished
Notice the coroutine starts running about 1 ms after creating it and the HTTP request takes about 0.5 seconds to complete; since 0.5 value is close to what you were seeing I believe you were measuring the time to complete the request and not the time to start the request.
I have a function similar to the following:
def wrapper(fn, client):
async def helper(*args, **kwargs):
id_ = fn(*args, **kwargs)
while True:
await asyncio.sleep(60)
status = client.get_status(id_)
if status == 'OK':
break
return status
return helper
Basically it takes a function, which launches a compute job somewhere. The coroutine then asks the status of this compute job every 60 seconds and if the status is OK then it breaks from the infinite loop.
To test it I am using:
classs Test_wrapper(unittest.TestCase):
def test_wrapper(self):
client = get_client()
aio_fn = wraper(fn, client)
coro = aio_fn(arg1, arg2, arg3=arg3, arg4=arg4)
loop = asyncio.get_event_loop()
loop.run_until_complete(coro)
loop.close()
I have also tested variations of the alternatives shown in this answer by I do not get it to work. I have run the wraper outside pytest and it works.
I have tested this wrapper running the code it is suppose to run, but the tests keep failing. Do you have any clue?
No error message or exeption is displayed when it fails, it only says that it fails.
Inside one of the handlers I am doing the following:
async def get(self):
client = httpclient.AsyncHTTPClient()
url = 'some url here'
request = httpclient.HTTPRequest(url=url, streaming_callback=self.on_chunk, request_timeout=120)
result = await client.fetch(request)
self.write("done")
#gen.coroutine
def on_chunk(self, chunk):
self.write(chunk)
yield self.flush()
The requests can sometimes be quite large and the client may leave while the request is still in progress of being fetched and pumped to the client. If this happens an exception will appear in the on_chunk function when self.write() is attempted. My question is how do I abort the remaining download if my client went away ?
If your streaming_callback raises an exception, the client request should be aborted. This will spam the logs with stack traces, but there's not currently a cleaner way to do it. You can override on_connection_close to detect when the client has disconnected and set an attribute on self that you can check in on_chunk.
I want my lambda function to return the response of another lambda function invoked via AWS API Gateway.
Both functions are deployed by Lambda Chalice to different APIs.
When the first function sends a request to the 2nd functions API endpoint, I am getting an error response saying that "The request could not be satisfied".
Any help is appreciated.
Edit to include some code as requested; shortened for brevity:
#app.route('/verify_user_token', methods=['GET'], cors=True)
def verify_user_token():
request = app.current_request
params = request.query_params or {}
# do your things here; if all goes well:
r = requests.get(ANOTHER_AWS_API_GATEWAY_ENDPOINT_URL, data=params)
return r.text