I have configured a lambda function as EventBridge rule target and I have configured a Dead Letter Queue on the EventBridge rule to capture exceptions.
Now, if the lambda function fails, Event Bridge does not recognize that failure as an error.
Since the EventBridge invocation to the Lambda is asynchronous, for EventBridge it is enough to reach the lambda to consider the event as successfull,but in this way I am not able to track and retry events once lambda fails.
Anyone know a way to make the EventBridge to the Lamdba request synchronous or another way to be able to Retry the events if the Lambda code fails after the invocation?
one option is to make the SQS to be rule target and use SQS event to trigger the lambda. when failure occurs, the lambda won't flag the event done so as to keep the event in the SQS. retry will auto happen after a configured period (SQS configuration). also you can configure dead letter queue after the retention time expires
EventBridge guarantees the delivery of the event to the lambda function but is not aware of what happens post that. It's lambda invocation vs lambda execution. Eventbridge successfully delivered the message to the lambda service, so it's a successful invocation.
For Lambda, EventBridge calls invokeAsync API. So, if this API sends a success response, EventBridge will assume the delivery was successful. Any failures within Lambda from async to sync is not visible to EventBridge. We should configure retries and create DLQs in our Lambda functions to make sure the events are not lost in case the Lambda function fails to execute. We could in fact configure the same DLQ used by the EventBridge to be used by the Lambda as well so that all eventual failures land in a single place.
AWS has a dedicated documentation page for this, which states the following for asynchronous invocation:
Lambda retries function errors twice. If the function doesn't have enough capacity to handle all incoming requests, events might wait in the queue for hours or days to be sent to the function. You can configure a dead-letter queue on the function to capture events that weren't successfully processed. For more information, see Asynchronous invocation.
So that means that as long as your Lambda functions handler function does return an error, the AWS Lambda service should retry to run your Lambda again.
Therefore, you might not need EventBridge to retry your event.
See: Error handling and automatic retries in AWS Lambda
Related
I am using CloudWatch log subscription filters to get logs from a specific log group and send them to a Lambda function, which after processing will send the results to another service. I wonder if there's any possibility to send failed events by Lambda to a Dead Letter Queue, noting that in the above settings, we have no SNS/SQS setup to trigger the Lambda.
Destinations gives you the ability to handle the Failure of function
invocations along with their Success. When a function invocation
fails, such as when retries are exhausted or the event age has been
exceeded (hitting its TTL), Destinations routes the record to the
destination resource for every failed invocation for further
investigation or processing.
To configure destination in Lambda function, Kindly refer
I have a lambda producer which putRecords to a kinesis steam. Sometimes while writing to kinesis I get Internal service failure. What is the best way to handle such cases where lambda fails to write to kinesis ? I have a retry mechanism on my producer lambda but even after retry attempts it fails to write in some cases.
A good approach could be to use the DeadletterQueue with Lambda functions. You can configure SNS/SQS to create a queue and write all the failed events there and retry to process.
https://aws.amazon.com/about-aws/whats-new/2016/12/aws-lambda-supports-dead-letter-queues/
We have a SQS queue subscribe to SNS Topic which publishes about 1-5 million events per month. I want to know which of these combinations - SNS->SQS->Lambda vs SNS->SQS->SQSConsumer would benefit me for such use-cases.
I understand the maintain difference between them is Event driven Vs Pull Driven. A lambda is triggered for each message that comes into a queue so that is an event driven architecture, an SQSConsumer has to constantly poll for messages. You have to have constant up time for a poller like that vs a lambda that is only triggered once a message is received.
I have couple of questions here :
Why SNS->SQS-> Lambda is considered Event driven, when lambda has to poll the SQS queue similar to what SQSConsumer does?
Followup : When Lambda is also constantly polling, then why lambda is considered to be more cost efficient than SQSConsumer?
If you ignore the 'internals' of how Amazon SQS with AWS Lambda is implemented, simply think of it as SQS directly triggering the Lambda function. This is a serverless model, whereas using an SQS consumer requires code to be running on a computer somewhere. Lambda will automatically scale, so it is more cost effective than having computing infrastructure waiting around for events (and costing money even when it isn't used).
So, it's really a decision about whether to use a serverless architecture.
You could also subscribe the AWS Lambda function direction to the Amazon SNS topic, without using Amazon SQS in the middle.
For AWS Lambda handlers, does it matter if the code is async vs sync (eg async def vs def), with respect to the concurrency performance of multiple calls to that Lambda function? That is, if I write a synchronous function, will that function be called "asynchronously" anyway due to the way AWS internally handles Lambda invocations from separate calls? Or will the function start processing UserA, and queue UserB's request until it has completed UserA?
Backstory: I'm writing some Python + SQLAlchemy Lambda functions. SQLAlchemy 1.4 just released asyncio support, but there will be some re-write overhead on my part. I'm wondering if I can just write traditional synchronous Python code without concurrency consideration on Lambda (where concurrency is something I'd definitely need to consider for custom Flask / FastAPI code).
If you have more than one request to invoke a lambda at the same time the service will attempt to run multiple instances of the lambda. You may run into throttling due to the concurrency limits on either the lambda itself, or the account. Lambda does not queue requests to the same lambda function. A function is never processing more than one request at a time. Even in cases where the system is providing a queuing mechanism to handle the requests, they are not a thread queue, like you might have with a server. They are really just internal SQS queues that are invoking the lambda with one request at a time.
I have a lambda function written in java that listens to sqs events and tries to do some kind of processing on those sqs messages.
As per lambda documentation, if lambda code throws a runtimeException then, lambda would retry the same message twice before it sends it back to the queue. However, I don't see that behavior.
I only see it processing the message just once.
Here is a snippet from the relevant lambda code in this case.
#Override
public Boolean handleRequest(SQSEvent sqsEvent, Context context) {
try{
........some queue message processing.....
}
catch(Exception ex){
throw new RuntimeException("exception occurred");
}
}
Is this not good enough for lambda to retry the message 2 more times? I did check the cloudwatch to see what lambda logs and it just has logs from the very first processing only and not the retries.
Can someone tell me what did I miss here because of which it does not work as expected.
You are missing the throws clause in handleRequest. If you don't have that then, lambda would just swallow the exception
public Boolean handleRequest(SQSEvent sqsEvent, Context context) throws RuntimeException
Other than that, what Thales Munussi has told you about synchronous polling is absolutely right. When you hook sqs with lambda, lambda polls sqs which keeps an open connection between the two hence making it a synchronous connection
As per aws documentation, lambda doesn’t retry in such synchronous cases. Setting up a dlq and retires in sqs itself is your best recourse
Keep in mind that lambda would send the message back to the queue after a runtime exception is thrown in your java code
Based on the redrive setting in your sqs, the sqs will generate the same event based on redrive number.
Once lambda fails to process successfully for redrive number of times, message is sent to the DLQ from the main queue
The documentation says it retries two more times if the invocation is asynchronous. SQS is a poll-based system. Lambda will poll the Queue and all of its invocations will be synchronous.
For poll-based AWS services (Amazon Kinesis, Amazon DynamoDB, Amazon
Simple Queue Service), AWS Lambda polls the stream or message queue
and invokes your Lambda function synchronously.
What you can do is configure a DLQ on your source SQS queue in case your message fails so you can either analyse it further or process the message again based on the logic you have configured.
EDIT
The OP is not able to see the messages in the DLQ for somehow. I have attached images to show it works.
Lambda sqs-test is triggered by a new message in SQS queue sqs-test
These are the Queues (sqs-test-dlq is configured as a DLQ for sqs-test).
This is the code for the Lambda function:
This is the configuration for sqs-test
And this is the redrive policy
After the messages failed in the Lambda function, they were successfully sent to the configured DLQ:
You must be missing some basic configuration because it works seamlessly as the images above show.