How to know which Lambda sent a message to the Dead Letter Queue? - aws-lambda

Question

I'm using a SNS Topic as a Dead Letter Queue to handle errors thrown by multiple Lambdas. In the error messages, there are the following attributes :
RequestID,
ErrorCode,
ErrorMessage,
However, I can't easily find which Lambda threw the error, since nothing related to it appear in the message (eg: ARN, function name...)
Although it's possible to look up the request ID on CloudWatch, or to create multiple topics, there should be a much easier way to find which Lambda threw the error. Below is the structure of the received message:
{
"Records": [
{
"EventSource": "aws:sns",
"EventVersion": "1.0",
"EventSubscriptionArn": "",
"Sns": {
"Type": "Notification",
"MessageId": "",
"TopicArn": "",
"Subject": null,
"Message": "",
"Timestamp": "",
"SignatureVersion": "",
"Signature": "",
"SigningCertUrl": "",
"UnsubscribeUrl": "",
"MessageAttributes": {
"RequestID": {
"Type": "String",
"Value": ""
},
"ErrorCode": {
"Type": "String",
"Value": "200"
},
"ErrorMessage": {
"Type": "String",
"Value": "test"
}
}
}
}
]
}
Is there any way to add information, such as the ARN, on the Lambda which triggered this error message?

Answer 1

You can use AWS CloudTrail to identify which Lambda executed:
https://docs.aws.amazon.com/lambda/latest/dg/logging-using-cloudtrail.html

Answer 2

You should have one DLQ per Lambda function. This will let you know where the dead letter is coming from.

Answer 3

I ended up configuring:
One unique SNS topic for every lambda DLQ.
A lambda listening to the above topic and storing the request ID in S3
A trail on CloudTrail logging every lambda invoke
A lambda matching the failed request ID in S3 and the cloudtrail logs. The latter provide the name of the failed lambda.
This infrastructure might seem a bit complicated but it's working very well. It allows to only add one unique onError: ... line of code in every lambda configuration file.

Answer 4

Not sure if you figured out a solution or not, but i solved a somewhat similar problem with a little help from boto3.
To give some more context about my setup:
I had 2 lambda functions (lambdaA and lambdaB) listening to a SNS topic which gets updated by S3 object creation events.
Both lambda functions use one SNS topic as DLQ which is subscribed by a single dlq lambda (lambdaDLQ).
In the lambdaDLQ's received message, you get a Message attribute which has the message received by the error throwing lambda functions (lambdaA or lambdaB) which contains the information about SNS message received by them.
So the steps i used to figure out the error throwing lambda function were:
Parse the Message attribute from lambdaDLQ's event into json.
Get the EventSubscriptionArn from the above parsed message. This holds the subscription details of the error throwing lambda.
Using boto3 to get Protocol, Endpoint, etc. Which give the name of the error throwing lambda.
sns_client = boto3.client('sns')
arn = 'arn_goes_here'
response = sns_client.get_subscription_attributes(SubscriptionArn=arn)
In case your error throwing lambda functions do not listen to a SNS topic, you might need to change this a little bit.