I am writing an application in Lambda that is invoked by SQS messages.
I would like to be able to tell the difference between an invocation resulting from a "duplicate" message vs one resulting from a previous failure/retry (both SQS and Lambda will retry in case of failure).
Is the messageId the same for duplicate messages, or just the body? If they are different I might be able to track a messageId against a key from the body to identity duplicates.
TIA.
Ideally, you would want to store the message id some kind of database once you successfully process the message. And the next time around if you get a message with a message id already present in your store, you will know that this has been successfully processed.
Related
The AWS SQS -> Lambda integration allows you to process incoming messages in a batch, where you configure the maximum number you can receive in a single batch. If you throw an exception during processing, to indicate failure, all the messages are not deleted from the incoming queue and can be picked up by another lambda for processing once the visibility timeout has passed.
Is there any way to keep the batch processing, for performance reasons, but allow some messages from the batch to succeed (and be deleted from the inbound queue) and only leave some of the batch un-deleted?
The problem with manually re-enqueueing the failed messages to the queue is that you can get into an infinite loop where those items perpetually fail and get re-enqueued and fail again. Since they are being resent to the queue their retry count gets reset every time which means they'll never fail out into a dead letter queue. You also lose the benefits of the visibility timeout. This is also bad for monitoring purposes since you'll never be able to know if you're in a bad state unless you go manually check your logs.
A better approach would be to manually delete the successful items and then throw an exception to fail the rest of the batch. The successful items will be removed from the queue, all the items that actually failed will hit their normal visibility timeout periods and retain their receive count values, and you'll be able to actually use and monitor a dead letter queue. This is also overall less work than the other approach.
Considerations
Only override the default behavior if there has been a partial batch failure. If all the items succeeded, let the default behavior take its course
Since you're tracking the failures of each queue item, you'll need to catch and log each exception as they come in so that you can see what's going on later
I recently encountered this problem and the best way to handle this without writing any code from our side is to use the FunctionResponseTypes property of EventSourceMapping. Using this we just have to pass the list of failed message Id and the event source will handle to delete the successful message.
Please checkout Using SQS and Lambda
Cloudformation template to configure Eventsource for lambda
"FunctionEventSourceMapping": {
"Type": "AWS::Lambda::EventSourceMapping",
"Properties": {
"BatchSize": "100",
"Enabled": "True",
"EventSourceArn": {"Fn::GetAtt": ["SQSQueue", "Arn"]},
"FunctionName": "FunctionName",
"MaximumBatchingWindowInSeconds": "100",
"FunctionResponseTypes": ["ReportBatchItemFailures"] # This is important
}
}
After you configure your Event source with above configuration it should look something like below
Then we just have to return the response in the below-mentioned format from our lambda
{"batchItemFailures": [{"itemIdentifier": "85f26da9-fceb-4252-9560-243376081199"}]}
Provide the list of failed message Ids in batchIntemFailures list
If your lambda runtime environment is in python than please return dict in the above mentioned format for java based runtime you can use aws-lambda-java-event
Sample Python code
Advantages of this approach are
You don't have to add any code to manually delete the message from SQS queue
You don't have to include any third party library or boto just for deleting the message from the queue it will help you to reduce your final artifact size.
Keep it simple an stupid
On a side note make sure your lambda have the required permission on sqs to get and delete the message.
Thanks
One option is to manually send back the failed messages to the queue, and then replying with a success to the SQS so that there are no duplicates.
You could do something like setting up a fail count, so that if all messages failed you can simply return a failed status for all messages, otherwise if the fail count is < 10 (10 being the max batch size you can get from SQS -> Lambda event) then you can individually send back the failed messages to the queue, and then reply with a success message.
Additionally, to avoid any possible infinite retry loop, add a property to the event such as a "retry" count before sending it back to the queue, and drop the event when "retry" is greater than X.
We're moving from Mandrill to SparkPost. We figured that SparkPost's transmission is the closest thing to Mandrill's send-template message call.
Mandrill responded to those calls with a list of ids and statuses for each email. On the other hand SparkPost returns a single id and summary statistics (number of emails that were sent and number of emails that failed). Is there some way to get those ids and statuses out of the transmission response or at all?
you can get the message IDs for messages sent using the tranmissions API two ways:
Query the message events API, which allows you to filter by recipients, template IDs, campaign IDs, and other values
Use webhooks - messages are sent to your endpoint in batches, and each object in the batch contains the message ID
Which method you choose really depends on your use case. It's essentially poll (message events) vs. push (webhooks). There is no way to get the IDs when you send the transmission because they are sent asynchronously.
Querying message events API, while a viable option, would needlessly complicate our simple solution. On the other hand we very much want to use Webhooks, but not knowing which message they relate to would be troublesome...
The missing link was putting our own id in rcpt_meta. Most of the webhooks we care about do contain rcpt_meta, so we can substitute message_id with that.
I'm stacked too in this problem..
using rcpt_meta solution would be perfect if substitution would work on rcpt_meta but it's not the case.
So, in case of sending a campaign, I cannot specify all recipients inline but have to make a single API call for every message, wich is bad for - say - 10/100k recipients!
But now, all transmission_id are unique for every SINGLE recipient, so I have the missing key and rcpt_meta is not needed anymore.
so the key to be used when receiving a webhook is composed:
transmission_id **AND** rcpt_to
The Event Queue is a box that get messages and handles them.
Could you please say , where are the messages that PostMessage and SendMessage are stored? and how to tell who send what message?
What variables can be inspected to get the information?
Where are the messages stored?
They are stored in an internal data structure associated with a thread. Note that each thread has zero or one message queues. A message queue is not created automatically for a thread, but created on demand when the thread calls a function that requires a message queue.
Note also that sent messages, those delivered by SendMessage do not appear in the queue. Sent messages are synchronous and so not queued.
How to tell who send what message?
In general that is not possible: Can I determine which process sent my window a message?
What variables can be inspected to get the information?
You cannot. You can use PeekMessage to find out whether or not the queue contains a specific message, or find the first message in a given range. But there is no functionality to dump the entire message queue.
1) I'm interested to learn if it is possible to keep the messages that were delivered using Spring Integration. I'm already using the mongo persistent storage (ConfigurableMongoDbMessageStore), but only failed messages remain in the collection. Ideally, I want all messages to remain with the functionality to list them and retry them.
I would use a field "status" or similar to identify queued, succesful or failed messages. Not sure if this field exists already, but I'm guessing something similar must be in place.
2) Also, when a message fails and is persited, there is a lot more data in the message. This data is serialised, so I'm curious how I can extract the original message and retry it.
3) The goal is to create an interface in the webapp where all queued messages can be seen, and retried. Not only failed messages, but also succesful deliveries (useful for testing).
I looked everywhere for an answer to this, but could not find it.
Thanks
I'd say it isn't good design for queue component.
Right it returns failed messages to the queue back for the future redelivery, but good message should be removed from the queue to avoid duplication on the next poll from queue.
No, there is no "status" field on the message, because you use store as a queue.
BTW Spring Integration provides separete implementation for queue channels: MongoDbChannelMessageStore.
You can achieve it with separate parallel Mongo collection and store your message twice: for the queue and for the future analysis. Here you can introduce "status" field and control it, when message successful or not.
From here you can introduce you UI to manage that collection and provide actions like send, retry. Remove the message from here and send it again to those two collections.
HTH
I have a single AMQ queue that receives simple messages with string body. Consider I'm sending CLSIDs as message bodies. CLSIDs could be not unique, but I'd like to reject all messages with not unique bodies and keep only single instance of such messages in the queue. Is there any simple way to do it?
Currently I'm using a workaround. Messages from the queue are consumed by some processor that tries to insert bodies into a simple DB table with UNIQUE constraint applied to message_body field. If processor inserts the messages succesfuly - it's assigned to exchange.out.body and sent to other queue. If ConstraintViolationException is thrown - nothing is resent to other queue.
I would like to know does AMQ support something similar out of the box?
I believe you can write an interceptor for activemq where you can perform certain actions on messages. Check out: http://activemq.apache.org/interceptors.html
That being said, in my personal opinion this is bad practice. ActiveMQ is a messaging system which should only be responssible for transport of the message. All logic can beter be performed using your application ( either make sure the sender cannot send the same message more then once OR , create an intermediate consumer which indeed matches the received body with a database that contains already seen message bodies BEFORE, routing the message to the actual receiver queue)