I have an application using jms that sends data to an ActiveMQ Artemis queue. I got an exception with this message:
The transaction was rolled back on failover however commit may have been successful
This exception is basically telling me that the message may or may not have reached the queue so I don't know if I need to send the message again. Whats the best way to handle an exception like this when:
I cannot send duplicate messages to applications on the other end of the queue.
and
I cannot skip a message.
I can't state it better than the ActiveMQ Artemis documentation:
When sending messages from a client to a server, or indeed from a server to another server, if the target server or connection fails sometime after sending the message, but before the sender receives a response that the send (or commit) was processed successfully then the sender cannot know for sure if the message was sent successfully to the address.
If the target server or connection failed after the send was received and processed but before the response was sent back then the message will have been sent to the address successfully, but if the target server or connection failed before the send was received and finished processing then it will not have been sent to the address successfully. From the senders point of view it's not possible to distinguish these two cases.
When the server recovers this leaves the client in a difficult situation. It knows the target server failed, but it does not know if the last message reached its destination ok. If it decides to resend the last message, then that could result in a duplicate message being sent to the address. If each message was an order or a trade then this could result in the order being fulfilled twice or the trade being double booked. This is clearly not a desirable situation.
Sending the message(s) in a transaction does not help out either. If the server or connection fails while the transaction commit is being processed it is also indeterminate whether the transaction was successfully committed or not!
To solve these issues Apache ActiveMQ Artemis provides automatic duplicate messages detection for messages sent to addresses.
See more details about how to configure and use duplicate detection in the ActiveMQ Artemis documentation.
Related
I’m using Masstransit request/response pattern. So I’ve a requester application and a consumer application, which is working very well. I didn’t configure any retry/redelivery as if there is any error happen into consumer , the requester will handle it or might be send another request. So far so good.
But if the consumer application crash and restart in the middle of the process , the consumer try to take the message from queue and start reprocessing it, which is not intended for my case. Because the requester will get error response (or timeout) when the consumer application crashed. I know that MessageRetry in MassTransit is entirely in-memory.
My question is, can we somehow stop consumer to reprocess message on application restart ? OR we need to remove the pending message from service bus queues?
There is no connection between a message sent by the request client and the request client. Using the request timeout as the default value, MassTransit sets the TimeToLive of the message to that same value. The transport should remove that message once the TimeToLive has expired.
If the consumer application crashes consuming a message, that message will remain on the queue. If that message repeatedly causes your application to crash, you could check the Redelivered property that is on the ReceiveContext (a property on ConsumeContext) and possibly handle that message another way if you believe the message is causing the process to crash.
Of course, the real solution is to fix the consumer so it doesn't crash the process...
UPDATE
You could configure the receive endpoint with a DeliveryCount of 1 if you want Azure Service Bus to move the message to the dead-letter queue if the consuming process crashes.
I'm evaluating NATS for migrating an existing msg based software
I did not find documentation about msg timeout exception and overload.
For Example:
After Subscriber has been chosen , Is it aware of timeout settings posted by Publisher ? Is it possible to notify an additional time extension ?
If the elected subscriber is aware that some DBMS connection is missing and cannot complete It could be possible to bounce the message
NATS server will pickup another subscriber and will re-post the same message ?
Ciao
Diego
For your first question: It seems to me that you are trying to publish a request message with a timeout (using the nc.Request). If so, the timeout is managed by the client. Effectively the client publishes the request message and creates a subscription on the reply subject. If the subscription doesn't get any messages within the timeout it will notify you of the timeout condition and unsubscribe from the reply subject.
On your second question - are you using a queue group? A queue group in NATS is a subscription that specifies a queue group name. All subscriptions having the same queue group name are treated specially by the server. The server will select one of the queue group subscriptions to send the message to rotating between them as messages arrive. However the responsibility of the server is simply to deliver the message.
To do what you describe, implement your functionality using request/reply using a timeout and a max number of messages equal to 1. If no responses are received after the timeout your client can then resend the request message after some delay or perform some other type of recovery logic. The reply message should be your 'protocol' to know that the message was handled properly. Note that this gets into the design of your messaging architecture. For example, it is possible for the timeout to trigger after the request recipient received the message and handled it but before the client or server was able to publish the response. In that case the request sender wouldn't be able to tell the difference and would eventually republish. This hints that such type of interactions need to make the requests idempotent to prevent duplicate side effects.
I built a small server (golang) to grab messages from a RabbitMQ and deliver them to connected browsers via a Websocket.
It works quite well, though has one caveat: the messages are acknowledged when delivered to the browser via the websocket. For most messages that is ok but some messages might be very important. If the user's browser received those but the user didn't SEE the message, it would be lost if the browser was closed or reloaded.
Is there a way to ack a message at a later time based on its message id (from the Delivery struct)?
The use case would be that some messages are acked when the user acknowledges them explicitly and at that point the message id is sent back to the tool to be acknowledged with RabbitMQ.
Even if you can do this, it's bad design.
What happens to the message if the user doesn't see it? Does your web server infinitely hang on to it? Does it "nack" the message back to the queue?
Neither of these options are good.
Hang on to every message, and RabbitMQ will start having issues with thousands of unacknowledged messages from a lot of users. Nack the message back to the queue and you'll thrash the message round in circles, spiking CPU resources on the web server and the RMQ server, as well as network traffic between the two.
The better solution to this problem is to store the message in a database, after pulling it out of RabbitMQ. When it gets sent to / viewed by the browser, update the database to reflect that.
From a yet-unpublished article I've written:
Store the message in a database.
Add a field to the database record that says who this message belongs
to. When the user reconnects later, query the database for any
messages that this user needs to see and send them along at that time.
The full process started above, then becomes this:
User's browser connects to SignalR/Socket.io/Pusher/websockets on web
server
Web server checks a queue for updates that happen during a long
running process
When a message for a logged in user comes in
If the
user is logged in, broadcast the message through the websocket to the
user
If the user is not logged in, store the message in a database
When the user logs in again, query the database and send all waiting
messages
It's what you would have done before the idea of a message
queue came in to play, right? It should be what you would do now that
you have a message queue, as well.
I want to send data to a server from a client. Only the last message is important to the server. If the server comes up after a failure I only want the server to get the last message from the client.
While the server is down I want the client to keep processing and send messages or atlest put them in a queue(with the length of one message).
I try to use NetMQ/ZeroMQ for this. How can it be done?
Thanks!
First use PubSub where the client is the publisher, with PubSub you will only get messages while you online, if the subscriber (server in your case) was down it missed all the messages (like a radio)
ZeroMQ also has a feature called Conflate (NetMQ doesn't have it yet, you might want to port it), take a look at the following question:
ZeroMQ: I want Publish–Subscribe to drop older messages in favor of newer ones
Also description of conflate from ZeroMQ documentation:
ZMQ_CONFLATE: Keep only last message If set, a socket shall keep only one message in its inbound/outbound queue, this message being the last message received/the last message to be sent. Ignores 'ZMQ_RCVHWM' and 'ZMQ_SNDHWM' options. Does not support multi-part messages, in particular, only one part of it is kept in the socket internal queue.
Is there any way we can know when a consumer disconnects from a queue or when a queue is deleted?
The requirement is as follows:
I'm building a system in which multiple clients can subscribe to certain events from the system. All clients create their own queue and registers themselves with the system using some sort of authentication. The system, as the events are generated, filters the events and forwards them to clients who are eligible for them.
I have implemented a POC for most part of it and it works well. An issue that I'm not able to fix is that, if a client just disconnects from the queue (due to program termination or so), the registration still exists and the system keeps trying to push messages to that client.
So we would like to be notified when a client disconnects or a queue gets deleted so that we can remove that client's registration data and no longer push messages to him.
Let your publisher utilize Confirms (aka Publisher Acknowledgements) and make client queue be exclusive and transient, so only one client at a time will be consuming from one queue and after it disconnection it will be deleted.
If you publish message that get routed to only one queue and that queue gone (assume you utilize publisher confirms and publish message with mandatory flag set) publisher will be notified that message cannot be routed with that message returned back to it, so you can stop publishing messages.
For details see How Confirms Work section in RabbitMQ blog post "Introducing Publisher Confirms" and Confirms (aka Publisher Acknowledgements) official docs.