We have a JMS queue and multiple competing clients are reading from this queue.
Once the message is taken and successfully processed, we want to send the acknowledge to delete ( i.e. CLIENT ACKNOWLEDGE )
However, we want to make sure that if one client has picked the message another client should not take it from the queue.
Does activeMQ provide this feature out of the box using some configuration ?
Moreover:
If the message processing failed after picking the message, so it could not be acknowledged back, in this scenario we should like other client thread to pickup the message. Is it possible out of the box with configuration , may be specifying timeout values ?
Regards,
JE
You need to take some time to understand the difference between a Topic and a Qeueue in order to understand why the first question is not an issue.
For the second question it depends a bit on the ACK mode you are using and how you are processing messages sync or async. Normally for processing where you want to control redeliveries you would do the work inside of a transaction and if the processing fails the message would be redelivered when the TX is rolled back. ActiveMQ supports redelivery policies both client side and broker side that control how many time a message will be redelivered before sent to a DLQ.
Related
I created a SpringBoot/Spring AMQP project where I configured a listener on a RabbitMQ queue. Question: Is there any way to leave the message in the queue? Let me explain: I consume the message and do some things (eg save on db), if something goes wrong I would like to be able to reconsume the message.
Thanks in advance
You need to think about configuring your listener container with transactions, so when DB call fails, the transaction is going to be rolled back and an AMQP message will not be acked on RabbitMQ.
See docs for more info: https://docs.spring.io/spring-amqp/docs/current/reference/html/#transactions
I don't know about the "Spring" way of accomplishing this, but what you describe is the normal behavior for AMQP consumers that do not automatically acknowledge.
From the documentation:
In automatic acknowledgement mode, a message is considered to be successfully delivered immediately after it is sent.
When you turn off automatic acknowledgment, your consumer must explicitly acknowledge the message, otherwise it will not be dequeued (or as you put it, it will be left "in the queue"). You will then need to simply ACK the message at the very end of your operation, when you are certain that your operation succeeded (and perhaps coordinated with your database transaction).
There is always the question of what to do first; acknowledge first or commit your database transaction first? Without adding complexity, you must choose what's best depending on what failure mode is less problematic for you, i.e. Would you rather tolerate a duplicated message or a missing message?
I have a HornetQ based JMS provider and the consumer is attached to the provider in the CLIENT_ACKNOWLEDGE_MODE . The message.acknowledge () snippet though is under an if else . Thus the consumer would sometimes would not send an acknowledgement to the server in case there was an application layer failure in processing the message .So there are two questions here -
1)Will more messages that are queued in the server will keep on
flowing to the consumer even though the consumer did not acknowledge
on of the messages as stated earlier
2)Will the unacknowledged message flow down again on restarting the
consumer .
These are some of my observations on the questions I have asked
1)The messages keep on flowing down to the consumer as per the
consumer logs even though it did not acknowledge one of the messages
to the server due to an application layer failure (Note , there was
no uncaught exception as such , just that the consumer did not
acknowledge) .
2)Secondly , on restarting the consumer as well the message did not
flow down again from the server which is surprising .
Can someone please clarify this behaviour?
It has hard to fully determine what your application is doing in terms of message acknowledgement but my guess is that you are continuing to acknowledge messages after the failed attempt to acknowledge the message in question. In that case because you are using the Client Acknowledge mode the next Acknowledge will also apply to the previous message as that is how client mode works.
Session.CLIENT_ACKNOWLEDGE: A client acknowledges a message by calling the message’s acknowledge method. In this mode, acknowledgment takes place on the session level: Acknowledging a consumed message automatically acknowledges the receipt of all messages that have been consumed by its session. For example, if a message consumer consumes ten messages and then acknowledges the fifth message delivered, all ten messages are acknowledged.
So if you read that carefully you will see how consumers in a Session acknowledging messages while in client Acknowledge mode could even affect one another.
For a better answer you'd need to break down the chain of event further so that in became more clear what is going on.
Hi I am new to Camel and have a design question related to JMS queues.
I am receiving set of data. These data have a reference date. These data are sent every 15 minutes by a batch process.
I have to process the received data and forward them to another route.
If a given data cannot be processed, I need to reprocess it. And I have to ensure it is processed before the next data set is processed.
So I was thinking of creating a JMS route to receive these data before processing. Then process it. Then send it to another queue.
FTP --> Process data rows (A) --> JMS Queue --> Processor (B) --> direct:call
If processor B fails I want the data to be processed before the next data set is sent by FTP. (because second data set may contain an update of the data of the first dataset)
So I was thinking using a queue, to make sure they are always processed in the order they are being received.
But my experience with JMS, without Camel, is that once the object is consumed from the queue it is not in the queue anymore.
Is it also the case with Camel?
In this case to I have to retry to process the data, or put them back in the queue?
This "recovery" part is not clear to me and I'd like to understand the patterns that do support this.
Many thanks for your help
Gilles
This part "once the object is consumed from the queue it is not in the queue anymore." is not fully correct. Actually, when you are subscribing to the queue and getting a message you need to process it and send acknowledge back to the JMS broker. If acknowledge is successful then the message will be removed from the queue. But if acknowledge will be not successful or if your process will die and connection to the broker will break then the message will not be removed from the queue and will be passed to another consumer.
Often most of the JMS libraries are using mode when acknowledgement is sent right when message was received by consumer but you always have possibility to change this mode and send acknowledgement manually when your processing part will be finished successfully.
What about camel jms (http://camel.apache.org/jms.html) you can use endpoint option "acknowledgementModeName" which has some different possible values like:
AUTO_ACKNOWLEDGE (default) - acknowledgement will be sent right after corresponded "from" in your route
CLIENT_ACKNOWLEDGE - allows the application to control when the acknowledgment is sent and if there are no exceptions will be thrown during exchange processing then message will be acknowledged and removed from queue.
I'm designing a quite complicated system and was wondering what the best way is to put a jms consumer (activemq, vm protocol, non persitent) inside a netty handler.
Let me explain, i have several clients connecting to my netty server using websockets. For every client connection i create a jms consumer that listens for interesting messages on one or more topics. If a interesting message arrives i need to do a extra step (additional filtering) before sending the message to the client using the websocket.
Is the following a good way to do this:
inside a SimpleChannelInboundHandler i declare a private non static consumer
the consumer is initialized in channelActive
the consumer is destroyed in channelInactive
when a message is received by consumer i do the extra filter a send it using ctx.channel().write()
In this setup i'm a bit worried that the consumer might turn into slow consumer and slow everything down, cause the websocket goes over the internet.
I came up with a more complex one to decouple the "receiving of message by consumer" and "sending of message through a websocket".
inside a SimpleChannelInboundHandler i declare a private non static consumer
the consumer is initialized in channelActive
the consumer is destroyed in channelInactive
when a message is received by consumer i put it in a blockedqueue
every minute i let a thread (created for every client) look in the queue and send the found messages to the client using ctx.channel().write().
At this point i'm a bit worried about the extra thread per client.
Or is there maybe a better way to accomplish this task?
This is a classic slow consumer problem and the first step to resolving it is to determine what the appropriate action is when a slow consumer is detected. If it is acceptable that the slow consumer misses messages then the solution is some variation on dropping messages or unsubscribing them from the feed. For example, if it's acceptable that the client misses messages then, when one is received from JMS, check if the channel is writable. If it isn't, drop the message. If you want to give yourself a bit more of a buffer (although OS buffers are quite large) you can track the number of write completion future's that haven't completed (ie the messages haven't been written to the OS send buffer) and drop messages if there are too many outstanding write requests.
If the client may not miss messages, and is consistently slow, then the problem is more difficult. One option might be to divert messages to a JMS queue with a specific header value, then open a new consumer that reads messages from that queue using a JMS selector. This will put more load on the JMS server but might be appropriate for temporary slowness and hopefully it won't interfere with you main topic feeds. Alternatively you might want to stash the messages in a different store, such as a database, so you can poll for messages when they can be sent. If you do this right a single polling thread can cope with many clients (query for clients which have outstanding messages, then for each client, load a bunch of messages). However this isn't as convenient as using JMS.
I wouldn't go with option 2 because the blocking queue is only going to solve the problem temporarily, and you can achieve the same thing by tracking how many write operations are waiting to complete.
I created a AQ in oracle and wrote 2 JMS consumers in Java to listen to the queue. I have observed sometimes that if I produce some message in to queue; the count of dequeued messages from queue is greater than what enqueued. It means that some messages are consumed twice.
I have created queue with property:- multiple_consumers => FALSE
And JMS consumers are working in CLIENT_ACKNOWLEDGE mode
Please help me learn the possible reasons for such behavior and it's solution. So, that I can replicate the problem and solve above issue and ensure that the number of message enqueued is equal to number of message dequeued in case of multiple JMS consumers listening to same AQ .
Without having seen your code, CLIENT_ACKNOWLEDGE typically says you are sending acknowledgements manually. If you do not send an ack, the message won't get deleted and the broker will try to redeliver it at a later stage (like when you restart the connection or similar). This might be the cause of your concern.