My program will be receiving messages rather slowly; and I want to them to persist in the queue until I have receive all of them and acknowledge all of them. I don't know if I have enough messages until I receive a bunch of them.
My question: will the queue block, waiting for the acknowledgement from the first message before delivering the second?
Well I ran a test one this using the sample producer/consumer code. The consumer actually has some code (if you switch over to ClientAcknowledge). It receives a bunch of messages (10 of them) and only acks the last one.
When setting the acknowledge mode to Session.CLIENT_ACKNOWLEDGE you can get as many messages you need. The messages will be locked on the server, so no other consumer can retrieve them meanwhile. So the answer is no, the queue won't block (even thu there might be provider-specific settings that can do that, which I don't know).
However, you can acknowledge only all at once. So when you have received 10 messages, and you acknowledge one of them (doesn't matter which), all messages will be acknowledged.
Check for your reference Controlling Message Acknowledgment
Related
I created a SpringBoot/Spring AMQP project where I configured a listener on a RabbitMQ queue. Question: Is there any way to leave the message in the queue? Let me explain: I consume the message and do some things (eg save on db), if something goes wrong I would like to be able to reconsume the message.
Thanks in advance
You need to think about configuring your listener container with transactions, so when DB call fails, the transaction is going to be rolled back and an AMQP message will not be acked on RabbitMQ.
See docs for more info: https://docs.spring.io/spring-amqp/docs/current/reference/html/#transactions
I don't know about the "Spring" way of accomplishing this, but what you describe is the normal behavior for AMQP consumers that do not automatically acknowledge.
From the documentation:
In automatic acknowledgement mode, a message is considered to be successfully delivered immediately after it is sent.
When you turn off automatic acknowledgment, your consumer must explicitly acknowledge the message, otherwise it will not be dequeued (or as you put it, it will be left "in the queue"). You will then need to simply ACK the message at the very end of your operation, when you are certain that your operation succeeded (and perhaps coordinated with your database transaction).
There is always the question of what to do first; acknowledge first or commit your database transaction first? Without adding complexity, you must choose what's best depending on what failure mode is less problematic for you, i.e. Would you rather tolerate a duplicated message or a missing message?
We have a JMS queue and multiple competing clients are reading from this queue.
Once the message is taken and successfully processed, we want to send the acknowledge to delete ( i.e. CLIENT ACKNOWLEDGE )
However, we want to make sure that if one client has picked the message another client should not take it from the queue.
Does activeMQ provide this feature out of the box using some configuration ?
Moreover:
If the message processing failed after picking the message, so it could not be acknowledged back, in this scenario we should like other client thread to pickup the message. Is it possible out of the box with configuration , may be specifying timeout values ?
Regards,
JE
You need to take some time to understand the difference between a Topic and a Qeueue in order to understand why the first question is not an issue.
For the second question it depends a bit on the ACK mode you are using and how you are processing messages sync or async. Normally for processing where you want to control redeliveries you would do the work inside of a transaction and if the processing fails the message would be redelivered when the TX is rolled back. ActiveMQ supports redelivery policies both client side and broker side that control how many time a message will be redelivered before sent to a DLQ.
I'm designing a quite complicated system and was wondering what the best way is to put a jms consumer (activemq, vm protocol, non persitent) inside a netty handler.
Let me explain, i have several clients connecting to my netty server using websockets. For every client connection i create a jms consumer that listens for interesting messages on one or more topics. If a interesting message arrives i need to do a extra step (additional filtering) before sending the message to the client using the websocket.
Is the following a good way to do this:
inside a SimpleChannelInboundHandler i declare a private non static consumer
the consumer is initialized in channelActive
the consumer is destroyed in channelInactive
when a message is received by consumer i do the extra filter a send it using ctx.channel().write()
In this setup i'm a bit worried that the consumer might turn into slow consumer and slow everything down, cause the websocket goes over the internet.
I came up with a more complex one to decouple the "receiving of message by consumer" and "sending of message through a websocket".
inside a SimpleChannelInboundHandler i declare a private non static consumer
the consumer is initialized in channelActive
the consumer is destroyed in channelInactive
when a message is received by consumer i put it in a blockedqueue
every minute i let a thread (created for every client) look in the queue and send the found messages to the client using ctx.channel().write().
At this point i'm a bit worried about the extra thread per client.
Or is there maybe a better way to accomplish this task?
This is a classic slow consumer problem and the first step to resolving it is to determine what the appropriate action is when a slow consumer is detected. If it is acceptable that the slow consumer misses messages then the solution is some variation on dropping messages or unsubscribing them from the feed. For example, if it's acceptable that the client misses messages then, when one is received from JMS, check if the channel is writable. If it isn't, drop the message. If you want to give yourself a bit more of a buffer (although OS buffers are quite large) you can track the number of write completion future's that haven't completed (ie the messages haven't been written to the OS send buffer) and drop messages if there are too many outstanding write requests.
If the client may not miss messages, and is consistently slow, then the problem is more difficult. One option might be to divert messages to a JMS queue with a specific header value, then open a new consumer that reads messages from that queue using a JMS selector. This will put more load on the JMS server but might be appropriate for temporary slowness and hopefully it won't interfere with you main topic feeds. Alternatively you might want to stash the messages in a different store, such as a database, so you can poll for messages when they can be sent. If you do this right a single polling thread can cope with many clients (query for clients which have outstanding messages, then for each client, load a bunch of messages). However this isn't as convenient as using JMS.
I wouldn't go with option 2 because the blocking queue is only going to solve the problem temporarily, and you can achieve the same thing by tracking how many write operations are waiting to complete.
I have a persistent queue, non-transacted, client-acknowledge, the consumers read with jms.prefetchPolicy.queuePrefetch=1&wireFormat.maxInactivityDuration=50000
and once a consumer processes a message, it ack's the message.
If the consumer reads the message, and before it can send an ack, the process terminates abruptly, what happens in ActiveMq? (What ActiveMq parameters come into play here?)
How is that different than if the the consumer will take 10 minutes to process the message (so the consumer task is alive and working), how does ActiveMq know the message is still being worked on? (Does it monitor the TCP/IP connection, if the connection dies, it assumes the message will not be Ack'ed?)
How do I determine if a message is a "poison pill", i.e. it makes the consumers crash? (the redelivery count seems to be valid if the consumer task does not die; is there an internal counter in the message that says "it was been read n times without being successfully ack'ed?")
As an experiment, I sent 6 messages, one of them being a "poison pill" (kills the consumer before the consumer can send the ack), with 2 simultaneous consumers running (and automatically restarting consumers to bring the count to 2 whenever a consumer dies). Looking at the queue (using jconsole, I enable jmx using broker.setUseJmx(true)), 4 messages were delivered, 2 are in-flight. Why would there be 2 in-flight instead of just one?
I've been reading the ActiveMq and JMS specs for a while without clear/conclusive answers, so any insights on what parameters come into play, and if there are any known bugs, will be greatly useful.
This is purely based on my understanding of JMS - may not be completely correct:
If the consumer reads the message, and before it can send an ack, the process terminates abruptly, what happens in ActiveMq
My understanding is that since this happens in the context of a session with the JMS provider, JMS provider knows if the session is no longer active or has failed and any message not acknowledged as part of the session will be redelivered when the session is re-established.
How do I determine if a message is a "poison pill", i.e. it makes the consumers crash?
Like you have mentioned, the JMS provider keeps track of the # of times the message was redelivered possibly in the header of the message
4 messages were delivered, 2 are in flight
Not sure about this point
We have a situation in front of us and its explained as below:
We have durable subscriber subscribed to a topic. This durable subscriber is a perl script which is run by a daemon.
The perl script uses stomp to connect to the broker.
The perl script wakes up every 5 mins, checks for messages in the topic and processes them in a batch by pre-fetching the messages.
The subscriber uses a client acknowledgement and acknowledges only the last message of the batch.
We are using AMQ 5.5 with kahaDB persistence.
Now what we see is,
Even though the messages are processed in a batch and the last message is acknowledged the inflight count does not come down.
Enqueue count, Dequeue count and Dispatch count do not match.
The journal files are not getting cleaned up.
I do understand that the journal files would be cleaned up once the references to the messages are lost or removed (i.e. the messages are consumed). But does it have to do anything with the various count attributes I see on the topic?
Also should I expect the inflight count to come down to 0 if client crashes and then consumes all messages after come back live?
Please let me know if there could be any other reason that could cause the journal files to stay back.
Thank you
Hari
Actually I was disconnecting immediately after sending the ack.
Waiting for ack's receipt before disconnecting solved the issue.