I am currently exploring de-duplication strategies within Active MQ. Artemis supports duplicate detection, but I'm not sure about ActiveMQ 5
Is it possible to prevent a message from being placed on a queue if it currently exists on the queue in ActiveMQ 5?
Messages which are no longer on the queue and have been so in the past will be allowed back on the queue.
The underlying capability I am trying to achieve is flow control in which multiple messages of the same value are not placed on the queue as to remove duplicate processing.
Based on the documentation, I have tried using the message property defined _AMQ_DUPL_ID, but I am still experiencing duplication. I suspect this may not be supported in ActiveMQ 5 and am unsure what alternative option is. I'm open to suggestions.
NOTE: The Active MQ instance being used is provided by Amazon MQ.
As you suspect, ActiveMQ 5.x doesn't support automatic duplicate detection. This is only supported in ActiveMQ Artemis. That said, messages are not removed from the broker's duplicate ID cache when the message is consumed from the queue. This is because in most cases a duplicate sent after the message is consumed is still considered a duplicate.
You may be able to implement some kind of duplicate detection in a broker plugin, but I have no idea of Amazon MQ supports adding custom plugins. It's more likely that you'll have to implement duplicate detection in the clients themselves.
Related
First, let me explain what I have tried with classic ActiveMQ which worked perfectly for my requirements:
I have few Queues with naming templates and each queue represents a Tenant(customer). The naming pattern is like queue.<<tenant-id>>.event which, here, I used test1 to test5 for simplicity.
Multiple producers are putting messages on these different queues based on which tenants are requesting it.
My ActiveMQ queues look like this in the web console:
Queues in the classic ActiveMQ
Then I started the Spring JMS listener with the wildcard to be able to read from all of these queues with one listener. the code is like this:
#JmsListener(destination = "queue.>")
public void receiveMessage(Event event) {
//Process the event message
}
What I have observed which I cannot configure Artemis to do the same is:
Listening on ActiveMQ Queues with wildcard did not create a new queue(listener Queue)
Consuming the messages with a wildcard listener would actually reduce the number of pending messages in the actual queues.
The wildcard listener would actually quite fairly read messages from all queues. It still does respect the FIFO on each queue but would not respect it cross queues. For example, when I put 100 messages in the queue.test1.event and only then add 100 messages in queue.test2.event, then if I start the wildcard listener, it starts to read messages fairly from both queues, although all the messages in queue.test2.event queue are basically added after the 100 messages in queue.test1.event.
I need features #2 and #3. The first one is just the observation which I think is the root cause of my problem in Artemis.
Now, what happened when I moved to Artemis is:
The wildcard pattern is a little different but I did the same scenario. The listener looks like:
#JmsListener(destination = "queue.#")
public void receiveMessage(Event event) {
//Process the event message
}
As you see the wildcard template is changed to queue.# to be able to read from all those queues.
My Artemis queues look like this in the web console:
Queues in the Artemis,
My observation on the web console shows I cannot achieve the same here:
As you see in the picture, the number of message count in the original queues, in which I put the messages, are still kept, despite only 44 of them are remained for processing(looks at the message count of queue.#) and the rest has been already read by the wildcard listener.
This can cause a storage issue for me since all of my messages are persisted and I can't play with the message expiry too.
As you see in the picture, the listener created another Queue named queue.# which seems Artemis is internally copying the messages from the other ones into it.
Not a problem and just an observation.
It respects the FIFO across all queues, which I guess is because Artemis is doing the copy from the original queues to the wildcard one.
This creates a huge problem for me. Although I still want it to respect the FIFO inside each queue, I also want it to start consuming messages from other queues. Because, if one customer is processing huge tasks, it should not block others to continue theirs.
PS1: I restrict the listeners in both tests to just consume one message at a time to be able to test it properly.
PS2: If you wonder why I don't use classic ActiveMQ if it does exactly what I need, The answer is: Apache will make Artemis its Major version(once it reached a certain level of maturity) in the future and I would like to be aligned with the roadmap.Quote from its website:
Once Artemis reaches a sufficient level of feature parity with the "Classic" code-base it will become the next major version of ActiveMQ
PS3: I am using spring-boot and its starter packages to connect and put/consume messages.
PS4: I am using the default configuration for both solutions and installations.
Simply put, ActiveMQ Artemis doesn't support wildcard consumers. It only supports wildcard addresses which have similar but different semantics (as explained in the answer on this question of yours).
Feel free to open an issue to request this feature be implemented.
I have an ActiveMQ broker (version 5.14.3) running, which is being used by producers and consumers of different origins.
For some reason recently a lot of empty queues have been automatically created labeled either communication_{SOME_HASH} or communication_{NAME_OF_ORIGIN_MACHINE}_{SOME_PROCESS_ID}. These queues are all empty queues without any enqueued or dequeued messages, which I assume are useless but are still generated for some reason.
Has somebody come across the same issue and can either explain why these queues are being generated or point out how I can avoid this behavior?
These are not prefixes for any type of temporary or advisory based destinations that ActiveMQ would ever create so the fault must lay in your own application. You would need to debug this from your own side to find out where they are coming from. Since ActiveMQ creates destinations whenever a Producer or Consumer is created on a Topic or Queue unless security policies prevent that your applications may be creating these resources accidentally an never using them.
I am working with someone who is trying to achieve a load-balancing behavior using JMS Queues with IBM Websphere MQ. As such, they have multiple Camel JMS consumers configured to read from the same Queue. Despite that this behavior is undefined according to the JMS spec (last time I looked anyway), they expect a sort of round-robin / load-balancing behavior. And, while the spec leaves this undefined, I'm led to believe that the normal behavior of Websphere MQ is to deliver the message to only one of the consumers, and that it may do some type of load-balancing. See here, for example: When multi MessageConsumer connect to same queue(Websphere MQ),how to load balance message-consumer?
But in this particular case, it appears that both consumers are receiving the same message.
Can anyone who is more of an expert with Websphere MQ shed any light on this? Is there any situation where this behavior is expected? Is there any configuration change that can alleviate this?
I'm leaning towards telling everyone here to use the native Websphere MQ clustering facility and go away from having multiple consumers pointing at the same Queue, but that will be a big change for them, so I'd love to discover a way to make this work.
Not that I'm a fan of relying on anything that's undefined, but if they're willing to rely on IBM specific behavior, I'll leave that up to them.
The only way for them to both receive the same messages are:
There are multiple copies of the message.
The apps are browsing the message without a lock, then circling back to delete it.
The apps are backing out a transaction and making the message available again.
The connection is severed before the app acknowledges the message.
Having multiple apps compete for messages in a queue is a recommended practice. If one app goes down the queue is still served. In a cluster this is crucial because the cluster will continue to direct messages to the un-served queue instance until it fills up.
If it's a Dev system, install SupportPac MA0W and tell it to trace just that one queue and you will be able to see exactly what is happening.
See the JMS spec in section 4.4. The provider must never deliver a second copy of an acknowledged message. Exception is made for session handling in 4.4.13 which I cover in #4 above. That's pretty unambiguous and part of the official spec so not an IBM-specific behavior.
I'm using ActiveMQ along with Mule (a kind of ESB based on Spring).
We got a fast producer and a slow consumer.
It's synchronous configuration with only one consumer.
Here the configuration of the consumer in spring style: http://pastebin.com/vweVd1pi
The biggest requirement is to keep the order of the messages.
However, after hours of running this code, suddenly, ActiveMQ skips 200 messages, and send the next ones.The 200 messages are still there in the activeMQ, they are not lost.
But our client (Mule), does have some custom code to check the order of the messages, using an unique identifier.
I had this issue already a few month ago. We change the consumer by using the parameter "jms.prefetchPolicy.queuePrefetch=1". It seemed to have worked well and to be the fix we needed unti now when the issue reappeared on another consumer.
Is it a bug, or a configuration issue ?
I can't talk about the requirement from a Mule perspective, but there are a couple of broker features that you should take a look at. There are two ways to guarantee message ordering in ActiveMQ:
Message groups are a way of ensuring that a set of related messages will be consumed by the same consumer in the order that they are placed on a queue. To use it you need to specify a JMSXGroupID header on related messages, and assign them an incrementing JMSXGroupSeq number. If a consumer dies, remaining messages from that group will be sent to another single consumer, while still preserving order.
Total message ordering applies to all messages on a topic. It is configured on the broker on a per-destination basis and requires no particular changes to client code. It comes with a synchronisation overhead.
Both features allow you to scale out to more than one consumer.
I need some help with topics and selectors.
I have a scenario with a topic having multiple durable subscribers (each with a selector)
not all messages going into the topic aren't read by the consumers - because of unmatching selectors.
This is correct behavior.
However the problem occurs when the unmatched messages reach a certain quantity threshold, because at that point no other messages are being delivered to the consumers
activemq tries to dispatch those old unmatchable messages, but since there is no consumer for them everything is stuck
can anybody help with this?
My setup is ActiveMq 5.5
is there some configuration option, or is it just a flawed design?
I'd say this is a flawed design given there are better alternatives and perhaps a bug in ActiveMQ.
First question: is your producer publishing to this topic setting the JMSExpiration header on those messages?
If yes, the first thing I'd do is create a Jira issue detailing the scenario you described above, because it does seem incorrect that ActiveMQ will continue hold on to and continue send messages for which no selectors apply.
As for flawed design, the minute you hear yourself saying "I need durable subscribers" and you are using ActiveMQ, you should immediately turn to using Virtual Destinations instead. Virtual Destinations have the advantages of topics in that a producer can send a message to a destination and have that message propagated to N number of other destinations for consumption, but doesn't have the disadvantages that come with having durable subscribers on a topic. Read more about Virtual Destinations here.
This is related to the way that ActiveMQ handles sparse selectors. The current implementation doesn't page into the store to look for message matching sparse selectors so you need to make some configuration changes to try and work around this. You can set the maxBrowsePageSize in the configured destination policy, the default is 400. See this page.
Virtual destinations in ActiveMQ is probably the better choice, it almost always is when thinking about using durable subscribers. You could however add some message expiration to you messages and configure the policy to expire messages on inactive durable subscribers if you use a SNAPSHOT version of ActiveMQ 5.6.
It does seem like a bug, (or dashedly inconvenient at the least) but there is a work around using Virtual Destinations.
Quote:
Note that making a topic virtual does add a small CPU overhead when
sending messages to the topic but it is fairly small. From version
5.4, dispatch from virtual topics to subscription queues can be
selectorAware such that only messages that match one of the existing
subscribers are actually dispatched. Using this option prevents the
build up of unmatched messages when selectors are used by exclusive
consumers.