I want to automatically delete all the messages from queues that where inactive for a specified amount of time (no new messaged was arrived on that time).
I don't want to explicitly empty the queue from code nor call purge explicitly as described here.
The configuration described here is also not appropriate to my case, since it deletes automatically only empty queues and my queues are not empty.
Is there any known ActiveMQ configuration that can do that task automatically?
I never had such requierement and I don't know if such functionality exists in activemq, however, there is two options you might be interested with :
1) If you want to purge messages on inactive queues because they a no longer relevant, you could set the time to live on each messages ( setTimeToLive() method on producer side)
2) If you need that exact behavior, then you could develop your own plugin. Indeed activemq brokers are fairly extensible (see : http://activemq.apache.org/developing-plugins.html)
Hope it helps.
Related
First, let me explain what I have tried with classic ActiveMQ which worked perfectly for my requirements:
I have few Queues with naming templates and each queue represents a Tenant(customer). The naming pattern is like queue.<<tenant-id>>.event which, here, I used test1 to test5 for simplicity.
Multiple producers are putting messages on these different queues based on which tenants are requesting it.
My ActiveMQ queues look like this in the web console:
Queues in the classic ActiveMQ
Then I started the Spring JMS listener with the wildcard to be able to read from all of these queues with one listener. the code is like this:
#JmsListener(destination = "queue.>")
public void receiveMessage(Event event) {
//Process the event message
}
What I have observed which I cannot configure Artemis to do the same is:
Listening on ActiveMQ Queues with wildcard did not create a new queue(listener Queue)
Consuming the messages with a wildcard listener would actually reduce the number of pending messages in the actual queues.
The wildcard listener would actually quite fairly read messages from all queues. It still does respect the FIFO on each queue but would not respect it cross queues. For example, when I put 100 messages in the queue.test1.event and only then add 100 messages in queue.test2.event, then if I start the wildcard listener, it starts to read messages fairly from both queues, although all the messages in queue.test2.event queue are basically added after the 100 messages in queue.test1.event.
I need features #2 and #3. The first one is just the observation which I think is the root cause of my problem in Artemis.
Now, what happened when I moved to Artemis is:
The wildcard pattern is a little different but I did the same scenario. The listener looks like:
#JmsListener(destination = "queue.#")
public void receiveMessage(Event event) {
//Process the event message
}
As you see the wildcard template is changed to queue.# to be able to read from all those queues.
My Artemis queues look like this in the web console:
Queues in the Artemis,
My observation on the web console shows I cannot achieve the same here:
As you see in the picture, the number of message count in the original queues, in which I put the messages, are still kept, despite only 44 of them are remained for processing(looks at the message count of queue.#) and the rest has been already read by the wildcard listener.
This can cause a storage issue for me since all of my messages are persisted and I can't play with the message expiry too.
As you see in the picture, the listener created another Queue named queue.# which seems Artemis is internally copying the messages from the other ones into it.
Not a problem and just an observation.
It respects the FIFO across all queues, which I guess is because Artemis is doing the copy from the original queues to the wildcard one.
This creates a huge problem for me. Although I still want it to respect the FIFO inside each queue, I also want it to start consuming messages from other queues. Because, if one customer is processing huge tasks, it should not block others to continue theirs.
PS1: I restrict the listeners in both tests to just consume one message at a time to be able to test it properly.
PS2: If you wonder why I don't use classic ActiveMQ if it does exactly what I need, The answer is: Apache will make Artemis its Major version(once it reached a certain level of maturity) in the future and I would like to be aligned with the roadmap.Quote from its website:
Once Artemis reaches a sufficient level of feature parity with the "Classic" code-base it will become the next major version of ActiveMQ
PS3: I am using spring-boot and its starter packages to connect and put/consume messages.
PS4: I am using the default configuration for both solutions and installations.
Simply put, ActiveMQ Artemis doesn't support wildcard consumers. It only supports wildcard addresses which have similar but different semantics (as explained in the answer on this question of yours).
Feel free to open an issue to request this feature be implemented.
I have an ActiveMQ broker (version 5.14.3) running, which is being used by producers and consumers of different origins.
For some reason recently a lot of empty queues have been automatically created labeled either communication_{SOME_HASH} or communication_{NAME_OF_ORIGIN_MACHINE}_{SOME_PROCESS_ID}. These queues are all empty queues without any enqueued or dequeued messages, which I assume are useless but are still generated for some reason.
Has somebody come across the same issue and can either explain why these queues are being generated or point out how I can avoid this behavior?
These are not prefixes for any type of temporary or advisory based destinations that ActiveMQ would ever create so the fault must lay in your own application. You would need to debug this from your own side to find out where they are coming from. Since ActiveMQ creates destinations whenever a Producer or Consumer is created on a Topic or Queue unless security policies prevent that your applications may be creating these resources accidentally an never using them.
I have to run two instances of the same application that read messages from 'queue-1' and write them back to another queue 'queue-2'.
I need my messages inside the two queues to be ordered by specific property (sequence number) which is initially added to every message by producer. As per documentation, inside queue-1 the order of messages will be preserved as messages are sent by a single producer. But because of having multiple consumers that read, process and send the processed messages to queue-2, the order of messages inside queue-2 might be lost.
So my task is to make sure that messages are delivered to queue-2 in the same order as they were read from queue-1. I have implemented re-sequencer pattern from Apache camel to re-order messages inside queue-2. The re-sequencer works fine but results to data transfer overhead as the camel routes run locally.
Thinking about doing it in a better way, I have three questions:
Does artemis inherently supports re-ordering of messages inside a
queue using a property such as sequence number.
Is it possible to run the routes inside the server? If yes, can you
give an example or give a link to the documentation?
Some artemis features such as divert (split) requires modifying
broker configuration (broker.xml file), is there a way to do them
programmatically and dynamically so I can decide when to start
diverting message? I know this can be accomplished by using camel,
but I want everything to be running in the server.
Does artemis inherently supports re-ordering of messages inside a queue using a property such as sequence number.
No. Camel is really the best solution here in my opinion.
Is it possible to run the routes inside the server? If yes, can you give an example or give a link to the documentation?
You should be able to do the same kind of thing in Artemis as in ActiveMQ 5.x using a web application with a Camel context. The 5.x doc is here.
Some artemis features such as divert (split) requires modifying broker configuration (broker.xml file), is there a way to do them programatically and dynamically so I can decide when to start diverting message?
You can use the Artemis management methods to create, modify, and delete diverts programmatically (or administratively) at runtime. However, these modifications will be volatile (i.e. they won't survive a broker restart).
Is there a way to configure a JMS message queue to drop all received messages automatically? (In other words: for the sender it must look like the message arrived ok, but the queue should silently drop it. That is, the queue should behave like "/dev/null") In case that matters, I'm most interested in IBM MQ, however, if JMS specification offers this possibility, that would be even better.
Since I already tried searching, and did not find such a possibility, I suspect the answer is "no". So an answer that confirms that this is the case (and I'm not missing anything) is also valid for me.
Use case: test a system, that sends messages to a real JMS queue, which (in the test setup) are not read out by any other program. (I do not want to mock the JMS queue, to keep the test environment as close to production as possible. For the same reason, it is also not a valid solution to e.g. set "time to live" for the messages on the client side.)
How about making a queue which you put to being an alias to a topic with noone listening on it?
e.g.:
def qalias(MYQUEUE) target(DUMMYTOPIC) targtype(TOPIC)
def topic(DUMMYTOPIC) topicstr(DUMMYTOPIC)
Then you can put to the queue, it becomes a publish and if noone is listening on the queue it vanishes. amqsput MYQUEUE QM for example will work fine.
When you want to then consume the messages, change the QALIAS to either point to a real queue, or replace the QALIAS definition with a QLOCAL for example.
The only downside is if anyone is subscribed to every topic, but for testing purposes you can
If you want a different approach where you can control (turn off/on) the flow of messages being deleted/removed, then an MQ Service and my 'emtyq' program is another option.
Here's a write up I did a while ago on the subject: http://www.capitalware.com/rl_blog/?p=3680
I'm using ActiveMQ along with Mule (a kind of ESB based on Spring).
We got a fast producer and a slow consumer.
It's synchronous configuration with only one consumer.
Here the configuration of the consumer in spring style: http://pastebin.com/vweVd1pi
The biggest requirement is to keep the order of the messages.
However, after hours of running this code, suddenly, ActiveMQ skips 200 messages, and send the next ones.The 200 messages are still there in the activeMQ, they are not lost.
But our client (Mule), does have some custom code to check the order of the messages, using an unique identifier.
I had this issue already a few month ago. We change the consumer by using the parameter "jms.prefetchPolicy.queuePrefetch=1". It seemed to have worked well and to be the fix we needed unti now when the issue reappeared on another consumer.
Is it a bug, or a configuration issue ?
I can't talk about the requirement from a Mule perspective, but there are a couple of broker features that you should take a look at. There are two ways to guarantee message ordering in ActiveMQ:
Message groups are a way of ensuring that a set of related messages will be consumed by the same consumer in the order that they are placed on a queue. To use it you need to specify a JMSXGroupID header on related messages, and assign them an incrementing JMSXGroupSeq number. If a consumer dies, remaining messages from that group will be sent to another single consumer, while still preserving order.
Total message ordering applies to all messages on a topic. It is configured on the broker on a per-destination basis and requires no particular changes to client code. It comes with a synchronisation overhead.
Both features allow you to scale out to more than one consumer.