I understand that Topics have the additional features of subscriptions and filters which Queues do not.
In which case, when would I absolutely need to use a queue over a topic?
For consistency, could I use a topics everywhere including as a replacement for queues?
A topic is not a replacement for a queue. The combination of a topic and a subscription is. A topic is allowing to “replicate” the same message to multiple subscriptions. A subscription what actually holds messages. A subscription is identical to a queue in its attributes and behaviour. You could replace a queue with a topic+subscription combo if you’d like, generating 2 entities per use case instead of a single queue. Just keep in mind there’s a finite number of entities per namespace.
Related
ActiveMQ 5.15.13
Context: I have a single queue with multiple Consumers. I want to stop some consumers from processing certain messages. This has to be dynamic, I don't want to create separate queues for this. This works without any problems. e.g. Consumer1 ignores Stocks -> Consumer1 can process all invoices and Consumer2 can process all Stocks
But if there is a large number of messages already in the Queue (of one type, e.g. stocks) and I send a message of another type (e.g. invoices), Consumer1 won't process the message of type invoices. It will instead be idle until Consumer2 has processed all Stocks messages. It does not happen every time, but quite often.
Is there any option to change the order of the new messages coming into the queue, such that an idle consumer with matching selector picks up the new message?
Things I've already tried:
using a PendingMessageLimitStrategy -> it seems like it does not work for queues
increasing the maxPageSize and maxBrowsePageSize in the hope that once all Messages are in RAM, the Consumers will search for their messages.
Exclusive Consumers aren't an option since I want to be able to use more than one Consumer per message type.
Im pretty sure that there is some configuration which allows this type of usage. I'm aware that there are better solutions for this issue, but sadly I can't use them easily due to other constraints.
Thanks a lot in advance!
EDIT: I noticed that when I'm refreshing on the localhost queue browser, the stuck messages get executed immediately. It seems like this action performs some sort of queue refresh where the messages get filtered based on their selector again. So I just need this action whenever a new message enters the queue...
This is a 'window' problem where the next set of 'stocks' data needs to be processed before the 'invoicing' data can be processed.
The gotcha with window problems like this is that you need to account for the fact that some messages may never come through, or a consumer may never come back online either. Also, eventually you will be asked 'how many invoices or stocks are left to be processed'-- aka observability.
ActiveMQ has you covered-- check out wild-card destinations and consumers.
Produce 'stocks' to:
queue://data.stocks.input
Produce 'invoices' to:
queue://data.invoices.input
You then setup consumes to connect:
queue://data.*.input
note: the wildard '*'.
ActiveMQ will match queues based on the wildcard pattern, and then process data accordingly. As a bonus, you can still use a selector.
I use ActiveMQ's composite queue in front of physical queues because of the ability to set permissions differently on the producer and consumer side. And this works like designed.
I also I want to use a composite queue in front of topics. In this way I can use the same permission mechanism like with the above mentioned queuing concept.
Is there a disadvantage for using composite queue in front of a topic regarding for example a potential decrease of performance? Are there other disadvantages which I have to take into account when working constructs like composite queue -> topic?
The performance impact would be negligible for most workloads. Workloads short of 100's of client connections and 100M's of messages per day is usually a blip for modern hardware and ActiveMQ.
This sounds like a policy along the lines of an 'alias naming' for destinations. This pattern exists in other products, and is definitely a valid use case for Composite Destinations in ActiveMQ-- you are well within the lines of intended use for that feature.
Disadvantage wise-- nothing jumps out you should be good.
I am trying to create an "alerting" application that has to process information from multiple kafka topics. There exist thousands of topics, but realistically only a few hundred need to be processed at any given time based on the alerting configuration. If I continuously update my topics list with "subscribe" then the latency of rebalancing may delay my alerts.
How can I efficiently implement a consumer group that subscribes to a set of constantly changing topics?
I'd say the answer to this today is to use assign() instead of subscribe and manually add in the new topic partitions removing any unused ones as you need to. Though it might be helpful for you to take a step back and ask if it makes more sense for the number of topics to be static and identify things to monitor by keys. That might make your life easier.
I have read in the ActiveMQ documentation, that subtopics can be created by using wildcards. So for instance I could create the topics:
physicalEnvironmet.Conditions
physicalEnvironmet.Infrastructure
physicalEnvironmet.Location
I could then register to either one of the topics, or to all (physicalEnvironmet.>)
But how is it working for more complex structures, like this:
Would the topic for Flickering be called:
physicalEnvironmet.Conditions.Light.Flickering
And could I still have a precise selection, like only subscribing to topics considered with light:
physicalEnvironmet.Conditions.Light.>
So basically I am asking If there is a level restriction for subtopics and If there is maybe a more easy way to create hierarchical topic orders.
In my 10+ yrs of messaging, every hierarchal topic structure ends up being replaced, b/c the taxonomy never works out. Your overall message pattern suggests a moderate total volume, so I suggest a flexible event model where you use fields to define the variance vs topic names eventType="Environmental" sensorType="Light". This allows you to add new ones and then have the option of filtering out what clients want and do not want without having to mess with the broker.
Another option is to use JMS headers to do the same. This would allow you to use selectors to do broker-side filtering.
I am trying to implement an application(Java) which will subscribe to different message types (XMLs) from other different applications via TIBCO EMS. Each of these message types will have a specific purpose. I am of the opinion that I should have multiple queues with multiple subscribers in my application, however, the TIBCO guy is adamant that there should be only one queue where all of these messages will be published and I will have one subscriber and the subscriber then should have logic to different tasks based on the XML received.
Which approach is better? One with multiple queues and subscribers OR the one queue and one subscriber? Please let me know reasons for the choice.
Thanks!
-Naveen
In general, if the same application is reading all the messages, it is much cleaner for that application to have a single input queue instead of multiple input queues. With multiple then the application will need to have logic to know which order to process the queues and so on. With one input queue, the messaging system can deal with the order of the messages - whether FIFO or by priority etc, and the application can just read the next message and process it.
Use unique message header for each type of xml while sending the message. And use message selectors / filters while receiving the same, so that it can be routed / delegated to the respective handler based on the header value. This way, you will be able to handle different type of xml messages by single queue as well.