I read in ActiveMQ page, using JMX we can monitor queues in activemq. Is it possible to plot the graphs of messages count vs time, basically rate at which messages are arriving at queue. Thanks
Yes, if you can monitor the values, you can plot it. If you want something out of the box, you can download Hawt.io and connect to AMQ. There is a chart feature that allows plotting various values.
Related
I worked a little with the ActiveMQ scheduler plugin. This simplifies scheduling messages for delivery with a delay at low volume, but as I get into the 100ks of messages the system breaks down in two key ways.
It's very slow (compared to queues) to enqueue messages in the scheduler.
Attempting to view the schedules in the dashboard crashes the ActiveMQ instance.
The existing scheduler feels a little bolted on and does not perform as expected. So, rethinking the problem I would like to have a jobs and jobs-scheduled queue. Messages sent to the jobs-scheduled queue will have a ttl header with the unix timestamp for when it should be delivered. A process will run on a cron job which will take messages from the jobs-scheduled queue and send it to the jobs queue using a selector to just pick out the messages with an elapsed ttl convert_string_expressions:ttl < %(now)s.
My two questions are:
Will this strategy work for delaying messages at scale or will I find scaling pains around the selector? These messages will be persisted if that makes a difference.
Is there an existing feature in ActiveMQ that will allow me to send messages from one queue to another with a selector query?
ActiveMQ is a message broker not a job scheduler so what you are trying to do is really outside the scope of the what the broker is intended to do. Yes ActiveMQ does have a scheduled message feature but this is not intended for large scale job queue type work, it is a simple feature to provide some minimal delayed delivery.
What you are looking for sounds more like Quartz or some other batch job scheduling library. You could develop your own Job scheduler implementation for ActiveMQ or do something in a plugin but you are really trying to run against the grain of what a broker is meant to do which is deliver messages as quickly as possible in a decoupled manner.
Side note-- potentially off-topic.
I've had to solve a similar situation in the past where it made a lot of sense to load up the queues with messages ahead of time to cut down on the total transfer time.
I solved it by using Camel routes and a side-channel activation. Camel allows you to programmatically start and stop routes, so you can load up a queue with no consumers for the data for a given time period. Then using a dedicated queue for control you send the 'start' message. The control route receives the 'start' message, and then activates the main data processing route. You then need to configure some sort of 'stop' message semantic to be ready for the next time periods run.
Effectively, you get the delayed behavior pattern with much more control over scheduling and cut down on the data-to-queue loading time problem. You can also solve the scaling problem by loading the data across more than one queue.
I assume that there are hundreds and thousands IoT devices that publish the data to the (broker)MQTT cluster via the MQTT protocol, behind the broker i have the data processing module which subscribe the data from the broker and maintain a status table for all these devices. The number of the devices is still rising, therefor I have to scale out the broker cluster and data processing module accordingly, for the MQTT broker such as Kafka/Rabbit MQ/Hive MQ can be scaled out very easily, but for the data processing module I'm not quite sure whether there is any best practice, or any framework/architecture can achieve this very easily:
I assume I have to create many daemon processes with hundreds and thousands threads to listen on the MQTT broker, the question is how to scale out these services dynamically?
Thanks.
One way of doing this would be using Node.js as it uses an event-driven approach and you don't have to deal with threads, etc.
I found this library for Node.js which is specific to MQTT:
https://www.npmjs.com/package/mqtt
You can use this to subscribe to different topics.
You may also find this project interesting:
http://nodered.org/
The other solution can be using Apache Kafka which has scalability as an important feature. However, the problem here is that Kafka does not support MQTT out of the box and has its own conventions. Therefore, there is a need for some sort of adapter to make them work together. For that, take a look at this:
using mqtt protocol with kafka as a message broker
I am creating a hosted system where multiple customers can send messages. I am receiving thoses messages on a JMS queue.
Now, all processing is done in a similar way and I want my process to poll all incoming queues for messages and handle them. Is there a way in WSO2 ESB to subscribe to multiple queues?
If not possible, the workaround would be to create a seperate listener process for each queue and have this post the message to a central processing queue. But that seems to be a less clean solution (and I think it will scale worse than listening to multiple queues).
Any ideas on this?
If changes to activeMQ server is possible ie. if OP is able to influence the configuration to the server, something like ActiveMQ diverts could do the trick.
<divert name="prices-divert">
<address>jms.queue.ABC</address>
<forwarding-address>jms.queue.theone</forwarding-address>
<exclusive>true</exclusive>
</divert>
<divert name="prices-divert">
<address>jms.queue.xyz</address>
<forwarding-address>jms.queue.theone</forwarding-address>
<exclusive>true</exclusive>
</divert>
Basically, multiple diverts that converge the messages from multiple queues to the single queue. This method has advantage over the reading and writing to single queue-as mentioned by the OP and would in my view scale well as it is inbuilt feature.
You can define a sequence with all the required logic in it and then call it from multiple proxy services (each listening to a specific queue). Otherwise you can try something similar to this sample.
I have a single 'point to point' IBM MQ queue receiving messages from multiple producers. My application consumes the messages from the queue. I am using spring 'jmstemplate" and "DefaultMessageListenerContainer" to consume the messages asynchronously.
My application is running on 2 jvms, meaning there are 2 listeners active on each jvm listening to the same queue.
Coming to my questions, If a message arrives...
1) How does the listeners know that the message arrived in the queue?
2) Out of the two listeners, which one will receive the message ? What is the approach followed to distribute the messages to the listeners?
3) Can i scale to 'N' count of listeners for a singe queue? If i grow to 10 listeners, how does the scaling work? how are the messages distributed to listeners?
4) How does the MQ server make sure that the same message is not sent to multiple listeners?
May be these are simple questions, but not able to drill down on how the above scenarios works. Please share your thoughts...
That's a function of the IBM client library; the listener container simply polls the JMS API waiting for a message; by default, it uses a 1 second receive timeout; with TRACE level logging, you will see log messages showing this activity. The timeout can be modified by setting receiveTimeout on the container.
That is indeterminate from the client's perspective; the IBM broker knows how many consumers are registered and picks one. Some brokers allow configuration of a pre-fetch; this can help performance under high volume but can hurt performance under low volume.
Yes; the Spring Listener Container can dynamically scale the listeners based on load; you can configure min/max consumers and Spring will adjust within those bounds as necessary. Each listener is a separate consumer, as far as the broker is concerned so the work is distributed according to the broker's algorithms.
That's a function of the IBM broker (and part of the JMS contract).
If using transactions and a message is rolled back onto the queue; there is no guarantee that the same listener will get the re-delivered message.
in order to reach high performance production of messages with jms with transactions enabled, one needs to control the amount of messages being sent on each transaction, the larger the number the higher the performance are,
is it possible to control transactions in such a way using spring integration ?
one might suggest using an aggregator, but that defeats the purpose because i dont want to have one message containing X smaller messages on the queue, but actually X messages on my queue..
Thanks !
I'm not aware of your setup, but I'd bump up the concurrent consumers on the source than try to tweak the outbound adapter. What kind of data source is pumping in this volume of data ? From my experience, usually the producer lags behind the publisher - unless both are JMS / messaging resources - like in the case of a bridge. In which case you will mostly see a significant improvement by bumping up the concurrent consumers, because you are dedicating n threads to receive messages and process them in parallel, and each thread will be running in its own "transaction environment".
It's also worthwhile to note that JMS does not specify a transport mechanism, and its unto the broker to choose the transport. If you are using activemq you can try experimenting with open wire vs amqp and see if you get the desired throughput.