I'm new to ActiveMQ. I'm using it (and Apache Camel) for batch processing that ends up communicating with web services.
My question is how does ActiveMQ control how asynchronous it really is? In other words, if it can process 20 messages at the same time but the bottleneck is the web services on the other end, how can I control that? Can I slow ActiveMQ down?
Thanks!
if you're using apache camel 2.4+, you can use the throttler with camel to control message flow to endpoints - you can change the limit dynamically as of camel 2.8 - hope it helps.
The Camel throttler would be a good solution. It does however keep the messages in memory in the Camel application at the time. However it allows to react precisely.
Another alternative is the throttling in-flight route policy you can configure on the Camel ActiveMQ JMS consumer. That policy can be configured with a upper/lower water mark settings. Then the policy will automatic suspend/resume the AMQ consumer accordingly. You can extend this logic and use your custom metrics instead.
You can read about this policy here: http://camel.apache.org/routepolicy
This approach would then not keep any messages in memory, as it will control on the AMQ consumer side to suspend/resume it to "throttle" the flow.
You could also set a pre-fetch limit, preventing AMQ from dispatching messages on a specific transport to its consumers based on the amount of message_delivered responses it gets back from the client. Here is a reference
Related
We have configured our ActiveMQ message broker as a Spring Boot project and there's another Spring Boot application (let's call it service-A) that has a listener configured to listen to some topics using #JmsListener annotation. It's a Spring Cloud microservice appilcation.
The problem:
It is possible that service-A can have multiple instances running.
If we have 2 instances running, then any message coming on topic gets listened to twice.
How can we avoid every instance listening to the topic?
We want to make sure that the topic is listened to only once no matte the number of service-A instances.
Is it possible to run the microservice in a cluster mode or something similar? I also checked out ActiveMQ virtual destinations but not too sure if that's the solution to the problem.
We have also thought of an approach where we can decide who's the leader node from the multiple instances, but that's the last resort and we are looking for a cleaner approach.
Any useful pointers, references are welcome.
What you really want is a shared topic subscription which was added in JMS 2. Unfortunately ActiveMQ 5.x doesn't support JMS 2. However, ActiveMQ Artemis does.
ActiveMQ Artemis is the next generation broker from ActiveMQ. It supports most of the same features as ActiveMQ 5.x (including full support for OpenWire clients) as well as many other features that 5.x doesn't support (e.g. JMS 2, shared-nothing high-availability using replication, last-value queues, ring queues, metrics plugins for integration with tools like Prometheus, duplicate message detection, etc.). Furthermore, ActiveMQ Artemis is built on a high-performance, non-blocking core which means scalability is much better as well.
So using JMS and ActiveMQ, I can be sure that my message sent from my Spring Boot application using JmsTemplate will reach it's destination application even if that destination application is down at the time I send the message to ActiveMQ. As when the destination application starts up, it grabs the message from the queue. Great!
However.
What happens if my Spring Boot application tries to send a JMS message to a queue on the ActiveMQ server, but the ActiveMQ server is down at that point or the network is down and I get a connection refused exception?
What is the recommended way to make sure my application keeps trying to re-sends the message to ActiveMQ until it is successful? Is this something I have to develop into my application myself? Are there any nifty Spring tools or annotations which do this for me? Any advice on best practice or how I should be handling this scenario?
You can try Spring-Retry. Has lots of fine grain controls for it:
http://www.baeldung.com/spring-retry
https://github.com/spring-projects/spring-retry
If it is critical that you don't lose this message, you will want to save it to some alternative persistent store (e.g. filesystem, local mq server) along with whatever retry code you come up with. But for those occasional network glitches or a very temporary mq shutdown/restart, Spring-Retry alone should do the trick.
Couple of approaches I can think of
1. You can set up another ActiveMq as fallback. In your code you don't have to do anything, just change your broker url from
activemq.broker.url=tcp://amq01.blah.blah.com:61616
to
activemq.broker.url=failover:(tcp://amq01.blah.blah.com:61616,tcp://amq02.blah.blah.com:61616)?randomize=false
The rest is automatically taken care of. i.e. when one of them is down, the messages are sent to other.
Another approach is to send to a internal queue (like seda, direct) when activemq is down and read from there.
Adding failover to the url is one appropriate way.
And another reasonable way is to making sure activemq always online , as activemq has the master-slave mode(http://activemq.apache.org/masterslave.html) to get high availability.
I want my system to guarantee there is no data loss even if the system is shutting down.
What this mean is that the system must not miss the request message. So, I will change the way that accept http reqeust. Now, I am using http gateway/webservice gateway in spring integration. But, This isn't receive the message even if the system dies. So, I want to add the queue between the http client and the http receiver. So, I want to use a queue channel. Here is the question.
① I have to install other queue program such as activemq or rabbitmq and have to connect to the queue channel in spring integration?
② and which one is the best combination with spring integration? I heard that rabbit mq is the best one.
please give me a elaborate explanation. thanks.
First of all you description isn't clear...
If you don't want to lose messages from the QueueChannel use some Persistence MessageStore, like JdbcChannelMessageStore:
http://docs.spring.io/spring-integration/docs/latest-ga/reference/html/system-management-chapter.html#message-store
From other side there are channel wrappers for the AMQP as well as for JMS:
http://docs.spring.io/spring-integration/docs/latest-ga/reference/html/amqp.html#d4e5846
http://docs.spring.io/spring-integration/docs/latest-ga/reference/html/jms.html#jms-channel
Which really provide the same persistence durability, fault tollerant options for your use-case.
Re. activemq VS rabbitmq. I can say by my own expiriance that the last one is better, by configuration, usage from Spring Integration (Spring AMQP is under the shell). And its performance is really better.
All other info you can find in the Internet.
I am currently working on an application with Spring Integration. The application requires guaranteed delivery and the option that the it will be functional for a specific amount of time that the external systems are unavailable without losing messages. Channels will be JMS backed with expiration time. I would like to understand which is the best practice for redelivery with Spring Integration. We are having the following options:
The application's integration flow has a number of outbound message gateways that requires RPC calls with external system. Statefull retry advice can be used. After the max attemps is reached for specific runtime exceptions the message will be addressed to a recovery channel. The recovery channel will use a delayer and will then address the message back to the original channel. After X times that the message will reach the recovery channel it will be addressed to the error channel where it will be simply logged without further processing. The delayer component in this case should use the jdbc message store option.
Another option would be to use the standard JMS option for redelivery. In this case the redelivery policy will not be implemented on Spring Integration but on the JMS provider side.
Which is the best practice for message redelivery with Spring Integration?
I'd say like this: don't reinvent the wheel!
If there is already some similar solution on the matter, just use it as is with its specific configuration.
Right, if JMS has that solution, just go ahead.
There is need, of course, to get deal with DLQ in case of message expiration or redelivery exhausting. But the concept is here.
I have a app server that runs multiple camel routes that are reading messages from a JMS Queue, each route runs with different selectors.
We have an app server for each group of clients. So at the end there are multiple connections created to the queue. And that seems to be affecting the read and write performance on the queue.
I tried to use a connection pool, (by the way i am using WMQ), the closest I got is to use spring's CachingConnectionFactory (as described in this post how to configure (spring) JMS connection Pool for WMQ).
I was hoping that the number of connections will reduce from each app server, but that did not happen. Even though i set the size to 5 on the CCFactory, i see 10 connections created to the queue. According to the documentation on CCFactory, explicit closing of the connections is required. Is it possible or does it even make sense to close the connection that is used by a camel route that is reading from a queue? Or is there a way to tell camel routes to share the connections? I thought the use of connection pool and factory will do that, but i don't see that from happening.
Can I achieve what I want to achieve?
Thanks in advance.
what you can do is have only one route reading from the queue instead of many, and then use some camel conditional logic to redirect the message to the correct processing route via direct endpoints, see also this example.
This way there would only one consumer for the queue, instead of multiple consumers each running it's own selector.