We are using activemq for micro services message broker. My query is about how to build failover mechanism for consumer service. If the consumer service is down , how to proceed further. Can we do somethign like hystrix or any other failover/circuit breaker mechanism for such scenarios.
Short answer is 'yes'. The details on how to implement depend on your use case. Recommended architecture:
Publish to virtual topic
Option A: Multi-threaded consumers
2. Multiple competing consumers reading from the queue that is used as the subscriber from the virtual topic
Option B: Single-threaded HA consumers
2. Multiple competing consumers reading from the queue that is used as the subscriber from the virtual topic using an exclusive consumer to ensure only one thread is processing data and the other consumers are hot-standby
Related
We have configured our ActiveMQ message broker as a Spring Boot project and there's another Spring Boot application (let's call it service-A) that has a listener configured to listen to some topics using #JmsListener annotation. It's a Spring Cloud microservice appilcation.
The problem:
It is possible that service-A can have multiple instances running.
If we have 2 instances running, then any message coming on topic gets listened to twice.
How can we avoid every instance listening to the topic?
We want to make sure that the topic is listened to only once no matte the number of service-A instances.
Is it possible to run the microservice in a cluster mode or something similar? I also checked out ActiveMQ virtual destinations but not too sure if that's the solution to the problem.
We have also thought of an approach where we can decide who's the leader node from the multiple instances, but that's the last resort and we are looking for a cleaner approach.
Any useful pointers, references are welcome.
What you really want is a shared topic subscription which was added in JMS 2. Unfortunately ActiveMQ 5.x doesn't support JMS 2. However, ActiveMQ Artemis does.
ActiveMQ Artemis is the next generation broker from ActiveMQ. It supports most of the same features as ActiveMQ 5.x (including full support for OpenWire clients) as well as many other features that 5.x doesn't support (e.g. JMS 2, shared-nothing high-availability using replication, last-value queues, ring queues, metrics plugins for integration with tools like Prometheus, duplicate message detection, etc.). Furthermore, ActiveMQ Artemis is built on a high-performance, non-blocking core which means scalability is much better as well.
We are using microservice architecture in our project. We deploy each service to a cluster by using Kubernetes. Services are developed by using Java programming language and Spring Boot framework.Three replicas exist for each service. Services communicate with each other using only events. RabbitMQ is used as a message queue. One of the services is used to send an email. The details of an email are provided by another service with an event. When a SendingEmail event is published by a service, three replicas of email service consume the event and the same email is sent three times.
How can I prevent that sending emails by other two services?
I think it depends on how you work with Rabbit MQ.
You can configure the rabbit mq with one queue for these events and make spring boot applications that represent the sending servers to be "Competing" Consumers.
If you configure it like this, only one replica will get an event at a time and only if it fails to process it the message will return to the queue and will become available to other consumers.
From what you've described all of them are getting the message, so it works like a pub-sub (which is also a possible way of work with rabbit mq, its just not good in this case).
I am running multiple instances of the same Spring Boot 2.0.4 Application, for scaling purposes, that consume messages from an ActiveMQ queue using the following:
#JmsListener(destination = "myQ")
Only the first consumer receives messages and if I stop the first consumer the second instance starts receiving the messages. I want each consumer to consume a message, not the same message, in a round robin fashion. But only the first consumer consumes messages.
It sounds like you want a JMS Topic rather than a Queue. You should also research durable subscriptions, shared subscriptions, and durable topics before you settle on the configuration you need for your setup.
See:
JMS API Programming Model (Search for JMS Message Consumers)
Queues vs Topics
Durable Queues and Topics
Is there a way to implement a topology with ActiveMQ, where P is a publisher, s_a is a subscriber of service A and s_b1 and s_b2 are subscribers of service B. The latter are set in a cluster for load balancing (so s_b1 or s_b2 gets a message but not both).
Is there a way to combine publish-subscribe with a peer-to-peer messaging so one of the subscribers would be a queue which two consumers are listening on?
Thanks,
Gil
Yes this can be done.
You may want to look at Apache ActiveMQ Artemis which implements JMS 2.0 and support what you ask for out of the box. JMS 2.0 allows multiple subscribers per topic to load balance a cluster.
For ActiveMQ and JMS 1.0 you can use Virtual Destinations instead. They work with naming conventions.
If you name your topic: VirtualTopic.StockPrice and publish messages to it, you will be able to consume from queues named Consumer.Consumer1.VirtualTopic.StockPrice, Consumer.Consumer2.VirtualTopic.StockPrice etc. Consumer1 can be anything.
You can reconfigure ActiveMQ to use other names for Virtual Destinations (prefix, suffix etc).
How to configure cluster of Consumers in ActiveMQ?
I created a simple embedded ActiveMQ application with two consumers of one Queue, consumers are working in separate threads. But when I send a message to the Queue, JMS delivers it to first consumer no matter how long it sleeps after receiving.
I think you're trying to explain that the first consumer is receiving all the messages. There is a FAQ entry for this type of problem available here:
http://activemq.apache.org/i-do-not-receive-messages-in-my-second-consumer.html
Bruce