In RabbitMQ documentation, it is mentioned that:
All data/state required for the operation of a RabbitMQ broker is replicated across all nodes. An exception to this are message queues, which by default reside on one node, though they are visible and reachable from all nodes. To replicate queues across nodes in a cluster, use a queue type that supports replication. This topic is covered in the Quorum Queues guide.
If we are using springboot amqp classic queue and we need to start using a cluster of RabbitMQ where data is replicated across nodes for a lowest risk of data loss, what changes needs to be done to the code to start using a quorum queue?
When defining the queue, by default, the type is classic queue, to choose the quorum type instead, just add the type of queue as argument:
#Bean
public Queue eventsQueue() {
Map<String, Object> args = new HashMap<>();
args.put("x-queue-type", "quorum");
return new Queue(queueName, true, false, false, args);
}
In addition to the above, make sure you point your spring boot rabbit mq to the cluster not to one node. This can be done by changing spring.rabbitmq.host configuration in application.properties to spring.rabbitmq.addresses=[comma separated ip:port]
A Classic Queue has a master running somewhere on a node in the cluster, while the mirrors run on other nodes. This works the very same way for Quorum Queues, whereby the leader, by default, runs on the node the client application that created it was connected to, and followers are created on the rest of the nodes in the cluster.
In the past, replication of queues was specified by using policies in conjunction with Classic Queues. Quorum queues are created differently, but should be compatible with all client applications which allow you to provide arguments when declaring a queue. The x-queue-type argument needs to be provided with the value quorum when creating the queue.
Related
My application create with SpringBoot and is in cluster (two different istance openshit)
Every istance has one consumer that read message of topic in replication factory.
I would like to find a mechanism to block the reading of a message into topic in replication factory if it has already been read by one of the two consumers
Example:
CONSUMER CLIENT A -- READ MSG_1 --> BROKER_1
- Offset increase
- Commit OK
CONSUMER CLIENT B --> NOT READ MSG_1 --> BROKER_1
-- Correct beacause already commit
Now BROKER_1 is show and new lead is BROKER_2
How can I block the already read message into BROKER_2?
Thanks all!
Giuseppe.
Replication factor doesn't control if/how consumers read messages. The partition count does. If the topic only has one partition, then only one consumer instance is able to read messages, and all other instances are "blocked". And if the message is already read and commited then it doesn't matter which broker is the leader because the offsets are maintained per topic, not per replica
If you have more than one partition and you still want to block consumers from being able to read data, then you'll need to implement some external, coordinated lock via Zookeeper, for example
Just to make the scenario simple.
number of consumers == number of partitions == Kafka broker numbers
If deploy the consumers on the same machines where the brokers are, how to make each consumer only consume the messages locally? The purpose is to cut all the network overhead.
I think we can make it if each consumer can know the partition_id on their machines, but I don't know how? or is there other directions to solve this problem?
Thanks.
bin/kafka-topics.sh --zookeeper [zk address] --describe --topic [topic_name] tells you which broker hosts the leader for each partition. Then you can use manual partition assignment for each consumer to make sure it consumes from a local partition.
Probably not worth the effort because partition leadership can change and then you would have to rebalance all your consumers to be local again. You can save the same amount of network bandwidth with less effort by just reducing the replication factor from 3 to 2.
Maybe you could use the Admin Client API.
First you can use the describeTopics() methods for getting information about topics in the cluster. From the DescribeTopicResult you can access to TopicPartitionInfo with information about partitions for each topic. From there you can access to the Node through the leader(). Node contains the host() and you can check if it's the same as the host your consumer is running or id() and the consumer should have the information about the broker-id running on the same machine (in general it's an information you can define upfront). More infor on Admin Client API at the following JavaDoc :
https://kafka.apache.org/0110/javadoc/index.html?org/apache/kafka/clients/admin/AdminClient.html
We are using ActiveMq in our application. We need to implement priority consumer mechanism on the queue. We have multiple consumers and we want to treat them as master/slave. So, the master consumer would have higher priority and will consume all messages and once master gets down then the consumer (with higher priority) will consume all requests from the queue.
To Implement this scenario, we came across the mechanism of priority queues. So, we initialized the queues by this way:
javax.jms.Queue queue = queueSession.createQueue("myQueue" + "?consumer.priority=" + 127);
and remaining slaves nodes were assigned lesser priorities.
But this mechanism is not working expectedly, sometimes it works fine that we get all requests on the consumer with the highest priority but sometimes consumer with lesser priority also start consuming messages while the consumer with the highest priority is working fine.
We have tried it with consumer.exclusive=true option as well, with no luck.
Note: We are running all components (ActiveMQ, producer and consumer application) locally on the same machine for now, so no network delays involved. And we are running consumers with default prefetch policy.
Is there any other approach to implement this scenario using activeMq or are we missing any configuration.
Requirement:
1)I need to fetch data stored in Mongo DB through Java Application and using topic exchange & binding keys, created 3 queues on RabbitMQ. I have implemented everything up to this point.
The problem starts from the 2nd point onwards.
2) When the messages should be available to multiple consumers from all the 3 queues. But when first consumer consumes the messages from 3 queues it will not be available for the rest of the consumers. How to make messages highly available to multiple consumers.
Is there any ways to achieve this or is this requirement has any alternate solutions to it.
All your consumers must provide their own unique queue and bind it to the same exchange.
There is no such a Topic abstraction in AMQP, like it is with JMS.
Even if we can publish message through the topic or fanout exchange, the message will be placed to the queue as single entry, so only one consumer will be able to pick it up from the the.
The config for my proposition may look like:
<queue id="commandQueue" name="#{node.id}.command"
auto-delete="true"/>
<fanout-exchange name="commandsExchange">
<bindings>
<binding queue="commandQueue"/>
</bindings>
</fanout-exchange>
<amqp:inbound-channel-adapter id="commandConsumer"
queue-names="#{commandQueue.name}"
channel="commandChannel"/>
With that all my application instances bind their unique queue (based on the node.id abstraction) to the same commandsExchange. And the published message to the commandsExchange will be delivered to all my nodes.
auto-delete="true" helps me to avoid extra messages for my queue, if node is dead.
HTH
We knowthat, in kafka 0.7, we can specify zk.connect for producer, so producer can find the additions and removals of broker. But in kafka 0.8, we can't specify zk.connect for producer. Can producer in kafka 0.8 find that? If not, the scalability of the system is not worse than the 0.7 version?
You can still use a ZooKeeper client to retrieve the broker list:
ZkClient zkClient = new ZkClient("localhost:2108", 4000, 6000, new BytesPushThroughSerializer());
List<String> brokerList = zkClient.getChildren("/brokers/ips");
According to that, you do not have to "hardcode" the broker list on client side and you are flexible as far as the system architecture is concerned. But anyway, this would add the ZooKeeper dependency again which is in fact an disadvantage for producer in several environments.
If you want to get a detailed view to the so called "cluster metadata API" solution, check out this link: https://issues.apache.org/jira/browse/KAFKA-369
Best
pre
P.S.: Sorry for reposting this to your other question - but the answer fits on both ;-)
Little confused what exactly you are looking for, in 0.8 we must specify the list of brokers in the metadata.broker.list property
Properties props = new Properties();
props.put("metadata.broker.list", "broker1:9092,broker2:9092");
from the kafka consumer example they say
The property, “metadata.broker.list” defines where the Producer can find a one or more Brokers to determine the Leader for each topic. This does not need to be the full set of Brokers in your cluster but should include at least two in case the first Broker is not available. No need to worry about figuring out which Broker is the leader for the topic (and partition), the Producer knows how to connect to the Broker and ask for the meta data then connect to the correct Broker.
By saying additions do you mean adding new node to your cluster ?