I got rocketmq clusters named A and B, they are in network net-A and net-B separately. How could I sync their topics and messages.
You should define them in the same broker groups, say master is in net-A and slave is in net-B.
Thanks for your attention to RocketMQ.
Sorry for your inconvenience. We are still working on the replicator of RocketMQ. The improvement would be released soon in the next few versions.
At the moment, we recommend you make them a master-slave mode and define them in the same broker group. This enables the slave node to sync its topics and messages with the master node.
Related
I have two network envionments (such as NETWORK -A and NETWORK -B). Now, I deployed rocketmq-a in the NETWORK-A and deployed rocketmq-b in the NETWORK-B, how to communicate rocketmq-a and rocketmq-b directly?
According to your comment, you have two rocketmq clusters and one message should replicate to another rocketmq cluster.
So this is a message replication.
you have two choices:
Implement a send message hook
Use a messageStore plugIn that extend AbstractPluginMessageStore and load it using broker configuration
Both of them needs to implement replication by yourself.
However, if you make them the same broker group, it is very easy.
Just make rocketmq-b as a slave of rocketmq-a, and deploy them in different machine room.
Then rocketmq-b will only provide read operations and always replicate the data from master
Does RocketMQ support master-slave auto switch?
I have try it in v3.5.8, but it does not work. So, i just want ask someone who can gave accuracy answer
No, Rocketmq does not support this for sending messages.
Howerver, slave can take the position for pulling messages when master is out of service. And auto switch to slave for reading has been implemented.
So problems only exist in the scenario of sending messsages, if you need to have high-availability on writing, please put more master with diffrent group name in service.
you can use Rocketmq Version 5.0.0
I want to run multiple Kafka brokers on Mesos as Marathon jobs. I'm thinking what to use as a broker ID. I know that some people are using IP based broker ID, but I read that this approach is not optimal for situations when migrating the brokers to different machines. Any insights?
Related to that, is there a way to do a rolling deploy of Kafka brokers in a way that Marathon waits for the old broker instances to replicate the data to the new ones before killing them.
I created a Docker image which can start on Marathon and also scale up and down. To get a unique broker id, I concatenate the last octet of the IP address and the port number which Marathon devises to the Docker container.
I understood that this shouldn't be a problem, because normally the clients connect via ZooKeeper, which itself has the nodes and their connection info, so you shouldn't need to take care of the details. Or I misunderstood the question...
Have a look at
http://kafka.apache.org/documentation.html#replication
I have some messages flowing through the MQ cluster by using cluster and alias queues. Some queues are defined multiple times, though the loadbalancing mechanism is used.
What is the propper way to extract one QM from the cluster without disturbing the whole message flow? Disabling the cluster-receiver channel, cluster-sender channels, or else?
Use the
suspend qmgr
command.
This suspends the queue manager from the cluster.
command reference
Hi I create three RabbitMQ servers running in cluster on EC2
I want to scale out RabbitMQ cluster base on CPU utilization but when I publish message only one server utilizes CPU and other RabbitMQ-server not utilize CPU
so how can i distribute the load across the RabbitMQ cluster
RabbitMQ clusters are designed to improve scalability, but the system is not completely automatic.
When you declare a queue on a node in a cluster, the queue is only created on that one node. So, if you have one queue, regardless to which node you publish, the message will end up on the node where the queue resides.
To properly use RabbitMQ clusters, you need to make sure you do the following things:
have multiple queues distributed across the nodes, such that work is distributed somewhat evenly,
connect your clients to different nodes (otherwise, you might end up funneling all messages through one node), and
if you can, try to have publishers/consumers connect to the node which holds the queue they're using (in order to minimize message transfers within the cluster).
Alternatively, have a look at High Availability Queues. They're like normal queues, but the queue contents are mirrored across several nodes. So, in your case, you would publish to one node, RabbitMQ will mirror the publishes to the other node, and consumers will be able to connect to either node without worrying about bogging down the cluster with internal transfers.
That is not really true. Check out the documentation on that subject.
Messages published to the queue are replicated to all mirrors. Consumers are connected to the master regardless of which node they connect to, with mirrors dropping messages that have been acknowledged at the master. Queue mirroring therefore enhances availability, but does not distribute load across nodes (all participating nodes each do all the work).