I have two network envionments (such as NETWORK -A and NETWORK -B). Now, I deployed rocketmq-a in the NETWORK-A and deployed rocketmq-b in the NETWORK-B, how to communicate rocketmq-a and rocketmq-b directly?
According to your comment, you have two rocketmq clusters and one message should replicate to another rocketmq cluster.
So this is a message replication.
you have two choices:
Implement a send message hook
Use a messageStore plugIn that extend AbstractPluginMessageStore and load it using broker configuration
Both of them needs to implement replication by yourself.
However, if you make them the same broker group, it is very easy.
Just make rocketmq-b as a slave of rocketmq-a, and deploy them in different machine room.
Then rocketmq-b will only provide read operations and always replicate the data from master
Related
I got rocketmq clusters named A and B, they are in network net-A and net-B separately. How could I sync their topics and messages.
You should define them in the same broker groups, say master is in net-A and slave is in net-B.
Thanks for your attention to RocketMQ.
Sorry for your inconvenience. We are still working on the replicator of RocketMQ. The improvement would be released soon in the next few versions.
At the moment, we recommend you make them a master-slave mode and define them in the same broker group. This enables the slave node to sync its topics and messages with the master node.
I assume that there are hundreds and thousands IoT devices that publish the data to the (broker)MQTT cluster via the MQTT protocol, behind the broker i have the data processing module which subscribe the data from the broker and maintain a status table for all these devices. The number of the devices is still rising, therefor I have to scale out the broker cluster and data processing module accordingly, for the MQTT broker such as Kafka/Rabbit MQ/Hive MQ can be scaled out very easily, but for the data processing module I'm not quite sure whether there is any best practice, or any framework/architecture can achieve this very easily:
I assume I have to create many daemon processes with hundreds and thousands threads to listen on the MQTT broker, the question is how to scale out these services dynamically?
Thanks.
One way of doing this would be using Node.js as it uses an event-driven approach and you don't have to deal with threads, etc.
I found this library for Node.js which is specific to MQTT:
https://www.npmjs.com/package/mqtt
You can use this to subscribe to different topics.
You may also find this project interesting:
http://nodered.org/
The other solution can be using Apache Kafka which has scalability as an important feature. However, the problem here is that Kafka does not support MQTT out of the box and has its own conventions. Therefore, there is a need for some sort of adapter to make them work together. For that, take a look at this:
using mqtt protocol with kafka as a message broker
I am new to spring batch. I want to run spring batch jobs using remote chunking & partitioning technique on multiple servers without using jms.
I want to use HTTP Invoker or RMI rather than using jms.
But, All examples of remote chunking & partitioning use jms.
I can't find examples that use HTTP Invoker or RMI.
I wonder if it is possible..
English is not my mother language.. please excuse any errors on my part
You can use any form of communication you want for remote partitioning. However, remote chunking does require persistent communication which is why JMS is typically used.
The reason you see JMS for remote partitioning is because it's easier to configure a clustered environment with JMS than it is for HTTP. The reason for that is everyone (master and all the slaves) only need to know where the queue is to talk to. Using HTTP as a communication mechanism requires the master and slaves to know a lot more. The master needs to know how to evenly distribute the partitions over all the slaves and where to send the requests to for each slave. All the slaves also need to know where the master is. JMS's centralized distribution model also allows you to dynamically add new slaves during processing where HTTP would require you to have some way to register a new slave with the master.
The reason persistent communication is required for remote chunking is that there is nothing in the remote partition model to prevent an item from being processed twice since it's sent over the wire (remote partitioning just sends descriptions of the data across and the job repository prevents data from being processed twice).
You can read more about the difference between the two in my answer here: Difference between spring batch remote chunking and remote partitioning
I am considering the feasibility that if we can replace our message-queue-middleware with ØMQ.
I have two set of servers.
The first set of the servers, they don't talk to another server from the same set, they only append the requests into specific message-queue.
The 2nd set of the servers, they don't talk to another server from the same set, they only receive the requests from specific message-queue to handle the requests.
It looks like a producer-consumer model.
And I think it can be replaced by the ØMQ's freelance pattern http://zguide.zeromq.org/page:all#Brokerless-Reliability-Freelance-Pattern.
But the questions are:
How to support dynamic discovery for both server & clients?
How to support dynamic discovery for both server & clients?
There are probably a hundred ways you could implement that, and greatly depend on your situation. If all the servers will always be on the same LAN you could bootstrap using the broadcast address on the local network and ask all responders who they are. Quick and dirty.
I would personally implement a bootstrap service that everyone knows about. They all can ask this always-available service for who is 'online' for the type of server they're after.
Another option, you could also use pub-sub. This would require a central publisher. newly connecting nodes would notify the publisher who would notify all other nodes of the new join, possibly including the new nodes ID, ip:port (if desired) etc. All nodes will still be able to communicate if the publisher crashes since its only used for global notifications, and a backup publisher could be used to make the system failsafe. Each node can also send heartbeats to publisher, with publisher notifying all other nodes when a node leaves/crashes.
Issue:
Having multiple consumer applications active specification attached to a single MQ on distributed VM servers is causing a null payload in a MQ Message.
Note: See notes at the bottom. No issue with mq.
Details:
I have 3 Websphere applications deployed across 2 VM servers. 1 application is a publisher and the other 2 applications are consumers attached to a single MQ Manager, and MQ.
2 consumer applications are pulling off the messages and processing them. The consumer application on the separate server receives a null payload. I have confirmed that its seems to be an issue having multiple application server instances attached to MQ. Confirmed by deploying the publisher on server 2 with consumer 2, then consumer 1 fails.
Question:
Has anyone tried attaching multiple MDB applications deployed on separate server instances bind to one Queue Manager and one MQ?
Specifications:
Websphere 7, EJB 3.0 MDB's,Transactions turned off,Queue in a queue installed on another machine.
Goal:
Distributed computing, scaling up against large number of messages.
I'm thinking this is a configuration issue but not 100% sure where to look. I had read that you could use MQLink, but I don't see why I would need to use service bus integration.
Supporting Doucmentation:
[MQ Link][1
UPDATE: I fixed the problem and it was a related to a combination of class loader issue with a duplicate classes. See Solution Notes below I added.
EDIT HISTORY:
- Clarified specifications, clarified question and added overall goal.
- reference notes to solution.
Has anyone tried attaching multiple MDB applications deployed on
separate server instances bind to one Local MQ?
Multiple MDB applications deployed on separate servers, connecting to one Queue Manager(but different queue) is a normal scenario and we have it everywhere and all applications work fine.
But, I suspect what you are doing is: Multiple MDB applications deployed on separate servers, connecting to one Queue Manager and listening to the same queue.
In this case one message will be received by one consumer only.
You need to create separate queue for each application and create subscriptions for each for the topic being published by your publisher.
Addition:
I suspect, for load balancing the problem you may be facing is that, when your first application gets the message, it doesn't issue a commit. So, there will be an uncommited message in the queue, which may be stopping your other application from getting message from the queue. When your first application finishes its processing, it issues a commit, but then again it is ready to pick the message and hence it again issues a get.
In my architecture, we have implemented load balancing using multiple queue managers like below:
You create 3 queue managers, say GatewayQM, App1QM and App2QM.
Keep the three queue managers in the same cluster.
Create an alias queue(shared in cluster) in GatewayQM and ask your putting app to put message in the gateway queue.
Now create one local cluster queue in each of App1QM and App2QM. Read from these queues via your applications App1 and App2 respectively.
This implementation provides you better security and serves a perfect load balancer.
This specific problem was caused by a code issue and the combination of class loading being set to "Parent First" in the Websphere console. On one node it would work and the other nodes in a cluster would fail, I think this was caused by the "Parent First" setting.
More important, in terms of my configuration in a cluster binding multiple active specifications to a single MQ to provide distributed computing is a correct solution.
However "points" due go to "nitgeek" solution references above if you are looking for a extremely high volume solution. Its important to understand that a single MQ can have a very high depth and takes a lot to fully utilize one. My current configuration is a good starting point for quick configuration and distributed processing using Multiple MDB's.