How do I setup a message queue that automatically sends all it's messages to another server?
I'm working on a proof of concept for a system that needs to run on multiple servers, writing to local message queues, then have a central service on another server running that reads its local queue to pick up all the messages from the other servers.
From what I've read I believe this is possible, but I'm not seeing how to set it up...
Thanks
When your application send a message to a remote computer, the msmq service actually write the message to a local queue ( temporary outgoing queue). So practically the behaver of msmq is exactly what you want. Can you elaborate more about your scenario?
Update to comment: There is one problem. You can't create a remote queue.
Related
Can somebody help to understand the basic of these 3 queues with example.when do we use all 3
Perhaps a more simple explanation: Think of a local queue as a queue which exists on the queue manager it is defined.. you can PUT and GET messages off a local queue. A remote queue is like a pointer to a queue on another queue manager which is usually on different host. Therefore messages can be PUT to it (and they will usually arrive on a local queue at that remote host) but you cannot GET messages from a remote queue.
Simply put, a queue manager only ever hosts messages on local or transmition queues on that queue manager. If you want to go to another queue manager, you can use definitions which tell the queue manager that the 'put' is running on, how to route the message to a destination queue manager - however this ends up with a message on a transmit queue which is then picked up and sent down a channel towards that destination. Alias queues are just an opportunity to use a different name for another queue. Remote queues are definitions on one queuemanager with information about where the message should be routed.
MQ documentation:
https://www.ibm.com/support/knowledgecenter/SSFKSJ_8.0.0/com.ibm.mq.explorer.doc/e_queues.htm
Another description:
https://www.scribd.com/doc/44887119/Different-Types-Queues-in-Websphere-MQ
A queue is known to a program as local if it is owned by the queue manager to which the program is connected; the queue is known as remote if it is owned by a different queue manager. The important difference between these two types of queue is that you can get messages only from local queues. (You can put messages on both types of queue.)
References:
https://www.ibm.com/support/knowledgecenter/en/SSFKSJ_7.0.1/com.ibm.mq.csqzal.doc/fg10950_.htm
https://www.ibm.com/support/knowledgecenter/SSFKSJ_8.0.0/com.ibm.mq.explorer.doc/e_queues.htm
An application which is connected to a local QueueManager which host a queue, So that queue will be local queue for that application. An application which is connected to a QueueManager which is located remotely and that host a queue that queue will be remote queue. we should always keeps in mind that we always read the message from the local queue. A message put on the remote queue, will be routed to local queue through the object defined on the local queue thrugh the channel and the transmission queue.
I am new to mqtt and would like to get my head around something.
I need to get messages from (subscribe to) topics from a remote mosquitto broker. The documentation for the service says I need to run a mosquitto broker on my server.
If I understand correctly then a script that uses the mqtt gem and manages to connect using something like this:
MQTT::Client.connect(conn_opts) do |c|
# The block will be called when you messages arrive to the topic
c.get('test') do |topic, message|
puts "#{topic}: #{message}"
end
end
IS a broker? Do I need to run mosquitto on my machine or can I get away with just a script and mqtt?
The doc describes the architecture and includes these lines:
The 3rd party platform needs an MQTT broker installed that will allow
communication with the different boxes on our servers. The broker on our servers will
initiate the connection and provide the credentials to allow
bidirectional communication.
The architecture I have in mind is a scheduled background process, using ruby-mqtt, that will spawn, connect with the remote mosquitto server and pull down new messages in batches before finishing. Does this sound like a reasonable approach for getting messages from a remote mosquitto broker?
I have a sneaky suspicion there is something I am not getting... any help/direction will be appreciated. Thanks!
No, you do not need a local MQTT server, you can connect directly to the remote server from your ruby script.
It is typical to keep the MQTT client running all the time, and not just download periodically using cron. Although I imagine that could work, providing you are using QoS 1/2 and disable clean sessions, so that messages are retained on the remote server. Despite its name, MQTT is not a message queuing protocol, it is a publish/subscribe protocol, so it is possible at the remote server will not allow you to build up a large pool of messages.
It may however be desirable to have a local MQTT server (eg mosquitto):
* You local MQTT server could deal with the storing of messages to disk until ruby is ready for them
* It allows multiple local clients to receive the same message without the remote server having to send it over the network multiple times
* Multiple local clients can send messages to each other, even when the remote network is down
Also be warned that ruby-mqtt doesn't support QoS 1 properly yet and also doesn't support persisting of messages or automatic reconnects, so a local mosquitto instance could solve some of those problems for you.
I'm trying to figure out under what conditions I would want to implement a remote queue versus a local one for 2 endpoint applications.
Consider this scenario: App A on Server A needs to send messages to App B on Server B via MQServer1.
It seems like the simplest configuration would be to create a single local queue on MQServer1 and configure AppA to put messages to the local queue while configuring AppB to get messages from the same local queue. Both AppA and AppB would connect to the same Queue Manager but execute different commands.
What sort of circumstances would require the need to install another MQ server (e.g. MQServer2) and configure a remote queue on MQServer1 which instead sends the messages from AppA over a channel to a local queue on MQServer2 to be consumed by AppB?
I believe I understand the benefit of remote queuing but I'm not sure when it's best used over a more simpler design.
Here are some problems with what you call the simpler design that you don't have with remote queuing:-
Time Independance - Server1 has to be available all the time, whereas with a remote queue, once the messages have been moved to Server B, Server A and Server 1 don't need to be online when App B wants to get its messages.
Network Efficiency - with two client applications putting or getting from a central queue, you have two inefficient network hops, instead of one efficient channel batched network connection from Server A to Server B (no need for Server 1 in the middle)
Network Problems - No network, no messages. Whereas when they are stored locally, any that have already arrived can be processed even while the network is down. Likewise, the application putting messages is also not held up by a network problem, the messages sit on the transmit queue easy to be moved, and the application can get on with the next thing.
Of course your applications should be written so that they aren't even aware of the difference, and it's just configuration changes that switch you from one design to the other.
Here we can have separate Queue Manager for both the application.Application A will send the message on to the queue defined on local Queue Manager, which in turn transmit it to the Transmission queue via defined channels (Need to do configuration for that in the QueueManager) which in turn send it to the Local queue of the Application B.
This is very basic question about IBM WebSphere MQ V7.
Regarding the Transmission Queue, my understanding is it is only used with remote queue that resides in the same queue manager. Therefore, if I want to put message to the queue, I need to put it to remote queue.
It is like this.
App --> Remote queue --> Transmission Queue
My question is:
Is it possible to put the message directly into transmission queue like this?
App --> Transmission Queue
--Modified on 2014.03.17 --
I found a way to put message directly into transmission queue. I do not know this is a common use, but in order to do that I needed to prepend MQXQH to the message. I tried and confirmed it works. See the Infocenter reference here.
Do not ever put directly to a transmission queue. It is dangerous if you do not know what you are doing.
You should put your message to a remote queue. A remote queue is not the same as a local queue. A remote queue is simply a pointer to a queue on another queue manager.
Although it is possible to put messages directly on the XMitQ, there is considerable risk in allowing that to occur so most admins will prevent applications from directly accessing that queue. As you have found, it is possible to construct a message with the transmission queue header and behind that a normal message with the MQMD and payload. (This is, in fact, excatly how the MCA works.)
The problem here is that the QMgr does not check the values in the MQMD residing in the payload so you can put mqm as the MQMD.UserID and then address the message to the remote command queue and grant yourself admin access to that remote QMgr.
Security-conscious administrators typically use two security controls to prevent this. First, they disallow direct access to the XMitQ. That helps for outbound messages. More importantly, they set the MCAUSER of their RCVR/RQSTR/CLUSRCVR channels to a non-admin user ID that is not authorized to put messages onto any sensitive queues.
The other issue is, of course, that what you describe completely defeats WMQ's name resolution. By embedding routing into the app, you prevent the administrator from adjusting channel weights, cluster settings, failover and load distribution at the network level. Need to redistribute traffic? Redeploy the code. Not a good plan.
So for security reasons and because you paid a lot of money to get WMQ's reliability - much of which comes from dynamic addressing and name resolution features - coding apps to write directly to the XMitQ is strongly discouraged.
You should not directly be using the transmission queue. Its used by the message channel agent (MCA) as temporary storage when sending messages across to a remote queue manager.
This is distributed queuing - i.e you publish a message to Queue Manager A, and want it routed to a local queue on Queue Manager B. So you define a reference on QM-A referring to the local queue on QM-B. This reference is the 'remote queue definition'.
The remote queue definition specifies the transmission queue name. The transmission queue is bound to the MCA, which in turn knows about the remote QM.
We have 2 MQ Queue Managers, and one WAS server. We need to send message from QM01 to QM02 and then from QM02 to WAS server.
For doing this we have built a sender channel between QM01 and QM02, message is placed over a remote queue which has a definition of another remote queue for the WAS Service Integration Bus queue.
Is there any harm in sending message as following;
Remote Queue on QM01 ==> Remote Queue on QM02 ==> Queue on SI
Bus on WAS
Yes, this is a perfectly valid method of hopping messages from one queue manager to another, especially if the queue managers are not clustered. There is no harm to the message or the environment with this approach, though it does has a little more administrative overhead as opposed to a cluster-based solution.