I'd like set up a battery of tests for testing new installations of QMs. Most of them will be in a "remote" setup. I'd like to be able to send a message to the new QM and receive a response. Is there any way to make a "loop" just with normal configuration (i.e. without using exits) ?
My former employer was a bank and we had many connections to clearing
houses and other vendors. On many of these I was able to have the business partner set up loopback queues on their end. These consisted of a QRemote on my QMgr that pointed to a QRemote on the other side, which pointed to a QLocal on the originating QMgr on my side. When a message is PUT to the QRemote it flowed to the other side, landed on the QRemote there, then was directed back to the QLocal.
If the message didn't make it back, we would then start our inbound channel (which was defined as a RQSTR on our side). If this caused the channel to start and the message arrived then we knew the channel initiator on the other side was down but at least we could keep the channel running until it was fixed. But if this didn't work we knew there was some problem on their side other than channel triggering.
And, yes, this was production. We figured that non-transactional instrumentation messages are valid in Production environments. As far as we were concerned, this was the same as a ping or cluster control message. Of course, we had a web screen and other instrumentation to kick them off so usage of the utility was controlled and were recorded.
There are similar methods in which QMgr aliasing is used and this can greatly cut down on the number of loopback queues needed in a large network. Do NOT be tempted to do this! When we use QRemotes that specify a fully-qualified destination, all they can do is send and receive loopback messages on the prescribed queue. But if a QRemote is used without an RQNAME then it can be used to send messages to any queue on the adjacent QMgr.
Let's say you have QMA and QMB:
runmqsc QMA
DEFINE QLOCAL(QMB.LOOPBACK) REPLACE
DEFINE QREMOTE(QMB.LOOPBACK.RMT) +
XMITQ(QMB) +
RQMNAME(QMB) +
RNAME(QMB.QMA.LOOPBACK.RMT) +
REPLACE
END
runmqsc QMB
DEFINE QREMOTE(QMB.QMA.LOOPBACK.RMT) +
XMITQ(QMA) +
RQMNAME(QMA) +
RNAME(QMB.LOOPBACK) +
REPLACE
Now you can go to the server hosting QMA and do:
echo Hello World | amqsput QMB.LOOPBACK.RMT QMA
Note that the name of the loopback reflector on QMB has both QMgr names in it. This is because you probably want to set up lopback from both sides and cannot use QMB.LOOPBACK.RMT both as a reflector and to initiate loopbacks. In that case the object inventory looks more like this:
runmqsc QMA
DEFINE QLOCAL(QMB.LOOPBACK) REPLACE
DEFINE QREMOTE(QMB.LOOPBACK.RMT) +
XMITQ(QMB) +
RQMNAME(QMB) +
RNAME(QMB.QMA.LOOPBACK.RMT) +
REPLACE
DEFINE QREMOTE(QMA.QMB.LOOPBACK.RMT) +
XMITQ(QMB) +
RQMNAME(QMB) +
RNAME(QMA.LOOPBACK) +
REPLACE
END
runmqsc QMB
DEFINE QREMOTE(QMB.QMA.LOOPBACK.RMT) +
XMITQ(QMA) +
RQMNAME(QMA) +
RNAME(QMB.LOOPBACK) +
REPLACE
DEFINE QLOCAL(QMA.LOOPBACK) REPLACE
DEFINE QREMOTE(QMA.LOOPBACK.RMT) +
XMITQ(QMA) +
RQMNAME(QMA) +
RNAME(QMA.QMB.LOOPBACK.RMT) +
REPLACE
END
Note that all of the objects sort out based on the name of the remote QMgr. Some people prefer names like LOOPBACK.QMB.RMT to make all the loopback queues cluster together in a list of objects or a backup.
All of this is a great target for automation since the names of the objects can all be derived from the names of the QMgrs.
I can think of a couple of ways to do this. It rather depends on whether you want to include the MQPUT and MQGET on the remote system target queue.
Define the target queue as a QREMOTE pointing back to the originating system again with a real QLOCAL as the final destination back on the original system.
Use something like SupportPac MA01 (Q program) to echo the message back when it lands on a real QLOCAL queue on the remote system. The -e option let's you echo the message back to the ReplyToQ and ReplyToQMgr fields.
Related
I have two clusters CLUSMES and CLUSHUBS. Each cluster has two queue managers.
Cluster CLUSMES has QMGRS: QMGR1A and QMGR1B
Cluster CLUSHUBS has QMGRS: QMGR3A and QMGR3B
There is a Gateway QMGR: QMGR2, which forms the overlap and is a partial repository in each MQ cluster.
Request messages are sent out from either QMGR1A/B to either QMGR3A/B via QMGR2 which acts as a cluster Load balance to QMGR3A/B (This works fine) and a reply is expected back to the sending QMGR.
All channel connectivity is in place and fully functional. The problem is how to return the message from where it came from. The replying QMGR connects to QMGR3A/B and issues a put. I will either get a REMOTE_QMGR not found (MQRC 2087) or a MQ Object not found (MQRC 2085) depending on how I have it configured. The message header of the message contains the ReplytoQueue and ReplyToQMgr properly. I would like to have the replying application just issue a put and have it delivered to the proper queue in CLUSMES, but this is proving to be extremely difficult. I have played with Remote QMGR Alias and QAlias on the GateWay Qmgr: QMGR2, but no luck. There is got to be a simple trick to this and there are plenty of examples, but I have not been able to implement one successfully. A clear cut example of what my return path should be would be most helpful. Keep in mind, that the ReplyToQMgris in the MQMD and resolution needs to occur from that. I need resolution to occur at the QMGR2 level, where both clusters are known. Concrete full suggestions appreciated.
MQ Definitions on the QMGR1A/B, where the REPLY is expected:
DEFINE QLOCAL('SERVER.REPLYQ') CLUSTER('CLUSMES')
On QMGR2 ( The Gateway for message hoping)
DEFINE NAMELIST(CLUSTEST) NAMES(CLUSMES,CLUSHUBS)
DEFINE QALIAS(SERVER.ALIAS.REPLYQ) TARGQ(SERVER.REPLYQ) CLUSTER(CLUSTEST) DEFBIND(NOTFIXED)
DEFINE QREMOTE(QMGR1A) RNAME(' ') RQMNAME(QMGR1A) XMITQ('') CLUSTER(CLUSHUBS)
DEFINE QREMOTE(QMGR1B) RNAME(' ') RQMNAME(QMGR1B) XMITQ('') CLUSTER(CLUSHUBS)
On MQMGR3A/B QALIAS(SERVER.ALIAS.REPLYQ) cluster queue. The Gateway QMGR could not resolve the baseQ: mqrc_unknown_alias_base_q 2082
This was the configuration when trying to resolve it using the cluster.
When the request message is sent by the application it would specify ReplyToQMgr of either QMGR1A and QMGR1B and ReplytoQueue with the name of the queue that is present on QMGR1A and QMGR1B, the reply queue need not be clustered.
On gateway queue manager QMGR2 you would define the following objects:
DEFINE QREMOTE(QMGR1A) RNAME('') RQMNAME(QMGR1A) XMITQ('') CLUSTER(CLUSHUBS)
DEFINE QREMOTE(QMGR1B) RNAME('') RQMNAME(QMGR1B) XMITQ('') CLUSTER(CLUSHUBS)
This would allows any queue manager in the cluster CLUSHUBS to route reply messages back to QMGR1A and QMGR1B via gateway queue manager QMGR2.
If you want to limit queues on QMGR1A and QMGR1B that queue managers in the CLUSHUBS cluster can put to you would need to take a different approach. Let me know if that is what you need and I will update my answer with some suggestions.
In TIBCO EMS, which I am familiar with, there is a feature called "destination bridges".
Queues and Topic can be bridged (linked) so that the 2nd destination can become client of the the first. (Queue to queue, Topic to queue, Queue to topic, topic to topic)
For instance, a topic can be bridged to a queue, which in essence will become a durable subscriber of the messages submitted to the topic. Clients can subscribe to the topic OR read from the queue. This example is a way to load balance the reading of a pub/sub for multiple clients (readers of the queue).
This "bridge" feature can also involve message selectors and destination name wilcards.
So, a QUEUE X can be the client of TOPIC.* with condition CUST_ID(a JMS attribute)>30.
In that case, all message submitted to TOPIC.A OR TOPIC.B fitting the criteria would end up in QUEUE X. All this does not involve anything but simple EMS configuration.
I don't know enough about Websphere MQ, and I need a similar behavior. Will I have to develop a processing program outside of MQ, or can the feature within the product sufice ?
Note : I have already gone through MQ documentation and found about the "Alias queues" feature. Since the feature should really be called "Shortcut queue" and does not involve 2 destinations... I don't think it could help me...
Thanks!
Edit : For reference, the command (DEF SUB) enabling this in MQ is documented here
Edit 2 : The selected answer cover the "Topic -> Queue" pattern from TIBCO EMS "Destination bridge" featuire. Please note that the "Q->Q", T->T and Q->T" patterns are not covered here.
Easy! Define your queue to receive the subscription and then define a durable administrative subscription.
DEF QL(MY.SUSCRIBER.QUEUE)
DEF SUB('MY.SUBSCRIPTION') +
TOPICSTR('SOME/TOPIC/#') +
DEST('MY.SUSCRIBER.QUEUE') +
SELECTOR('JMSType = 'car' AND color = 'blue' AND weight > 2500') +
REPLACE
The Infocenter has a section on Selector Syntax and a page for the DEFINE SUB command.
Fairly new to zeromq and trying to get a basic pub/sub to work. When I run the following (sub starting before pub) the publisher finishes but the subscriber hangs having not received all the messages - why ?
I think the socket is being closed but the messages have been sent ? Is there a way of ensuring all messages are received ?
Publisher:
import zmq
import random
import time
import tnetstring
context=zmq.Context()
socket=context.socket(zmq.PUB)
socket.bind("tcp://*:5556")
y=0
for x in xrange(5000):
st = random.randrange(1,10)
data = []
data.append(random.randrange(1,100000))
data.append(int(time.time()))
data.append(random.uniform(1.0,10.0))
s = tnetstring.dumps(data)
print 'Sending ...%d %s' % (st,s)
socket.send("%d %s" % (st,s))
print "Messages sent: %d" % x
y+=1
print '*** SERVER FINISHED. # MESSAGES SENT = ' + str(y)
Subscriber :-
import sys
import zmq
import tnetstring
# Socket to talk to server
context = zmq.Context()
socket = context.socket(zmq.SUB)
socket.connect("tcp://localhost:5556")
filter = "" # get all messages
socket.setsockopt(zmq.SUBSCRIBE, filter)
x=0
while True:
topic,data = socket.recv().split()
print "Topic: %s, Data = %s. Total # Messages = %d" % (topic,data,x)
x+=1
In ZeroMQ, clients and servers always try to reconnect; they won't go down if the other side disconnects (because in many cases you'd want them to resume talking if the other side comes up again). So in your test code, the client will just wait until the server starts sending messages again, unless you stop recv()ing messages at some point.
In your specific instance, you may want to investigate using the socket.close() and context.term(). It will block until all the messages have been sent. You also have the problem of a slow joiner. You can add a sleep after the bind, but before you start publishing. This works in a test case, but you will want to really understand what is the solution vs a band-aid.
You need to think of the PUB/SUB pattern like a radio. The sender and receiver are both asynchronous. The Publisher will continue to send even if no one is listening. The subscriber will only receive data if it is listening. If the network goes down in the middle, the data will be lost.
You need to understand this in order to design your messages. For example, if you design your messages to be "idempotent", it doesn't matter if you lose data. An example of this would be a status type message. It doesn't matter if you have any of the previous statuses. The latest one is correct and message loss doesn't matter. The benefits to this approach is that you end up with a more robust and performant system. The downsides are when you can't design your messages this way.
Your example includes a type of message that requires no loss. Another type of message would be transactional. For example, if you just sent the deltas of what changed in your system, you would not be able to lose the messages. Database replication is often managed this way which is why db replication is often so fragile. To try to provide guarantees, you need to do a couple things. One thing is to add a persistent cache. Each message sent needs to be logged in the persistent cache. Each message needs to be assigned a unique id (preferably a sequence) so that the clients can determine if they are missing a message. A second socket (ROUTER/REQ) needs to be added for the client to request the missing messages individually. Alternatively, you could just use the secondary socket to request resending over the PUB/SUB. The clients would then all receive the messages again (which works for the multicast version). The clients would ignore the messages they had already seen. NOTE: this follows the MAJORDOMO pattern found in the ZeroMQ guide.
An alternative approach is to create your own broker using the ROUTER/DEALER sockets. When the ROUTER socket saw each DEALER connect, it would store its ID. When the ROUTER needed to send data, it would iterate over all client IDs and publish the message. Each message should contain a sequence so that the client can know what missing messages to request. NOTE: this is a sort of reimplementation of Kafka from linkedin.
Using WebSphere MQ I want to setup up a topic that uses queues, so that when an application or inbound cluster connection attempts to put a message to a “queue”, it
actually uses a topic, and publishes it to 2 subscriptions, which are
themselves 2 separate queues. In essence, I want to go from an inbound
queue by name, but map it to 2 separate queues, like
AF_TO_DAAS is the inbound topic/which today is an actual
cluster queue alias
=>Goes to AF_TO_APP1 and
=>Goes to AF_TO_APP2
Sort of like if these were queues on a distribution list I suppose.
Those two things are local queues.
I’m getting lost in the /topic/node business mapping it to subscriptions
and model queues and what not …
WebSphere MQ allows an alias to point to a queue or a topic. It also provides a means to create a durable subscription administratively. To accomplish this setup, you connect the dots by replacing the existing alias with one that points to a topic. Then use two administrative subscriptions to route publications to the two (or more) queues.
* First, define the topic
DEFINE TOPIC('AF_TO_DAAS.TOPIC') +
TOPICSTR('AF_TO_DAAS') +
REPLACE
* Now, create an alias over the topic.
* Sending apps think this is a queue.
DEFINE QALIAS('AF_TO_DAAS') +
TARGET('AF_TO_DAAS.TOPIC') +
TARGTYPE(TOPIC) +
REPLACE
* Queues for the two recvr apps
DEFINE QLOCAL('AF_TO_APP1') +
REPLACE
DEFINE QLOCAL('AF_TO_APP2') +
REPLACE
* Now set up adminsitrative subs to route
* messages to the two app queues.
DEFINE SUB('AF_TO_DAAS.SUB') +
TOPICSTR('') +
TOPICOBJ('AF_TO_DAAS.TOPIC') +
DEST('AF_TO_APP1') +
PSPROP(NONE) +
REPLACE
DEFINE SUB('AF_TO_APP2.SUB') +
TOPICSTR('') +
TOPICOBJ('AF_TO_DAAS.TOPIC') +
DEST('AF_TO_APP2') +
PSPROP(NONE) +
REPLACE
Because this is a publication, the messages will contain a property Top which contains the topic string. The PSPROP(NONE) in the subscription entries causes this to be suppressed so the messages look like the original publication.
Note also that the MQMD.MsgID on the publications is different than it is on the original message.
I have setup MQ7.0 On three system: A, B and C. A -> B there is sender-receiver channel (A.B) and B->C, there is sender-receiver channel(B.C). Transmission queue names are same.Name of queue managers as QMA,QMB and QMC respectively.
Now I have a queue QC on system C.
I have defined a remote queue definition on system A as:
define qremote(RQ) rname(QC) rqmname(third) xmitq(A.B)
I have a define remote queue mgr definition on system B as:
define qremote(third) rname('') rqmname(QMC) xmitq(B.C)
now what I expected when I put a message to remote queue RQ on A, it should travel to system B with transmission header "QC on third" through channel on tx queue(A.B)..on system B since third is aliased as QMC so this message should be put to tx B.C and should reach Queue QC on system C.
But after I put message, it is lost. Is my understanding wrong about remote queue mgr alias. Can someone give better example.
Try this:
On QMA
DEF QR(QMC) RQMNAME(' ') RNAME(' ') XMITQ(QMB)
DEF QR(QC) RQMNAME('QMC') RNAME(' ') XMITQ('QMB')
DEF QL(QMB) USAGE(XMITQ) TRIGTYPE(FIRST) TRIGGER
* Channel def goes here
The QRemote is a QMgr alias. It specifies the path to QMC goes through the QMB XMitQ. Once this is defined, anything destined for QMC will resolve to the QMB XMitQ. The blank RNAME and RQMNAME is what makes this a QMgr alias and not a classic remote queue definition. At this point if you use a tool like the Q program which allows you to specify the QMgr to connect to separately from the QMgr used for the OPEN command, you don't even need a QRemote. ANY mesage destined for QMC will now resolve to QMB.
However, if you wanted to use something less sophisticated like amqsput then you really do need a QRemote. The DEF QR(QC accomplishes this. If you are willing to use QRemotes then you don't really need the QMgr alias. This just says "If someone opens a queue named QC then address it to QC on QMC and put it on the QMB XMitQ.
On QMB
DEF QL(QMC) USAGE(XMITQ) TRIGTYPE(FIRST) TRIGGER
* Channel def goes here
On QMB when the mssages arrive addressed to QMC, they will automatically resolve to the XMitQ without any further definitions or guidance.
On QMC
DEF QL(QC)
Finally, the messages arrive on QMC addressed to QC so you need a QLocal for them to land in.
Name resolution is a bit complicated but once you understand it, you pretty much understand WMQ addressing. I'd recommend sitting down with the Name Resolution Infocenter topic and reconciling back to the behavior you are seeing.
By the way, the discrepancy in your definitions appears to be that your QRemote used the name of the channel instead of the XMitQ. At one point you say "Transmission queue names are same.Name of queue managers as QMA,QMB and QMC respectively." So I expect on B to find an XMitQ named QMC serving the QMB.QMC channel. But later you define the QMgr alias as define qremote(third) rname('') rqmname(QMC) xmitq(B.C) which should be XMITQ(QMC).