How can we browse all the messages in a WebSphere MQ queue in one API call using java?
Here is the code which I'm using. Here I'm using this code a for loop until q depth is reached.
MQGetMessageOptions gmo=new MQGetMessageOptions();
gmo.options = MQC.MQGMO_WAIT | MQC.MQGMO_BROWSE_NEXT ;
//System.out.println("Status: "+i);
MQMessage out=new MQMessage();
out.format =MQC.MQFMT_XMIT_Q_HEADER;//MQC.MQFMT_REF_MSG_HEADER;
mqCon.getQue().get(out,gmo);
System.out.print(i);
How can I get all messages without using for loop? It's taking a long time to browse 10,000 messages.
How can I get all messages without using for loop?
Use a while loop. Sorry, could not resist a slightly snarky answer on that one. WMQ does not have an API call analogous to the SQL select statement. Messaging and databases share some traits but address fundamentally different requirements.
It's taking a long time to browse 10,000 messages.
Take a look at the Performance SupportPacs. These are published on the SupportPacs main page and have names beginning with MP. Find the one for your platform and MQ version and it will list different scenarios for putting and getting messages as well as performance tuning recommendations.
I would also ask why a normal app needs to browse 10,000 messages. The QMgr will select messages for you based on MsgID, Correlation ID or property and this is much faster than browsing all the messages in order for the application to find the ones of interest. Occasionally people need to browse all messages on a queue to archive the queue or to debug a problem, but this is the exception rather than the rule. If a Production app regularly browses all messages on a queue, then the queues may have been inappropriately used as a database.
How can I get all messages without using for loop?
MQGetMessageOptions getOptions = new MQGetMessageOptions();
getOptions.options = MQC.MQGMO_BROWSE_NEXT + MQC.MQGMO_NO_WAIT + MQC.MQGMO_FAIL_IF_QUIESCING;
MQMessage message = new MQMessage();
byte[] b = null;
while(true)
{
try
{
queue.get(message, getOptions);
b = new byte[message.getMessageLength()];
message.readFully(b);
System.out.println(new String(b));
message.clearMessage();
}
catch (IOException e)
{
System.out.println("IOException: " + e.getMessage());
break;
}
catch (MQException e)
{
if (e.completionCode == 2 && e.reasonCode == MQException.MQRC_NO_MSG_AVAILABLE)
System.out.println("All messages read.");
else
System.out.println("MQException: Completion Code = " + e.completionCode + " : Reason Code = " + e.reasonCode);
break;
}
}
I have posted many Java/MQ samples here:
http://www.capitalware.biz/mq_code_java.html
If you want just browse messeges (not retrive them), you can use javax.jms.QueueBrowser.
Is pretty fast...
import javax.jms.*
...
public ArrayList<Message> browse() {
...
QueueBrowser queueBrowser = queueSession.createBrowser((javax.jms.Queue) lookupQueue());
Enumeration enums = queueBrowser.getEnumeration();
while (enums.hasMoreElements()) {
Object objMsg = enums.nextElement();
if (objMsg instanceof TextMessage) {
TextMessage message = (TextMessage) objMsg;
Log4j.trace("Text message: " + i + ". MSG:" + message.getText() + " MSG id:"
+ message.getJMSMessageID() + " MSG dest:" + message.getJMSDestination());
} else if (objMsg instanceof ObjectMessage) {
ObjectMessage message = (ObjectMessage) objMsg;
Log4j.trace("Object Message: " + i + ". MSG" + " MSG id:" + message.getJMSMessageID()
+ " MSG dest:" + message.getJMSDestination());
}
}
}
...
Just noticed the message format (out.Format) you are using. MQFMT_XMIT_Q_HEADER is used for messages that are sent to a transmit queue. Messages in transmit queue are not generally read by applications. MQ uses a transmit queue to send messages from one queue manager to another queue manager in a MQ network. I hope you are not browsing messages in a transmission queue.
For applications, message format would typically depend on the receiving application. For example if the receiving application is CICS based, then format would be MQFMT_CICS, for IMS it would be MQFMT_IMS. If text/string type of data is expected then you could use MQFMT_STRING. For administering MQ using PCF messages, then format can be MQFMT_PCF.
Related
I have a project to transfer file using IBM MQ. There are 10000 clients and one data center. The largest file size is almost 8MB. The MQ cluster contains three MQ managers which are at different Windows server. Each MQ manager have 5 channels for client and 5 channel for data center. There are two cases for testing. Clients are evenly distributed to MQ manager in each case. Do not lose any file is the most important thing in these cases.
Case 1:
Every client send 50 files to data center at the same time. The files size are between 150KB to 5MB.
In this case, the sum of file size one client send is almost 80MB.
Case 2 :
Data center send the 10 identical files to every client at the same time. In this case, I create a topic named `myTopic` and 10000 clients subscribe this topic. Data center send 10 identical files to the topic.
MQ Mangers have a heavy load. I already set some attribute in IBM MQ:
Queue Manager:
Max handles: 100000
Maximum message length: 100MB
Max channels: 10000
Max channels: 10000
Is there any attribute that could increase the performance?
5/11 update:
First, I have modified the situation of case 2 above. I have a data center server that has a 4 core CPU and 32G RAM. I use 4 clients server to simulate 10000 clients, and each client server has 4 core CPU and 16G RAM.
In case 1, it take about 37 minutes when 1000 clients send files to the data center. There are not enough memory on data center server when data center receive files from 2000 clients. I find there are 20G memory used for buffer/cache. Here is my java code used to receive files:
try {
String filePath = ConfigReader.getInstance().getConfig("filePath");
MQMessage mqMsg = new MQMessage();
mqMsg.messageId = CMQC.MQMI_NONE;
mqMsg.correlationId = CMQC.MQCI_NONE;
mqMsg.groupId = CMQC.MQGI_NONE;
int flag = 1;
while (true) {
try {
MQQueueManager queueManager = new MQQueueManager("QMGR1");
int option = CMQC.MQTOPIC_OPEN_AS_SUBSCRIPTION | CMQC.MQSO_DURABLE;
MQTopic subscriber = queueManager.accessTopic("", "myTopic", option, null, "datacenter");
subscriber.get(mqMsg);
if (mqMsg.getDataLength() != 0) {
String fileName = filePath + "_file" + flag + ".txt";
byte[] b = new byte[mqMsg.getDataLength()];
mqMsg.readFully(b);
System.out.println("Receive " + fileName + ", complete time: " + System.currentTimeMillis());
Path path = Paths.get(fileName);
System.out.println("Write " + fileName + ", start time: " + System.currentTimeMillis());
Files.write(path, b);
System.out.println("Write " + fileName + ", complete time: " + System.currentTimeMillis());
flag++;
}
} catch (MQException e) {
// e.printStackTrace();
if (e.reasonCode != 2033) {
e.printStackTrace();
}
} finally {
mqMsg.clearMessage();
mqMsg.messageId = CMQC.MQMI_NONE;
mqMsg.correlationId = CMQC.MQCI_NONE;
mqMsg.groupId = CMQC.MQGI_NONE;
}
}
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
I use byte array to read message and write it to disk. Is it possible that the byte array does not release memory and takes 20G memory?
In case 2, I find if I send a 5MB file to myTopic that has 1000 subscribers on MQ manager01, MQ manager01 take a lot of time to sync with cluster member. The disks on the MQ servers are very busy. There are another problem: Sometimes I get only 7 seconds to send a 5MB file, sometimes it takes 90 seconds. Here is my java code to send files:
try {
MQQueueManager queueManager = new MQQueueManager("QMGR1");
MQTopic publisher = queueManager.accessTopic("myTopic", "", CMQC.MQTOPIC_OPEN_AS_PUBLICATION,
CMQC.MQOO_OUTPUT);
System.out.println("---- start publish , time: " + System.currentTimeMillis() + " ----");
publisher.put(InMemoryDataProvider.getInstance().getMessage("my5MBFile"));
System.out.println("---- end publish , time: " + System.currentTimeMillis() + " ----");
publish.getPublisher().close();
} catch (MQException e) {
System.out.println("threadNum: " + publish.getThreadNo() + " publish error");
if (e.reasonCode != 2033) {
e.printStackTrace();
}
}
A couple of things.
MQ has FTE which transfers files for you. I think it does it using non persistent messages, so you avoid the disk overhead.
You might try checking your .ini files for parameters like ClntRcvBuffSize=0
see here.
0 says use the operating system values.
TCP used to send some data in short packets (64KB chunk), then wait till the packets have been acknowledged, and send more. If the connection is reliable, then you get higher throughput by sending bigger logical packets, a technique known as Dynamic Right Sizing. See here
it works best when the connection is long lived and sending a lot if data. For example the first few chunks may be 64KB, then increase it a bit to 128KB chunks, eventually up to 100MB ( or more) if needed.
You need to set both ends.
Depending on platform, you can use Netstat replacement ss command to display the various window sizes.
For your QM to QM channels specify a large batchsz and batchlim - though this may make your disk IO worse as the data gets to the remote end faster.
I'm using NetMQ (Nuget 3.3.2.2) on .NET 4.5 and I have a single fast generator process with a PUSH socket, and a single slow consumer process using a PULL socket. If I send enough messages to hit the sending HWM, the sending process blocks the thread indefinitely.
Some contrived (generator) code which illustrates the problem:
using (var ctx = NetMQContext.Create())
using (var pushSocket = ctx.CreatePushSocket())
{
pushSocket.Connect("tcp://127.0.0.1:42404");
var template = GenerateMessageBody(i);
for (int i = 1; i <= 100000; i++)
{
pushSocket.SendMoreFrame("SampleMessage").SendFrame(Messages.SerializeToByteArray(template));
if (i % 1000 == 0)
Console.WriteLine("Sent " + i + " messages");
}
Console.WriteLine("All finished");
Console.ReadKey();
}
On my configuration, this will usually report it has sent about 5000 or 6000 messages, and will then simply block. If I set the send HWM set to a large value (or 0), then it sends all of the messages as expected.
It looks like it's waiting to receive another command before it tries again, here: (SocketBase.TrySend)
// Oops, we couldn't send the message. Wait for the next
// command, process it and try to send the message again.
// If timeout is reached in the meantime, return EAGAIN.
while (true)
{
ProcessCommands(timeoutMillis, false);
From what I've read in the 0MQ guide, blocking on a full PUSH sockeet is the correct behaviour (and is what I want it to do), however I would expect it to recover once the consumer has cleared its queue.
Short of using some sort of TrySend pattern and dealing with the block myself, is there some option I can set or some other facility I can use to have the PUSH socket attempt to resend blocked messages periodically?
I have a mail reader class which sets the FetchProfile and later does a msg.getContent.
I want to do both reading of header and content in one call, basically download the full mail in one call. Because I have observed msg.getcontent makes a call to the server to get the body/content , if we can download the full mail in one call, a call to the server can be saved.
Is this possible?
The code is similar to this
inbox.open(Folder.READ_ONLY);
/* Get the messages which is unread in the Inbox */
Message messages[] = inbox.search(new FlagTerm(
new Flags(Flag.SEEN), false));
/* Use a suitable FetchProfile */
FetchProfile fp = new FetchProfile();
fp.add(FetchProfile.Item.ENVELOPE);
fp.add(FetchProfile.Item.CONTENT_INFO);
inbox.fetch(messages, fp);
for (int i = 0; i < messages.length; i++) {
System.out.println("MESSAGE #" + (i + 1) + ":");
Message message = messages[i];
**String content = message.getContent();**
System.out.println("Content : " + content);
}
Appreciate any help.
Thanks and Regards
Raaghu.K
If you want the entire message in one call, and don't need to use any of the features of the IMAP protocol, you have two choices:
Use POP3 instead of IMAP.
Use the Message.writeTo method to write the message content to a file or byte array and process it from there, e.g., using the MimeMessage constructor that takes an InputStream. (This makes a local copy of the entire message.)
I've written a Continuous JMS Message reveiver :
Here, I'm using CLIENT_ACKNOWLEDGE because I don't want this thread to acknowledge the messages.
(...)
connection.start();
session = connection.createQueueSession(true, Session.CLIENT_ACKNOWLEDGE);
queue = session.createQueue(QueueId);
receiver = session.createReceiver(queue);
While (true) {
message = receiver.receive(1000);
if ( message != null ) {
// NB : I can only pass Strings to the other thread
sendMessageToOtherThread( message.getText() , message.getJMSMessageID() );
}
// TODO Implement criteria to exit the loop here
}
In another thread, I'll do something as follows (after successful processing) :
This is in a distinct JMS Connection executed simultaneously.
public void AcknowledgeMessage(String messageId) {
if (this.first) {
this.connection.start();
this.session = this.connection.createQueueSession( false, Session.AUTO_ACKNOWLEDGE );
this.queue = this.session.createQueue(this.QueueId);
}
QueueReceiver receiver = this.session.createReceiver(this.queue, "JMSMessageID='" + messageId + "'");
Message AckMessage = receiver.receive(2000);
receiver.close();
}
It appears that the message is not found (AckMessage is null after timeout) whereas it does exist in the Queue.
I suspect the message to be blocked by the continuous input thread.. indeed, when firing the AcknowledgeMessage() alone, it works fine.
Is there a cleaner way to retrieve 1 message ? based on its QueueId and messageId
Also, I feel like there could be a risk of memory leak in the continuous reader if it has to memorize the Messages or IDs during a long time.. justified ?
If I'm using a QueueBrowser to avoid impacting the Acknowledge Thread, it looks like I cannot have this continuous input feed.. right ?
More context : I'm using ActiveMQ and the 2 threads are 2 custom "Steps" of a Pentaho Kettle transformation.
NB : Code samples are simplified to focus on the issue.
Well, you can't read that message twice, since you have already read it in the first thread.
ActiveMQ will not delete the message as you have not acknowledge it, but it won't be visible until you drop the JMS connection (I'm not sure if there is a long timeout here as well in ActiveMQ).
So you will have to use the original message and do: message.acknowledge();.
Note, however, that sessions are not thread safe, so be careful if you do this in two different threads.
I am using IBM Websphere MQ. I have the queue manager and queue name. Now, I want to check whether the queue has any messages in it?
I did not work on this before. Pleas help
Please let me know if you need further information!
Thanks
The below code is .NET / amqmdnet - but you might try and convert this in the meantime until a Java dev sees your post.
To see if there is a message on the queue, without actually taking it off the queue, use MQC.MQOO_BROWSE on the Queue and IBM.WMQ.MQC.MQGMO_BROWSE_FIRST as the option
You'll get MQRC_NO_MSG_AVAILABLE if the queue is empty.
MQMessage queueMessage = new MQMessage();
MQQueueManager queueManager = new MQQueueManager(qmName, channelName, connName);
MQQueuequeue = queueManager.AccessQueue(qName,
MQC.MQOO_BROWSE + MQC.MQOO_FAIL_IF_QUIESCING);
MQGetMessageOptions opt = new MQGetMessageOptions();
opt.Options = IBM.WMQ.MQC.MQGMO_BROWSE_FIRST;
queueMessage.CorrelationId = IBM.WMQ.MQC.MQMI_NONE;
queueMessage.MessageId = IBM.WMQ.MQC.MQMI_NONE;
queue.Get(queueMessage, opt);
String sMessage = queueMessage.ReadString(queueMessage.DataLength);
To peek the next message use IBM.WMQ.MQC.MQGMO_BROWSE_NEXT;
To actually read the message OFF the queue, use MQC.MQOO_INPUT_SHARED on the AccessQueue.
The answer didn't show how to check for MQRC_NO_MSG_AVAILABLE. Here is my solution. If there are better ones please let me know.
try
{
queue.Get(queueMessage, opt);
String sMessage = queueMessage.ReadString(queueMessage.DataLength);
}
catch (MQException err)
{
if (err.ReasonCode.CompareTo(MQC.MQRC_NO_MSG_AVAILABLE) == 0)
return true;
}
For Windows machine
It depends on where your queue manager is.
You could use MQUtilities - ih03 pack - which has rfhUtil.exe (Local Qm) and rfhUtilC.exe (for remote qm)
For Local QM , it is straight forward you need to place appropriate values and hit browse, it will show you Queue Depth.
For Remote QM, Place /TCP/(PortNo) for queue manager name and queue for queue name. Hit browse and you will get to know the queue depth.
For Unix/Ubuntu/Linux versions - There is a product called MQVisualEdit which is similar to this one.