We have a BatchJob application which is configured in Websphere Application Server (8.0.0.7). The application processes the requests put in the source MQ queue. Through WAS, the MQ queue is polled to see if there are any new requests available for processing.
We have been recently notified by a MQ resource that, there is high CPU resource utilization due to the MQ channel used by our application. When looked at the numbers, the MQGETS and MQINQS requests are humongous. This is not a 1 off incidence. It has always been like that since the day our application was installed. So I believe there is some configuration at Websphere that is causing this high volume of MQGETS and MQINQS requests.
Can somebody give any pointers which configs need to be checked? I am from application development side, so don't have in-detailed knowledge about WAS.
Thanks in advance.
Related
I have a Spring Boot based messaging app sending/receiving JMS messages to/from IBM MQ queue manager.
Basically, it uses MQConnectionFactory to organize connection to IBM MQ and a JmsPoolConnectionFactory from messaginghub:pooledjms to enable JMS connection pool, which is removed from MQConnectionFactory in IBM MQ 7.x
The app uses two different appoach to work with JMS. A "correct" one runs a JMSListener to receive messages and then sends a response on each message using JmsTemplate.send(). And there is a second "troubling" approach, where the app sends requests using JmsTemplate.send() and waits for response using JmsTemplate.readByCorrelId() until received or timed out.
I say troubling because this makes JMS sessions last longer if the response is delayed and could easily exhaust IBM MQ connection limit. Unfortunately, I cannot rewrite the app at the moment to the first approach to resolve the issue.
Now I want to restrict the number of connections in the pool. Of course, the delayed requests will fail but IBM MQ connection limit is more important at the moment, so this is kind of appropriate. The problem is that even if I disable the JmsPoolConnectionFactory, it seems that MQConnectionFactory still opens multiple connections to the query manager.
While profiling the app I see multiple threads RvcThread: com.ibm.mq.jmmqi.remote.impl.RemoteTCPConnection#12433875[...] created by JMSCCMasterThreadPool and corresponding connections to the query manager in MQ Explorer. I wonder why there are many of them in spite of the connection pooling is removed from MQConnectionFactory? I suppose it should open and reuse a single connection then but it is not true in my test.
Disabling "troubling" JmsTemplate.readByCorrelId() and leaving only "correct" way in the app removes these multiple connections (and the waiting threads of course).
Replacing JmsPoolConnectionFactory with SingleConnectionFactory has not effect on the issue.
Is there any way to limit those connections? Is it possible to control max threads in the JMSCCMasterThreadPool as a workaround?
Because it affects other applications your MQ admins probably want you to not exhaust the overall Queue Manager's connection limit (MaxChannels and MaxActiveChannels parameters in qm.ini). They can help you by defining an MQ channel exclusively used by your application. By this, they can limit the number of connections of your application with the MAXINST / MAXINSTC channel parameter. You will get an exception when this number is exhausted which is appropriate as you say. Other applications won’t be affected anymore.
What is the difference between a Websphere Message Broker and a Queue Manager. I guess the queue manager puts messages in the queue, takes messages out of the queue, moves messages to backout queues etc. So what is the job of the broker?
Does it sit between the publisher and the Queue Manager or between the consumer and the Queue Manager?
Websphere MQ is a software which uses the AMQ(Asynchronous messaging protocol). You can achieve asynchronous messaging between your applications via Websphere MQ, which will make your infrastructure loosely coupled(Applications can keep working even though other applications are down in the infrastructure).
But the applications in your infrastructure may not be able to understand each others' message formats, and hence just sending the message to the target application may not be enough. You may require transformation of the message.
You can do it by writing your own program using the Websphere MQ API.
Your program should be able to do the below things:
Pick message from a specific queue (using MQGET)
Should be able to understand the message. That is say it's an XML message. Then your program must be able to parse the XML and read the
data in it.
After reading the input message you will make your output message based on the requirements.
Then you will either publish the message or put the message in some specific queue(say TargetQ), so that the target application can get
the message. Target application will then get the message either by
issuing MQGET on the TargetQ or subscribing to the topic which was
published from your application.
But writing your own program will take a lot of development time and effort and also may be a bit complex.
So, IBM provided its own software to do the job, which is "Websphere Message Broker".
WMB allows you to create programs very easily and a lot faster.
Appropriate nodes in WMB will do all above steps for you. In fact it provides lot many features than the above steps.
Websphere MQ still doesn't have an HTTP listener. But, a message broker does. It allows you to host web services and have HTTP based flows etc that too in a secure way(Supports SSL).
MQ is providing you the infrastructure for messaging: queues and topics - IBM MQ
IBM Integration Bus (formerly known as WebSphere Message Broker) allows you to apply the common EAI patterns, e.g. Routing, Transformation
Hope that helps.
Best,
Patrick
I want to add just two points: Message Broker (now IIB) includes a set of optimized and fast parsers (XML, CSV, etc) and useful mapping nodes (msg-msg, msg-db). MQ is also used for internal configuration messages coming from the Configuration Manager.
WebSphere MQ is a solution for application-to-application communication services regardless of where your applications or data reside. Whether on a single server, separate servers of the same type, or separate servers of different architecture types, WebSphere MQ facilitates communications between applications by sending and receiving message data via messaging queues. Applications then use the information in these messages to interact with Web browsers, business logic, and databases. WebSphere MQ provides a secure and reliable transport layer for moving data unchanged in the form of messages between applications but it is not aware of the content of the messages. WebSphere MQ uses a set of small and standard application programming interfaces (APIs) that support a number of programming languages, including Visual Basic, NATURAL, COBOL, Java, and C across all platforms.
WebSphere Message Broker is built to extend WebSphere MQ, and it is capable of understanding the content of each message that it moves through the Broker. Customers can define the set of operations on each message depending on its content. The message processing nodes supplied with WebSphere Message Broker are capable of processing messages from various sources, such as Java Message Service (JMS) providers, HyperText Transfer Protocol (HTTP) calls, or data read from files. By connecting these nodes with each other, customers can define linked operations on a message as it flows from one application to its destination.
Message Broker can do the following:
Matches and routes communications between services
Converts between different transport protocols
Transforms message formats between requestor and service
Identifies and distributes business events from disparate sources
Together, WebSphere MQ and WebSphere Message Broker deliver a comprehensive publish and subscribe facility, connecting Message Broker’s broad transport and format support to WebSphere MQ’s messaging backbone. WebSphere Message Broker extends the WebSphere MQ publish and subscribe functionality with advanced function such as content-based publish and subscribe by means of an enhanced Publication node. The two products share a common publish and subscribe domain for topic- and content-based operations
MQ is mainly for the transforming the messages from on system to another system.
WMB(websphere message broker) will sit between QMGR's and transforming message along with change content of the message format as per the system requirement/Business Logic implementation.
Srinu D
Currently using WebLogic and Distributed Queues. And I know from the documentation that Distributed Queues allow you to retrieve a connection to any of the Queues across a cluster by using the Global JNDI name. It seems one of the main pieces of functionality Distributed Queue gives you is load balanced connections across multiple managed servers. So we have 4 Managed Servers (two on each physical, that communicate over multicast), and each Managed Server has an individual JMS Server which is configured to it's own Data Store.
I am 99% certain I already know the answer to this, but it appears that if you wanted to do a Consume a message off of a Queue, and that Queue exists on each Mgd Server in the Cluster, you cannot technically pull a Message off of any of the Queues (you can only pull the Message off the Queue to which you are connected to). So if I have a Message on Mgd Server 4, and I connect to Mgd Server 1, I won't see the messages on the Queue from Mgd Server 4.
So is there a way in Java EE or WLS to consume a message from all the nodes of a Queue (across the Cluster). Like a view into every instance of the Queue on each Mgd Server? It doesn't appear so and the documentation makes it seem like this is not possible, as well as this video (around minute 5):
http://www.youtube.com/watch?v=HAKixK_wp0Q
No you cannot consumer a message that is delivered to one managed server when your client is connected to another managed server of the same cluster.
Here's how it works.
When using UDT, wls provides a JNDI name that resolves internally into 4 distinct JNDI names for each of the managed server, the JMS servers on each of the managed servers are distinct.
When using the UDQ JNDI name when you post a message, it gets to one of the 4 managed servers using the algorithm you chose and other configuration done in your connection factory.
When a message consumer listens to the UDQ it gets pinned to the JMS server on one of the managed servers. It has no visibility about messages in the other servers.
Usually UDQ is used in scenarios where you want the message to be consumed concurrently by more than one managed server. You would normally deploy a MDB to the cluster, meaning the MDB will be deployed to each of the managed server and each of these will be able to consume the messages from their local JMS server.
I believe you can if your message store is config'd to use a database. If so, then I would think removing an item from the queue would remove it from the shared db table. I.e. all JMS servers are pointing to the same db instance and table. That should be pretty easy to test, too.
I have set up uniform distributed queue with weblogic server 12c. I am trying to achieve order of delivery and high availability with jms distributed queue. In my prototpe testing deployment I have two managed servers in the cluster, let us say managed_server1 and managed_server2. Each of this managed server hosts jms server namely jms server1 and jms server2. I have configured the jms servers with jdbc persistent store. I have enabled server affinity.
I have a producer running such as java queuproducer t3::/managed_server1. I send out 4 messages. From the weblogic monitoring console I see there are 4 messages in the queu since there are no consumers to the queue yet.
Now I shut down managed_server1.
Bring up a consumer to listen on java queuconsumer t3://managed_server2. This consumer cannot consume message since the producer send all the messages to jms server1, and it is down.
Bring up managed server 1, start a consumer to listen to t3://managed_server1 I can get all messages.
Here is my problem say if the managed_server1 went down then there it never came back up, do i loose all my messages. Also if there is another producer sending messages to java queuproducer t3://managed_server2 then order of messages based on the time between these producers are not guanranteed.
I am a little lost, am I missing something. Can unit of order help me to overcome this. Or should I use distributed topic instead of distributed queue, where all the jms server will receive all the messages from producers, but if one jms server where my consumre is listening fails there is only one consumer in my application, when I switch over to other jms server, I might be starting to get messages from the beginning not from where I left off.
Any suggestions regarding the same will be helpful.
Good Question !
" Here is my problem say if the managed_server1 went down then there it never came back up, do i loose all my messages. "
Ans - no you do not loose all your messages, they are stored in the JDBC store configured for the JMS server deployed on managed server 1. If you want the Messages sent to managed_server1 to be consumed from managed_server2 you need to configure JMS migration.
" Also if there is another producer sending messages to java queuproducer t3://managed_server2 then order of messages based on the time between these producers are not guanranteed. Can unit of order help me to overcome this."
Ans - If you want the messages to be consumed strictly in a certain order, then you will have to make use of unit of order (UOO). when messages are sent using UOO, they are sent to one of the several UDQ destinations, if midway that destination fails, and migration is enabled the messages are migrated to the next UDQ destination and new UDQ messages are also delivered to the new destination.
Useful links -
http://www.youtube.com/watch?v=B9J7q5NbXag
http://www.youtube.com/watch?v=_W3EJ8p35lI
Hope this helps.
How can you create a durable architecture environment using MQ Client and server if the clients don't allow you to persist messages nor do they allow for assured delivery?
Just trying to figure out how you can build a salable / durable architecture if the clients don't appear to contain any of the necessary components required to persist data.
Thanks,
S
Middleware messaging was born of the need to persist data locally to mitigate the effects of failures of the remote node or of the network. The idea at the time was that the queue manager was installed locally on the box where the application lives and was treated as part of the transport stack. For instance you might install TCP and WMQ as a transport and some apps would use TCP while others used WMQ.
In the intervening 20 years, the original problems that led to the creation of MQSeries (Now WebSphere MQ) have largely been solved. The networks have improved by several nines of availability and high availability hardware and software clustering have provided options to keep the different components available 24x7.
So the practices in widespread use today to address your question follow two basic approaches. Either make the components highly available so that the client can always find a messaging server, or put a QMgr where the application lives in order to provide local queueing.
The default operation of MQ is that when a message is sent (MQPUT or in JMS terms producer.send), the application does not get a response back on the MQPUT call until the message has reached a queue on a queue manager. i.e. MQPUT is a synchronous call, and if you get a completion code of OK, that means that the queue manager to which the client application is connected has received the message successfully. It may not yet have reached its ultimate destination, but it has reached the protection of an MQ Server, and therefore you can rely on MQ to look after the message and forward it on to where it needs to get to.
Whether client connected, or locally bound to the queue manager, applications sending messages are responsible for their data until an MQPUT call returns successfully. Similarly, receiving applications are responsible for their data once they get it from a successful MQGET (or JMS consumer.receive) call.
There are multiple levels of message protection are available.
If you are using non-persistent messages and asynchronous PUTs, then you are effectively saying it doesn't matter too much whether the messages reach their destination (although they generally will).
If you want MQ to really look after your messages, use synchronous PUTs as described above, persistent messages, and perform your PUTs and GETs within transactions (aka syncpoint) so you have full application control over the commit points.
If you have very unreliable networks such that you expect to regularly fail to get the messages to a server, and expect to need regular retries such that you need client-side message protection, one option you could investigate is MQ Telemetry (e.g. in WebSphere MQ V7.1) which is designed for low bandwidth and/or unreliable network communications, as a route into the wider MQ.