I am dropping messages on an MDB in a loop and I can see in my logs that I keep running out of available connections.
Caused by: com.sun.messaging.jms.JMSException: MQRA:CFA:allocation failure:createConnection:Error in allocating a connection. Cause: In-use connections equal max-pool-size and expired max-wait-time. Cannot allocate more connections.
at com.sun.messaging.jms.ra.ConnectionFactoryAdapter._allocateConnection(ConnectionFactoryAdapter.java:209)
at com.sun.messaging.jms.ra.ConnectionFactoryAdapter.createConnection(ConnectionFactoryAdapter.java:162)
at com.sun.messaging.jms.ra.ConnectionFactoryAdapter.createConnection(ConnectionFactoryAdapter.java:144)
After dropping each message on the queue I am closing all the connections but still I do not know how I run out of available connections.
I am thinking of increasing the pool size instead. But havent been able to find that setting.
Can anyone guide me how to change that setting for the Glassfish server.
For an MDB you can set this via the MaxPoolSize of the activation spec. This property is the "Maximum size of server session pool internally created by the resource adapter for achieving concurrent message delivery. This should be equal to the maximum pool size of MDB objects".
Related
I have a Spring Boot based messaging app sending/receiving JMS messages to/from IBM MQ queue manager.
Basically, it uses MQConnectionFactory to organize connection to IBM MQ and a JmsPoolConnectionFactory from messaginghub:pooledjms to enable JMS connection pool, which is removed from MQConnectionFactory in IBM MQ 7.x
The app uses two different appoach to work with JMS. A "correct" one runs a JMSListener to receive messages and then sends a response on each message using JmsTemplate.send(). And there is a second "troubling" approach, where the app sends requests using JmsTemplate.send() and waits for response using JmsTemplate.readByCorrelId() until received or timed out.
I say troubling because this makes JMS sessions last longer if the response is delayed and could easily exhaust IBM MQ connection limit. Unfortunately, I cannot rewrite the app at the moment to the first approach to resolve the issue.
Now I want to restrict the number of connections in the pool. Of course, the delayed requests will fail but IBM MQ connection limit is more important at the moment, so this is kind of appropriate. The problem is that even if I disable the JmsPoolConnectionFactory, it seems that MQConnectionFactory still opens multiple connections to the query manager.
While profiling the app I see multiple threads RvcThread: com.ibm.mq.jmmqi.remote.impl.RemoteTCPConnection#12433875[...] created by JMSCCMasterThreadPool and corresponding connections to the query manager in MQ Explorer. I wonder why there are many of them in spite of the connection pooling is removed from MQConnectionFactory? I suppose it should open and reuse a single connection then but it is not true in my test.
Disabling "troubling" JmsTemplate.readByCorrelId() and leaving only "correct" way in the app removes these multiple connections (and the waiting threads of course).
Replacing JmsPoolConnectionFactory with SingleConnectionFactory has not effect on the issue.
Is there any way to limit those connections? Is it possible to control max threads in the JMSCCMasterThreadPool as a workaround?
Because it affects other applications your MQ admins probably want you to not exhaust the overall Queue Manager's connection limit (MaxChannels and MaxActiveChannels parameters in qm.ini). They can help you by defining an MQ channel exclusively used by your application. By this, they can limit the number of connections of your application with the MAXINST / MAXINSTC channel parameter. You will get an exception when this number is exhausted which is appropriate as you say. Other applications won’t be affected anymore.
We are getting this error in every alternate day.I know that increasing the channel max depth will resolve the issue. Is there any option we could close the connection after the messages have been pushed to the queues? Can we do that in that way?
Assuming the connections that datapower is using are MQI connections (ie client connecting to a SVRCONN channel) you can try setting a disconnect interval on the SVRCONN channel that is being used by datapower. This forces connections that haven't been used for the specified interval to close.
The MQSC command to alter the interval would be this:
ALTER CHANNEL(channel_name) CHLTYPE(SVRCONN) DISCINT(number_of_seconds)
Further information about DISCINT is available at the IBM Knowledge Center:
https://www.ibm.com/support/knowledgecenter/SSFKSJ_9.1.0/com.ibm.mq.ref.con.doc/q081860_.htm
I am looking for detailed description on how IBM MQ Client or listener get the messages from MQ Server when new messages are placed on MQ Queue or Topic.
How the connection between MQ Client and MQ Server are created?
Does MQ Client initiates the connection with Server or does server initiates the connection to its consumer?
In case we have connection pool defined on MQ Client, how client knows that it has to create more connections with Server as the messages are increasing on Server? How Client know about the messages on Server?
Is there a communication from Server to Client which tells client that new messages have arrived?
I am looking for these details not the details on how this is setup or how to setup MQ Channels or Listeners. I am looking for how it works behind the scene.
If someone can point me to the right direction or documentation, it would be great.
It's hard to speak definitively about how the IBM WebSphereMQ client & server work since they're closed source, but based on my experience with other messaging implementations I can provide a general explanation.
A JMS connection is initiated by a client to a server. A JMS client uses a javax.jms.ConnectionFactory to create a javax.jms.Connection which is the connection between the client and server.
Typically when a client uses a pool the pool is either filled "eagerly" (which means a certain number of connections are created when the pool is initialized to fill it to a certain level) or "lazily" (which means the pool is filled with connections one-by-one as clients request them from the pool). If a client requests a connection from the pool and if all the connections in the pool are being used and the maximum size of connections allowed by the pool hasn't been reached then another connection will be created. If the pool has reached its maximum allowed size (i.e no more connections can be created) then the client requesting a connection will have to wait for another client to return their connection to the pool at which point the pool will then give it to the waiting client.
A JMS client can find out about messages on the server in a few different ways.
If the JMS client wants to ask the server occasionally about the messages it has on a particular queue it can create a javax.jms.Consumer and use the receive() method. This method can wait forever for a message to arrive on the queue or it can take a timeout parameter so that if a message doesn't arrive in the specified timeout the call to receive() will return.
If the JMS client wants to receive a message from a particular queue as soon as the message arrives on the queue then it can create a javax.jms.MessageListener implementation and register it on the queue. When such a listener is registered on a queue then when a message arrives on the queue the server will send the message to the listener. This is sometimes referred to as a "callback" since the server is "calling back" to the client.
The first thing you should do is take a course on JMS/IBM MQ or go to the new IBM conference called: Integration Technical Conference
Ok, now to your questions:
How the connection between MQ Client and MQ Server are created?
You simply issue the createQueueConnection method of QueueConnectionFactory class and specify the credentials.
QueueConnection conn = cf.createQueueConnection("myUserId", "myPwd");
Does MQ Client initiates the connection with Server or does server initiates the connection to its consumer?
MQ client application starts the connection - always.
In case we have connection pool defined on MQ Client, how client knows that it has to create more connections with Server as the messages are increasing on Server? How Client know about the messages on Server?
It is up to the team's architect or lead developer to understand message flow and message patterns. Hence, they will know what to set the pool count at. Also, lots and lots of testing too. Some client applications will only require a pool count of 10 whereas other applications may need a pool count of 50 because it is a heavy flow.
Is there a communication from Server to Client which tells client that new messages have arrived?
You use the createReceiver method of the QueueSession class to retrieve a message. Set a timeout value for the createReceiver method rather than continuously polling the queue manager.
Again, some training on the use of JMS/IBM MQ is strongly recommended.
Hi Our EMS server is used by other clients for putting message. But some time they dont close connections and number of connections is reaching maximum limit of the server. Is there any way where we can restrict the number of connections for the client based on emsusername provided to the client or based on the host name from where client is creating connection. Is there any configuration we can do for client specific connections restriction.
No, there is no such provision in EMS server or client libraries where you can restrict the number of consumer/producer clients based on their user names or other properties. You can have a look at the JAAS and JACI provision supported by EMS which can be used to write your own JAVA authentication custom modules which run in JVM within EMS server. You can find more information about JAAS and JACI on Oracles documentation site.
Have you looked into the server_timeout_client_connection setting ?
From the doc :
server_timeout_client_connection = limit
In a server-to-client connection, if the server does not receive a heartbeat for a
period exceeding this limit (in seconds), it closes the connection.
We recommend setting this value to approximately 3 times the heartbeat interval, as it is specified in client_heartbeat_server.
Zero is a special value, which disables heartbeat detection in the server (although
clients still send heartbeats).
We are running a performance test on a WebSphere 8.5.5.1 application server that calls many external SOAP services. Running netstat we notice that the server is only creating a max of 50 outbound connections.
We are trying to increase this value but cannot find the correct property. We have increased the Default pool but this doesn't seem to apply.
The WebContainer pool size is also set to higher than 50 and we can see this pool grow. Is there some hidden pool that is defaulting to 50?
You'll need to configure com.ibm.websphere.webservices.http.maxConnection property based on this:
http://www-01.ibm.com/support/knowledgecenter/api/content/SSEQTP_8.5.5/com.ibm.websphere.base.iseries.doc/ae/rwbs_httptransportprop.html