Cannot obtain exclusive access to locked queue - spring

I have an anonymous and exclusive queue defined like this:
#Bean
public SimpleMessageListenerContainer responseMessageListenerContainer(){
SimpleMessageListenerContainer container = new SimpleMessageListenerContainer(simpleRoutingConnectionFactory());
container.setQueues(responseAnonymousQueue());
container.setMessageListener(rabbitTemplate());
container.setAcknowledgeMode(AcknowledgeMode.AUTO);
container.setMessageConverter(jsonMessageConverter());
return container;
}
#Bean
public Queue responseAnonymousQueue() {
return new MyAnonymousQueue();
}
Sometimes I get this error en rabbitmq log:
=ERROR REPORT==== 12-Apr-2016::15:13:42 === Channel error on connection <0.6899.0> (XX.XXX.57.174:51716 -> 192.168.100.145:5671,
vhost: '/', user: 'XXXX_USER'), channel 1:
{amqp_error,resource_locked,
"cannot obtain exclusive access to locked queue ' XXXX_USER-broad-1457bb43-6487-4252-b21a-a5a92d19e0dc' in vhost '/'",
'queue.declare'}
So the client can’t declare the queue and it can’t receive the messages from the AMQP server.
It happens after this message:
=WARNING REPORT==== 12-Apr-2016::15:11:51 === closing AMQP connection <0.6810.0> (XX.XXX.57.174:17959 -> 192.168.100.145:5671):
connection_closed_abruptly
=INFO REPORT==== 12-Apr-2016::15:13:41 === accepting AMQP connection <0.6899.0> (XX.XXX.57.174:51716 -> 192.168.100.145:5671)
I can’t reproduce it (I have tried closing the connection from rabbitmq and removing the network cable, but the application reconnect well again), so I don’t know exactly why is this happening.
It is supposed that private and exclusive queues are deleted with the closing of the connection, so why is this happening? How can I catch this exception and recover from it?
Thanks

You are correct, exclusive queues are deleted when the connection that declared it; this implies that that connection is still open and it wasn't declared by the connection you see in the log.
When your system is in that condition, go to the admin UI where you can explore the queue and which connection owns it.
e.g. Exclusive owner 127.0.0.1:60113
If that shows the closed connection (XX.XXX.57.174:17959 in the example above) you should reach out to the rabbitmq guys on the rabbitmq-users google group, this does not appear to be a spring-amqp issue;
EDIT
FYI, if passive queue declaration fails for any reason, by default the consumer will try 3 times at 5 second intervals, then give up and stop the container.
There are two properties on the container that can be used to adjust this - declarationRetries and failedDeclarationRetryInterval (default 3 and 5000 respectively). If you are using <rabbit:listener-container /> configuration, there are equivalent attributes on the namespace.

Check the queue name should not bound with an exchange otherwise you will get this error. Or I think queue name already exists.

Related

Make Spring RabbitMQ fail on missing exchange

There is nice property spring.rabbitmq.listener.simple.missing-queues-fatal=true
it makes SimpleMessageListenerContainer and whole application fail when I set it a non existent queue - which is what I want. I don't want to have an application running in invalid state.
I can't find similar solution for exchanges like
spring.rabbitmq.listener.simple.missing-exchanges-fatal
I get in logs several
ERROR 432430 --- [ 127.0.0.1:5672] o.s.a.r.c.CachingConnectionFactory : Shutdown Signal: channel error; protocol method: #method<channel.close>(reply-code=404, reply-text=NOT_FOUND - no exchange 'some-non-existent-exchange' in vhost '/', class-id=40, method-id=30)
and application starts. I would like it to fail. How can I do it?
How can I make spring boot/rabbit fail when trying to bind to any of non existing queue or exchange?
The exchange does nothing with listener. We need it along side with a routing key when we produce data into AMQP. The listener has that option to fail because it is out of end-user control and starts automatically when application is ready. The exchange is used in the RabbitTemplate when you send a data. See publisherReturns option on the CachingConnectionFactory for use-case like yours to handle such an error in case of missed exchange:
https://docs.spring.io/spring-amqp/docs/current/reference/html/#cf-pub-conf-ret
https://www.rabbitmq.com/confirms.html
You also add a ReturnsCallback into your RabbitTemplate to catch an unrouted message and its reason and handle such an error respectively: in your case stop the app, e.g. System.exit(1);.

MQRC_UNKNOWN_ALIAS_BASE_Q when connecting with IBM MQ cluster using CCDT and Spring Boot JMSTemplate

I have a Spring Boot app using JMSListener + IBMConnectionFactory + CCDT for connecting an IBM MQ Cluster.
A set the following connection properties:
- url pointing to a generated ccdt file
- username (password not required, since test environment)
- queuemanager name is NOT defined - since it's the cluster's task to decide, and a few google results, including several stackoverflow ones indicate that in my case qmgr must be set to empty string.
When my Spring Boot JMSListener tries to connect to the queue, the following MQRC_UNKNOWN_ALIAS_BASE_Q error occurs:
2019-01-29 11:05:00.329 WARN [thread:DefaultMessageListenerContainer-44][class:org.springframework.jms.listener.DefaultMessageListenerContainer:892] - Setup of JMS message listener invoker failed for destination 'MY.Q.ALIAS' - trying to recover. Cause: JMSWMQ2008: Failed to open MQ queue 'MY.Q.ALIAS'.; nested exception is com.ibm.mq.MQException: JMSCMQ0001: IBM MQ call failed with compcode '2' ('MQCC_FAILED') reason '2082' ('MQRC_UNKNOWN_ALIAS_BASE_Q').
com.ibm.msg.client.jms.DetailedInvalidDestinationException: JMSWMQ2008: Failed to open MQ queue 'MY.Q.ALIAS'.
at com.ibm.msg.client.wmq.common.internal.Reason.reasonToException(Reason.java:513)
at com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:215)
In the MQ error log I see the following:
01/29/2019 03:08:05 PM - Process(27185.478) User(mqm) Program(amqrmppa)
Host(myhost) Installation(Installation1)
VRMF(9.0.0.5) QMgr(MyQMGR)
AMQ9999: Channel 'MyCHL' to host 'MyIP' ended abnormally.
EXPLANATION:
The channel program running under process ID 27185 for channel 'MyCHL'
ended abnormally. The host name is 'MyIP'; in some cases the host name
cannot be determined and so is shown as '????'.
ACTION:
Look at previous error messages for the channel program in the error logs to
determine the cause of the failure. Note that this message can be excluded
completely or suppressed by tuning the "ExcludeMessage" or "SuppressMessage"
attributes under the "QMErrorLog" stanza in qm.ini. Further information can be
found in the System Administration Guide.
----- amqrmrsa.c : 938 --------------------------------------------------------
01/29/2019 03:15:14 PM - Process(27185.498) User(mqm) Program(amqrmppa)
Host(myhost) Installation(Installation1)
VRMF(9.0.0.5) QMgr(MyQMGR)
AMQ9209: Connection to host 'MyIP' for channel 'MyCHL' closed.
EXPLANATION:
An error occurred receiving data from 'MyIP' over TCP/IP. The connection
to the remote host has unexpectedly terminated.
The channel name is 'MyCHL'; in some cases it cannot be determined and so
is shown as '????'.
ACTION:
Tell the systems administrator.
Since the MQ error log contains QMgr(MyQMGR), which MyQMGR value I did not set in the connection properties, I assume the routing seems to be fine: the MQ Cluster figured out a qmgr to use.
The alias exists and points to an existing q. Bot the target q and the alias are added to the cluster via the CLUSTER(clustname) command.
What can be wrong?
Short Answer
MQ Clustering is not used for a consumer application to find a queue to GET messages from.
MQ Clustering is used when a producer application PUTs messages to direct them to a destination.
Further reading
Clustering is used when messages are being sent to help provide load balancing to multiple instances of a clustered queue. In some cases people use this for hot/cold failover by having two instances of a queue and keeping only one PUT(ENABLED).
If an application is a producer that is putting messages to a clustered queue, it only needs to be connected to a queue manager in the cluster and have permissions to put to that clustered queue. MQ based on a number of different things will handle where to send that message.
Prior to v7.1 there was only two ways to provide access to remote clustered queues:
Using a QALIAS:
Define a local QALIAS which has a TARGET set to the clustered queue name
Note this QALIAS does not itself need to be clustered.
Grant permission to put to the local QALIAS.
Provide permissions to PUT to the SYSTEM.CLUSTER.TRANSMIT.QUEUE.
The first option allows for granting granular access to an application for specific clustered queues in the cluster. The second option allows for the application to put to any clustered queue in the cluster or any queue on any clustered queue manager in the cluster.
At 7.1 IBM added a new optional behavior, this was provided with the setting ClusterQueueAccessControl=RQMName in the Security stanza of the qm.ini. If this is enabled (it is not the default), then you can actually provide permission for the app to PUT to the remote clustered queues directly without the need for a local QALIAS.
What clustering is not for is consuming applications such as your example of a JMSListener.
An application that will consume from any QLOCAL (clustered or not) must be connected to the queue manager where the QLOCAL is defined.
If you have a situation where there are multiple instances of a clustered QLOCAL that are PUT(ENABLED), you would need to ensure you have consumers connected directly to each queue managers that an instance is hosted on.
Based on your comment you have a CCDT with an entry such as:
CHANNEL('MyCHL') CHLTYPE(CLNTCONN) QMNAME('MyQMGR') CONNAME('node1url(port1),node2url(port2)')
If there are two different queue managers with different queue manager names listening on node1url(port1) and node2url(port2), then you have different ways to accomplish this from the app side.
When you specify the QMNAME to connect to the app will expect the name to match the queue manager you connect to unless it meets one of the following:
If you specify *MyQMGR it will find the channel or channels with QMNAME('MyQMGR') and pick one and connect and will not enforce that the remote queue manager name must match.
If in your CCDT you have QNAME(''), it is set to NULL, then in your app you can specify a empty queue manager name or only a space and it will find this entry in the CCDT and will not enforce that the remote queue manager name must match.
In your app you specify the queue manager name as *, MQ will use any channel in the CCDT and will not enforce that the remote queue manager name must match.
One limitation of CCDT is that channel name must be unique in the CCDT. Even if the QMNAME is different you can't have a second entry with the same channel name.
When you connect you are hitting the entry with two CONNAME's and getting connected to the first IP(port), you would only get to the second IP(port) if at connect time the first is not available, MQ will try the second, or if you are connected and have RECONNECT enabled and then the first goes down MQ will try to connect to the first then second.
If you want to have both clustered queue PUT(ENABLED) to receive traffic then you want to be able to specifically connect to each of the two queue managers to read those queues.
I would suggest you add a new channel on each queue manager that has a different QM specific name that is also different from the existing name, something like this:
CHANNEL('MyCHL1') CHLTYPE(CLNTCONN) QMNAME('MyQMGR1') CONNAME('node1url(port1)')
CHANNEL('MyCHL2') CHLTYPE(CLNTCONN) QMNAME('MyQMGR2') CONNAME('node2url(port2)')
This would be in addition to the existing entry.
For your putting components you can continue to use the channel that can connect to either queue manager.
For your getting components you can configure at least two of them, one to connect to each queue manager using the new queue manager specific CCDT entries, this way both queues are being consumed.

client-mode="true" and retryInterval on the inbound adapter with Client Connection factory

In spring Documentation --> 32.6 TCP Adapters it is mentioned that we use clientMode = "true" then the inbound adapter is responsible for the connection with external server.
I have created a flow in which the TCP Adapter with client connection factory makes connection with external server the code for the flow is :
IntegrationFlow flow = IntegrationFlows.from(Tcp.inboundAdapter(Tcp.nioClient(hostConnection.getIpAddress(),Integer.parseInt(hostConnection.getPort()))
.serializer(customSerializer)
.deserializer(customSerializer)
.id(hostConnection.getConnectionNumber())).clientMode(true).retryInterval(1000).errorChannel("testChannel").id(hostConnection.getConnectionNumber()+"adapter"))
.enrichHeaders(f->f.header("CustomerCode",hostConnection.getConnectionNumber()))
.channel(directChannel())
.handle(Jms.outboundAdapter(ConnectionFactory())
.destination(hostConnection.getConnectionNumber()))
.get();
theFlow = this.flowContext.registration(flow).id(hostConnection.getConnectionNumber()+"outflow").register();
I have created multiple flow by iterating over the list of connections and
iterate the above code in for loop and register them in flowcontext with unique ID.
My clients are created successfully with no issue and then establish there connection as supported by topology.
Issue :
I have counted the number of client connection created successfully so I have counted that 7 client connection (7 Integration flow) made successfully and they initiate connection from themselves.
when I create 8th client connection (8th flow created and registered successfully) but the .clientMode(true) is not working means the client don't initiate connection itself after first failure means it try for the first time to make connection if connected successfully then no issue but in case of failure it don't retry again.
Also my other created clients i.e 7 clients connection which are created successfully they also stopped initiating connection from itself when they got disconnected.
Note: There is no issue with flow only the TCP Adapters they stop initiating the connection
The flow is created and registered successfully as there is no issue it is because when I run a control bus command #adapter_id.retryConnection() it got connected with the server.
I don't understand that what is the issue with my flow that i couldn't initiate a connection after a particular count i.e seven or is there limitation in creating number of clients.
One possibility is the taskScheduler's thread pool is exhausted - that shouldn't happen with the above configuration, but it depends on what else is in the application. Take a thread dump (e.g. jstack) to see what the taskScheduler threads are doing.
See the documentation for information about how to configure the threads in the scheduler. However, if it solves it, you should really figure out what task(s) are using scheduler threads for long tasks.
Also turn on DEBUG logging to see if it provides any clues.

Multi Instance MQ set up

I am trying to set up Multi Instance MQ. I have configured NFS and able to see the active and stand by instances using dspmq -x. However my doubt is after this setup how to proceed further.
How can I configure the channel(s). I got CONNAME property should be use for this. e.g. CONNAME(<ip><port>,<ip><port>). I am not aware how to do it exactly.
How many listeners I need to start.
For normal mode of MQ, I follow the below steps:
1. crtmqm QM
2. strmqm QM
3. runmqsc QM
4. runmqlsr -m QM -t tcp -p 1125
5. runmqsc QM
6. define channel(SYSTEM.ADMIN.SVRCONN) chltype(SVRCONN) mcauser('mqm')
For multi-instance MQ, what changes I have to do for the below steps. There are many good documents available for Multi Instance MQ set up, but most of these are limited to how to configure the queue manager with multi-instance. Could anybody please guide me for the remaining steps.
Any guidance is much appreciated.
EDIT
Thanks Shashi and Morag for the guidance.
I have created different listeners on port 1600 in both the servers. I have created the below channel:
DEFINE CHANNEL(MYCHANNEL) CHLTYPE(CLNTCONN) TRPTYPE(TCP)
CONNAME('IP11600),IP2(1600)')
DEFINE CHANNEL(MYCHANNEL) CHLTYPE(SVRCONN) TRPTYPE(TCP) MCAUSER(' ')
Below is the stand alone java code I am using to put a message in queue.
public class MutilInstanceMq
{
public static void main(String[] args)
{
sendMsg("Test Message");
}
public static void sendMsg(String msg)
{
MQQueueConnectionFactory connectionFactory = null;
QueueConnection queueConn = null;
QueueSession queueSession = null;
QueueSender queueSender = null;
TextMessage message = null;
try
{
connectionFactory = new MQQueueConnectionFactory();
connectionFactory.setConnectionNameList("<IP1>(1600), <IP2>(1600)");
connectionFactory.setTransportType(WMQConstants.WMQ_CM_CLIENT);
connectionFactory.setQueueManager("MYQM1");
connectionFactory.setChannel("MYCHANNEL");
queueConn = connectionFactory.createQueueConnection(" ","password");
queueSession = queueConn.createQueueSession(false, Session.AUTO_ACKNOWLEDGE);
queueSender = queueSession.createSender(queueSession.createQueue("MYQ"));
queueSender.setDeliveryMode(DeliveryMode.NON_PERSISTENT);
message = queueSession.createTextMessage(msg);
message.setJMSCorrelationID("12345");
queueSender.send(message);
queueConn.close();
}
catch (Exception e)
{
e.printStackTrace();
}
}
}
With the above code I end up with the below exception while creating the connection.
> com.ibm.msg.client.jms.DetailedJMSException: JMSWMQ0018: Failed to
> connect to queue manager 'MYQM1' with connection mode 'Client' and
> host name 'IP1(1600),IP2(1600)'. Check the queue manager is
> started and if running in client mode, check there is a listener
> running. Please see the linked exception for more information.
Detailed Print stack trace is:
com.ibm.msg.client.jms.DetailedJMSException: JMSWMQ0018: Failed to connect to queue manager 'MYQM1' with connection mode 'Client' and host name 'IP1(1600),IP2(1600)'. Check the queue manager is started and if running in client mode, check there is a listener running. Please see the linked exception for more information.
at com.ibm.msg.client.wmq.common.internal.Reason.reasonToException(Reason.java:608)
at com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:236)
at com.ibm.msg.client.wmq.internal.WMQConnection.<init>(WMQConnection.java:440)
at com.ibm.msg.client.wmq.factories.WMQConnectionFactory.createV7ProviderConnection(WMQConnectionFactory.java:7062)
at com.ibm.msg.client.wmq.factories.WMQConnectionFactory.createProviderConnection(WMQConnectionFactory.java:6453)
at com.ibm.msg.client.jms.admin.JmsConnectionFactoryImpl.createConnection(JmsConnectionFactoryImpl.java:295)
at com.ibm.mq.jms.MQConnectionFactory.createCommonConnection(MQConnectionFactory.java:6230)
at com.ibm.mq.jms.MQQueueConnectionFactory.createQueueConnection(MQQueueConnectionFactory.java:144)
at MutilInstanceMq.sendMsg(MutilInstanceMq.java:40)
at MutilInstanceMq.main(MutilInstanceMq.java:17)
Caused by: com.ibm.mq.MQException: JMSCMQ0001: WebSphere MQ call failed with compcode '2' ('MQCC_FAILED') reason '2543' ('MQRC_STANDBY_Q_MGR').
at com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:223)
... 8 more
Caused by: com.ibm.mq.jmqi.JmqiException: CC=2;RC=2543;AMQ9204: Connection to host 'IP2(1600)' rejected. [1=com.ibm.mq.jmqi.JmqiException[CC=2;RC=2543;AMQ9487: Remote queue manager is a standby queue manager instance. [3=MYCHANNEL]],3=IP2(1600),5=RemoteConnection.analyseErrorSegment]
at com.ibm.mq.jmqi.remote.internal.RemoteFAP.jmqiConnect(RemoteFAP.java:2010)
at com.ibm.mq.jmqi.remote.internal.RemoteFAP.jmqiConnect(RemoteFAP.java:1227)
at com.ibm.msg.client.wmq.internal.WMQConnection.<init>(WMQConnection.java:355)
... 7 more
Caused by: com.ibm.mq.jmqi.JmqiException: CC=2;RC=2059;AMQ9204: Connection to host 'IP1(1600)' rejected. [3=IP1(1600)]
at com.ibm.mq.jmqi.remote.internal.RemoteFAP.jmqiConnect(RemoteFAP.java:1980)
... 9 more
Where I am going wrong? Meanwhile I have turned off channel authentication and able to connect to the queue.
To answer your explicit questions:
If you want to use the comma-separated CONNAME method, then you'll end up with something like this. CONNAME('machine1(1234),machine2(1234)'). You should have the same port number in both.
You have two options here. Either use runmqlsr on each machine - so you have two listeners, on the same port. The purpose of the one on the primary machine is for connections to be able to connect to the queue manager. The purpose of the one on the standby machine is to reject connections which try to connect there more quickly than they could be rejected by TCP working out that there is no listener. It also means that the error reported by the connection for the reason it could not connect is much more explicit - "This machine is the standby" rather than "no TCP listener here".
The alternate option is to define a LISTENER object and have it controlled by the queue manager, this way it will only run where the queue manager runs. This keeps the configuration all within the queue manager, but does mean you don't get the benefits outlined above.
The steps are well documented in Knowledge Center here: http://www-01.ibm.com/support/knowledgecenter/SSFKSJ_7.1.0/com.ibm.mq.doc/fa70161_.htm?lang=en. How to verify the multi-instance setup is described here: http://www-01.ibm.com/support/knowledgecenter/SSFKSJ_7.1.0/com.ibm.mq.doc/fa70163_.htm?lang=en.

Reconnecting with MQ - issues

I'm looking forward to implement a somewhat intelligent MQ comms module, which should be tolerant for the outages in the network connection. Basically it should try to reconnect each 5 seconds if connection was lost.
The problem is the following. I use the following code for reading:
queueMessage = new MQMessage();
queueMessage.Format = MQC.MQFMT_STRING;
queueGetMessageOptions = new MQGetMessageOptions();
queueGetMessageOptions.Options = MQC.MQGMO_SYNCPOINT + MQC.MQGMO_WAIT + MQC.MQGMO_FAIL_IF_QUIESCING;
queueGetMessageOptions.WaitInterval = 50;
producerQueue.Get(queueMessage, queueGetMessageOptions);
msg = queueMessage.ReadBytes(queueMessage.MessageLength);
(Of course I successfully connect to the queuemanager before etc.)
I got the following issue: when this routine runs, but at the time of .Get there's no connection, the code simply hangs and stays in the .Get.
I use a timer to see if there's a timeout (in theory even that shouldn't be necessary, is that right?) and at the timeout I try to reconnect. BUT when this timeout expires, I still see that the queuemanager reports that it's connected, while its clearly not (no physical connection is present anymore). This issue has popped up since I use SYNCPOINT, and I experience the same when I cut connection during writing, or in this case I try to force a Disconnect on the queuemanager. So please help, what settings shall I use to avoid getting stuck in Get and Put and rather have an MQException thrown or something controllable?
Thanks!
UPDATE: I used the following code to connect to the QueueManager.
Hashtable props = new Hashtable();
props.Add(MQC.HOST_NAME_PROPERTY, Host);
props.Add(MQC.PORT_PROPERTY, Port);
props.Add(MQC.CHANNEL_PROPERTY, ChannelInfo);
if(User!="") props.Add(MQC.USER_ID_PROPERTY, User);
if(Password!="") props.Add(MQC.PASSWORD_PROPERTY, Password);
props.Add(MQC.TRANSPORT_PROPERTY, MQC.TRANSPORT_MQSERIES_MANAGED);
queueManager = new MQQueueManager(QueueManagerName, props);
producerQueue = queueManager.AccessQueue(
ProducerQueueName,
MQC.MQOO_INPUT_AS_Q_DEF // open queue for input
+ MQC.MQOO_FAIL_IF_QUIESCING); // but not if MQM stopping
consumerQueue = queueManager.AccessQueue(
ConsumerQueueName,
MQC.MQOO_OUTPUT + MQC.MQOO_BROWSE + MQC.MQOO_INPUT_AS_Q_DEF // open queue for output
+ MQC.MQOO_FAIL_IF_QUIESCING); // but not if MQM stopping
Needless to say that normally the code works well. Read/Write, connect/disconnect works as it should, I only have to figure out the current issue.
Thanks!
What version of MQ are you using? For automatic reconnection to work the queue manager need to be at least at MQ v701 and MQ .NET client needs to be a MQ v7.1 level.
Assuming you are using MQ v7.1 .NET client, you need to specify reconnect option during connection create. You will need to enable reconnection by adding something like:
props.Add(MQC.CONNECT_OPTIONS_PROPERTY, MQC.MQCNO_RECONNECT);
Reconnection can be enabled/disabled from mqclient.ini file also.
But what is surprising is why the Get/Put are hanging when there is no network connection. Hope you are not connecting a queue manager running on the same machine as your application. There is no need to set any timer or something like that. You can issue MQ calls and if there is anything wrong with connection, an exception will be thrown.
Update:
I think you are referring to IsConnected property of MQQueueManager class. The documentation says the value of this property: "If true, a connection to the queue manager has been made, and is not known to be broken. Any calls to IsConnected do not actively attempt to reach the queue manager, so it is possible that physical connectivity can break, but IsConnected can still return true. The IsConnected state is only updated when activity, for example, putting a message, getting a message, is performed on the queue manager.
If false, a connection to the queue manager has not been made, or has been broken, or has been disconnected."
As you can see a True value does not mean the connection is still ON. My suggestion would be to call a method, Put/Get and handle any exception thrown.
Put/Get/Disconnect calls hanging appears to be a problem. My suggestion would be raise a PMR with IBM.

Resources