Set brokerDurSubQueue property in Spring Boot+JMS+IBM MQ Durable Topic listener - spring-boot

I am trying to listen message through Spring boot application using IBM MQ topic subscription.
Available info (Provided by MQ Admin):
Topic name
Host
Port
QueueManager
BrokerDurableSubscriptionQueue
I am trying to set BrokerDurableSubscriptionQueue property in MQConnectionFactory.
I can find mqConnectionFactory.setBrokerSubQueue(queueName) which I guess can be used for Non-Durable Subscription.
But I cannot find similar property for Durable subscription.
However I can see MQTopic class has setBrokerDurSubQueue property but I am not sure how can I make use of MQTopic object in my case.
I am using below code:
MQConnectionFactory:
#Bean
public MQTopicConnectionFactory topicConnectionFactory(){
MQTopicConnectionFactory mqTopicConnectionFactory= new MQConnectionFactory();
mqTopicConnectionFactory.setHostName(); //mq host name
mqTopicConnectionFactory.setPort(); // mq port
mqTopicConnectionFactory.setQueueManager(); //mq queue manager
mqTopicConnectionFactory.setChannel(); //mq channel name
mqTopicConnectionFactory.setTransportType(1);
mqTopicConnectionFactory.setSSLCipherSuite(); //tls cipher suite name
return mqTopicConnectionFactory;
}
#Bean
public JmsListenerContainerFactory<?> topicListenerFactory(MQTopicConnectionFactory mqtopicConnectionFactory,
DefaultJmsListenerContainerFactoryConfigurer configurer)
{
DefaultJmsListenerContainerFactory factory = new DefaultJmsListenerContainerFactory();
configurer.configure(factory, mqtopicConnectionFactory);
factory.setPubSubDomain(true);
factory.setSubscriptionDurable(true);
return factory;
}
Listener:
#JmsListener(
destination = "someTopic",
subscription = "someTopic",
containerFactory = "topicListenerFactory"
)
public void receiveMessage(String msg) {
repository.save(msg);
}

Background:
When you provide a specific queue for IBM MQ to use when you subscribe to a topic it is called a unmanaged subscription because MQ is not managing the underlying queue since you have provided it.
When a queue is not provided it is called a managed subscription, in this case MQ creates a queue for you to hold the published messages.
If it is a non-durable subscription the queue created is a temporary dynamic queue with a name like:
SYSTEM.MANAGED.NDURABLE.<8 hex characters>
If it is a durable subscription the queue created is a permanent dynamic queue with a name like:
SYSTEM.MANAGED.DURABLE.<8 hex characters>
What you have uncovered is that the IBM MQ classes for JMS API only supports managed subscriptions.
Suggestions:
I can suggest two options if you want to use IBM MQ classes for JMS API to receive messages published to a topic on a specific queue:
Have a MQ admin setup an administrative subscription on the queue manager. You can do this a few different ways. The example below would be using a MQSC command.
DEFINE SUB('XYZ') TOPICSTR('SOME/TOPIC') DEST(SOME.QUEUE)
Create a utility app using IBM MQ classes for Java that can open a queue and create a durable subscription with a provided queue, the only purpose of this app would be to subscribe and unsubscribe a provided queue, it would not be used to consume any of the published messages.
For both options above you would have the IBM MQ classes for JMS API application open the queue to consume the published message, for all purposes it would not know or need to know the messages were published to a topic. The message will still contain JMS headers showing the topic string where the message was published, so you can inquire this if required. You could also subscribe multiple topics to a single queue if you like.

Related

Message sent using JMS producer is not received in MQTT receiver in the same SpringBoot app

I'm just starting with ActiveMQ Artemis and have Artemis 2.17.0 installed on my machine. Created SpringBoot test app where both JMS and MQTT publishers and receivers exist. Created also small RestController so I can send messages using both JMS and MQTT producers. Receivers are quite simple and just create a log message to console. Now when I create a message using MQTT producer, both JMS and MQTT receivers get and log message to console. But when I send a message using JMS producer, the message is being received only in JMS receiver, no MQTT message in console. Tried several times. Implementation is ok I think as MQTT producer example is working fine. Is there any limitation for routing messages among protocols in Artemis in this way? Or what kind of problem it can be?
Code info about JMS implementation: https://dmarko.tcl-digitrade.com/post/2021/activemq-artemis-spring-boot/
Code info about MQTT implementation: https://dmarko.tcl-digitrade.com/post/2021/activemq-artemis-mqtt/
Apache ActiveMQ Artemis has a flexible addressing model that supports both Point-to-Point and Publish-Subscribe patterns.
By default, Spring Boot creates a JmsTemplate configured to transmit Point-to-Point while MQTT uses a Publish-Subscribe pattern, so the JMS and MQTT receivers are using different messaging patterns and this is causing your issue.
To configure a JmsTemplate for the Publish-Subscribe pattern set spring.jms.pub-sub-domain=true through Boot’s application.properties or set the JmsTemplate pubSubDomain to true, ie:
jmsTemplate.setPubSubDomain(true);
To configure a JmsListener for the Publish-Subscribe pattern set spring.jms.pub-sub-domain=true through Boot’s application.properties or set the JmsListenerContainerFactory pubSubDomain to true, ie:
#Bean
public JmsListenerContainerFactory<?> topicConnectionFactory(ConnectionFactory connectionFactory, DefaultJmsListenerContainerFactoryConfigurer configurer) {
DefaultJmsListenerContainerFactory factory = new DefaultJmsListenerContainerFactory();
configurer.configure(factory, connectionFactory);
factory.setPubSubDomain(true);
return factory;
}
#JmsListener(destination = "${prices.mqtt.east}", containerFactory = "topicConnectionFactory")
public void receiveFromTopic(String message) {
...
}

How to make topic subscription in IBM MQ AMQP unmodifiable or unalterable by client program? [duplicate]

We have programmatically created subscriber to IBM MQ AMQP TOPIC with createDurableSubscriber by providing clientId and subscriber name.
We start the program so it subscribes to TOPIC and stop the program. Then send the msgs to topic and again start the receiver program again but we cannot receive the msgs sent and loose the messages which should not happen in case of durable subscription..
We can see amqp topic and its durable subscription when subscriber is connected using mqsc commands DISPLAY TOPIC, DISPLAY TPSTATUS, DISPLAY TPSTATUS SUB, DISPLAY SUB SUBID but not when subscriber program is stopped. We have defined attribute DEFPSIST(YES) and client(producer to topic) is sending persistent messages.
Where are the messages gone as we cannot see messages in durable queues of subscriber? Does it depends on expiry attribute?
Output of DISPLAY SUB SUBID for our subscriber when it is connected.
AMQ8096: WebSphere MQ subscription inquired.
SUBID("hex sub id")
SUB(:private:CLINET01:TOPIC01) TOPICSTR(TOPIC01)
TOPICOBJ(SYSTEM.BASE.TOPIC) DISTYPE(RESOLVED)
DEST(SYSTEM.MANAGED.DURABLE.5F6B5C2524FB9AED)
DESTQMGR(qm.name) PUBAPPID( )
SELECTOR( ) SELTYPE(NONE)
USERDATA(010)
PUBACCT(***************************************************)
DESTCORL(***************************************************)
DESTCLAS(MANAGED) DURABLE(YES)
EXPIRY(0) PSPROP(MSGPROP)
PUBPRTY(ASPUB) REQONLY(NO)
SUBSCOPE(ALL) SUBLEVEL(1)
SUBTYPE(API) VARUSER(FIXED)
WSCHEMA(TOPIC) SUBUSER(mqm)
CRDATE(2020-09-28) CRTIME(04:14:09)
ALTDATE(2020-09-28) ALTTIME(04:14:09)
Subscriber id has private(not sure why) and client id but not subscriber name which is sub4
import javax.jms.Connection;
import javax.jms.ConnectionFactory;
import javax.jms.MessageConsumer;
import javax.jms.MessageProducer;
import javax.jms.Topic;
import javax.jms.Queue;
import javax.jms.Session;
import javax.jms.TextMessage;
import java.lang.String;
import javax.jms.Destination;
import javax.naming.Context;
import org.apache.qpid.jms.JmsConnectionFactory;
import javax.jms.DeliveryMode;
import javax.naming.InitialContext;
import javax.jms.Message;
public class AMQPQueueExample1 implements Runnable {
private static final int DELIVERY_MODE = DeliveryMode.PERSISTENT;
public void run(){
try{
Connection connection = null;
Context context = new InitialContext();
ConnectionFactory connectionFactory = (ConnectionFactory) context.lookup("myFactoryLookup");
connection = connectionFactory.createConnection();
connection.setClientID("123");//("WHATS_MY_PURPOSE3"); // Why do we need clientID while publishing the TOPIC from consumer / publisher
Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
Topic priceTopic = (Topic) context.lookup("myTopicLookup1");
MessageConsumer subscriber1 = session.createDurableSubscriber(priceTopic,"sub420"); //"sub3");
System.out.println("TOPIC "+priceTopic);
connection.start();
while(true){
TextMessage message1 = (TextMessage) subscriber1.receive(1000);
if(message1!=null)
System.out.println("Subscriber 1 received : " + message1.getText());
}
}catch(Exception e){
e.printStackTrace();
}
}
public static void main(String[] args) {
AMQPQueueExample1 amp=new AMQPQueueExample1();
Thread thread = new Thread(amp);
thread.start();
}
}
Values are taken from jndi.properties file for context factory and provider url.
It looks from the comments like you're using MQ 8.0.0.5? If that's the case then Apache Qpid JMS clients aren't supported with that version of MQ. I believe with that version a very basic non-durable subscribe might work, but any other JMS methods are unlikely to.
I suspect what is happening is the AMQP 1.0 flows from Qpid JMS aren't fully understood by that version of MQ, so the expiry of the subscription is being set to 0 rather than unlimited.
MQ 9.2 added support for more of the JMS 2.0 spec - although not every JMS feature. There is more information about the methods which are supported here:
https://www.ibm.com/support/knowledgecenter/SSFKSJ_9.2.0/com.ibm.mq.dev.doc/q125050_.htm
Creating durable subscribers and/or consumers should work as you expect.
An article from Matthew Whitehead "MQ Light messaging from Microsoft®.NET™ (Part 4)" states the following:
AMQP channels don’t support setting an unlimited expiry time for MQ Light subscriptions. While it is possible to create subscriptions that have a very long time-to-live, it isn’t possible to create subscriptions to exist forever.
If you wish to create subscriptions that never expire you can do so by creating an MQ administered subscription and having MQ Light clients join and leave the subscription. This can also help to ensure that any messages published to a topic before the first subscribers have connected, aren’t lost completely. Read my previous article on joining MQ Light clients to administered subscriptions.
Related AMQP Fields
To provide the expiry capabilities described above MQ Light uses 2 features of AMQP 1.0:
Source Timeout
Source Expiry Policy
The source timeout is used to specify the time in seconds that the subscription will expire.
The source expiry policy is used to determine what causes the expiry timer to begin. MQ AMQP channels only support an expiry policy of link-detach, which means the timer starts as soon as the last link detaches from the subscription.
I searched and couldn't find a reference on how to set Source Timeout or Source Expiry Policy in Apache QPID, but the linked blog references setting expiry via a administratively defined subscription. Based on the info in your question I think you can just define something like this ahead of time. I have not specified EXPIRY because this will pick up EXPIRY(UNLIMITED) from SYSTEM.DEFAULT.SUB:
DEFINE SUB(':private:CLINET01:TOPIC01') TOPICOBJ(SYSTEM.BASE.TOPIC) TOPICSTR('TOPIC01') DESTCLAS(MANAGED)
When you then connect your AMQP subscriber it will resume this existing subscription with the expiry set to UNLIMITED.

Redeliver message to MQ when using #JmsListener

I'm using #EnableJms and #JmsListener annotation to register a queue listener in my application, basing on this tutorial. I'm connecting to IBM MQ, getting connection factory using jndi. I have read about acknowledge modes etc. but still it's a new thing to me. And my problem is that the message is not being returned to a queue (the listener is never called again).
#Bean
public DefaultJmsListenerContainerFactory jmsListenerContainerFactory() {
DefaultJmsListenerContainerFactory factory
= new DefaultJmsListenerContainerFactory();
factory.setConnectionFactory(connectionFactory());
factory.setSessionTransacted(true);
factory.setSessionAcknowledgeMode(Session.AUTO_ACKNOWLEDGE); //I have tried also CLIENT_ACKNOWLEDGE
return factory;
}
#JmsListener(containerFactory = "jmsListenerContainerFactory", destination = "myQueue")
#SendTo("secondQueue")
public String testListener(String message){
if(true) throw new NullPointerException();
else return message;
}
Any help would be much appreciated.
I would have also a second question. From what I understand if I would like to implement any operation on a database, only way to rollback a commit (if something went wrong after this) would be to create a transaction manager? If not, I would need to detect a duplicated message.
First set the acknowledgement mode to Session.CLIENT_ACKNOWLEDGE
and when receiving the messages, if it's processed correctly then just call message.acknowledge() method or else don't call.
It will automatically stay in the queue and you don't need to resend it.
You need to use
import javax.jms.Message
I created simple Spring Boot app and Docker container of IBM MQ to test your case.
I found good instructions in this tutorial: https://developer.ibm.com/tutorials/mq-jms-application-development-with-spring-boot/
And in your case this environment behaves as expected: endless cycle of receive message -> NullPointerException -> return message -> ...
Than I found feature of IBM MQ called "Backout Queues & Thresholds", you'll found the explanation in this blog post: https://community.ibm.com/community/user/imwuc/browse/blogs/blogviewer?BlogKey=28814801-083d-4c80-be5f-90aaaf81cdfb
Briefly, it is possible to restrict number of times message returned to queue after exception, and after this limit send message to another queue.
May be in your case this feature used on your destination queue.

Spring JMS + IBM MQ: How to set message buffer size or wait timeout?

I'm unable to process large messages from IBM MQ and get the below error:
JMSCMQ0001: WebSphere MQ call failed with compcode '1' ('MQCC_WARNING') reason '2080' ('MQRC_TRUNCATED_MSG_FAILED')
I'm using the DefaultListenerContainer and not consuming via a MessageConsumer using IBM MQ Java API classes directly. I believe by using IBM MQ JMS API you can specific options before retrieving the message from the queue. But how do I do that with DefaultListenerContainer, is there a system property I can set for these?
If using IBM MQ JMS API(I'm not consuming message like this, pasted just for reference):
MQGetMessageOptions mqGetMessageOptions = new MQGetMessageOptions();
mqGetMessageOptions.waitInterval = ipreoProperties.getMqReceiveWaitTime();
mqGetMessageOptions.options = MQC.MQGMO_WAIT | MQC.MQPMO_SYNCPOINT | MQC.MQGMO_ACCEPT_TRUNCATED_MSG;
Below is my Java Config for the IBM MQ Connection:
#Bean
public CachingConnectionFactory ipreoMQCachingConnectionFactory() {
CachingConnectionFactory cachingConnectionFactory = new CachingConnectionFactory();
//Not defining MQQueueConnectionFactory as separate bean as Spring boot's auto-configuration finds two instances
//of ConnectionFactory and throws ambiguous implementation exception
//One implementation is CachingConnectionFactory and other one would be MQQueueConnectionFactory if defined separately
MQQueueConnectionFactory mqConnectionFactory = new MQQueueConnectionFactory();
try {
mqConnectionFactory.setHostName(env.getRequiredProperty(AppEnvPropertyConstants.JmsConstants.IPREO_MQ_HOSTNAME));
mqConnectionFactory.setQueueManager(env.getRequiredProperty(AppEnvPropertyConstants.JmsConstants.IPREO_MQ_QUEUE_MGR));
mqConnectionFactory.setPort(env.getRequiredProperty(AppEnvPropertyConstants.JmsConstants.IPREO_MQ_PORT, Integer.class));
mqConnectionFactory.setChannel(env.getRequiredProperty(AppEnvPropertyConstants.JmsConstants.IPREO_MQ_CHANNEL));
//mqConnectionFactory.setTransportType(WMQConstants.WMQ_CM_CLIENT);
//Setting connection mode as Client so it doesn't complain for native IBM MQ libraries
mqConnectionFactory.setIntProperty(CommonConstants.WMQ_CONNECTION_MODE, CommonConstants.WMQ_CM_CLIENT);
} catch (JMSException exception) {
exception.printStackTrace();
}
cachingConnectionFactory.setTargetConnectionFactory(mqConnectionFactory);
//Setting session caching size as 10, don't think we need more
cachingConnectionFactory.setSessionCacheSize(10);
cachingConnectionFactory.setReconnectOnException(true);
return cachingConnectionFactory;
}
public DefaultMessageListenerContainer ipreoDealActivityListenerContainer() {
DefaultMessageListenerContainer factory = new DefaultMessageListenerContainer();
factory.setConnectionFactory(ipreoMQCachingConnectionFactory());
factory.setDestinationName(env.getRequiredProperty(AppEnvPropertyConstants.JmsConstants.IPREO_DEAL_QUEUE_NAME));
factory.setMessageListener(ipreoDealActivityListener());
factory.setSessionAcknowledgeMode(Session.AUTO_ACKNOWLEDGE);
return factory;
}
#Bean
public MessageListener ipreoDealActivityListener() {
return new IpreoDealActivityListener();
}
Appreciate your help, thanks.
Adding a late response as it might be useful to someone.
In my case, when the java client had this exception, we noticed the actual message size was larger than the default 4 MB buffer size.
The Java API does not provide a hook to change buffer size. Hence, the buffer size has to be updated at the MQ server level.
First, we increased the message size in queue properties - It did not work.
Then, we increased the message size property at the MQ channel level as well, which finally resolved the issue.
To summarise, increase the buffer size at the MQ server for queue & the channel both.
On a client connection to a queue manager you can limit the size of messages on both the server and client side. I've seen this error before when the client side limit was smaller then the size of the message.
I don't know how you can set the message size limit directly in the JMS client, but you could use a Client Channel Definition Table. It's a file containing the details for connecting to queue managers, created on a queue manager and then copied to the client host. You need to reference the file by issuing setCCDTURL on the connection factory (setting the host, port and channel is not required when using a CCDT, the CCDT will specify those).
When the CCDT is created on the queue manager the appropriate message size limit needs to be set on the client channel.
The server side limit is set on the server connection channel.
Within the JMS client code handling of the receive buffer us handled automatically; the theory is that specific error should never be received by a JMS Application.
The first snippet of code is the Java Classes API and this could get that error.
How big actually are these messages? What level of the JMS client code are you using - make sure that it is the latest version. And certainly one of the 7.5 or 8 releases.
This answer also has some more information on this.

Send message to MQ using Jmeter

I want to send message to a remote IBM MQ using Jmeter for performance testing. I went through this link. But it requires the JNDI specific details like, QueueConnection Factory, JNDI Name Request queue, Initial Context Factory & Provider URL. Whereas the queu details i have are Qmanager, Qname, hostname, channel, port as given in the code shared in this link. Do these properties have any relation? Can i configure the Jmeter JMS test using the queue details i have?
Thanks in advance.
The first link you gave has a description using Java JMS/MQ and the second shows Java MQ (non-JMS).
JMS is just an abstraction layer. In simple terms, JMS is like giving everything a nick-name. A QCF (QueueConnectionFactory) is simply an object that has all of the information to connect to a queue manager.
i.e.
DEFINE QCF(myQCF) QMANAGER(MQWT1) CHANNEL(TEST.CHL) HOSTNAME(127.0.0.1) PORT(1415) TRANSPORT(CLIENT) FAILIFQUIESCE(YES)
A JMS queue is just a nick-name to an MQ queue.
DEFINE Q(test.q) QUEUE(TEST.Q1) QMANAGER(MQWT1) TARGCLIENT(JMS) FAILIFQUIESCE(YES)
Therefore, in your JMS code you simply reference your QCF (i.e. myQCF) and the JMS queue (i.e. test.q) and you are good to go.
Hashtable env = new Hashtable();
env.put(Context.INITIAL_CONTEXT_FACTORY, "com.sun.jndi.fscontext.RefFSContextFactory");
env.put(Context.PROVIDER_URL, ""file:/C:/JNDI-Directory");
try
{
Context ctx = new InitialContext(env);
QueueConnectionFactory cf = (QueueConnectionFactory) ctx.lookup("myQCF");
Queue q = (Queue) ctx.lookup("test.q");
}
catch (NamingException e)
{
System.err.println(e.getLocalizedMessage());
e.printStackTrace();
}
It can be done via the beanshell as well. You can directly access the queue manager via the api, or via exposing the queue via a jms binding. The first is more simple and does not require the MQ client installation.

Resources