I'm testing kafka broker down scenarios in my service. Is there any way to execute piece of code once the service became connected again to the broker ?
For example:-
When the producer fails to send a message or throws an exception the service stores the message in a temp table to resend it once reconnecting to the broker.
#Transactional
#EventListener
public void detailsObjectEventListener(EntityObjectEvent<DetailsEntity> event) {
StreamPayload<Details> payload = new StreamPayload<>(event.getType(), new Details(event.getSource()));
MessageChannel channel = BootstrapApplication.getMessageChannel(DetailsChannel.class);
boolean sent;
try {
sent = channel.send(MessageBuilder.withPayload(payload).build());
} catch (Exception ex) {
sent = false;
}
if (!sent) {
entityManager.persist(new FailedStreamEventEntity(event));
}
}
To handle the problem i made a cron job method to republish failed events again when the broker is UP.
#Scheduled(initialDelay = 10000, cron = "0 * * * * ?")
public void reSendFailedEvents() {
if (kafkaIndicator.health().getStatus().equals(Status.UP)) {
CriteriaBuilder builder = entityManager.getCriteriaBuilder();
Optional.of(FailedStreamEventEntity.class)
.map(builder::createQuery)
.map(query -> query.select(query.from(FailedStreamEventEntity.class)))
.map(entityManager::createQuery)
.map(TypedQuery::getResultList)
.map(List::stream)
.orElse(Stream.empty())
.peek(entityManager::remove)
.map(FailedStreamEventEntity::getEvent)
.forEach(BootstrapApplication.getEventPublisher()::publishEvent);
}
}
But, i believe its the worst solution to handle the problem. Also, i believe that there is a better way that can execute this code once the service reconnected to the broker. (Any help ?)
I have a JMS application deployed as a Docker image in AWS Fargate. Two services are running for the task. However the problem is I am getting this:
2021-03-24 05:15:43.022 ERROR 1 --- [ main] com.hp.ext.cpq.pubsub.SnsTopicPublisher : Exception happened in readJmsTopicPublishToSnsTopic --->javax.jms.JMSException: Duplicate durable subscription detected
This is the code I am using to create the durable subscriber:
SnsTopicPublisher asyncSubscriber = this.ctx.getBean(SnsTopicPublisher.class);
if (prop.getProperty("tibco.msgSourceType").equalsIgnoreCase("TOPIC")) {
dest_t = session.createTopic(prop.getProperty("tibco.msgSource"));
**TopicSubscriber topicSubscriber = session.createDurableSubscriber(dest_t, "pfpDurable");**
topicSubscriber.setMessageListener(asyncSubscriber);
logger.debug("Set Jms Topic Listener ---> asyncSubscriber");
}
if (prop.getProperty("tibco.msgSourceType").equalsIgnoreCase("QUEUE")) {
dest_q = session.createQueue(prop.getProperty("tibco.msgSource"));
MessageConsumer msgConsumer_p = session.createConsumer(dest_q);
msgConsumer_p.setMessageListener(asyncSubscriber);
logger.debug("Set Jms Queue Listener ---> asyncSubscriber");
}
I am getting the error for the marked line from AWS cloud watch logs.
Most likely you have a connection (and generally other JMS objects) leak. When the exception is thrown, you need to close resources in a finally {} block, similar to the JDBC pattern.
Also, you might want to look at using a Pooled Connection. This allows for open+close pattern on JMS connections without really closing connections to the server. Check out the activemq-jms-pool which is a JMS-standard pool (not ActiveMQ specific) that works with most JMS brokers, including Tibco and IBM MQ.
Connection connection = null;
Session session = null;
MessageConsumer messageConsumer = null;
try {
connection = connectionFactory.createConnection();
connection.start();
.. do some JMS
} catch (JMSException e) {
// handle errors
} finally {
if (messageConsumer != null) {
try { messageConsumer.close(); } catch (JMSException e) { logger.error("Error closing MessagingConsumer", e);
}
if (session != null) {
try { session.close(); } catch (JMSException e) { logger.error("Error closing Session", e);
}
if (connection != null) {
try { connection.close(); } catch (JMSException e) { logger.error("Error closing Connection", e);
}
}
Note: The JMS specification only allows 1 durable subscription per topic using the API you have in the code. The JMS v2.0 allows for a Shared Durable Subscription to support multiple consumers.
Working versions in the app
IBM AllClient version : 'com.ibm.mq:com.ibm.mq.allclient:9.1.1.0'
org.springframework:spring-jms : 4.3.9.RELEASE
javax.jms:javax.jms-api : 2.0.1
My requirement is that in case of the failure of a message processing due to say, consumer not being available (eg. DB is unavailable), the message remains in the queue or put back on the queue (if that is even possible). This is because the order of the messages is important, messages have to be consumed in the same order that they are received. The Java app is single-threaded.
I have tried the following
#Override
public void onMessage(Message message)
{
try{
if(message instanceOf Textmessage)
{
}
:
:
throw new Exception("Test");// Just to test the retry
}
catch(Exception ex)
{
try
{
int temp = message.getIntProperty("JMSXDeliveryCount");
throw new RuntimeException("Redlivery attempted ");
// At this point, I am expecting JMS to put the message back into the queue.
// But it is actually put into the Bakout queue.
}
catch(JMSException ef)
{
String temp = ef.getMessage();
}
}
}
I have set this in my spring.xml for the jmsContainer bean.
<property name="sessionTransacted" value="true" />
What is wrong with the code above ?
And if putting the message back in the queue is not practical, how can one browse the message, process it and, if successful, pull the message (so it is consumed and no longer on the queue) ? Is this scenario supported in IBM provider for JMS?
The IBM MQ Local queue has BOTHRESH(1).
To preserve message ordering, one approach might be to stop the message listener temporarily as part of your rollback strategy. Looking at the Spring Boot doc for DefaultMessageListenerContainer there is a stop(Runnable callback) method. I've experimented with using this in a rollback as follows.
To ensure my Listener is single threaded, on my DefaultJmsListenerContainerFactory I set containerFactory.setConcurrency("1").
In my Listener, I set an id
#JmsListener(destination = "DEV.QUEUE.2", containerFactory = "listenerTwoFactory", concurrency="1", id="listenerTwo")
And retrieve the DefaultMessageListenerContainer instance.
JmsListenerEndpointRegistry reg = context.getBean(JmsListenerEndpointRegistry.class);
DefaultMessageListenerContainer mlc = (DefaultMessageListenerContainer) reg.getListenerContainer("listenerTwo");
For testing, I check JMSXDeliveryCount and throw an exception to rollback.
retryCount = Integer.parseInt(msg.getStringProperty("JMSXDeliveryCount"));
if (retryCount < 5) {
throw new Exception("Rollback test "+retryCount);
}
In the Listener's catch processing, I call stop(Runnable callback) on the DefaultMessageListenerContainer instance and pass in a new class ContainerTimedRestart as defined below.
//catch processing here and decide to rollback
mlc.stop(new ContainerTimedRestart(mlc,delay));
System.out.println("#### "+getClass().getName()+" Unable to process message.");
throw new Exception();
ContainerTimedRestart extends Runnable and DefaultMessageListenerContainer is responsible for invoking the run() method when the stop call completes.
public class ContainerTimedRestart implements Runnable {
//Container instance to restart.
private DefaultMessageListenerContainer theMlc;
//Default delay before restart in mills.
private long theDelay = 5000L;
//Basic constructor for testing.
public ContainerTimedRestart(DefaultMessageListenerContainer mlc, long delay) {
theMlc = mlc;
theDelay = delay;
}
public void run(){
//Validate container instance.
try {
System.out.println("#### "+getClass().getName()+"Waiting for "+theDelay+" millis.");
Thread.sleep(theDelay);
System.out.println("#### "+getClass().getName()+"Restarting container.");
theMlc.start();
System.out.println("#### "+getClass().getName()+"Container started!");
} catch (InterruptedException ie) {
ie.printStackTrace();
//Further checks and ensure container is in correct state.
//Report errors.
}
}
I loaded my queue with three messages with payloads "a", "b", and "c" respectively and started the listener.
Checking DEV.QUEUE.2 on my queue manager I see IPPROCS(1) confirming only one application handle has the queue open. The messages are processed in order after each is rolled five times and with a 5 second delay between rollback attempts.
IBM MQ classes for JMS has poison message handling built in. This handling is based on the QLOCAL setting BOTHRESH, this stands for Backout Threshold. Each IBM MQ message has a "header" called the MQMD (MQ Message Descriptor). One of the fields in the MQMD is BackoutCount. The default value of BackoutCount on a new message is 0. Each time a message rolled back to the queue this count is incremented by 1. A rollback can be either from a specific call to rollback(), or due to the application being disconnected from MQ before commit() is called (due to a network issue for example or the application crashing).
Poison message handling is disabled if you set BOTHRESH(0).
If BOTHRESH is >= 1, then poison message handling is enabled and when IBM MQ classes for JMS reads a message from a queue it will check if the BackoutCount is >= to the BOTHRESH. If the message is eligible for poison message handling then it will be moved to the queue specified in the BOQNAME attribute, if this attribute is empty or the application does not have access to PUT to this queue for some reason, it will instead attempt to put the message to the queue specified in the queue managers DEADQ attribute, if it can't put to either of these locations it will be rolled back to the queue.
You can find more detailed information on IBM MQ classes for JMS poison message handling in the IBM MQ v9.1 Knowledge Center page Developing applications>Developing JMS and Java applications>Using IBM MQ classes for JMS>Writing IBM MQ classes for JMS applications>Handling poison messages in IBM MQ classes for JMS
In Spring JMS you can define your own container. One container is created for one Jms Destination. We should run a single-threaded JMS listener to maintain the message ordering, to make this work set the concurrency to 1.
We can design our container to return null once it encounters errors, post-failure all receive calls should return null so that no messages are polled from the destination till the destination is active once again. We can maintain an active state using a timestamp, that could be simple milliseconds. A sample JMS config should be sufficient to add backoff. You can add small sleep instead of continuously returning null from receiveMessage method, for example, sleep for 10 seconds before making the next call, this will save some CPU resources.
#Configuration
#EnableJms
public class JmsConfig {
#Bean
public JmsListenerContainerFactory<?> jmsContainerFactory(ConnectionFactory connectionFactory,
DefaultJmsListenerContainerFactoryConfigurer configurer) {
DefaultJmsListenerContainerFactory factory = new DefaultJmsListenerContainerFactory() {
#Override
protected DefaultMessageListenerContainer createContainerInstance() {
return new DefaultMessageListenerContainer() {
private long deactivatedTill = 0;
#Override
protected Message receiveMessage(MessageConsumer consumer) throws JMSException {
if (deactivatedTill < System.currentTimeMillis()) {
return receiveFromConsumer(consumer, getReceiveTimeout());
}
logger.info("Disabled due to failure :(");
return null;
}
#Override
protected void doInvokeListener(MessageListener listener, Message message)
throws JMSException {
try {
super.doInvokeListener(listener, message);
} catch (Exception e) {
handleException(message);
throw e;
}
}
private long getDelay(int retryCount) {
if (retryCount <= 1) {
return 20;
}
return (long) (20 * Math.pow(2, retryCount));
}
private void handleException(Message msg) throws JMSException {
if (msg.propertyExists("JMSXDeliveryCount")) {
int retryCount = msg.getIntProperty("JMSXDeliveryCount");
deactivatedTill = System.currentTimeMillis() + getDelay(retryCount);
}
}
#Override
protected void doInvokeListener(SessionAwareMessageListener listener, Session session,
Message message)
throws JMSException {
try {
super.doInvokeListener(listener, session, message);
} catch (Exception e) {
handleException(message);
throw e;
}
}
};
}
};
// This provides all boot's default to this factory, including the message converter
configurer.configure(factory, connectionFactory);
// You could still override some of Boot's default if necessary.
return factory;
}
}
I use xms.net 8.0.0.8 and I want to start multithread xms listener in web application.
I start processmessage code with using new Thread t=new Thread()...
but something goes wrong,and threads are stuck and not read message?If someone has multithread xms sample for web application can share or tell my mistake?Or is this any bug for multithread in xms?
public void ProcessMessage()
{
try
{
ConnectionFactory = WsHelper.CreateConnectionFactoryWMQ();
Connection = WsHelper.CreateConnection(ConnectionFactory);
if (QueueMessage.MessageCallType == MessageCallType.WithoutControl)
Session = WsHelper.CreateSessionWithAutoAcknowledge(Connection);
else
Session = WsHelper.CreateSessionWithSessionTransaction(Connection);
Destination = WsHelper.CreateDestination(Session, QueueMessage.QueueName);
MessageConsumer = WsHelper.CreateConsumer(Session, Destination, QueueMessage);
MessageListener messageListener = new MessageListener(NewMessageProcess);
MessageConsumer.MessageListener = messageListener;
this.Connection.Start();
while (true)
{
if (stopping)
{
MessageConsumer.Close();
return;
}
Thread.Sleep(1000);
}
}
catch(ThreadAbortException ex)
{
throw ex;
}
catch (Exception ex)
{
throw ex;
}
}
private void NewMessageProcess(IBM.XMS.IMessage msg)
{
None of that looks right. Multi-threading requires in-depth programming knowledge. Where is your ThreadStart? Make sure each thread is performing its own connection to the queue manager. Also, are you defining 1 MessageListener for all threads or does each thread have its own MessageListener? Because it looks like you are using 1 for all threads which defeats multi-threading.
Did you read the MQ Knowledge Center on MessageListener and how to handle mutlit-threading:
https://www.ibm.com/support/knowledgecenter/en/SSFKSJ_9.0.0/com.ibm.mq.xms.doc/xms_cmesdel_async.htm
Here is my scenario. I have few ActiveMQ (JBoss-AMQ) producers and consumers installed as services. In a server restart, what is the best practice of handling such a situation where a producer or a consumer service starts before the ActiveMQ broker service. In that case producer/client cannot establish a connection and starts to hang on as it is even after the broker service starts.
here's my code snippet of connection creation:
try {
connection = connectionFactory.createConnection();
connection.start();
LOGGER.info(STARTED_CONNECTION_WITH_THE_DESTINATION + destinationName);
session = createSession();
destination = session.createQueue(destinationName);
LOGGER.info(CREATED_QUEUE_IN_DESTINATION + destinationName);
if (isImageProcAgent) {
consumer = createConsumer();
LOGGER.info(CONSUMER_HAS_BEEN_INITIALIZED);
} else {
producer = session.createProducer(destination);
LOGGER.info(PRODUCER_HAS_BEEN_INITIALIZE);
}
} catch (MessagingException e) {
LOGGER.error(e);
} catch (JMSException e) {
LOGGER.error(e);
}
I'm new to JMS so appreciate your support.
This can be achieved by configuring a failover as this document explains.
according to my code snippet, the change I required it:
destination = session.createQueue("failover:"+destinationName);
producer = session.createProducer("failover:"+destination);