Single queue: concurrent message processing with multiple consumers - jms

I am new to jms. The goal is to process messages concurrently from a queue in an asynchronous listener's onMessage method by attaching a listener instance to multiple consumer's with each consumer using its own session and running in a separate thread, that way the messages are passed on to the different consumers for concurrent processing.
1) Is it possible to process messages concurrently from a single queue by creating multiple consumers ?
2) I came up with the below code, but would like to get your thoughts on whether the below code looks correct for what I want to accomplish.
public class QueueConsumer implements Runnable, MessageListener {
public static void main(String[] args) {
QueueConsumer consumer1 = new QueueConsumer();
QueueConsumer consumer2 = new QueueConsumer();
try {
consumer1.init("oms", "US.Q.CHECKOUT-ORDER.1.0.JSON");
consumer2.init("oms","US.Q.CHECKOUT-ORDER.1.0.JSON");
} catch (JMSException ex) {
ex.printStackTrace();
System.exit(-1);
}
Thread newThread1 = new Thread(consumer1);
Thread newThread2 = new Thread(consumer1);
newThread1.start();
newThread2.start();
}
private static String connectionFactoryName = null;
private static String queueName = null;
private static ConnectionFactory qcf = null;
private static Connection queueConnection = null;
private Session ses = null;
private Destination queue = null;
private MessageConsumer msgConsumer = null;
public static final Logger logger = LoggerFactory
.getLogger(QueueConsumer.class);
public QueueConsumer() {
super();
}
public void onMessage(Message msg) {
if (msg instanceof TextMessage) {
try {
//process message
} catch (JMSException ex) {
ex.printStackTrace();
}
}
}
public void run() {
try {
queueConnection.start();
} catch (JMSException e) {
e.printStackTrace();
System.exit(-1);
}
while (!Thread.currentThread().isInterrupted()) {
synchronized (this) {
try {
wait();
} catch (InterruptedException ex) {
break;
}
}
}
}
public void init(String factoryName, String queue2) throws JMSException {
try {
qcf = new JMSConnectionFactory(factoryName);
queueConnection = qcf.createConnection();
ses = queueConnection.createSession(false,
Session.CLIENT_ACKNOWLEDGE);
queue = ses.createQueue(queue2);
logger.info("Subscribing to destination: " + queue2);
msgConsumer = ses.createConsumer(queue);
msgConsumer.setMessageListener(this);
System.out.println("Listening on queue " + queue2);
} catch (Exception e) {
e.printStackTrace();
System.exit(-1);
}
}
private static void setConnectionFactoryName(String name) {
connectionFactoryName = name;
}
private static String getQueueName() {
return queueName;
}
private static void setQueueName(String name) {
queueName = name;
}
}

Yes absolutely
I only took a brief look and I noticed that you pass the wrong consumer to your second thread:
Thread newThread2 = new Thread(consumer1); // has to pass consumer2
beside of this, some variables such as ConnectionFactory are static and initialized multiple times/overriden. You only need one connection that could create multiple sessions and/or consumers.

Related to the code example you provided, It is not recommanded by Oracle to create low-level threads on a deployed application. Example for Weblogic :
Using Threads in WebLogic Server

Instead in the applicationcontext.xml, where you have made the bean of mail container, you can add concurrent consumer property which would be a better approach.
<bean id="jmsMailContainer" class="org.springframework.jms.listener.DefaultMessageListenerContainer">
<property name="concurrentConsumers">
<value>100</value>
</property>
<property name="connectionFactory" ref="connectionFactory"/>
<property name="destination" ref="mailDestination"/>
<property name="messageListener" ref="jmsMailConsumer"/>

Related

How to properly configure multiple DMLCs to listen to a single sqs queue?

We have an order managament system in which after every order state update we make an api call to our client to keep them updated. We do this by first sending a message to a sqs queue and inside a consumer we hit our clients api. The processing on consumer side usually takes about 300-350ms but The approximate age of oldest message in sqs dashboard is showing spikes that reach upto 50-60 secs.
Seeing this I thought that maybe one consumer is not enough for our load and I created multiple DMLC beans and multiple copies of our consumer class. I attached these consumer classes as listeners in these DMLCs. But I have not seen any improvement in approximate age of oldest message.
I am guessing that maybe only one of the DMLC is processing these messages and others are just sitting idle.
I added multiple DMLCs because there are other places in pur codebase where the same thing is used, But now I am not sure if this is the correct way to solve the problem.
My Consumer class looks like this:
#Component
#Slf4j
#RequiredArgsConstructor
public class HOAEventsOMSConsumer extends ConsumerCommon implements MessageListener {
private static final int MAX_RETRY_LIMIT = 3;
private final OMSEventsWrapper omsEventsWrapper;
#Override
public void onMessage(Message message) {
try {
TextMessage textMessage = (TextMessage) message;
String jmsMessageId = textMessage.getJMSMessageID();
ConsumerLogging.logStart(jmsMessageId);
String text = textMessage.getText();
log.info(
"Inside HOA Events consumer Request jmsMessageId:- " + jmsMessageId + " Text:- "
+ text);
processAndAcknowledge(message, text, textMessage);
} catch (JMSException e) {
log.error("JMS Exception while processing surge message", e);
}
}
private void processAndAcknowledge(Message message, String text, TextMessage textMessage) throws JMSException {
try {
TrimmedHOAEvent hoaEvent = JsonHelper.convertFromJsonPro(text, TrimmedHOAEvent.class);
if (hoaEvent == null) {
throw new OMSValidationException("Empty message in hoa events queue");
}
EventType event = EventType.fromString(textMessage.getStringProperty("eventType"));
omsEventsWrapper.handleOmsEvent(event,hoaEvent);
acknowledgeMessage(message);
} catch (Exception e) {
int retryCount = message.getIntProperty("JMSXDeliveryCount");
log.info("Retrying... retryCount: {}, HOAEventsOMSConsumer: {}", retryCount, text);
if (retryCount > MAX_RETRY_LIMIT) {
log.info("about to acknowledge the message since it has exceeded maximum retry limit");
acknowledgeMessage(message);
}
}
}
}
And my DMLC configuration class looks like this:
#Configuration
#SuppressWarnings("unused")
public class HOAEventsOMSJMSConfig extends JMSConfigCommon{
private Boolean isSQSQueueEnabled;
#Autowired
private HOAEventsOMSConsumer hoaEventsOMSConsumer;
#Autowired
private HOAEventsOMSConsumer2 hoaEventsOMSConsumer2;
#Autowired
private HOAEventsOMSConsumer3 hoaEventsOMSConsumer3;
#Autowired
private HOAEventsOMSConsumer4 hoaEventsOMSConsumer4;
#Autowired
private HOAEventsOMSConsumer5 hoaEventsOMSConsumer5;
#Autowired
private HOAEventsOMSConsumer6 hoaEventsOMSConsumer6;
#Autowired
private HOAEventsOMSConsumer7 hoaEventsOMSConsumer7;
#Autowired
private HOAEventsOMSConsumer8 hoaEventsOMSConsumer8;
#Autowired
private HOAEventsOMSConsumer9 hoaEventsOMSConsumer9;
#Autowired
private HOAEventsOMSConsumer10 hoaEventsOMSConsumer10;
public HOAEventsOMSJMSConfig(IPropertyService propertyService, Environment env) {
queueName = env.getProperty("aws.sqs.queue.oms.hoa.events.queue");
endpoint = env.getProperty("aws.sqs.queue.endpoint") + queueName;
JMSConfigCommon.accessId = env.getProperty("aws.sqs.access.id");
JMSConfigCommon.accessKey = env.getProperty("aws.sqs.access.key");
try {
ServerNameCache serverNameCache = CacheManager.getInstance().getCache(ServerNameCache.class);
if (serverNameCache == null) {
serverNameCache = new ServerNameCache();
serverNameCache.set(InetAddress.getLocalHost().getHostName());
CacheManager.getInstance().setCache(serverNameCache);
}
this.isSQSQueueEnabled = propertyService.isConsumerEnabled(serverNameCache.get(), false);
} catch (Exception e) {
this.isSQSQueueEnabled = false;
}
}
#Bean
public JmsTemplate omsHOAEventsJMSTemplate(){
SQSConnectionFactory sqsConnectionFactory;
if (endpoint.toLowerCase().contains("localhost")) {
sqsConnectionFactory =
SQSConnectionFactory.builder().withEndpoint(getEndpoint("sqs")).build();
} else {
sqsConnectionFactory = SQSConnectionFactory.builder()
.withAWSCredentialsProvider(awsCredentialsProvider)
.withNumberOfMessagesToPrefetch(10)
.withEndpoint(endpoint)
.build();
}
CachingConnectionFactory cachingConnectionFactory = new CachingConnectionFactory(sqsConnectionFactory);
JmsTemplate jmsTemplate = new JmsTemplate(cachingConnectionFactory);
jmsTemplate.setDefaultDestinationName(queueName);
jmsTemplate.setDeliveryPersistent(false);
jmsTemplate.setSessionTransacted(false);
jmsTemplate.setSessionAcknowledgeMode(SQSSession.UNORDERED_ACKNOWLEDGE);
return jmsTemplate;
}
#Bean
public DefaultMessageListenerContainer jmsListenerHOAEventsListenerContainer() {
SQSConnectionFactory sqsConnectionFactory;
if (endpoint.toLowerCase().contains("localhost")) {
sqsConnectionFactory = SQSConnectionFactory.builder()
.withEndpoint(getEndpoint("sqs"))
.build();
} else {
sqsConnectionFactory = SQSConnectionFactory.builder()
.withAWSCredentialsProvider(awsCredentialsProvider)
.withNumberOfMessagesToPrefetch(10)
.withEndpoint(endpoint)
.build();
}
DefaultMessageListenerContainer dmlc = new DefaultMessageListenerContainer();
dmlc.setConnectionFactory(sqsConnectionFactory);
dmlc.setDestinationName(queueName);
dmlc.setAutoStartup(isSQSQueueEnabled);
dmlc.setMessageListener(hoaEventsOMSConsumer);
dmlc.setSessionTransacted(false);
dmlc.setSessionAcknowledgeMode(SQSSession.UNORDERED_ACKNOWLEDGE);
return dmlc;
}
#Bean
public DefaultMessageListenerContainer jmsListenerHOAEventsListenerContainerNo2() {
SQSConnectionFactory sqsConnectionFactory;
if (endpoint.toLowerCase().contains("localhost")) {
sqsConnectionFactory = SQSConnectionFactory.builder()
.withEndpoint(getEndpoint("sqs"))
.build();
} else {
sqsConnectionFactory = SQSConnectionFactory.builder()
.withAWSCredentialsProvider(awsCredentialsProvider)
.withNumberOfMessagesToPrefetch(10)
.withEndpoint(endpoint)
.build();
}
DefaultMessageListenerContainer dmlc = new DefaultMessageListenerContainer();
dmlc.setConnectionFactory(sqsConnectionFactory);
dmlc.setDestinationName(queueName);
dmlc.setAutoStartup(isSQSQueueEnabled);
dmlc.setMessageListener(hoaEventsOMSConsumer2);
dmlc.setSessionTransacted(false);
dmlc.setSessionAcknowledgeMode(SQSSession.UNORDERED_ACKNOWLEDGE);
return dmlc;
}
#Bean
public DefaultMessageListenerContainer jmsListenerHOAEventsListenerContainerNo3() {
SQSConnectionFactory sqsConnectionFactory;
if (endpoint.toLowerCase().contains("localhost")) {
sqsConnectionFactory = SQSConnectionFactory.builder()
.withEndpoint(getEndpoint("sqs"))
.build();
} else {
sqsConnectionFactory = SQSConnectionFactory.builder()
.withAWSCredentialsProvider(awsCredentialsProvider)
.withNumberOfMessagesToPrefetch(10)
.withEndpoint(endpoint)
.build();
}
DefaultMessageListenerContainer dmlc = new DefaultMessageListenerContainer();
dmlc.setConnectionFactory(sqsConnectionFactory);
dmlc.setDestinationName(queueName);
dmlc.setAutoStartup(isSQSQueueEnabled);
dmlc.setMessageListener(hoaEventsOMSConsumer3);
dmlc.setSessionTransacted(false);
dmlc.setSessionAcknowledgeMode(SQSSession.UNORDERED_ACKNOWLEDGE);
return dmlc;
}
}
If this question is already answered somehwere else, then please point me towards that.

Stop RabbitMQ-Connection in Spring-Boot

I have a spring-boot application that pulls all the messages from a RabbitMQ-queue and then terminates. I use rabbitTemplate from the package spring-boot-starter-amqp (version 2.4.0), namely receiveAndConvert(). Somehow, I cannot get my application to start and stop again. When the rabbitConnectionFactory is created, it will never stop.
According to Google and other stackoverflow-questions, calling stop() or destroy() on the rabbitTemplate should do the job, but that doesn't work.
The rabbitTemplate is injected in the constructor.
Here is some code:
rabbitTemplate.setMessageConverter(new Jackson2JsonMessageConverter());
Object msg = getMessage();
while (msg != null) {
try {
String name = ((LinkedHashMap) msg).get(propertyName).toString();
//business logic
logger.debug("added_" + name);
} catch (Exception e) {
logger.error("" + e.getMessage());
}
msg = getMessage();
}
rabbitTemplate.stop();
private Object getMessage() {
try {
return rabbitTemplate.receiveAndConvert(queueName);
} catch (Exception e) {
logger.error("" + e.getMessage());
return null;
}
}
So, how do you terminate the connection to RabbitMQ properly?
Thanks for your inquiry.
You can call resetConnection() on the CachingConnectionFactory to close the connection.
Or close() the application context.
If I were to do it , I would use #RabbitListener to receive the messages and RabbitListenerEndpointRegistry to start and stop the listener. Sample Code is given below
#EnableScheduling
#SpringBootApplication
public class Application implements ApplicationRunner {
public static void main(String[] args) {
SpringApplication.run(Application.class, args);
}
public static final String queueName = "Hello";
#Bean
public Queue hello() {
return new Queue(queueName);
}
#Autowired
private RabbitTemplate template;
#Scheduled(fixedDelay = 1000, initialDelay = 500)
public void send() {
String message = "Hello World!";
this.template.convertAndSend(queueName, message);
System.out.println(" [x] Sent '" + message + "'");
}
#Autowired
RabbitListenerEndpointRegistry registry;
#Override
public void run(ApplicationArguments args) throws Exception {
registry.getListenerContainer( Application.queueName).start();
Thread.sleep(10000L);
registry.getListenerContainer( Application.queueName).stop();
}
}
#Component
class Receiver {
#RabbitListener(id= Application.queueName,queues = Application.queueName)
public void receive(String in) {
System.out.println(" [x] Received '" + in + "'");
}
}

Why spring bean can't be injected in quartz job?

I am now writing a quartz job to start at 3:00 am every day to save data in database and redis. But now I got a problem and can't get it solved even though I tried a lot of times.
The job:
public class QuartzScheduler implements ServletContextListener {
private static Logger logger = Logger.getLogger(QuartzScheduler.class);
#Autowired
private ExchangeRateConfigBean exchangeRateConfigBean;
private Scheduler scheduler = null;
#Override
public void contextInitialized(ServletContextEvent servletContext) {
try {
ExchangeRateConfig exchangeRateConfig = exchangeRateConfigBean.setUpConfigurationFromConfigFile();
if(!exchangeRateConfig.getJob_fire()) {
logger.info("ExchangeRate Job Info: Current Exchange Rate Job flag is not open. Wont start the job!");
return;
}
} catch (Exception e) {
logger.error("ExchangeRate Job Info: Something wrong when analyzing config.properties file!");
logger.error(e.getMessage());
logger.error(e.getStackTrace());
logger.error(e.getCause());
return;
}
try {
scheduler = StdSchedulerFactory.getDefaultScheduler();
scheduler.start();
logger.info("ExchangeRate Job Info: Exchange Rate job starting......");
} catch (SchedulerException ex) {
logger.error(ex);
}
}
#Override
public void contextDestroyed(ServletContextEvent servletContextEvent) {
try {
scheduler.shutdown();
} catch (SchedulerException e) {
logger.error(e);
}
}
In this job , I autowired the component ExchangeRateConfigBean, but when I debugged in, I saw that the instance is null.
the spring config below:
<bean id="exchangeRateConfigBean" class="com.letv.exchangerate.bean.impl.ExchangeRateConfigBeanImpl">
<constructor-arg ref="exchangeRateErrorBean" />
</bean>
<bean id="exchangeRateErrorBean" class="com.letv.exchangerate.bean.impl.ExchangeRateErrorBeanImpl">
</bean>
The config Bean below:
public class ExchangeRateConfigBeanImpl implements ExchangeRateConfigBean {
public ExchangeRateConfigBeanImpl(){}
public ExchangeRateConfigBeanImpl(ExchangeRateErrorBean exchangeRateErrorBean)
{
this.exchangeRateErrorBean = exchangeRateErrorBean;
}
private static Logger logger = Logger.getLogger(ExchangeRateConfigBeanImpl.class.getClass());
private ExchangeRateErrorBean exchangeRateErrorBean;
#Override
public ExchangeRateConfig setUpConfigurationFromConfigFile() throws IOException {
InputStream inputStream = null;
ExchangeRateConfig exchangeRateConfig = null;
try {
Properties prop = new Properties();
String propFileName = "config.properties";
inputStream = getClass().getClassLoader().getResourceAsStream(propFileName);
exchangeRateConfig = new ExchangeRateConfig();
if (inputStream != null) {
prop.load(inputStream);
} else {
logger.error("property file '" + propFileName + "' not found in the classpath");
throw new FileNotFoundException("property file '" + propFileName + "' not found in the classpath");
}
String job_fire_tmp = prop.getProperty("job_fire");
exchangeRateConfig.setJob_fire(job_fire_tmp.isEmpty() ? false : Boolean.parseBoolean(job_fire_tmp));
return exchangeRateConfig;
} catch (IOException e) {
logger.error(e.getStackTrace());
return exchangeRateConfig;
} finally {
inputStream.close();
}
}
In the config bean, I used the constructor injection to inject the error bean above. the error bean's code lists below:
public class ExchangeRateErrorBeanImpl implements ExchangeRateErrorBean{
public ExchangeRateErrorBeanImpl(){}
#Override
public ExchangeRateError getExchangeRateError(String content) {
if (content.isEmpty())
return null;
if (!content.contains(":"))
return null;
String[] strArray = content.split(":");
return new ExchangeRateError(strArray[0], strArray[1]);
}
}
Anyone can help? why I autowired the Config bean in the job class, it will create null instance? thx.
And before I post this, I have read this in detail, Why is my Spring #Autowired field null?
but it didn't save my time.
Your code shows QuartzScheduler as an instance of ServletContextListener. This will only create a Java bean. Your Quartz Scheduler class must be a bean in a Spring applicationContext. Only then will the other dependencies be autowired. Refer this for example - http://docs.spring.io/spring/docs/current/spring-framework-reference/html/scheduling.html#scheduling-quartz

How to subscribe to an existing advisory topic?

I have activemq5.3.2 running and I wanted to subscribe existing advisory topics using my java program. while, `jndi` lookup I am getting following error:
javax.naming.NoInitialContextException: Need to specify class name in environment or system property, or as an applet parameter, or in an application resource file:
java.naming.factory.initial
at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:657)
at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:259)
at javax.naming.InitialContext.getURLOrDefaultInitCtx(InitialContext.java:296)
at javax.naming.InitialContext.lookup(InitialContext.java:363)
at jmsclient.Consumer.<init>(Consumer.java:38)
at jmsclient.Consumer.main(Consumer.java:74)
Exception occurred: javax.jms.InvalidDestinationException: Don't understand null destinations
Please suggest where the problem is, or how could I use my topic name to look for?
package jmsclient;
import javax.jms.*;
import javax.naming.Context;
import javax.naming.InitialContext;
import javax.naming.NamingException;
import org.apache.activemq.ActiveMQConnectionFactory;
public class Consumer implements MessageListener {
private static int ackMode;
private static String clientTopicName;
private boolean transacted = false;
//private MessageConsumer messageConsumer;
static {
clientTopicName = "ActiveMQ.Advisory.Consumer.Queue.example.A";
ackMode = Session.AUTO_ACKNOWLEDGE;
}
#SuppressWarnings("null")
public Consumer()
{// TODO Auto-generated method stub
TextMessage message = null;
Context jndiContext;
//TopicConnectionFactory topicConnectionFactory = null;
TopicConnection topicConnection = null;
TopicSession topicSession = null;
ActiveMQConnectionFactory connectionFactory = new ActiveMQConnectionFactory("tcp://usaxwas012ccxra.ccmp.ibm.lab:61616");
try{
Topic myTopic = null;
try { jndiContext = new InitialContext();
myTopic = (Topic) jndiContext.lookup(clientTopicName);
} catch (NamingException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
topicConnection = connectionFactory.createTopicConnection();
topicConnection.start();
topicSession = topicConnection.createTopicSession(transacted, ackMode);
TopicSubscriber topicSubscriber = topicSession.createSubscriber(myTopic);
Message m = topicSubscriber.receive(1000);
if (m != null) {
if (m instanceof TextMessage) {
message = (TextMessage) m;
System.out.println("Reading message: " + message.getText());
}
}
} //try ends
catch (JMSException e) {
System.out.println("Exception occurred: " + e.toString());
} finally {
if (topicConnection != null) {
try {
topicConnection.close();
} catch (JMSException e) {}
}}}
public void onMessage(Message arg0) {
// TODO Auto-generated method stub
}
public static void main(String[] args) {
new Consumer();
}
}

How to create an active-mq ExceptionListener?

I have a project coded using Spring-hibernate-activeMq
What I would like to know is if I configured activeMq like I explained below, how should I implement the exception listener class of it? I know you don't understand well now but please give a look to my samples below.
Let me know if I implemented exception listener right or not. If not, please give an example how it must be. Thanks in advance.
Application context: (note that I didn't declare any bean for exception listener except the one the property of connectionFactory)
<bean id="connectionFactory"
class="org.springframework.jms.connection.CachingConnectionFactory"
depends-on="broker">
<constructor-arg ref="amqConnectionFactory"/>
<property name="reconnectOnException" value="true"/>
<property name="exceptionListener" ref="jmsExceptionListener"/>
<property name="sessionCacheSize" value="100"/>
</bean>
Jms exception listener class: (Note that I am trying to inject ConnectionFactory, I am not sure whether it is possible or not.. And the last thing, please check the constructor arguments of it, I am also not sure of it..)
#Component("jmsExceptionListener")
public class JMSExceptionListener implements ExceptionListener {
private final static Logger logger = LoggerFactory.getLogger(JMSExceptionListener.class);
#Autowired
private CachingConnectionFactory connection;
// private Connection connection = null;
private ExceptionListener exceptionListener = null;
public JMSExceptionListener() {
}
public JMSExceptionListener(CachingConnectionFactory connection, ExceptionListener exceptionListener) {
super();
this.connection = connection;
this.exceptionListener = exceptionListener;
}
public void onException(JMSException arg0) {
logger.error("JMS exception has occured.. ", arg0);
if(connection != null){
connection.onException(arg0);
}
if (exceptionListener != null) {
exceptionListener.onException(arg0);
}
}
}
I have manipulated exceptionListener class like the following:
public class JmsExceptionListener implements ExceptionListener {
private final static Logger logger = LoggerFactory.getLogger(JmsExceptionListener.class);
#Autowired
private MailService mailService;
private ExceptionListener exceptionListener = null;
private CachingConnectionFactory cachingConnectionFactory;
public JmsExceptionListener() {
}
public JmsExceptionListener(Connection connection, ExceptionListener exceptionListener) {
super();
this.exceptionListener = exceptionListener;
}
public synchronized void onException(JMSException e) {
logger.error("JMS exception has occurred: ", e);
sendErrorNotificationMail(e);
Exception ex = e.getLinkedException();
if (ex != null) {
logger.error("JMS Linked exception: ", ex);
}
if (exceptionListener != null) {
exceptionListener.onException(e);
}
}
public CachingConnectionFactory getCachingConnectionFactory() {
return cachingConnectionFactory;
}
public void setCachingConnectionFactory(CachingConnectionFactory cachingConnectionFactory) {
this.cachingConnectionFactory = cachingConnectionFactory;
}
private void sendErrorNotificationMail(Exception e) {
try {
mailService.sendJmsExceptionMail(e, ErrorMessageAccessor.get("core.jms.unexpected"));
} catch (ElekBusinessException e1) {
logger.error(ErrorMessageAccessor.get("generic.mailService.exp"), e);
}
}
}

Resources