I am learning JMS and came across this statement: http://docs.oracle.com/javaee/1.3/jms/tutorial/1_3_1-fcs/doc/advanced.html#1023387
The PERSISTENT delivery mode, the default, instructs the JMS provider
to take extra care to ensure that a message is not lost in transit in
case of a JMS provider failure. A message sent with this delivery mode
is logged to stable storage when it is sent.
If JMS Provider failure occurs then how the JMS Provider can ensure that a message is not lost?
What does it mean that:
"A message sent with this delivery mode is logged to stable storage when it is sent."
Please help me in understanding the JMS concept here.
It means the message with PERSISTENT delivery mode is not lost when a messaging provider goes down for any reason and comes up again. The messaging provider saves messages with PERSISTENT delivery mode to disk and when the message provides restarts, the message is read from the disk and brought into memory.
Hope this is clear.
You can do a simple test to understand the concept. Refer the tutorial here on how to create producer and consumer.
You will see producer.setDeliveryMode(DeliveryMode.NON_PERSISTENT);
Change it to producer.setDeliveryMode(DeliveryMode.PERSISTENT);
Now create two classes. One which calls only Producers and one only consumer.
public class AppOnlyProduce {
public static void thread(Runnable runnable, boolean daemon) {
Thread brokerThread = new Thread(runnable);
brokerThread.setDaemon(daemon);
brokerThread.start();
}
public static void main(String[] args) throws InterruptedException {
thread(new HelloWorldProducer(), false);
thread(new HelloWorldProducer(), false);
}
}
public class AppOnlyConsumer {
public static void thread(Runnable runnable, boolean daemon) {
Thread brokerThread = new Thread(runnable);
brokerThread.setDaemon(daemon);
brokerThread.start();
}
public static void main(String[] args) throws InterruptedException {
thread(new HelloWorldConsumer(), false);
thread(new HelloWorldConsumer(), false);
}
}
First run AppOnlyProduce. It will create two messages. Now run AppOnlyConsumer it will read two messages.
Now change back the line to producer.setDeliveryMode(DeliveryMode.NON_PERSISTENT);
Again run AppOnlyProduce. It will create two messages. Now run AppOnlyConsumer You will see that it waits for sometime for message and they say Received: null
In first case mode was persistent. So though Java program ended messages were persisted and made available when JMS started (this time by consumer)
In second case mode was not persistent. So messages vanished as soon as program ended.
Related
I implemented a spring batch project that reads from a weblogic Jms queue (Custom Item Reader not message driven), then pass the Jms message data to an item writer (chunk = 1) where i call some APIs and write in DataBase.
However, i am trying to implement parallel Jms processing, reading in parallel Jms messages and passing them to the writer without waiting for the previous processes to complete.
I’ve used a DefaultMessageListenerContainer in a previous project and it offers a parallel consuming of jms messages, but in this project i have to use the spring batch framework.
I tried using the easiest solution (multi-threaded step) but it
didn’t work , JmsException : "invalid blocking receive when another
receive is in progress" which means probably that my reader is
statefull.
I thought about using remote partitioning but then i have to read all
messages and put the data into step execution contexts before calling
the slave steps, which isn't really efficient if dealing with a large
number of messages.
I looked a little bit into remote chunking, i understand that it passes data via queue channels, but i can't seem to find the utility in reading from a Jms and putting messages in a local queue for slave workers.
How can I approach this?
My code:
#Bean
Step step1() {
return steps.get("step1").<Message, DetectionIncoherenceLiqJmsOut>chunk(1)
.reader(reader()).processor(processor()).writer(writer())
.listener(stepListener()).build();
}
#Bean
Job job(#Qualifier("step1") Step step1) {
return jobs.get("job").start(step1).build();
}
Jms Code :
#Override
public void initQueueConnection() throws NamingException, JMSException {
Hashtable<String, String> properties = new Hashtable<String, String>();
properties.put(Context.INITIAL_CONTEXT_FACTORY, env.getProperty(WebLogicConstant.JNDI_FACTORY));
properties.put(Context.PROVIDER_URL, env.getProperty(WebLogicConstant.JMS_WEBLOGIC_URL_RECEIVE));
InitialContext vInitialContext = new InitialContext(properties);
QueueConnectionFactory vQueueConnectionFactory = (QueueConnectionFactory) vInitialContext
.lookup(env.getProperty(WebLogicConstant.JMS_FACTORY_RECEIVE));
vQueueConnection = vQueueConnectionFactory.createQueueConnection();
vQueueConnection.start();
vQueueSession = vQueueConnection.createQueueSession(false, 0);
Queue vQueue = (Queue) vInitialContext.lookup(env.getProperty(WebLogicConstant.JMS_QUEUE_RECEIVE));
consumer = vQueueSession.createConsumer(vQueue, "JMSCorrelationID IS NOT NULL");
}
#Override
public Message receiveMessages() throws NamingException, JMSException {
return consumer.receive(20000);
}
Item reader :
#Override
public Message read() throws Exception {
return jmsServiceReceiver.receiveMessages();
}
Thanks ! i'll appreciate the help :)
There's a BatchMessageListenerContainer in the spring-batch-infrastructure-tests sub project.
https://github.com/spring-projects/spring-batch/blob/d8fc58338d3b059b67b5f777adc132d2564d7402/spring-batch-infrastructure-tests/src/main/java/org/springframework/batch/container/jms/BatchMessageListenerContainer.java
Message listener container adapted for intercepting the message reception with advice provided through configuration.
To enable batching of messages in a single transaction, use the TransactionInterceptor and the RepeatOperationsInterceptor in the advice chain (with or without a transaction manager set in the base class). Instead of receiving a single message and processing it, the container will then use a RepeatOperations to receive multiple messages in the same thread. Use with a RepeatOperations and a transaction interceptor. If the transaction interceptor uses XA then use an XA connection factory, or else the TransactionAwareConnectionFactoryProxy to synchronize the JMS session with the ongoing transaction (opening up the possibility of duplicate messages after a failure). In the latter case you will not need to provide a transaction manager in the base class - it only gets on the way and prevents the JMS session from synchronizing with the database transaction.
Perhaps you could adapt it for your use case.
I was able to do so with a multithreaded step :
// Jobs et Steps
#Bean
Step stepDetectionIncoherencesLiq(#Autowired StepBuilderFactory steps) {
int threadSize = Integer.parseInt(env.getProperty(PropertyConstant.THREAD_POOL_SIZE));
return steps.get("stepDetectionIncoherencesLiq").<Message, DetectionIncoherenceLiqJmsOut>chunk(1)
.reader(reader()).processor(processor()).writer(writer())
.readerIsTransactionalQueue()
.faultTolerant()
.taskExecutor(taskExecutor())
.throttleLimit(threadSize)
.listener(stepListener())
.build();
}
And a jmsItemReader with jmsTemplate instead of creating session and connections explicitly, it manages connections so i dont have the jms exception anymore:( JmsException : "invalid blocking receive when another receive is in progress" )
#Bean
public JmsItemReader<Message> reader() {
JmsItemReader<Message> itemReader = new JmsItemReader<>();
itemReader.setItemType(Message.class);
itemReader.setJmsTemplate(jmsTemplate());
return itemReader;
}
This is the first time i am using Kafka. i have a spring boot application and i am consuming messages from kafka topics and storing messages in DB. I have a requirement to handle DB fail over, if DB is down that message should not be committed and suspend consuming messages for some time and after some time listener can start consuming messages again. what is the better approach to do this.
i am using spring-kafka:2.2.8.RELEASE which is internally using kafka 2.0.1
Configure a ContainerStoppingErrorHandler and throw an exception from your listener.
https://docs.spring.io/spring-kafka/docs/2.2.13.RELEASE/reference/html/#container-stopping-error-handlers
You can restart the container later when you have detected that your DB is back online.
https://docs.spring.io/spring-kafka/docs/2.2.13.RELEASE/reference/html/#kafkalistener-lifecycle
EDIT
#SpringBootApplication
public class So62125817Application {
public static void main(String[] args) {
SpringApplication.run(So62125817Application.class, args);
}
#Bean
TaskScheduler scheduler() {
return new ThreadPoolTaskScheduler();
}
#Bean
public NewTopic topic() {
return TopicBuilder.name("so62125817").partitions(1).replicas(1).build();
}
}
#Component
class Listener {
private final TaskScheduler scheduler;
private final KafkaListenerEndpointRegistry registry;
public Listener(TaskScheduler scheduler, KafkaListenerEndpointRegistry registry,
AbstractKafkaListenerContainerFactory<?, ?, ?> factory) {
this.scheduler = scheduler;
this.registry = registry;
factory.setErrorHandler(new ContainerStoppingErrorHandler());
}
#KafkaListener(id = "so62125817.id", topics = "so62125817")
public void listen(String in) {
System.out.println(in);
// run this code if you want to stop the container and restart it in 60 seconds
this.scheduler.schedule(() -> {
this.registry.getListenerContainer("so62125817.id").start();
}, new Date(System.currentTimeMillis() + 60_000));
throw new RuntimeException("test restart");
}
}
There are two approaches which I can think of doing this:
First Approach: Let the auto-commit option for consuming messages be true. The configuration for this is enable.auto.commit. By default, this would be true, so you do not need to change anything. Whenever your DB operation fails, you can put the messages on a different topic say a topic named failed_events. When you do this, you can have the same application (Which populates the DB) running say once at a daily level to consume the message from failed_events topic and populate the DB again. This way you can keep track of how many times the DB write gets failed. One small thing to note is what if during this run also the DB is down, then what do you do. You can decide what to do in this case. Probably discard the message if it is Ok to do so, or do a certain number of retries.
Second approach: If it is very deterministic to know that for how long the DB would be down. And if the time period is very small, then it is better to do a sleep operation in the case of DB write failure. Say the application sleeps for 10 minutes before it retries again. You will not have to create a separate topic in this case.
The advantage of this approach is that you don't have to run a separate instance of the same application to fetch from a different topic. You could do all of them in one single application. Maintaining this becomes relatively easier.
The disadvantage of this approach is that if the DB is down for a very long period, say 1 day, Then you will end up losing the message.
I figured I would toss a question on here incase anyone has ideas. My MQ Admin created a new queue and alias queue for me to write messages to. I have one application writing to the queue, and another application listening on the alias queue. I am using spring jmsTemplate to write to my queue. We are seeing a behavior where the message is being written to the queue but then instantly being discarded. We disabled gets and to see if an expiry parameter was being set somehow, I used the jms template to set the expiry setting (timeToLive). I set the expiry to 10 minutes but my message still disappears instantly. A snippet of my code and settings are below.
public void publish(ModifyRequestType response) {
jmsTemplate.setExplicitQosEnabled(true);
jmsTemplate.setTimeToLive(600000);
jmsTemplate.send(CM_QUEUE_NAME, new MessageCreator() {
public Message createMessage(Session session) throws JMSException {
String responseXML = null;
try {
responseXML myJAXBContext.getInstance().toXML(response);
log.info(responseXML);
TextMessage message = session.createTextMessage(responseXML);
return message;
} catch (myException e) {
e.printStackTrace();
log.info(responseXML);
return null;
}
}
});
}
/////////////////My settings
QUEUE.PUB_SUB_DOMAIN=false
QUEUE.SUBSCRIPTION_DURABLE=false
QUEUE.CLONE_SUPPORT=0
QUEUE.SHARE_CONV_ALLOWED=1
QUEUE.MQ_PROVIDER_VERSION=6
I found my issue. I had a parent method that had the #Transactional annotation. I do not want my new jms message to be part of that transaction so I am going to add jmsTemplate.setSessionTransacted(false); before performing a jmsTemplate.send. I have created a separate jmsTempalte for sending my new message instead of reusing the one that was existing, which needs to be managed.
We are using JMS to process messages in a Java 1.8 SE environment. The messages originate from an Oracle (12) Advanced Queue.
We would like to read a message from a JMS queue, do some work based on it, and save the result in the database. We don’t want to lose any messages, and we don’t want to duplicate processing on any message. In other words, we’d like the processing of the JMS message and the associated database activity to be a single transaction.
We’ve read various articles about how to do this (including Transaction and redelivery in JMS, JMS Message Delivery Reliability and Acknowledgement Patterns, Reliable JMS with Transactions). The consensus seems to be to use JTA/XA, but we were hoping for something simpler.
We are using Oracle’s Advanced Queueing as our JMS provider, so we decided to see whether we could use the same database connection for both JMS and database activity, so that a single commit would work for both JMS and database activity. It seems to have worked.
In the code below, we create a QueueConnection using an existing SQL Connection when we initialize the JMS queue. After processing the message, committing the JMS session also commits the database changes.
We haven’t seen this approach discussed elsewhere, so we’re wondering if
We have a reliable solution that works for Oracle Advanced
Queueing,
We have a solution that just happens to work some of the
time for this version of Oracle Advanced Queueing,
We just got
really, really lucky on our test cases, and this approach is fraught
with peril
Please comment on whether our approach should be reliable or whether we should use JTA/XA.
public class OracleJmsQueue {
private DataSource dataSource;
protected Queue queue;
protected QueueConnection queueConnection;
protected QueueReceiver queueReceiver;
protected QueueSession queueSession;
private java.sql.Connection dbConnection = null;
protected void initQueueSession()
throws JMSException, SQLException {
// Connect to the database source of messages
DataSource dataSource = getDataSource();
dbConnection = dataSource.getConnection();
dbConnection.setAutoCommit(false);
queueConnection = AQjmsQueueConnectionFactory.createQueueConnection(
dbConnection);
queueSession =
queueConnection.createQueueSession(true, Session.SESSION_TRANSACTED);
queue = ((AQjmsSession)queueSession).getQueue(queueUser, queueName);
queueReceiver = queueSession.createReceiver(queue);
}
public void run() {
initQueueSession();
// code omitted
while (!Thread.currentThread().isInterrupted()) {
try {
Thread.sleep(200);
final Message message = queueReceiver.receiveNoWait();
if (message != null) {
processMessage(message); // alters DB tables
commitSession();
}
}
// catches omitted
}
}
protected void commitSession() throws JMSException {
logger.info("Committing " + queueName + " queue session");
queueSession.commit();
}
} // class OracleJmsQueue
It looks that your assumptions about JMS and OAQ are correct, given that processMessage uses the dbConnection class attribute.
https://docs.oracle.com/javaee/7/api/javax/jms/QueueConnection.html
So, answering your question: Yes, you have a reliable solution (assuming what I mentioned before).
ActiveMQ 5.10.0
Spring 4.1.2
I'm using Spring to access activeMQ and trying to peek at the queue before adding a new message onto the queue. The message is added successfully, but it does not show anything in the queue. Through the web interface, I see my messages are pending in the queue.
Thanks!
#Service
public class MessageQueueService{
private static final Logger logger = LoggerFactory.getLogger(MessageQueueService.class);
#Inject
JmsTemplate jmsTemplate;
#SuppressWarnings({ "rawtypes", "unchecked" })
public void testAddJob(){
jmsTemplate.send(new MessageCreator() {
public Message createMessage(Session session) throws JMSException {
IndexJob j1=new IndexJob();
j1.setOperation("post");
ObjectMessage om=session.createObjectMessage();
om.setObject(j1);
QueueBrowser qb=session.createBrowser((javax.jms.Queue) jmsTemplate.getDefaultDestination());
Enumeration<Message> messages=qb.getEnumeration();
logger.info("browsing "+qb.getQueue().getQueueName());
int i=0;
while(messages.hasMoreElements()) {
i++;
Message message=messages.nextElement();
logger.info(message+"");
}
logger.info("total record:"+i);
return om;
}
});
}
output:
2014-12-07 00:03:43.874 [main] INFO c.b.b.s.MessageQueueService - browsing indexJob
2014-12-07 00:03:43.878 [main] INFO c.b.b.s.MessageQueueService - total record:0
UPDATE: execute has a not yet well-documented parameter boolean startConnection. When it is set to "true", it seem to work. This is not a solution though -
String result=jms.execute(new SessionCallback<String>() {
#Override
public String doInJms(Session session) throws JMSException {
QueueBrowser queue=session.createBrowser((Queue)session.createQueue("indexJob"));
Enumeration<Message> messages=queue.getEnumeration();
String result="";
logger.info("Browse Queue: "+queue.getQueue().getQueueName());
while(messages.hasMoreElements()) {
Message message=messages.nextElement();
result+=message;
}
logger.info(result);
return result;
}
}, true);
Looking at org.springframework.jms.core.JmsTemplate.class source, most of the send methods are using execute() method with startConnection=false.
If the connection was not started, then how did the messages get added to the queue?
Does anyone know what this #param startConnection whether to start the Connection means?
This can be a somewhat confusing bit of JMS. The Connection start only refers to consumption of messages from the connection, not to producing. You are free to produce messages whenever you like, started or not, but if you want to consume or browse a destination you need to start the connection otherwise you will not get any messages dispatched to your consumers.
This purpose behind this is to allow you to create all your JMS resources prior to receiving any messages which might otherwise catch you in an state where you app isn't quite ready for them.
So in short, if you want to browse that message, you need to ensure the connection gets started.