Creating DefaultMessageListenerContainer runtime with different message selector - spring

Below is the code in which I am trying to create a dynamic Listenercontainer instance after the application has been deployed. Without the message selector(the commented code), I am able to see my messages being consumed. Once I am adding the setMessageSelector the message is not getting consumed.
I configured the producers to produce messages with two different message selectors, say color='RED' and another one color='BLUE'. I have wired the 'RED' one using the Spring XML configuration. And this configuration works without any issues. I am able to see the message being consumed by consumers. But when I am trying to create a dynamic bean with color='BLUE' it does not work. The same will work without any issues if I add it in spring XML
'''
DefaultMessageListenerContainer defaultMessageListenerContainer=new DefaultMessageListenerContainer();
defaultMessageListenerContainer.setAutoStartup(Boolean.FALSE);
defaultMessageListenerContainer.setMessageListener(this.getMessageListener());
//defaultMessageListenerContainer.setMessageSelector(this.getMessageSelector());
defaultMessageListenerContainer.setBeanName(this.getBeanName());
defaultMessageListenerContainer.setConnectionFactory(this.getConnectionFactory());
defaultMessageListenerContainer.setDestination((Destination) this.getApplicationContext().getBean("customDestination"));
defaultMessageListenerContainer.setSessionTransacted(Boolean.TRUE);
defaultMessageListenerContainer.setConcurrentConsumers(1);
defaultMessageListenerContainer.setMaxConcurrentConsumers(5);
defaultMessageListenerContainer.initialize();
defaultMessageListenerContainer.afterPropertiesSet();
defaultMessageListenerContainer.start();
System.out.println(defaultMessageListenerContainer.isRunning());
System.out.println(defaultMessageListenerContainer.isAcceptMessagesWhileStopping());
System.out.println(defaultMessageListenerContainer.isRegisteredWithDestination());
ConfigurableListableBeanFactory beanFactory = ((ConfigurableApplicationContext) applicationContext).getBeanFactory();
beanFactory.registerSingleton("jmsRequestListenerContainer", defaultMessageListenerContainer);
<bean id="jmsRequestListenerContainerdefault" class="org.springframework.jms.listener.DefaultMessageListenerContainer">
<property name="concurrentConsumers" value="1" />
<property name="maxConcurrentConsumers" value="5" />
<property name="cacheLevel" value="0"/>
<property name="connectionFactory" ref="queueConnectionFactory" />
<property name="destination" ref="customeDestination"/>
<property name="sessionTransacted" value="true"/>
<property name="messageListener" ref="jmsRequestListener" />
<property name="messageSelector" value="color='RED'"/>
</bean>
<jee:jndi-lookup id="queueConnectionFactory" jndi-name="java:/JmsXA"/>
'''
Am I missing something in the code while creating it dynamically after the application has launched?

This works fine for me...
#SpringBootApplication
public class So66359276Application {
public static void main(String[] args) throws InterruptedException {
ConfigurableApplicationContext ctx = SpringApplication.run(So66359276Application.class, args);
Thread.sleep(5_000);
ctx.close();
}
#Bean
public ApplicationRunner runner(JmsTemplate template, ConnectionFactory cf, GenericApplicationContext ctx) {
template.setDefaultDestinationName("foo");
return args -> {
createContainer("RED", new JmsRequestListener1(), cf, ctx);
createContainer("BLUE", new JmsRequestListener2(), cf, ctx);
IntStream.range(0, 10).forEach(i -> {
template.convertAndSend("foo" + i, msg -> {
msg.setStringProperty("color", "RED");
return msg;
});
});
IntStream.range(0, 10).forEach(i -> {
template.convertAndSend("bar" + i, msg -> {
msg.setStringProperty("color", "BLUE");
return msg;
});
});
};
}
private void createContainer(String color, MessageListener listener, ConnectionFactory cf,
GenericApplicationContext ctx) {
DefaultMessageListenerContainer container = new DefaultMessageListenerContainer();
container.setConnectionFactory(cf);
container.setSessionTransacted(true);
container.setDestinationName("foo");
container.setMessageSelector("color='" + color + "'");
container.setMessageListener(listener);
ctx.registerBean("container" + color, DefaultMessageListenerContainer.class, () -> container);
ctx.getBean("container" + color, DefaultMessageListenerContainer.class).start();
}
}
class JmsRequestListener1 implements MessageListener {
#Override
public void onMessage(Message message) {
try {
System.out.println("1:" + ((TextMessage) message).getText() + " - "
+ message.getStringProperty("color"));
}
catch (JMSException e) {
e.printStackTrace();
}
}
}
class JmsRequestListener2 implements MessageListener {
#Override
public void onMessage(Message message) {
try {
System.out.println("2:" + ((TextMessage) message).getText() + " - "
+ message.getStringProperty("color"));
}
catch (JMSException e) {
e.printStackTrace();
}
}
}
1:foo0 - RED
1:foo1 - RED
1:foo2 - RED
1:foo3 - RED
1:foo4 - RED
1:foo5 - RED
1:foo6 - RED
1:foo7 - RED
1:foo8 - RED
1:foo9 - RED
2:bar0 - BLUE
2:bar1 - BLUE
2:bar2 - BLUE
2:bar3 - BLUE
2:bar4 - BLUE
2:bar5 - BLUE
2:bar6 - BLUE
2:bar7 - BLUE
2:bar8 - BLUE
2:bar9 - BLUE
registerBean is a newer way of adding bean definitions at runtime, but using registerSingleTon should work too.

Related

Reply messages timing out - Spring AMQP

In our application around 1% of the transactions times out daily.
We clearly can see in the log that messages is being processed and sent back using reply-to mechanism but client not receiving the message and wait till the timeout then throws an error.
I'm unable to identify the issue that why these messages getting lost occasionally. Any help much appreciated.
Clint -> Rabbit -> Server -> Rabbit -> Client (timeout - 1% of transactions)
log lines:
client:
2021-11-28 23:03:43.317 WARN <sending msg to vhost <IP> with correlationId: 5cc5c40b-7193-1cf9-c66f-7c5d72901bdf , message properties correlationId: 10976>
Server:
2021-11-28 23:03:43.318 WARN <Received msg with correlationId: 5cc5c40b-7193-1cf9-c66f-7c5d72901bdf , message properties correlationId: 10976>
Server:
2021-11-28 23:03:57.830 WARN <sending msg to vhost <IP> with correlationId: 112f5771-93b9-6f8f-a353-b9a79bdd1438 , message properties correlationId: 10976>
client:
2021-11-28 23:04:43.317 ERROR - <An exception has occured: No reply received from 'assign' with arguments '[]' - perhaps a timeout in the template? org.springframework.remoting.RemoteProxyFailureException: No reply received from 'assign' with arguments '[]' - perhaps a timeout in the template?>
Client and Listener configuration:
<!-- client config -->
<rabbit:queue id="application.queue" name="API.request.queue"
declared-by="rabbitAdminConnectionFactory1"/>
<rabbit:direct-exchange name="#{ API +'.RequestDirectExchange'}" id="requestDirectExchange"
declared-by="rabbitAdminConnectionFactory1">
<rabbit:bindings>
<rabbit:binding queue="application.queue" key="API"/>
</rabbit:bindings>
</rabbit:direct-exchange>
<bean id="amqpTemplate" class="com.api.APIRabbitTemplate">
<property name="connectionFactory" ref="rabbitConnectionFactory"/>
<property name="routingKey" value="API"/>
<property name="exchange" value="#{ API +'.RequestDirectExchange'}"/>
<property name="queue" value="API.request.queue"/>
<property name="messageConverter" ref="rmessageConverter"/>
<property name="replyTimeout" value="60000"/>
<property name="receiveTimeout" value="60000"/>
<property name="retryTemplate" ref="retryTemplate"/>
</bean>
<!-- server listener config -->
<bean id="remotingAmqpTemplate1" class="com.API.APIRabbitTemplate">
<property name="connectionFactory" ref="rabbitConnectionFactory"/>
<property name="messageConverter" ref="rmessageConverter"/>
<property name="queue" value="$app{appName}.request.queue"/>
<property name="retryTemplate" ref="retryTemplate1"/>
</bean>
<bean id="amqpRemotingListener1" class="com.API.remoting.amqpRemotingListener">
<property name="amqpTemplate" ref="remotingAmqpTemplate1"/>
</bean>
<rabbit:listener-container connection-factory="rabbitConnectionFactory" auto-startup="false"
requeue-rejected="false" task-executor="taskExecutor"
concurrency="6" max-concurrency="6"
acknowledge="auto" prefetch="1"
transaction-size="1" channel-transacted="true">
<rabbit:listener ref="amqpRemotingListener1" queue-names="API.request.queue"
admin="rabbitAdminConnectionFactory1"/>
</rabbit:listener-container>
<rabbit:connection-factory id="rabbitConnectionFactory"
thread-factory="tfCommon1"
connection-factory="clientConnectionFactory"
addresses="$amqp{connectionURL1}"
username="$amqp{username}"
password="$amqp{password}"
channel-cache-size="$amqp{listeners.session.cache.size}"/>
<util:properties id="spring.amqp.global.properties">
<prop key="smlc.missing.queues.fatal">false</prop>
</util:properties>
EDIT:
public class ARabbitTemplate extends RabbitTemplate {
protected static Logger log = Logger.getLogger( AARabbitTemplate.class );
#Override
public <T> T execute(ChannelCallback<T> action) {
try {
return super.execute(action);
}catch (AmqpException ex){
//log
throw ex;
}
}
#Override
public void doSend(Channel channel, String exchange, String routingKey, Message message,
boolean mandatory, CorrelationData correlationData) throws Exception {
try {
super.doSend(channel, exchange, routingKey, message, mandatory, correlationData);
if(channel != null && channel.getConnection() != null && channel.getConnection().getAddress() != null) {
String IP = channel.getConnection().getAddress().getHostAddress();
String correlationId = message.getMessageProperties().getHeaders().get(MessagingConsts.CORRELATION_ID);
String message_CorrelationId = message.getMessageProperties().getCorrelationId();
log("sending msg to vhost " + IP + " with correlationId: " + correlationId + " , message properties correlationId: " + message_CorrelationId);
}
} catch (Exception ex) {
log.error("got exception while sending msg to ampq "+ ex.getLocalizedMessage());
throw ex;
}
}
#Override
public <R, S> boolean receiveAndReply(String queueName, ReceiveAndReplyCallback<R, S> callback,
ReplyToAddressCallback<S> replyToAddressCallback) throws AmqpException {
try {
return super.receiveAndReply(queueName, callback, replyToAddressCallback);
} catch (AmqpException amqpEx) {
throw amqpEx;
}
}
#Override
public Message receive() throws AmqpException {
try {
return super.receive();
} catch (AmqpException amqpEx) {
throw amqpEx;
}
}
#Override
public void onMessage(Message message) {
super.onMessage(message);
}
}
public class AAmqpRemotingListener implements MessageListener {
private AmqpTemplate amqpTemplate;
#Autowired
private MessageConverter messageConverter;
#Override
public void onMessage(Message message) {
String correlationId = message.getMessageProperties().getHeaders().get(MessagingConsts.CORRELATION_ID);
String message_CorrelationId = message.getMessageProperties().getCorrelationId();
log("Received msg with correlationId: " + correlationId + " , message properties correlationId: " + message_CorrelationId);
//rest of the code
//exporter.onMessage(message);
}
}
//Message Converter
public class SpringAmqpMessageConverter implements MessageConverter {
#Autowired
protected ObjectSerializer serializer;
#Override
public Message toMessage(Object object, MessageProperties messageProperties) throws MessageConversionException {
byte[] data = serializer.writeObject(object);
// build message
MessageBuilder messageBuilder = MessageBuilder.withBody(data);
String correlationId = context.getCorrelationId();
if (StringUtils.isEmpty(correlationId)) {
correlationId = (new UUID(ThreadLocalRandom.current().nextLong(), ThreadLocalRandom.current().nextLong())).toString();
}
messageBuilder.setHeader(MessagingConsts.CORRELATION_ID, correlationId);
messageBuilder.copyHeaders(messageProperties.getHeaders());
messageBuilder.setCorrelationIdIfAbsent(correlationId);
messageBuilder.setDeliveryModeIfAbsentOrDefault(MessageDeliveryMode.NON_PERSISTENT);
Message message = messageBuilder.build();
return message;
}
#Override
public Object fromMessage(Message message) throws MessageConversionException {
//
}
}

DefaultMessageListenerContainer not receiving all messages from ActiveMQ queue

ActiveMQ/Spring experts,
I am running into a very strange problem with ActiveMQ and DefaultMessageListenerContainer/SimpleMessageListenerContainer combination. We have a web application built using Spring (we are at 4.x). One of the transactions is related to processing a bulk upload of files and each file has a number of lines. Each line will become a message for processing.
A publisher publishes each line as a message to a persistent Queue. When examined through the ActiveMQ console we can see the message in the queue. To the same queue, we have a group of listeners configured using DefaultMessageListenerContainer(DMLC)/SimpleMessageListenerContainer(SMLC) (tried both) for consuming messages.
When the publisher publishes 100 messages sometimes only 99 or 98 messages are delivered but the rest are stuck in queue. The configuration is as follows:
ActiveMQ broker is running in standalone mode not networked or embedded in WildFly.
In the Spring application, we tried both DMLC and SMLC but both ran into this issue. Tried simpleMQConnectionFactory as well as PooledConnectionFactory and both times ran into same problem.
Tried setting the prefetch limit to "1" on the PooledConnectionFactory and ran into the same problem. Spring SMLC is throwing an exception when set to "0".
Max concurrent consumers is set to 50
When messages are stuck, if we restart WildFly the remaining messages in the queue are delivered to the consumers.
We are not using transacted sessions rather set the acknowledgModeName = "CLIENT_ACKNOWLEDGE"
We initialize the queue using a spring bean and use that for initializing the SMLC or DMLC
I am running out of options to try at this. If you share your experiences in this regard it is highly appreciated. This application is in production and the problem happens almost every other day sometimes multiple times in a day.
private void publishDMRMessage(DmrDTO p_dmrDTO, long jobID, int numDMRs) {
//Create a DMR message for each of the unique keys and publish it to
try {
DMRImportMessage message = new DMRImportMessage();
message.setDmrDTO(p_dmrDTO);
message.setDmrKey(p_dmrDTO.toString());
message.setDmrImportJobID(new Long(jobID));
message.setTask(Command.INITIALIZE_DMR_FORM);
message.setNumDMRForms(new Long(numDMRs));
sender.sendMessage(message);
} catch (Exception e) {
System.out.println(" JMS Exception = " + e.getMessage());
e.printStackTrace();
}
}
public class DMRMessageListener implements MessageListener {
private DMRImportManager manager;
private JMSMessageSender sender;
private DmrFormInitService formService;
private ProcessDMRValidationMessage validateService;
private ImportDmrService dmrService;
private static final Logger log = Logger.getLogger(DMRMessageListener.class);
public ImportDmrService getDmrService() {
return dmrService;
}
public void setDmrService(ImportDmrService dmrService) {
this.dmrService = dmrService;
}
public ProcessDMRValidationMessage getValidateService() {
return validateService;
}
public void setValidateService(ProcessDMRValidationMessage validateService) {
this.validateService = validateService;
}
public DmrFormInitService getFormService() {
return formService;
}
public void setFormService(DmrFormInitService formService) {
this.formService = formService;
}
public JMSMessageSender getSender() {
return sender;
}
public void setSender(JMSMessageSender sender) {
this.sender = sender;
}
public DMRImportManager getManager() {
return manager;
}
public void setManager(DMRImportManager manager) {
this.manager = manager;
}
public void onMessage(Message message) {
if (message instanceof ObjectMessage) {
try {
ObjectMessage objectMessage = (ObjectMessage) message;
DMRImportMessage dmrMessage = (DMRImportMessage)objectMessage.getObject();
log.info("============= Message Received =========================");
log.info("Message Type = " + dmrMessage.getTask() + " for JOB ID = " + dmrMessage.getDmrImportJobID());
log.info("Message Received === DMR ID = " + dmrMessage.getDmrID());
log.info("Message Received === DMR key = " + dmrMessage.getDmrKey());
log.info("============= Message Received =========================");
//Process the message
processDMRMessage(dmrMessage);
DMRProcessingStatus status = manager.updateStatus(dmrMessage);
if (status.isStatus()) {
log.info(" One stage is complete, the next stage should start for JOB ID = " + dmrMessage.getDmrImportJobID());
publishMessageForNextStepOfProcessing(dmrMessage, status);
}
}
catch (Exception ex) {
ex.printStackTrace();
throw new RuntimeException(ex);
}
}
else {
log.error(" ***** Received an invalid message -- NOT AN Object message so cannot be processed and will result in stuck jobs **** ");
throw new IllegalArgumentException("Message must be of type ObjectMessage");
}
//Send the next message in the chain
}
/**
* It will examine the message content and based on the message type it will invoke the appropriate
* service.
*
* #param dmrMessage DMRImportMessage
*/
private void processDMRMessage(DMRImportMessage dmrMessage) {
if (dmrMessage.getTask() == Command.INITIALIZE_DMR_FORM) {
Map<String, String> dmrInitResults = formService.initDmrForm(dmrMessage.getDmrDTO());
//Indicate in message that this DMR Key is not in ICIS
if (dmrInitResults != null) {
if (StringUtils.equalsIgnoreCase(dmrInitResults.get("wsUnscheduleDmrError"), "truee")) {
log.info("DMR Key is not in ICIS: " + dmrMessage.getDmrDTO().toString());
dmrMessage.setDmrKeyInICIS(false);
} else if (StringUtils.equalsIgnoreCase(dmrInitResults.get("wsDBDown"), "truee")) {
log.error("Web Service call failed for DMR Key: " + dmrMessage.getDmrDTO().toString());
}
}
}
try {
if (dmrMessage.getTask() == Command.IMPORT_DMR_PARAMETER) {
//Process the Parameter line
ParameterProcessingStatus status = dmrService.processLine(dmrMessage.getDmrImportJobID(), dmrMessage.getDmrParameterSubmission(), new Integer(dmrMessage.getLineNumber()), dmrMessage.getDmrKeysNotInICIS());
System.out.println("LINE = " + dmrMessage.getLineNumber() + " Status = " + status.isStatus());
dmrMessage.setProcessingStatus(status.isStatus());
dmrMessage.setDmrID(status.getDmrID());
dmrMessage.setDmrComment(status.getDmrComment());
return;
}
} catch(Exception e) {
log.error("An exception occurred during processing of line " + dmrMessage.getLineNumber() + " in job " + dmrMessage.getDmrImportJobID());
e.printStackTrace();
dmrMessage.setProcessingStatus(false);
dmrMessage.setDmrID(0L);
}
try {
if (dmrMessage.getTask() == Command.END_DMR_PARAMETER_IMPORT) {
//Process the Parameter line
//ParameterProcessingStatus status = dmrService.processLine(dmrMessage.getDmrImportJobID(), dmrMessage.getDmrParameterSubmission(), 100);
dmrMessage.setProcessingStatus(true);
dmrMessage.setDmrID(0L);
return;
}
} catch(Exception e) {
e.printStackTrace();
dmrMessage.setProcessingStatus(false);
dmrMessage.setDmrID(0L);
}
try {
if (dmrMessage.getTask() == Command.POST_PROCESS_DMR) {
//Validate DMRs
validateService.validateDMR(dmrMessage);
}
} catch (Exception e) {
e.printStackTrace();
}
}
private void publishMessageForNextStepOfProcessing(DMRImportMessage dmrMessage, DMRProcessingStatus status) throws JMSException {
log.info(" =========== Publish a message for next step of processing for JOB ID = " + dmrMessage.getDmrImportJobID());
if (dmrMessage.getTask() == Command.INITIALIZE_DMR_FORM) {
//Start the DMR Parameter Processing
sender.sendDMRControlMessage(this.createControlMessage(ProcessPhase.START_PARAMETER_PROCESSING, dmrMessage.getDmrImportJobID(), status.getDmrKeysNotInICIS()));
return;
}
if ((dmrMessage.getTask() == Command.IMPORT_DMR_PARAMETER)
|| (dmrMessage.getTask() == Command.END_DMR_PARAMETER_IMPORT)) {
//Start the DMR Validation Process
dmrService.postProcessParameters(dmrMessage.getDmrImportJobID(), status.getSuccessfulLines(), status.getErroredLines());
DMRImportControlMessage message = this.createControlMessage(ProcessPhase.START_DMR_VALIDATION, dmrMessage.getDmrImportJobID());
message.setDmrIDsWithComments(status.getDmrIDsWithComments());
sender.sendDMRControlMessage(message);
return;
}
if (dmrMessage.getTask() == Command.POST_PROCESS_DMR) {
//Start the next DMR import process
sender.sendDMRControlMessage(this.createControlMessage(ProcessPhase.START_DMR_FORM_INIT, dmrMessage.getDmrImportJobID()));
return;
}
log.info(" =========== End Publish a message for next step of processing for JOB ID = " + dmrMessage.getDmrImportJobID());
}
private DMRImportControlMessage createControlMessage(DMRImportControlMessage.ProcessPhase phase, Long jobID) {
return createControlMessage(phase, jobID, null);
}
private DMRImportControlMessage createControlMessage(DMRImportControlMessage.ProcessPhase p_phase, Long p_jobID, Set<DmrDTO> p_dmrDTOs) {
DMRImportControlMessage message = new DMRImportControlMessage();
message.setDmrImportJobID(p_jobID);
message.setPhase(p_phase);
if (p_dmrDTOs != null) {
message.setDmrKeysNotInICIS(p_dmrDTOs);
}
return message;
}
//Bean Configs.
<bean id="prefetchPolicy" class="org.apache.activemq.ActiveMQPrefetchPolicy">
<property name="queuePrefetch" value="0"/>
</bean>
<bean id="jmsFactoryPub" class="org.apache.activemq.ActiveMQConnectionFactory">
<constructor-arg index="0" value="tcp://localhost:61616" />
</bean>
<bean id="jmsFactoryReceive" class="org.apache.activemq.ActiveMQConnectionFactory">
<constructor-arg index="0" value="tcp://localhost:61616" />
<property name="prefetchPolicy" ref="prefetchPolicy" />
</bean>
<bean id="jmsFactoryControlMsg" class="org.apache.activemq.ActiveMQConnectionFactory">
<property name="brokerURL">
<value>tcp://localhost:61616</value>
</property>
</bean>
<bean id="dmrQueue"
class="org.apache.activemq.command.ActiveMQQueue">
<constructor-arg value="DMRQueue" />
</bean>
<bean id="dmrControlQueue"
class="org.apache.activemq.command.ActiveMQQueue">
<constructor-arg value="DMRControlQueue" />
</bean>
<bean id="jmsQueueTemplate" class="org.springframework.jms.core.JmsTemplate">
<property name="connectionFactory" ref="jmsFactoryPub" />
</bean>
<bean id="jmsQueueTemplateControlMsg" class="org.springframework.jms.core.JmsTemplate">
<property name="connectionFactory" ref="jmsFactoryControlMsg" />
</bean>
<bean id="messageCreator" class="net.exchangenetwork.netdmr.service.DMRMessageCreator">
</bean>
<bean id="dmrMessageListener" class="net.exchangenetwork.netdmr.service.DMRMessageListener">
<property name="manager" ref="dmrImportManager"/>
<property name="sender" ref="messagePublisher"/>
<property name="formService" ref="dmrFormInit"/>
<property name="validateService" ref="dmrValidator"/>
<property name="dmrService" ref="importDmrService"/>
</bean>
<bean id="messageSender" class="net.exchangenetwork.netdmr.service.JMSMessageSender">
<property name="jmsTemplate" ref="jmsQueueTemplate" />
<property name="sendQueue" ref="dmrQueue" />
<property name="creator" ref="messageCreator" />
</bean>
<bean id="messagePublisher" class="net.exchangenetwork.netdmr.service.JMSMessageSender">
<property name="jmsTemplate" ref="jmsQueueTemplateControlMsg" />
<property name="sendQueue" ref="dmrControlQueue" />
<property name="creator" ref="messageCreator" />
</bean>
<bean id="jmsContainer" class="org.springframework.jms.listener.DefaultMessageListenerContainer">
<property name="connectionFactory" ref="jmsFactoryReceive"/>
<!-- this is the queue we will listen on -->
<property name="destination" ref="dmrQueue" />
<property name="messageListener" ref="dmrMessageListener"/>
<property name="concurrentConsumers" value="60"/>
<property name="sessionAcknowledgeModeName" value="CLIENT_ACKNOWLEDGE"/>
<property name="errorHandler" ref="jmsErrorHandler"/>
<property name="exceptionListener" ref="jmsExceptionHandler"/>
<property name="receiveTimeout" value="0"/>
</bean>

Spring Batch read step running in loop

I came across a piece of code that reads some data as the following:
public class StudioReader implements ItemReader<List<Studio>> {
#Setter private AreaDao areaDao;
#Getter #Setter private BatchContext context;
private HopsService hopsService = new HopsService();
#Override
public List<Studio> read() throws Exception {
List<Studio> list = hopsService.getStudioHops();
if (!isEmpty(list)) {
for (Studio studio : list) {
log.info("Studio being read: {}", studio.getCode());
List areaList = areaDao.getArea(studio
.getCode());
if (areaList.size() > 0) {
studio.setArea((String) areaList.get(0));
log.info("Area {1} is fetched for studio {2}", areaList.get(0), studio.getCode());
}
this.getContext().setReadCount(1);
}
}
return list;
}
However when I run the job this read is running in a loop. I found from another stackoverflow answer that it is the expected behavior. My question then is what is the best solution given this particular example? Extend StudioReader from JdbcCursorItemReader ? I found one example that defines everything in the xml which I don't want. And here is the context.xml part for the reader:
<bean class="org.springframework.batch.core.scope.StepScope" />
<bean id="ItemReader" class="com.syc.studio.reader.StudioReader" scope="step">
<property name="context" ref="BatchContext" />
<property name="areaDao" ref="AreaDao" />
</bean>
And here is the job definition in xml:
<bean id="StudioJob" class="org.springframework.batch.core.job.SimpleJob">
<property name="steps">
<list>
<bean id="StudioStep" parent="SimpleStep" >
<property name="itemReader" ref="ItemReader"/>
<property name="itemWriter" ref="ItemWriter"/>
<property name="retryableExceptionClasses">
<map>
<entry key="com.syc.studio.exception.CustomException" value="true"/>
</map>
</property>
<property name="retryLimit" value="2" />
</bean>
</list>
</property>
<property name="jobRepository" ref="jobRepository" />
</bean>
Writer:
public void write(List<? extends Object> obj) throws Exception {
List<Studio> list = (List<Studio>) obj.get(0);
for (int i = 0; i <= list.size(); i++) {
Studio studio = list.get(i);
if (apiClient == null) {
apiClient = new APIClient("v2");
}
this.uploadXML(studio);
}
The read method after suggestion from #holi-java:
public List<Studio> read() throws Exception {
if (this.listIterator == null) {
this.listIterator = initializing();
}
return this.listIterator.hasNext() ? this.listIterator.next() : null;
}
private Iterator<List<Studio>> initializing() {
List<Studio> listOfStudiosFromApi = hopsService.getStudioLocations();
for (Studio studio : listOfStudiosFromApi) {
log.info("Studio being read: {}", studio.getCode());
List areaList = areaDao.getArea(studio.getCode());
if (areaList.size() > 0) {
studio.setArea((String) areaList.get(0));
log.info("Area {1} is fetched for studio {2}", areaList.get(0), studio.getCode());
}
this.getContext().setReadCount(1);
}
return Collections.singletonList(listOfStudiosFromApi).iterator();
}
spring-batch documentation for ItemReader.read assert:
Implementations must return null at the end of the input data set.
But your read method is always return a List and should be like this:
public Studio read() throws Exception {
if (this.results == null) {
List<Studio> list = hopsService.getStudioHops();
...
this.results=list.iterator();
}
return this.results.hasNext() ? this.results.next() : null;
}
if you want your read method return a List then you must paging the results like this:
public List<Studio> read() throws Exception {
List<Studio> results=hopsService.getStudioHops(this.page++);
...
return results.isEmpty()?null:results;
}
if you can't paging the results from Service you can solved like this:
public List<Studio> read() throws Exception {
if(this.results==null){
this.results = Collections.singletonList(hopsService.getStudioHops()).iterator();
}
return this.results.hasNext()?this.results.next():null;
}
it's better not read a list of items List<Studio>, read an item at a time Studio instead. when you read a list of item you possibly duplicated iterate logic between writers and processors as you have shown the demo in comments. if you have a huge of data list to processing you can combine pagination in your reader, for example:
public Studio read() throws Exception {
if (this.results == null || !this.results.hasNext()) {
List<Studio> list = hopsService.getStudioHops(this.page++);
...
this.results=list.iterator();
}
return this.results.hasNext() ? this.results.next() : null;
}
Maybe you need to see step processing mechanism.
ItemReader - read an item at a time.
ItemProcessor - processing an item at a time.
ItemWriter - write entire chunk of items out.

JMS Rollback & redelivery not honoring the RedeliveryDelay configuration

I would like to have my Camel routes transactional with ActiveMQ. Rollback and maximum re-deliveries work fine, but not re-delivery delay, which should be incremental.
For example, when I failed to process message (raising an exception), it's redelivered 3 times (as expected), but with no time between it (which is not).
My Spring configuration:
<context:annotation-config/>
<context:component-scan base-package="fr.dush.poc.springplaceholder"/>
<spring:camelContext>
<spring:package>fr.dush.poc.springplaceholder.routes</spring:package>
<spring:contextScan/>
</spring:camelContext>
<bean id="jmsTransactionManager" class="org.springframework.jms.connection.JmsTransactionManager">
<property name="connectionFactory" ref="jmsConnectionFactory"/>
</bean>
<bean id="PROPAGATION_REQUIRED" class="org.apache.camel.spring.spi.SpringTransactionPolicy">
<property name="transactionManager" ref="jmsTransactionManager"/>
</bean>
<bean id="PROPAGATION_REQUIRES_NEW" class="org.apache.camel.spring.spi.SpringTransactionPolicy">
<property name="transactionManager" ref="jmsTransactionManager"/>
<property name="propagationBehaviorName" value="PROPAGATION_REQUIRES_NEW"/>
</bean>
Spring configuration continue in configuration bean:
#Component
public class CamelFactories {
private static final Logger LOGGER = LoggerFactory.getLogger(CamelFactories.class);
public static final int REDELIVERY_DELAY = 1000;
public static final int BACK_OFF_MULTIPLIER = 2;
public static final int HOUR = 3600000;
public static final int MAXIMUM_REDELIVERY_DELAY = 2 * HOUR;
public static final int MAXIMUM_REDELIVERIES = 3;
#Bean(name = "jmsConnectionFactory")
public ActiveMQConnectionFactory createFactory() {
ActiveMQConnectionFactory factory = new ActiveMQConnectionFactory();
factory.setBrokerURL("tcp://localhost:61616");
RedeliveryPolicy policy = new RedeliveryPolicy() {
#Override
public long getNextRedeliveryDelay(long previousDelay) {
long nextDelay = super.getNextRedeliveryDelay(previousDelay);
LOGGER.warn("Previous delay={} ; This delay={} ", previousDelay, nextDelay);
return nextDelay;
}
};
policy.setMaximumRedeliveries(MAXIMUM_REDELIVERIES);
policy.setRedeliveryDelay(REDELIVERY_DELAY);
policy.setBackOffMultiplier(BACK_OFF_MULTIPLIER);
policy.setUseExponentialBackOff(true);
policy.setMaximumRedeliveryDelay(MAXIMUM_REDELIVERY_DELAY);
factory.setRedeliveryPolicy(policy);
return factory;
}
#Bean(name = "activemq")
public JmsComponent createJmsComponent(JmsTransactionManager transactionManager,
ActiveMQConnectionFactory connectionFactory) {
ActiveMQComponent component = new ActiveMQComponent();
component.setTransactionManager(transactionManager);
component.setConnectionFactory(connectionFactory);
component.setTransacted(true);
return component;
}
My route is quite simple:
public class CamelRouteBuilder extends SpringRouteBuilder {
#Override
public void configure() throws Exception {
Policy required = getApplicationContext().getBean("PROPAGATION_REQUIRED",
SpringTransactionPolicy.class);
from("activemq:queue:foo.bar")
.transacted()
.policy(required)
.log(LoggingLevel.INFO, "fr.dush.poc", "Receive message: ${body}")
.beanRef("serviceBean") // throw an exception
.to("mock:routeEnd");
}
}
And in my logs, I have this, 3 times with previous delay=0:
CamelFactories:36 - Previous delay=0 ; This delay=1000
It seems I'm not alone to have this issue, but I still didn't find solution...
Thanks,
-Dush
This is possibly resolved by setting cacheLevelName=CACHE_CONSUMER on the ActiveMQComponent. I had the same symptoms & this resolved it for me. On a related note, I also get out of order delivery of messages with a transacted component, unless I use CACHE_CONSUMER.
I still didn't find solution. But I found an alternative: retry API from CAMEL itself.
Configuration is very similar. Spring config example:
<redeliveryPolicyProfile id="infiniteRedeliveryPolicy"
asyncDelayedRedelivery="true"
redeliveryDelay="${camel.redelivery_delay}"
maximumRedeliveryDelay="${camel.maximum_redelivery_delay}"
maximumRedeliveries="${camel.infinite_redelivery}"
backOffMultiplier="${camel.back_off_multiplier}"
useExponentialBackOff="true"/>
<routeContext>
<route>
<!-- ... -->
<!-- Define behaviour in case of technical error -->
<onException redeliveryPolicyRef="infiniteRedeliveryPolicy">
<exception>java.lang.Exception</exception>
<handled>
<constant>false</constant>
</handled>
<log message="Message can't be processed for now. I'll retry later!" />
</onException>
</route>
</routeContext>
Consumers should be transactional if you want to keep not processed messages in the ActiveMQ queue, even if you shut down application.

Spring: import a module with specified environment

Is there anything that can achieve the equivalent of the below:
<import resource="a.xml">
<prop name="key" value="a"/>
</import>
<import resource="a.xml">
<prop name="key" value="b"/>
</import>
Such that the beans defined in resouce a would see the property key with two different values? The intention would be that this would be used to name the beans in the imports such that resource a.xml would appear:
<bean id="${key}"/>
And hence the application would have two beans named a and b now available with the same definition but as distinct instances. I know about prototype scope; it is not intended for this reason, there will be many objects created with interdepednencies that are not actually prototypes. Currently I am simply copying a.xml, creating b.xml and renaming all the beans using the equivalent of a sed command. I feel there must be a better way.
I suppose that PropertyPlaceholderConfigurers work on a per container basis, so you can't achieve this with xml imports.
Re The application would have two beans named a and b now available with the same definition but as distinct instances
I think you should consider creating additional application contexts(ClassPathXmlApplicationContext for example) manually, using your current application context as the parent application context.
So your many objects created with interdependencies sets will reside in its own container each.
However, in this case you will not be able to reference b-beans from a-container.
update you can postprocess the bean definitions(add new ones) manually by registering a BeanDefinitionRegistryPostProcessor specialized bean, but this solution also does not seem to be easy.
OK, here's my rough attempt to import xml file manually:
disclaimer: I'm very bad java io programmer actually so double check the resource related code :-)
public class CustomXmlImporter implements BeanDefinitionRegistryPostProcessor {
#Override
public void postProcessBeanFactory(
ConfigurableListableBeanFactory beanFactory) throws BeansException {
}
private Map<String, String> properties;
public void setProperties(Map<String, String> properties) {
this.properties = properties;
}
public Map<String, String> getProperties() {
return properties;
}
private void readXml(XmlBeanDefinitionReader reader) {
InputStream inputStream;
try {
inputStream = new ClassPathResource(this.classpathXmlLocation).getInputStream();
} catch (IOException e1) {
throw new AssertionError();
}
try {
Scanner sc = new Scanner(inputStream);
try {
sc.useDelimiter("\\A");
if (!sc.hasNext())
throw new AssertionError();
String entireXml = sc.next();
PropertyPlaceholderHelper helper = new PropertyPlaceholderHelper("${",
"}", null, false);
Properties props = new Properties();
props.putAll(this.properties);
String newXml = helper.replacePlaceholders(entireXml, props);
reader.loadBeanDefinitions(new ByteArrayResource(newXml.getBytes()));
} finally {
sc.close();
}
} finally {
try {
inputStream.close();
} catch (IOException e) {
throw new AssertionError();
}
}
}
private String classpathXmlLocation;
public void setClassPathXmlLocation(String classpathXmlLocation) {
this.classpathXmlLocation = classpathXmlLocation;
}
public String getClassPathXmlLocation() {
return this.classpathXmlLocation;
}
#Override
public void postProcessBeanDefinitionRegistry(
BeanDefinitionRegistry registry) throws BeansException {
XmlBeanDefinitionReader reader = new XmlBeanDefinitionReader(registry);
readXml(reader);
}
}
XML configuration:
<bean class="CustomXmlImporter">
<property name="classPathXmlLocation" value="a.xml" />
<property name="properties">
<map>
<entry key="key" value="a" />
</map>
</property>
</bean>
<bean class="CustomXmlImporter">
<property name="classPathXmlLocation" value="a.xml" />
<property name="properties">
<map>
<entry key="key" value="b" />
</map>
</property>
</bean>
this code loads the resources from classpath. I would think twice before doing something like that, anyway, you can use this as a starting point.

Resources