I am trying to configure below workflow all in annotations
Inbound channel adapter with poller (cron trigger) scheduled to run every 30 minutes
Poll the file form directory i.e. 10 files and move to stage directory
For each file need to invoke a batch job in parallel i.e. 10 Jobs should run in parallel with the different files polled
I am able to achieve everything but unable to configure downstream executor channel to run jobs in parallel.
Below is the reference implementation. Eveyrything is working i.e. job is launched file after the file but it needs to launch job for different files in parallel
Appreciate for any help on this
#InboundChannelAdapter (incoming channel, custompoller)
public MessageSource<File> pollFile ( Directory Scanner) {
}
public PollerMetadata custompoller(errorhandler) {
poller.trigger(cron for every 10 minutes)
}
#ServiceActivator(incoming channel)
public MessageHandler filewritertotempdiretory() {
outputchannel(tempdirchannel)
}
#ServiceActivator(inputChannel = tempdirchannel)
public MessageHandler tempdirfilehandler() {
MethodInvokingMessageHandler messageHandler = (launcher class, "methodname");
return messageHandler;
}
Poller Metadata. Read in some other SO that we should not put the task executor when setting poller on cron, is that true ?
also how can i make the messages polled (say 10 messages polled) execute in parallel i.e. add task executor in poller metadata
#Bean
public PollerMetadata preProcessPoller(MessagePublishingErrorHandler errorHandler) {
PollerMetadata poller = new PollerMetadata();
poller.setTrigger(new CronTrigger("0/15 * * * * ?"));
poller.setMaxMessagesPerPoll(Long.valueOf(maxMessagesPerPoll));
errorHandler.setDefaultErrorChannel(errorChannel());
poller.setErrorHandler(errorHandler);
return poller;
}
You need to show your complete PollerMetadata configuration.
Best guess is you haven't set maxMessagesPerPoll.
By default, for an inbound channel adapter, maxMessagesPerPoll is 1.
You can add a TaskExecutor to the poller metadata to run the messages in parallel, or make incoming channel an ExecutorChannel.
Related
I am creating a simple integration service with AWS S3. I am facing some difficulties when an exception occurs.
My requirement is to poll an S3 Bucket periodically and to apply some transformation whenever a file is newly placed into S3 Bucket. The below code snippet works fine, but when an exception occurs it continues to retry again and again. I do not want that to happen. Can someone help me here.,
The IntegrationFlow is defined as below.,
#Configuration
public class S3Routes {
#Bean
public IntegrationFlow downloadFlow(MessageSource<InputStream> s3InboundStreamingMessageSource) {
return IntegrationFlows.from(s3InboundStreamingMessageSource)
.channel("s3Channel")
.handle("QueryServiceImpl", "processFile")
.get();
}
}
Configuration file is as below.,
#Service
public class S3AppConfiguration {
#Bean
#InboundChannelAdapter(value = "s3Channel")
public MessageSource<InputStream> s3InboundStreamingMessageSource(S3RemoteFileTemplate template) {
S3StreamingMessageSource messageSource = new S3StreamingMessageSource(template);
messageSource.setRemoteDirectory("my-bucket-name");
messageSource.setFilter(new S3PersistentAcceptOnceFileListFilter(new SimpleMetadataStore(),
"streaming"));
return messageSource;
}
#Bean
public PollableChannel s3Channel() {
return new QueueChannel();
}
#Bean
public S3RemoteFileTemplate template(AmazonS3 amazonS3) {
return new S3RemoteFileTemplate(new S3SessionFactory(amazonS3));
}
#Bean(name = "amazonS3")
public AmazonS3 nonProdAmazonS3(BasicAWSCredentials basicAWSCredentials) {
ClientConfiguration config = new ClientConfiguration();
config.setProxyHost("localhost");
config.setProxyPort(3128);
return AmazonS3ClientBuilder.standard().withRegion(Regions.fromName("ap-southeast-1"))
.withCredentials(new AWSStaticCredentialsProvider(basicAWSCredentials))
.withClientConfiguration(config)
.build();
}
#Bean
public BasicAWSCredentials basicAWSCredentials() {
return new BasicAWSCredentials("access_key", "secret_key");
}
#Bean(name = PollerMetadata.DEFAULT_POLLER)
public PollerMetadata nonProdPoller() {
return Pollers.cron("* */2 * * * *")
.get();
}
}
AcceptOnceFileList filter that I have used here, helps me to prevent handling the same file for continuous retries. But, I do not want to use AcceptOnceFileList filter, because when a file is not processed on 1st attempt, I wish to retry on next Poll (usually it happens every 1 hour in Prod region). I tried to use filter.remove() method whenever the processing fails(in case of any exception), it again results in continuous retries.
I am not sure how to disable the continuous retries on failure. Where should I configure it?
I took a look at Spring Integration ( Retry Strategy). Same scenario, but a different integration. I am not sure how to set up this for my IntegrationFlow. Can someone help here? Thanks in advance
That story is different: it talks about a listener container for AMQP. You use a source polling channel adapter - the approach might be different.
You create two source polling channel adapters: one via that #InboundChannelAdapter, another via IntegrationFlows.from(s3InboundStreamingMessageSource). Both of them produces data to the same channel. Not sure if that is really intentional.
It is not clear what is that retry in your case unless you really do that manual filter.remove() call. In this case indeed it is going to retry. But this is a single, not controlled retry. It is going to retry again only if you call that filter.remove() again. So, if you do everything yourself, why is the question?
Consider to use a RequestHandlerRetryAdvice configured for that your handle() instead: https://docs.spring.io/spring-integration/docs/current/reference/html/messaging-endpoints.html#message-handler-advice-chain. This way you really going to pull the remote file only once and retry is going to be managed by the Spring Retry API.
UPDATE
So, after some Cron Expression learning I realized that your one is wrong:
* */2 * * * * - means every second of every even minute
Must be like this:
0 */2 * * * * - at the beginning of every even minute
Perhaps something similar is with your hourly cron expression on the prod...
I am using spring data redis to subscribe to the 'task' redis stream to process tasks.
For some reason redis stream consumer only spawns one thread and processes one message at a time sequentially even thought I explicitly provide a Threadpool TaskExecutor.
I expect it to delegate the creation of threads to the provided Threadpool and spawn a thread up to the Threadpool configured limits. I can see that it is using the give TaskExecutor, but it's not spawning more than one thread.
Even when I don't specify my own taskExecutor, and it internally defaults to SimpleAsyncTaskExecutor, the problem still continues. Tasks are processed sequentially one at a time, one after the other, even when they are long lasting task.
What am I missing here?
#Bean
public Subscription
redisTaskStreamListenerContainer(
RedisConnectionFactory connectionFactory,
#Qualifier("task") RedisTemplate<String, Task<TransportEnvelope>> redisTemplate,
#Qualifier("task") StreamListener<String, MapRecord<String, String, String>> listener,
#Qualifier("task") Executor taskListenerExecutor) {
StreamMessageListenerContainerOptions<String, MapRecord<String, String, String>>
containerOptions = StreamMessageListenerContainerOptions.builder()
.pollTimeout(Duration.ofMillis(consumerPollTimeOutInMilli))
.batchSize(consumerReadBatchSize)
.executor(taskListenerExecutor)
.build();
StreamMessageListenerContainer<String, MapRecord<String, String, String>> container =
StreamMessageListenerContainer.create(connectionFactory, containerOptions);
StreamMessageListenerContainer.ConsumerStreamReadRequest<String> readOptions
=
StreamMessageListenerContainer.StreamReadRequest
.builder(StreamOffset.create(streamName, ReadOffset.lastConsumed()))
//turn off auto shutdown of stream consumer if an error occurs.
.cancelOnError((ex) -> false)
.consumer(Consumer.from(groupId, consumerId))
.build();
Subscription subscription = container.register(readOptions, listener);
container.start();
return subscription;
}
#Bean
#Qualifier("task")
public Executor redisListenerThreadPoolTaskExecutor() {
ThreadPoolTaskExecutor threadPoolTaskExecutor = new ThreadPoolTaskExecutor();
threadPoolTaskExecutor.setCorePoolSize(30);
threadPoolTaskExecutor.setMaxPoolSize(50);
threadPoolTaskExecutor.setQueueCapacity(Integer.MAX_VALUE);
threadPoolTaskExecutor.setThreadNamePrefix("redis-listener-");
threadPoolTaskExecutor.setRejectedExecutionHandler(new ThreadPoolExecutor.CallerRunsPolicy());
return threadPoolTaskExecutor;
}
I am using Spring Integration and Scatter Gather handler (https://docs.spring.io/spring-integration/docs/5.3.0.M1/reference/html/scatter-gather.html) in order to send 3 parallel requests (using ExecutorChannels) to external REST APIs and aggregate their response into one single message.
Everything works fine until exception is thrown within Aggregator's aggregatePayloads method (AggregatingMessageHandler). In this scenario error message is successfully delivered to Messaging Gateway which initiated the flow ( caller ). However, ScatterGatherHandler thread remains in hanging state waiting for gatherer reply (I believe) which never arrives due to the exception within it. I.e each sequential call leaves one additional thread in "stuck" state and eventually Thread Pool runs out of available working threads.
My current Scatter Gather configuration:
#Bean
public MessageHandler distributor() {
RecipientListRouter router = new RecipientListRouter();
router.setChannels(Arrays.asList(Channel1(asyncExecutor()),Channel2(asyncExecutor()),Channel3(asyncExecutor())));
return router;
}
#Bean
public MessageHandler gatherer() {
AggregatingMessageHandler aggregatingMessageHandler = new AggregatingMessageHandler(
new TransactionAggregator(),
new SimpleMessageStore(),
new HeaderAttributeCorrelationStrategy("correlationID"),
new ExpressionEvaluatingReleaseStrategy("size() == 3"));
aggregatingMessageHandler.setExpireGroupsUponCompletion( true );
return aggregatingMessageHandler;
}
#Bean
#ServiceActivator(inputChannel = "validationOutputChannel")
public MessageHandler scatterGatherDistribution() {
ScatterGatherHandler handler = new ScatterGatherHandler(distributor(), gatherer());
handler.setErrorChannelName("scatterGatherErrorChannel");
return handler;
}
#Bean("taskExecutor")
#Primary
public TaskExecutor asyncExecutor() {
ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
executor.setCorePoolSize(4);
executor.setMaxPoolSize(10);
executor.setQueueCapacity(100);
executor.setThreadNamePrefix("AsyncThread-");
executor.initialize();
return executor;
}
So far the only solution that I found is to add RequiresReply and GatherTimeout values for ScatterGatherHandler like below:
handler.setGatherTimeout(120000L);
handler.setRequiresReply(true);
This will produce an exception and release ScatterGatherHandler's thread to the pull after specified timeout value and after aggregator's exception is delivered to the messaging gateway. I can see following message in the log:
[AsyncThread-1] [WARN] [o.s.m.c.GenericMessagingTemplate$TemporaryReplyChannel:] [{}] - Reply message received but the receiving thread has already received a reply: ErrorMessage
Is there any other way to achieve this? My main goal is to make sure that I am not blocking any threads in case of exception is thrown within aggregator's aggregatePayloads method.
Thank you.
Technically this is really an expect behavior. See docs: https://docs.spring.io/spring-integration/docs/current/reference/html/message-routing.html#scatter-gather-error-handling
In this case a reasonable, finite gatherTimeout must be configured for the ScatterGatherHandler. Otherwise it is going to be blocked waiting for a reply from the gatherer forever, by default.
There is really no way to break expectations from the BlockingQueue.take() from that ScatterGatherHandler code.
I implemented a spring batch project that reads from a weblogic Jms queue (Custom Item Reader not message driven), then pass the Jms message data to an item writer (chunk = 1) where i call some APIs and write in DataBase.
However, i am trying to implement parallel Jms processing, reading in parallel Jms messages and passing them to the writer without waiting for the previous processes to complete.
I’ve used a DefaultMessageListenerContainer in a previous project and it offers a parallel consuming of jms messages, but in this project i have to use the spring batch framework.
I tried using the easiest solution (multi-threaded step) but it
didn’t work , JmsException : "invalid blocking receive when another
receive is in progress" which means probably that my reader is
statefull.
I thought about using remote partitioning but then i have to read all
messages and put the data into step execution contexts before calling
the slave steps, which isn't really efficient if dealing with a large
number of messages.
I looked a little bit into remote chunking, i understand that it passes data via queue channels, but i can't seem to find the utility in reading from a Jms and putting messages in a local queue for slave workers.
How can I approach this?
My code:
#Bean
Step step1() {
return steps.get("step1").<Message, DetectionIncoherenceLiqJmsOut>chunk(1)
.reader(reader()).processor(processor()).writer(writer())
.listener(stepListener()).build();
}
#Bean
Job job(#Qualifier("step1") Step step1) {
return jobs.get("job").start(step1).build();
}
Jms Code :
#Override
public void initQueueConnection() throws NamingException, JMSException {
Hashtable<String, String> properties = new Hashtable<String, String>();
properties.put(Context.INITIAL_CONTEXT_FACTORY, env.getProperty(WebLogicConstant.JNDI_FACTORY));
properties.put(Context.PROVIDER_URL, env.getProperty(WebLogicConstant.JMS_WEBLOGIC_URL_RECEIVE));
InitialContext vInitialContext = new InitialContext(properties);
QueueConnectionFactory vQueueConnectionFactory = (QueueConnectionFactory) vInitialContext
.lookup(env.getProperty(WebLogicConstant.JMS_FACTORY_RECEIVE));
vQueueConnection = vQueueConnectionFactory.createQueueConnection();
vQueueConnection.start();
vQueueSession = vQueueConnection.createQueueSession(false, 0);
Queue vQueue = (Queue) vInitialContext.lookup(env.getProperty(WebLogicConstant.JMS_QUEUE_RECEIVE));
consumer = vQueueSession.createConsumer(vQueue, "JMSCorrelationID IS NOT NULL");
}
#Override
public Message receiveMessages() throws NamingException, JMSException {
return consumer.receive(20000);
}
Item reader :
#Override
public Message read() throws Exception {
return jmsServiceReceiver.receiveMessages();
}
Thanks ! i'll appreciate the help :)
There's a BatchMessageListenerContainer in the spring-batch-infrastructure-tests sub project.
https://github.com/spring-projects/spring-batch/blob/d8fc58338d3b059b67b5f777adc132d2564d7402/spring-batch-infrastructure-tests/src/main/java/org/springframework/batch/container/jms/BatchMessageListenerContainer.java
Message listener container adapted for intercepting the message reception with advice provided through configuration.
To enable batching of messages in a single transaction, use the TransactionInterceptor and the RepeatOperationsInterceptor in the advice chain (with or without a transaction manager set in the base class). Instead of receiving a single message and processing it, the container will then use a RepeatOperations to receive multiple messages in the same thread. Use with a RepeatOperations and a transaction interceptor. If the transaction interceptor uses XA then use an XA connection factory, or else the TransactionAwareConnectionFactoryProxy to synchronize the JMS session with the ongoing transaction (opening up the possibility of duplicate messages after a failure). In the latter case you will not need to provide a transaction manager in the base class - it only gets on the way and prevents the JMS session from synchronizing with the database transaction.
Perhaps you could adapt it for your use case.
I was able to do so with a multithreaded step :
// Jobs et Steps
#Bean
Step stepDetectionIncoherencesLiq(#Autowired StepBuilderFactory steps) {
int threadSize = Integer.parseInt(env.getProperty(PropertyConstant.THREAD_POOL_SIZE));
return steps.get("stepDetectionIncoherencesLiq").<Message, DetectionIncoherenceLiqJmsOut>chunk(1)
.reader(reader()).processor(processor()).writer(writer())
.readerIsTransactionalQueue()
.faultTolerant()
.taskExecutor(taskExecutor())
.throttleLimit(threadSize)
.listener(stepListener())
.build();
}
And a jmsItemReader with jmsTemplate instead of creating session and connections explicitly, it manages connections so i dont have the jms exception anymore:( JmsException : "invalid blocking receive when another receive is in progress" )
#Bean
public JmsItemReader<Message> reader() {
JmsItemReader<Message> itemReader = new JmsItemReader<>();
itemReader.setItemType(Message.class);
itemReader.setJmsTemplate(jmsTemplate());
return itemReader;
}
I have a requirement where my application should read messages from MQ and write using file outbound channel adapter. I want each of my output file should contain messages of every 10 mins of interval. Is there any default implementation exist, or any pointers to do so.
public #Bean IntegrationFlow defaultJmsFlow()
{
return IntegrationFlows.from(
//read JMS topic
Jms.messageDrivenChannelAdapter(this.connectionFactory).destination(this.config.getInputQueueName()).errorChannel(errorChannel()).configureListenerContainer(c ->
{
final DefaultMessageListenerContainer container = c.get();
container.setSessionTransacted(true);
container.setMaxMessagesPerTask(-1);
}).get())
.channel(messageProcessingChannel()).get();
}
public #Bean MessageChannel messageProcessingChannel()
{
return MessageChannels.queue().get();
}
public #Bean IntegrationFlow messageProcessingFlow() {
return IntegrationFlows.from(messageProcessingChannel())
.handle(Files.outboundAdapter(new File(config.getWorkingDir()))
.fileNameGenerator(fileNameGenerator())
.fileExistsMode(FileExistsMode.APPEND).appendNewLine(true))
.get();
}
First of all you could use something like a QueueChannel with the poller on the endpoint for the FileWritingMessageHandler with the fixedDelay for those 10 mins. However you should keep in mind that messages are going to be stored in the memory before poller does its work. So, once a crash of your application, the messages are lost.
On the other hand you can use a JmsDestinationPollingSource with similar poller configuration. This way, however, you need to configure it with the maxMessagesPerPoll(-1) to let it to pull as much messages from the MQ as possible during single polling task - once in 10 mins.
Another variant is possible with an aggregator and its groupTimeout option. This way you won't have an output message from the aggregator until 10 mins interval passes. However again: the store is in memory by default. I wouldn't introduce one more persistence storage just to satisfy a periodic requirement when we already have an MQ and we really can poll exactly that. Therefore I would go a JmsDestinationPollingSource variant.
UPDATE
Can you help me with how to set fixed delay in file outbound adapter.
Since you deal with the QueueChannel, you need to configure for the "fixed delay" a PollingConsumer endpoint. This one really belongs to the subscriber of that channel. Indeed it is a .handle(Files.outboundAdapter) part. Only what you are missing that Poller is an option of the endpoint, not a MessageHandler. Consider to use an overloaded handle()variant:
.handle(Files.outboundAdapter(new File(config.getWorkingDir()))
.fileNameGenerator(fileNameGenerator())
.fileExistsMode(FileExistsMode.APPEND).appendNewLine(true),
e -> e.poller(p -> p.fixedDelay(10000)))
Or a sample example for JMSDestinationPollingSource
#Bean
public IntegrationFlow jmsInboundFlow() {
return IntegrationFlows
.from(Jms.inboundAdapter(cachingConnectionFactory())
.destination("jmsInbound"),
e -> e.poller(p -> p.fixedDelay(10000)))
.<String, String>transform(String::toUpperCase)
.channel(jmsOutboundInboundReplyChannel())
.get();
}