Spring Integration call another handler method after aggregation - spring

Question

I am developing a system which will read and process file from a directory. Once all the file has been processed it will call a method which in turn generates a file. Also, it should route/process the files based on file name, I have used spring integration router for the same. Below is the code snippet of the Integration. My question is, this is not working if I remove any of the line .channel(aggregatorOutputChannel()) or .channel(confirmChannel()), also I have to keep the same channel .channel(aggregatorOutputChannel()) before and after the aggregator. Why do I need all 3 channel declaration? if this is wrong how to correct it.
I am using JDK 8, Spring 5, Spring boot 2.0.4.
#Configuration
#EnableIntegration
public class IntegrationConfig {
#Value("${agent.demographic.input.directory}")
private String inputDir;
#Value("${agent.demographic.output.directory}")
private String outputDir;
#Value("${confirmationfile.directory}")
private String confirmDir;
#Value("${input.scan.frequency: 2}")
private long scanFrequency;
#Value("${processing.waittime: 6000}")
private long messageGroupWaiting;
#Value("${thread.corepoolsize: 10}")
private int corepoolsize;
#Value("${thread.maxpoolsize: 20}")
private int maxpoolsize;
#Value("${thread.queuecapacity: 1000}")
private int queuedepth;
#Bean
public MessageSource<File> inputFileSource() {
FileReadingMessageSource src = new FileReadingMessageSource();
src.setDirectory(new File(inputDir));
src.setAutoCreateDirectory(true);
ChainFileListFilter<File> chainFileListFilter = new ChainFileListFilter<>();
chainFileListFilter.addFilter(new AcceptOnceFileListFilter<>() );
chainFileListFilter.addFilter(new RegexPatternFileListFilter("(?i)^.+\\.xml$"));
src.setFilter(chainFileListFilter);
return src;
}
#Bean
public UnZipTransformer unZipTransformer() {
UnZipTransformer unZipTransformer = new UnZipTransformer();
unZipTransformer.setExpectSingleResult(false);
unZipTransformer.setZipResultType(ZipResultType.FILE);
unZipTransformer.setDeleteFiles(true);
return unZipTransformer;
}
#Bean("agentdemographicsplitter")
public UnZipResultSplitter splitter() {
UnZipResultSplitter splitter = new UnZipResultSplitter();
return splitter;
}
#Bean
public DirectChannel outputChannel() {
return new DirectChannel();
}
#Bean
public DirectChannel aggregatorOutputChannel() {
return new DirectChannel();
}
#Bean("confirmChannel")
public DirectChannel confirmChannel() {
return new DirectChannel();
}
#Bean
public MessageHandler fileOutboundChannelAdapter() {
FileWritingMessageHandler adapter = new FileWritingMessageHandler(new File(outputDir));
adapter.setDeleteSourceFiles(true);
adapter.setAutoCreateDirectory(true);
adapter.setExpectReply(true);
adapter.setLoggingEnabled(true);
return adapter;
}
#Bean
public MessageHandler confirmationfileOutboundChannelAdapter() {
FileWritingMessageHandler adapter = new FileWritingMessageHandler(new File(confirmDir));
adapter.setDeleteSourceFiles(true);
adapter.setAutoCreateDirectory(true);
adapter.setExpectReply(false);
adapter.setFileNameGenerator(defaultFileNameGenerator() );
return adapter;
}
#Bean
public TaskExecutor taskExecutor() {
ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
executor.setCorePoolSize(corepoolsize);
executor.setMaxPoolSize(maxpoolsize);
executor.setQueueCapacity(queuedepth);
return executor;
}
#Bean
public DefaultFileNameGenerator defaultFileNameGenerator() {
DefaultFileNameGenerator defaultFileNameGenerator = new DefaultFileNameGenerator();
defaultFileNameGenerator.setExpression("payload.name");
return defaultFileNameGenerator;
}
#Bean
public IntegrationFlow confirmGeneration() {
return IntegrationFlows.
from("confirmChannel")
.handle(confirmationfileOutboundChannelAdapter())
.get();
}
#Bean
public IntegrationFlow individualProcessor() {
return flow -> flow.handle("thirdpartyIndividualAgentProcessor","processfile").channel(outputChannel()).handle(fileOutboundChannelAdapter());
}
#Bean
public IntegrationFlow firmProcessor() {
return flow -> flow.handle("thirdpartyFirmAgentProcessor","processfile").channel(outputChannel()).handle(fileOutboundChannelAdapter());
}
#Bean
public IntegrationFlow thirdpartyAgentDemographicFlow() {
return IntegrationFlows
.from(inputFileSource(), spec -> spec.poller(Pollers.fixedDelay(scanFrequency,TimeUnit.SECONDS)))
.channel(MessageChannels.executor(taskExecutor()))
.<File, Boolean>route(f -> f.getName().contains("individual"), m -> m
.subFlowMapping(true, sf -> sf.gateway(individualProcessor()))
.subFlowMapping(false, sf -> sf.gateway(firmProcessor()))
)
.channel(aggregatorOutputChannel())
.aggregate(aggregator -> aggregator.groupTimeout(messageGroupWaiting).correlationStrategy(new CorrelationStrategy() {
#Override
public Object getCorrelationKey(Message<?> message) {
return "xyz";
}
}))
.channel(aggregatorOutputChannel())
.handle("agentDemograpicOutput","generateAgentDemographicFile")
.channel(confirmChannel())
.get();
}
}
Below is the log
2018-09-07 17:29:20.003 DEBUG 10060 --- [ taskExecutor-2] o.s.integration.channel.DirectChannel : preSend on channel 'outputChannel', message: GenericMessage [payload=C:\thirdpartyintg\input\18237232_firm.xml, headers={replyChannel=org.springframework.messaging.core.GenericMessagingTemplate$TemporaryReplyChannel#1a867ae7, errorChannel=org.springframework.messaging.core.GenericMessagingTemplate$TemporaryReplyChannel#1a867ae7, file_name=18237232_firm.xml, file_originalFile=C:\thirdpartyintg\input\18237232_firm.xml, id=dd70999a-8b8d-93d2-1a43-a961ac2c339f, file_relativePath=18237232_firm.xml, timestamp=1536366560003}]
2018-09-07 17:29:20.003 DEBUG 10060 --- [ taskExecutor-2] o.s.i.file.FileWritingMessageHandler : fileOutboundChannelAdapter received message: GenericMessage [payload=C:\thirdpartyintg\input\18237232_firm.xml, headers={replyChannel=org.springframework.messaging.core.GenericMessagingTemplate$TemporaryReplyChannel#1a867ae7, errorChannel=org.springframework.messaging.core.GenericMessagingTemplate$TemporaryReplyChannel#1a867ae7, file_name=18237232_firm.xml, file_originalFile=C:\thirdpartyintg\input\18237232_firm.xml, id=dd70999a-8b8d-93d2-1a43-a961ac2c339f, file_relativePath=18237232_firm.xml, timestamp=1536366560003}]
2018-09-07 17:29:20.006 DEBUG 10060 --- [ taskExecutor-2] o.s.integration.channel.DirectChannel : postSend (sent=true) on channel 'outputChannel', message: GenericMessage [payload=C:\thirdpartyintg\input\18237232_firm.xml, headers={replyChannel=org.springframework.messaging.core.GenericMessagingTemplate$TemporaryReplyChannel#1a867ae7, errorChannel=org.springframework.messaging.core.GenericMessagingTemplate$TemporaryReplyChannel#1a867ae7, file_name=18237232_firm.xml, file_originalFile=C:\thirdpartyintg\input\18237232_firm.xml, id=dd70999a-8b8d-93d2-1a43-a961ac2c339f, file_relativePath=18237232_firm.xml, timestamp=1536366560003}]
2018-09-07 17:29:20.006 DEBUG 10060 --- [ taskExecutor-2] o.s.integration.channel.DirectChannel : postSend (sent=true) on channel 'firmProcessor.input', message: GenericMessage [payload=C:\thirdpartyintg\input\18237232_firm.xml, headers={replyChannel=org.springframework.messaging.core.GenericMessagingTemplate$TemporaryReplyChannel#1a867ae7, errorChannel=org.springframework.messaging.core.GenericMessagingTemplate$TemporaryReplyChannel#1a867ae7, file_name=18237232_firm.xml, file_originalFile=C:\thirdpartyintg\input\18237232_firm.xml, id=0e6dcb75-db99-1740-7b58-e9b42bfbf603, file_relativePath=18237232_firm.xml, timestamp=1536366559761}]
2018-09-07 17:29:20.007 DEBUG 10060 --- [ taskExecutor-2] o.s.integration.channel.DirectChannel : preSend on channel 'thirdpartyintgAgentDemographicFlow.channel#2', message: GenericMessage [payload=C:\thirdpartyintg\output\18237232_firm.xml, headers={file_originalFile=C:\thirdpartyintg\input\18237232_firm.xml, id=e6e2a30a-60b9-7cdd-84cc-4977d4c21c97, file_name=18237232_firm.xml, file_relativePath=18237232_firm.xml, timestamp=1536366560007}]
2018-09-07 17:29:20.008 DEBUG 10060 --- [ taskExecutor-2] o.s.integration.channel.DirectChannel : postSend (sent=true) on channel 'thirdpartyintgAgentDemographicFlow.channel#2', message: GenericMessage [payload=C:\thirdpartyintg\output\18237232_firm.xml, headers={file_originalFile=C:\thirdpartyintg\input\18237232_firm.xml, id=e6e2a30a-60b9-7cdd-84cc-4977d4c21c97, file_name=18237232_firm.xml, file_relativePath=18237232_firm.xml, timestamp=1536366560007}]
2018-09-07 17:29:20.009 DEBUG 10060 --- [ taskExecutor-2] o.s.integration.channel.DirectChannel : postSend (sent=true) on channel 'thirdpartyintgAgentDemographicFlow.subFlow#1.channel#0', message: GenericMessage [payload=C:\thirdpartyintg\input\18237232_firm.xml, headers={file_originalFile=C:\thirdpartyintg\input\18237232_firm.xml, id=13713de8-91ce-b1fa-f52d-450d3038cf9c, file_name=18237232_firm.xml, file_relativePath=18237232_firm.xml, timestamp=1536366559757}]
2018-09-07 17:29:26.009 INFO 10060 --- [ask-scheduler-9] o.s.i.a.AggregatingMessageHandler : Expiring MessageGroup with correlationKey[processdate]
2018-09-07 17:29:26.011 DEBUG 10060 --- [ask-scheduler-9] o.s.integration.channel.NullChannel : message sent to null channel: GenericMessage [payload=C:\thirdpartyintg\output\17019222_individual.xml, headers={file_originalFile=C:\thirdpartyintg\input\17019222_individual.xml, id=c654076b-696f-25d4-bded-0a43d1a8ca97, file_name=17019222_individual.xml, file_relativePath=17019222_individual.xml, timestamp=1536366559927}]
2018-09-07 17:29:26.011 DEBUG 10060 --- [ask-scheduler-9] o.s.integration.channel.NullChannel : message sent to null channel: GenericMessage [payload=C:\thirdpartyintg\output\18237232_firm.xml, headers={file_originalFile=C:\thirdpartyintg\input\18237232_firm.xml, id=e6e2a30a-60b9-7cdd-84cc-4977d4c21c97, file_name=18237232_firm.xml, file_relativePath=18237232_firm.xml, timestamp=1536366560007}]

Answer 1

First of all the RegexPatternFileListFilter should be first in the ChainFileListFilter. This way you won't overhead a memory in the AcceptOnceFileListFilter for files which you are not interested in.
You need .channel(confirmChannel()) in the end of the thirdpartyAgentDemographicFlow because this one is an input to your confirmGeneration flow.
I don't think that you .channel(aggregatorOutputChannel()) at all it has to implicit.
You also don't need that .channel(outputChannel()) in the sub-flows.
this is not working
Please, elaborate more: what error you get, how then it works etc...
You also can share some DEBUG logs for the org.springframework.integration to determine how your messages travel.
Also it would help a lot if your share some simple Spring Boot project on GitHub to let us to play with and reproduce according your provided instructions.
UPDATE
Also I've noticed that your aggregator is based on the groupTimeout(). To make it to send aggregated message to downstream you also need to configure there this:
/**
* #param sendPartialResultOnExpiry the sendPartialResultOnExpiry.
* #return the handler spec.
* #see AbstractCorrelatingMessageHandler#setSendPartialResultOnExpiry(boolean)
*/
public S sendPartialResultOnExpiry(boolean sendPartialResultOnExpiry) {
It is false by default, so your messages indeed are sent to the NullChannel.
See more info in the Docs: https://docs.spring.io/spring-integration/docs/current/reference/html/messaging-routing-chapter.html#agg-and-group-to