Recursively read files from remote server present in subdirectories with Spring Integration - spring

Question

I have working flow for getting files from single folder present in remote server using inbound adopter but i want for get files for all subfolder present in any remote server parent folder
I have code like this
#Bean
public SessionFactory<SftpClient.DirEntry> sftpSessionFactory() {
DefaultSftpSessionFactory factory = new DefaultSftpSessionFactory(true);
factory.setHost("localhost");
factory.setPort(port);
factory.setUser("foo");
factory.setPassword("foo");
factory.setAllowUnknownKeys(true);
factory.setTestSession(true);
return new CachingSessionFactory<>(factory);
}
#Bean
public SftpInboundFileSynchronizer sftpInboundFileSynchronizer() {
SftpInboundFileSynchronizer fileSynchronizer = new SftpInboundFileSynchronizer(sftpSessionFactory());
fileSynchronizer.setDeleteRemoteFiles(false);
fileSynchronizer.setRemoteDirectory("foo");
fileSynchronizer.setFilter(new SftpSimplePatternFileListFilter("*.xml"));
return fileSynchronizer;
}
#Bean
#InboundChannelAdapter(channel = "sftpChannel", poller = #Poller(fixedDelay = "5000"))
public MessageSource<File> sftpMessageSource() {
SftpInboundFileSynchronizingMessageSource source =
new SftpInboundFileSynchronizingMessageSource(sftpInboundFileSynchronizer());
source.setLocalDirectory(new File("sftp-inbound"));
source.setAutoCreateLocalDirectory(true);
source.setLocalFilter(new AcceptOnceFileListFilter<File>());
source.setMaxFetchSize(1);
return source;
}
#Bean
#ServiceActivator(inputChannel = "sftpChannel")
public MessageHandler handler() {
return new MessageHandler() {
#Override
public void handleMessage(Message<?> message) throws MessagingException {
System.out.println(message.getPayload());
}
};
}`
but instead of single folder i want get files for all subfolder present in foo directory
if possible please help with full code
#GaryRussell
Thanks you so much for you early response .I have done some changes according to your suggested code app is started but files is not getting picked up by application.
CompositeFileListFilter<LsEntry> compositeFileListFilter = new CompositeFileListFilter<>();
SftpPersistentAcceptOnceFileListFilter fileListFilter =
new SftpPersistentAcceptOnceFileListFilter(
(JdbcMetadataStore) context.getBean("metadataStore"), "REMOTE");
if (Constants.APP1.equals(appName) || Constants.APP2.equals(appName)) {
SftpRegexPatternFileListFilter regexPatternFileListFilter =
new SftpRegexPatternFileListFilter(Pattern.compile("^IL.*"));
compositeFileListFilter.addFilter(regexPatternFileListFilter);
}
compositeFileListFilter.addFilter(fileListFilter);
return IntegrationFlows.fromSupplier(
() -> sftpEnvironment.getSftpGLSIncomingDir(), // remote dir
e -> e.autoStartup(true).poller(pollerMetada()))
.handle(
Sftp.outboundGateway(sftpSessionFactory(), Command.MGET, "payload")
.options(Option.RECURSIVE)
.filter(compositeFileListFilter)
.fileExistsMode(FileExistsMode.IGNORE)
.localDirectoryExpression("'/tmp/' + #remoteDirectory")) // re-create tree locally
.split()
.log()
.get();
#GaryRussell
I have changed my code this new way it is partially processing files mean one example out of 10 file only process 5 or 6 files. I am not able to figure the main issue in that.and I also have also some open challenges which i am mentioning below
Its able to read files form remote subdirectories and store in local directory but I want to process these file in some other sftpChannel, if posible without storing locally
I also want to apply some deduplication technique using data base which will help me to avoid duplicate file processing.
public class SFTPPollerService {
#Bean
public SessionFactory<LsEntry> sftpSessionFactory() {
DefaultSftpSessionFactory factory = new DefaultSftpSessionFactory(true);
//code
return factory;
}
//OLD code
// #Bean
// public SftpInboundFileSynchronizer sftpInboundFileSynchronizer() {
// SftpInboundFileSynchronizer fileSynchronizer =
// new SftpInboundFileSynchronizer(sftpSessionFactory());
// fileSynchronizer.setDeleteRemoteFiles(sftpEnvironment.isDeleteRemoteFiles());
// fileSynchronizer.setRemoteDirectory(sftpEnvironment.getSftpGLSIncomingDir());
// fileSynchronizer.setPreserveTimestamp(true);
// CompositeFileListFilter<LsEntry> compositeFileListFilter = new
// CompositeFileListFilter<>();
// SftpPersistentAcceptOnceFileListFilter fileListFilter =
// new SftpPersistentAcceptOnceFileListFilter(
// (JdbcMetadataStore) context.getBean("metadataStore"), "REMOTE");
// if (Constants.app2.equals(appName)
// || Constants.app1.equals(appName)) {
// SftpRegexPatternFileListFilter regexPatternFileListFilter =
// new SftpRegexPatternFileListFilter(Pattern.compile("*.txt"));
// compositeFileListFilter.addFilter(regexPatternFileListFilter);
// }
// compositeFileListFilter.addFilter(fileListFilter);
// fileSynchronizer.setFilter(compositeFileListFilter);
// return fileSynchronizer;
// }
//
// #Bean
// #InboundChannelAdapter(channel = "sftpChannel", poller = #Poller("pollerMetada"))
// public MessageSource<File> sftpMessageSource() {
// SftpInboundFileSynchronizingMessageSource source =
// new SftpInboundFileSynchronizingMessageSource(sftpInboundFileSynchronizer());
// source.setLocalDirectory(new File(sftpEnvironment.getSftpLocalDir()));
//
// source.setAutoCreateLocalDirectory(true);
//
// try {
// source.setLocalFilter(
// (FileSystemPersistentAcceptOnceFileListFilter)
// context.getBean("filelistFilter"));
// } catch (Exception e) {
// LOG.error(
// "Exception caught while setting local filter on
// SftpInboundFileSynchronizingMessageSource",
// e);
// }
// source.setMaxFetchSize(sftpEnvironment.getMaxFetchFileSize());
//
// return source;
// }
//new Code
#Bean
public IntegrationFlow sftpInboundFlow() {
CompositeFileListFilter<LsEntry> compositeFileListFilter = new CompositeFileListFilter<>();
SftpPersistentAcceptOnceFileListFilter fileListFilter =
new SftpPersistentAcceptOnceFileListFilter(
(JdbcMetadataStore) context.getBean("metadataStore"), "REMOTE");
if (Constants.app2.equals(appName) || Constants.app1.equals(appName)) {
SftpRegexPatternFileListFilter regexPatternFileListFilter =
new SftpRegexPatternFileListFilter(Pattern.compile("(subDir | *.txt)"));
compositeFileListFilter.addFilter(regexPatternFileListFilter);
}
fileListFilter.setForRecursion(true);
FileSystemPersistentAcceptOnceFileListFilter fileSystemPersistentAcceptOnceFileListFilter = (FileSystemPersistentAcceptOnceFileListFilter) context.getBean(
"filelistFilter");
compositeFileListFilter.addFilter(fileListFilter);
// IntegrationFlow ir =
// IntegrationFlows.from(
// Sftp.inboundAdapter(sftpSessionFactory())
// .preserveTimestamp(true)
// .remoteDirectory(sftpEnvironment.getSftpGLSIncomingDir())
// .deleteRemoteFiles(sftpEnvironment.isDeleteRemoteFiles())
// .filter(compositeFileListFilter)
// .autoCreateLocalDirectory(true)
// .localDirectory(new File(sftpEnvironment.getSftpLocalDir())),
// e -> e.autoStartup(true).poller(pollerMetada()))
// .handle(handler())
// .get();
return IntegrationFlows.fromSupplier(
() -> sftpEnvironment.getSftpGLSIncomingDir(), // remote dir
e -> e.autoStartup(true).poller(pollerMetada()))
.handle(
Sftp.outboundGateway(sftpSessionFactory(), Command.MGET, "payload")
.options(Option.RECURSIVE)
.fileExistsMode(FileExistsMode.IGNORE)
.regexFileNameFilter("(dsv[0-9]|.*.xml)")
// .filter(compositeFileListFilter)
.localDirectoryExpression("'user/localDir/test/'"))
// .handle(handler())
// .patternFileNameFilter(".*\\.xml")) // re-create tree locally
.split()
.channel("sftpChannel")
// .handle(handler())
.log()
.get();
}
#Bean
public PollerMetadata pollerMetada() {
PollerMetadata pm = new PollerMetadata();
ExpressionEvaluatingTransactionSynchronizationProcessor processor =
new ExpressionEvaluatingTransactionSynchronizationProcessor();
ExpressionParser parser = new SpelExpressionParser();
Expression exp = parser.parseExpression("payload.delete()");
processor.setAfterRollbackExpression(exp);
TransactionSynchronizationFactory tsf = new DefaultTransactionSynchronizationFactory(processor);
pm.setTransactionSynchronizationFactory(tsf);
List<Advice> advices = new ArrayList<>();
advices.add(compoundTriggerAdvice());
pm.setAdviceChain(advices);
pm.setTrigger(compoundTrigger());
pm.setMaxMessagesPerPoll(sftpEnvironment.getMaxMessagesPerPoll());
return pm;
}
#Bean
public CronTrigger cronTrigger() {
if (LOG.isDebugEnabled()) {
return new CronTrigger(sftpEnvironment.getPollerCronExpressionWhenDebugModeIsEnabled());
} else {
return new CronTrigger(sftpEnvironment.getPollerCronExpression());
}
}
#Bean
public PeriodicTrigger periodicTrigger() {
return new PeriodicTrigger(sftpEnvironment.getPeriodicTriggerInMillis());
}
#Bean
public CompoundTrigger compoundTrigger() {
return new CompoundTrigger(cronTrigger());
}
#Bean
public CompoundTriggerAdvice compoundTriggerAdvice() {
return new CompoundTriggerAdvice(compoundTrigger(), periodicTrigger());
}
#Bean
public FileSystemPersistentAcceptOnceFileListFilter filelistFilter(MetadataStore datastore) {
return new FileSystemPersistentAcceptOnceFileListFilter((JdbcMetadataStore) datastore, "INT");
}
#Bean
public PlatformTransactionManager transactionManager() {
return new org.springframework.integration.transaction.PseudoTransactionManager();
}
#Bean
DataSource dataSource() throws SQLException {
OracleDataSource dataSource = new OracleDataSource();
dataSource.setUser(databaseProperties.getOracleUsername());
dataSource.setPassword(databaseProperties.getOraclePassword());
dataSource.setURL(databaseProperties.getOracleUrl());
dataSource.setImplicitCachingEnabled(true);
dataSource.setFastConnectionFailoverEnabled(true);
return dataSource;
}
/**
* Creates a {#link JdbcMetadataStore} for the de-duplication logic.
*
* <p>This method uses the "REGION" column of the metadatastore table to differentiate between
* multiple apps. The value of the "REGION" column is set equal to the app-name.
*
* #return a JDBC metadata store
* #throws SQLException in case an exception occurs during connection to SQL database
*/
#Bean
public MetadataStore metadataStore() throws SQLException {
JdbcMetadataStore jdbcMetadataStore = new JdbcMetadataStore(dataSource());
if (!Constants.app2.equals(appName)) {
jdbcMetadataStore.setRegion(appName);
}
return jdbcMetadataStore;
}
#Bean
#ServiceActivator(inputChannel = "sftpChannel")
public MessageHandler handler() {
return message -> {
File file = (File) message.getPayload();
FileDto fileDto = new FileDto(file);
fileHandler.handle(fileDto);
LOG.info("controller is here ");
try {
if (sftpEnvironment.isDeleteLocalFiles()) {
Files.deleteIfExists(Paths.get(file.toString()));
}
} catch (IOException e) {
// TODO retry/report/handle gracefully
LOG.error(String.format("MessageHandler had error message=%s", message), e);
}
};
}
}

Answer 1

The synchronizer doesn't support recursion. Use the outbound gateway with a recursive mget command instead.
https://docs.spring.io/spring-integration/docs/current/reference/html/sftp.html#sftp-outbound-gateway
Using the mget Command
mget retrieves multiple remote files based on a pattern and supports the following options:
-P: Preserve the timestamps of the remote files.
-R: Retrieve the entire directory tree recursively.
-x: Throw an exception if no files match the pattern (otherwise, an empty list is returned).
-D: Delete each remote file after successful transfer. If the transfer is ignored, the remote file is not deleted, because the FileExistsMode is IGNORE and the local file already exists.
The message payload resulting from an mget operation is a List<File> object (that is, a List of File objects, each representing a retrieved file).
EDIT
Here is an example using the java DSL...
#SpringBootApplication
public class So75180789Application {
public static void main(String[] args) {
SpringApplication.run(So75180789Application.class, args);
}
#Bean
IntegrationFlow flow(DefaultSftpSessionFactory sf) {
return IntegrationFlows.fromSupplier(() -> "foo/*", // remote dir
e -> e.poller(Pollers.fixedDelay(5000)))
.handle(Sftp.outboundGateway(sf, Command.MGET, "payload")
.options(Option.RECURSIVE)
.fileExistsMode(FileExistsMode.IGNORE)
.localDirectoryExpression("'/tmp/' + #remoteDirectory")) // re-create tree locally
.split()
.log()
.get();
}
#Bean
DefaultSftpSessionFactory sf(#Value("${host}") String host,
#Value("${username}") String user, #Value("${pw}") String pw) {
DefaultSftpSessionFactory sf = new DefaultSftpSessionFactory();
sf.setHost(host);
sf.setUser(user);
sf.setPassword(pw);
sf.setAllowUnknownKeys(true);
return sf;
}
}