Download the eligible file from SFTP server by using spring Integration using Spring boot project - spring

Question

My current project is based on Spring Integration. I am developing this project by using spring Boot.
My goal is to use Spring Integration to complete the below task.
Connect to SFTP
check if directory is created in the local at a specific folder
check the eligible file extension of the file specific to (CSV & XLSX)
Down load all the content from SFTP remote directory to local directory & need to track the file transfer start time and file transfer end time.
Read the file from local directory line by line and extract the certain column info.
Can you give me some suggestions ?
And how can I get the transfer start time ?
Note : This requirement i have to develop as an rest api. Please provide some guidance how i can achieve this by using spring integration?
Thanks. :)
public class SftpConfig {
#Value("${sftp.host}")
private String sftpHost;
#Value("${sftp.port:22}")
private int sftpPort;
#Value("${sftp.user}")
private String sftpUser;
#Value("${sftp.password:#{null}}")
private String sftpPasword;
#Value("${sftp.remote.directory:/}")
private String sftpRemoteDirectory;
#Value("${sftp.privateKey:#{null}}")
private Resource sftpPrivateKey;
#Value("${sftp.privateKeyPassPhrase:}")
private String privateKeyPassPhrase;
#Value("${sftp.remote.directory.download.filter:*.*}")
private String sftpRemoteDirectoryDownloadFilter;
#Value("${sftp.remote.directory.download:/}")
private String sftpRemoteDirectoryDownload;
#Value("${sftp.local.directory.download:${java.io.tmpdir}/localDownload}")
private String sftpLocalDirectoryDownload;
/*
* The SftpSessionFactory creates the sftp sessions. This is where you define
* the host , user and key information for your sftp server.
*/
// Creating session for Remote Destination SFTP server Folder
#Bean
public SessionFactory<ChannelSftp.LsEntry> sftpSessionFactory() {
DefaultSftpSessionFactory factory = new DefaultSftpSessionFactory(true);
factory.setHost(sftpHost);
factory.setPort(sftpPort);
factory.setUser(sftpUser);
if (sftpPrivateKey != null) {
factory.setPrivateKey(sftpPrivateKey);
factory.setPrivateKeyPassphrase(privateKeyPassPhrase);
} else {
factory.setPassword("sftpPassword");
}
factory.setAllowUnknownKeys(true);
return new CachingSessionFactory<ChannelSftp.LsEntry>(factory);
}
/*
* The SftpInboundFileSynchronizer uses the session factory that we defined above.
* Here we set information about the remote directory to fetch files from.
* We could also set filters here to control which files get downloaded
*/
#Bean
public SftpInboundFileSynchronizer SftpInboundFileSynchronizer () {
SftpInboundFileSynchronizer synchronizer = new SftpInboundFileSynchronizer();
return null;
}

Answer 1

If you take a look into the SftpStreamingMessageSource instead: https://docs.spring.io/spring-integration/docs/current/reference/html/sftp.html#sftp-streaming, plus use a FileSplitter to read that file line by line (which supports a "first like as header", too), you won't need to worry about transferring the file to local dir. You will do everything with the remote content on demand.
On the other hand, since you talk about a REST API, you probably are going to have some #RestController or Spring Integration HTTP Inbound Gateway: https://docs.spring.io/spring-integration/docs/current/reference/html/http.html#http-inbound, then you need to think about using an SftpOutboundGateway with an MGET command: https://docs.spring.io/spring-integration/docs/current/reference/html/sftp.html#sftp-outbound-gateway.
If you still need to track a download time for every single file, you need to consider to use that gateway twice: with a LIST command and NAME_ONLY, and the second time for GET command. At this point you can add ChannelInterceptor for input and output channels of the second gateway, so you will have a file name info to correlate and catch start and stop time before and after this gateway.